Table data processing method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, and relates to a table data processing method, which comprises the following steps: matching a header field of a table to be processed with a standard field, taking the unmatched header field as a field to be replaced, and querying the field to be replaced in a preset synonym mapping table; if the to-be-replaced field exists, replacing the to-be-replaced field with a corresponding standard field according to the synonym mapping table; if the to-be-replaced fields do not exist, candidate words of the to-be-replaced fields are generated, and target synonyms are screened out from the candidate words; and establishing a mapping relationship among the target synonym, the to-be-replaced field and the standard field to obtain an updated synonym mapping table, and replacing the to-be-replaced field according to the mapping relationship. The invention further provides a table data processing device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-processed table can be stored in a block chain. According to the method, data import and export failures caused by mismatching of the headers can be reduced, and the data processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a table data processing method, device, computer equipment and storage medium. Background Art

[0002] With the development of computer technology, the requirements for data processing efficiency are getting higher and higher. At present, tables are used to record relevant data for different businesses. In many business processes, the import and export of data is a common and critical link. This may involve migrating data from one system to another, or converting data between different formats. The easyexcel library currently used provides import and export functions, but it relies on annotations to identify and process headers. This method may not be flexible enough when facing complex or dynamically changing business needs, and may encounter various problems, such as: headers are not recognized, headers are wrong, and headers are not dynamically replaced. When it is necessary to dynamically replace or adjust the header, the existing solutions usually require modifying the code and restarting the server, which not only increases the complexity of technical maintenance, but also affects the response speed and efficiency of the business, thereby affecting the overall work efficiency. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a table data processing method, device, computer equipment and storage medium to solve the technical problem in the prior art that data import and export may not be flexible enough when facing complex or dynamically changing business needs, resulting in low business response speed and efficiency, thereby affecting overall work efficiency.

[0004] In order to solve the above technical problems, the present application provides a table data processing method, which adopts the following technical solution:

[0005] Receiving a form to be processed, and extracting a header field of the form to be processed;

[0006] Acquire the standard field corresponding to the header field according to the table type of the table to be processed, match the header field with the standard field, and obtain a matching result;

[0007] Determine the unmatched header field as a field to be replaced according to the matching result, and determine whether the field to be replaced exists in a preset synonym mapping table;

[0008] If it exists, then replace the to-be-replaced field with the corresponding standard field according to the synonym mapping table to obtain a standard target table;

[0009] If it does not exist, generating candidate synonyms for the field to be replaced, and obtaining a target synonym based on the similarity between the candidate synonyms and the standard field;

[0010] Establishing a mapping relationship between the target synonym, the field to be replaced and the standard field, storing the mapping relationship in the synonym mapping table, and obtaining an updated synonym mapping table;

[0011] Replacing the to-be-replaced fields with corresponding standard fields according to the updated synonym mapping table to obtain a standard target table;

[0012] The table data in the standard target table is processed.

[0013] In order to solve the above technical problems, the embodiment of the present application further provides a table data processing device, which adopts the following technical solution:

[0014] An extraction module, used for receiving a form to be processed and extracting a header field of the form to be processed;

[0015] A matching module, used for acquiring the standard field corresponding to the header field according to the table type of the table to be processed, matching the header field with the standard field, and obtaining a matching result;

[0016] A determination module, configured to determine, according to the matching result, an unmatched header field as a field to be replaced, and determine whether the field to be replaced exists in a preset synonym mapping table;

[0017] A first replacement module, used for replacing the to-be-replaced field with a corresponding standard field according to the synonym mapping table if it exists, to obtain a standard target table;

[0018] A synonym generation module, used for generating candidate synonyms for the field to be replaced if none exists, and obtaining a target synonym based on the similarity between the candidate synonyms and the standard field;

[0019] A mapping module, used to establish a mapping relationship between the target synonym, the field to be replaced and the standard field, store the mapping relationship in the synonym mapping table, and obtain an updated synonym mapping table;

[0020] A second replacement module, used to replace the to-be-replaced field with a corresponding standard field according to the updated synonym mapping table to obtain a standard target table;

[0021] The processing module is used to process the table data in the standard target table.

[0022] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0023] The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the table data processing method as described above when executing the computer-readable instructions.

[0024] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0025] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the table data processing method described above are implemented.

[0026] Compared with the prior art, this application has the following beneficial effects:

[0027] The present application provides a table data processing method, which extracts the header field of a table to be processed, obtains the standard field corresponding to the header field, matches the header field with the standard field, obtains the header field with inconsistent matching as the field to be replaced, and searches the field to be replaced in a preset synonym mapping table; if the field to be replaced exists in the synonym mapping table, the field to be replaced is replaced with the corresponding standard field according to the synonym mapping table to obtain a standard target table; if the field to be replaced does not exist in the synonym mapping table, candidate synonyms of the field to be replaced are generated, and target synonyms are obtained according to the similarity between the candidate synonyms and the standard field; a mapping relationship between the target synonym, the field to be replaced and the standard field is established in the synonym mapping table to obtain an updated synonym mapping table, and the field to be replaced is replaced with the corresponding standard field according to the updated synonym mapping table to obtain a standard target table, and the table data in the standard target table is processed. This application ensures the consistency of header fields by identifying table fields that do not match standard fields in the table as fields to be replaced, and replaces the headers according to the synonyms corresponding to the fields to be replaced, thereby allowing the system to process data from different sources and improve data compatibility; reducing data import and export failures due to header mismatches and improving data processing efficiency; the system can adapt to changing business needs without frequent code modifications and server restarts; ensuring that business processes will not be interrupted due to data format issues, improving business continuity and stability; in addition, ensuring that data can be correctly identified and used, improving data accessibility and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0029] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0030] Figure 2 is a flow chart of an embodiment of a table data processing method according to the present application;

[0031] Figure 3 is a structural schematic diagram of an embodiment of a table data processing device according to the present application;

[0032] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0034] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0036] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0037] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0038] This application provides a table data processing method, which can be applied to Figure 1 In the system architecture 100 shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0039] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0040] Terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers and desktop computers, etc.

[0041] The server 105 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal devices 101 , 102 , and 103 .

[0042] It should be noted that the table data processing method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the table data processing device is generally arranged in the server / terminal device.

[0043] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0044] Continue to refer Figure 2 , shows a flow chart of an embodiment of a table data processing method according to the present application, comprising the following steps:

[0045] Step S201, receiving a form to be processed, and extracting header fields of the form to be processed.

[0046] The pending forms are forms that need to be analyzed, sorted, imported or exported in business scenarios. The forms contain business data that need to be processed. For example, in the financial field, the pending forms may contain financial data, transaction records, market analysis results, etc. The pending forms may be worksheets in spreadsheet software (such as Excel), tables in databases, or any other form of data collection, which can be uploaded by users or automatically obtained from set channels.

[0047] In this embodiment, the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the to-be-processed form via a wired connection or a wireless connection. It should be noted that the wireless connection method may include but is not limited to 3G / 4G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0048] The table to be processed contains two types of data, one is the table header, and the other is the sample data corresponding to the table header. Among them, the table header contains multiple data fields, namely the table header fields, each of which can be located in a cell, and the table header field can represent the nature of the data in the column where it is located, and the table header field can be the name or type of the data in the column where it is located. In this embodiment, after obtaining the table to be processed, the table to be processed is analyzed to identify and extract the table header fields.

[0049] It should be emphasized that in order to further ensure the privacy and security of the pending forms, the above-mentioned pending forms can also be stored in a node of a blockchain.

[0050] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0051] Step S202: acquiring the standard field corresponding to the header field according to the table type of the table to be processed, matching the header field with the standard field, and obtaining a matching result.

[0052] Among them, the table fields can be determined according to the data content, data structure, header fields, file name or title, data source, etc. Taking the financial field as an example, table types include but are not limited to financial statements, sales data tables, customer relationship management tables, inventory management tables, project management tables, order processing tables, supply chain management tables, market research tables, risk management tables, etc.

[0053] In this embodiment, each form type corresponds to a corresponding template, which includes standard fields of the form type. The standard fields corresponding to the extracted header fields can be obtained according to the form type, and the standard fields are matched with the header fields.

[0054] The matching between standard fields and header fields can be done by string matching, edit distance, word vector similarity, regular expression matching, etc.

[0055] In some optional implementations, the standard field and the header field are matched by a string matching method. The step of matching the header field with the standard field to obtain a matching result includes:

[0056] Construct the first string of the header field and the second string of the standard field;

[0057] Constructing a comparison matrix based on the first character string and the second character string;

[0058] Calculate a matching window value according to a first length of the first character string and a second length of the second character string;

[0059] Based on the comparison matrix and the matching window value, the number of matching characters and the number of matching character substitutions are calculated;

[0060] Calculate the string similarity according to the first length, the second length, the number of matching characters, and the number of matching character transpositions;

[0061] Determine the common prefix length of the first character string and the second character string, and calculate the matching degree according to the common prefix length and the character string similarity;

[0062] The matching result is obtained according to the matching degree.

[0063] Among them, assuming that the header field is X, construct the first string str corresponding to the header field x , the standard field is Y, construct the second string str corresponding to the standard field y For example, if the header field X is {name}, then the first string str x =“Name”.

[0064] Assume that the first length of the first string is m, the second length of the second string is n, and the constructed comparison matrix is ​​M(X, Y) m×n Specifically, according to the first length len1 of the first string and the second length len2 of the second string, the matching window value is calculated. The calculation formula of the matching window value is as follows:

[0065]

[0066] Based on the comparison matrix and the matching window value, the number of matching characters and the number of matching character permutations are calculated. Among them, the calculation of the number of matching characters s is: if the first string is str x and the second string is str y If the distance between the same characters in the matching window is less than the matching window value MW, it means that the character matches; in the matching process, the matched characters need to be excluded. If a matching character is found, it is necessary to jump out of this match and match the next character to obtain the matching character set;

[0067] For the calculation of the number of matching character transpositions t: determine the first string is str x and the second string is str y Whether the order of the matching character set is consistent, if not, half of the transposition number is the matching character transposition number t.

[0068] In this embodiment, the number of matching characters s and the number of matching character transpositions t satisfy the following formula:

[0069]

[0070] The string similarity is calculated based on the first length, the second length, the number of matched characters, and the number of matched character transpositions. The string similarity is Jaro Distanc, and the calculation formula is as follows:

[0071]

[0072] The common prefix length of the first character string and the second character string is determined, and the matching degree is calculated according to the common prefix length and the character string similarity, and the matching degree is used as the matching result. The calculation formula of the matching degree is as follows:

[0073] Sim(X,Y)=D w =D j +(l×p×(1-D j ))

[0074] Among them, l represents the common prefix length of the first string and the second string, but the maximum is 4; p is a constant for adjusting the score, which cannot exceed 0.25, otherwise D may appear.w When it is greater than 1, Winkler defines this constant as 0.1.

[0075] In this embodiment, the value of Sim(X,Y) reflects the similarity between the two fields, and the larger the value, the higher the similarity, and the more consistent the two fields are. If the value of Sim(X,Y) is 1, it means that the header field is consistent with the standard field, that is, the header field matches the standard field, otherwise, the header field does not match the standard field.

[0076] By comparing the header fields with the standard fields, we can improve the consistency and availability of data, enhance the quality of data analysis, improve the efficiency of comparison, and avoid wasting resources.

[0077] Step S203: determine the unmatched header fields as fields to be replaced according to the matching results, and determine whether the fields to be replaced exist in the preset synonym mapping table.

[0078] When there are fields in the header that do not match the standard fields, in order to avoid the header being unable to be recognized or being recognized incorrectly during the process of processing the table data, the unmatched header fields need to be replaced, that is, the unmatched header fields are used as the fields to be replaced.

[0079] In this embodiment, the preset synonym mapping table is a data structure for storing standard fields and their corresponding synonyms. The steps for creating the preset synonym mapping table include: determining the scope of vocabulary that the mapping table needs to cover, such as terms or general vocabulary in a specific field (such as finance, medicine); obtaining vocabulary that needs to be mapped through dictionaries, domain experts, corpora, etc.; for each vocabulary, determining its synonyms through manual analysis, synonym dictionaries, semantic networks or machine learning models; defining a list or set to store each vocabulary and its synonyms, for example, in the form of key-value pairs, that is, each vocabulary and its synonyms in the synonym mapping table are stored in a mapping relationship.

[0080] Check whether there is a field to be replaced in the preset synonym mapping table. If so, it means that the field to be replaced is a synonym of the corresponding standard field. If not, a synonym for the field to be replaced needs to be generated.

[0081] Step S204: if it exists, the field to be replaced is replaced with the corresponding standard field according to the synonym mapping table to obtain a standard target table.

[0082] If the field to be replaced exists in the synonym mapping table, the standard field corresponding to the field to be replaced is determined according to the mapping relationship between the field to be replaced and the standard field, and a suitable replacement tool is selected for replacement, such as SQL statements, Python scripts, etc. For example, in SQL, UPDATE statements and CASE expressions may be used; in Python, the replace method of the pandas library may be used.

[0083] Step S205: if it does not exist, then generate candidate synonyms for the field to be replaced, and obtain the target synonym based on the similarity between the candidate synonyms and the standard field.

[0084] If the field to be replaced does not exist in the preset synonym mapping table, a corresponding synonym is generated according to the field to be replaced, and matching and replacement are performed according to the generated synonym.

[0085] In some optional implementations, the step of generating candidate synonyms for the field to be replaced includes:

[0086] Get the description information of the field to be replaced from the preset field dictionary;

[0087] The description information is input into the trained synonym generation model to generate candidate synonyms.

[0088] In this embodiment, the preset field dictionary is used to define and describe the properties and rules of the fields in the data set, including field name, field description, data type, index information, associated fields, metadata, data source, and remarks information.

[0089] The field to be replaced is matched in the preset field dictionary, and the description information corresponding to the matched field is used as the description information of the field to be replaced. The description information may include a brief description of the field to illustrate its meaning or purpose.

[0090] When matching in the preset field dictionary, you can first use field exact matching. If field exact matching cannot match a field that is completely consistent with the field to be replaced, then use semantic fuzzy matching to match at the semantic level; you can also perform field exact matching and semantic fuzzy matching at the same time and output the matching result. Then, the description information of the field to be replaced is processed through the trained synonym generation model to generate candidate synonyms corresponding to the field to be replaced.

[0091] This embodiment generates synonyms for the field to be replaced by using a preset field dictionary and a synonym generation model, which can enrich vocabulary resources, enhance the relevance of text, enhance the model's ability to understand text, and improve the accuracy of generating synonyms.

[0092] In some optional implementations of this embodiment, the synonym generation model includes a knowledge layer, an embedding layer, a view layer, a mask attention layer, and an output layer. The description information is input into the trained synonym generation model, and the step of generating candidate synonyms includes:

[0093] The knowledge layer queries the entity relationship corresponding to the description information from the preset knowledge base, and injects the entity relationship into the description information to form a sentence tree, which is then input into the embedding layer and the view layer respectively;

[0094] The sentence tree is positionally embedded through the embedding layer to obtain an embedded coding vector;

[0095] Build the visible matrix of the sentence tree through the view layer;

[0096] Input the embedded coding vector and the visible matrix into the mask attention layer for attention calculation and output the attention enhanced features;

[0097] The attention-enhanced features are predicted through the output layer to output candidate synonyms.

[0098] Among them, the knowledge base refers to a data warehouse or knowledge graph built in the field of artificial intelligence that covers knowledge in a specific industry. This concept involves integrating and organizing a large amount of structured and unstructured information about a specific industry, including but not limited to technical standards, industry trends, market data, professional terminology, and the relationship between related entities.

[0099] Identify the entities in the description information, query the preset knowledge base based on the entities, and query the entity relationships related to the description information. The knowledge layer is responsible for injecting the queried entity relationships into the description information to form a sentence tree. The sentence tree completes the background information of the sentence and solves the problem that the word vector deviates from the core semantics due to the lack of knowledge background in the description information as a single sentence.

[0100] In this embodiment, forming a sentence tree is divided into two steps: knowledge query (K-Query) and knowledge injection (K-Inject).

[0101] Furthermore, the knowledge query function of the knowledge layer is called to identify all entities corresponding to the description information, and the triples corresponding to each entity are queried in the knowledge base; the knowledge injection function of the knowledge layer is called to embed the triples into the corresponding positions in the description information to obtain the sentence tree.

[0102] K-Query is responsible for querying the relationship and value corresponding to each entity in each description information from the knowledge base, that is, the triple. The specific query process is as follows:

[0103] E = K_Query(s, K);

[0104] The function K_Query represents the description information s to query the knowledge base K and obtain the triple set E = {(w i , r i0 , w i0 ),……,(w i , r ik , w ik )}.

[0105] K-Inject is responsible for embedding the triple set E into the corresponding position in the description information s to form a sentence tree. Each triple constitutes a branch. The matrix sentence tree output by the knowledge layer is: t = {w1, w2, w3, ..., w i (r i0 , w i0 ), ..., w i (r ik , w ik ),……,w n}.

[0106] Sentence trees cannot be directly used as sequence inputs. They are converted into text sequences through the embedding layer. The embedding layer includes the token embedding layer, the soft-position embedding layer, and the segment embedding layer. The token embedding layer maps each token in a sentence into a vector representation with a dimension of H. There is a special token [CLS] at the beginning of each sentence, which is mainly used for sentence classification. The soft-position embedding layer uses soft positions to encode embedded values ​​and relationships, which is different from the position encoding of entities. The segment embedding layer is used to distinguish between two sentence fragments.

[0107] Specifically, the sentence tree is input into the embedding layer for segment embedding operation, soft position embedding operation and word embedding operation respectively to obtain the corresponding sentence encoding vector, position encoding vector and word encoding vector; the sentence encoding vector, position encoding vector and word encoding vector are summed to obtain the embedded encoding vector.

[0108] Through the token embedding layer, each word in the sentence tree is embedded to obtain a sentence encoding vector; through the soft-position embedding layer, each word in the sentence tree is soft-position embedded to obtain a position encoding vector, which retains the trunk position information of the sentence describing the information; through the segment embedding layer, each word in the sentence tree is embedded to obtain a word encoding vector, and the sentence encoding vector, position encoding vector and word encoding vector are summed to obtain an embedded encoding vector, wherein the embedded encoding vector retains the structural information of the tree structure, which can strengthen the feature representation and thus better obtain the semantic features.

[0109] Since the triples in the sentence tree may affect the meaning of the description information, in order to prevent the influence of knowledge noise on the sentence, a visible matrix M is constructed to limit each word to only see the context and knowledge related to itself.

[0110] Assume that the two tokens are w i and w j , M ij = 0, indicating that in the visible matrix w i For w j It is visible, M ij = -∞, indicating that in the visible matrix w i For w j is invisible.

[0111] In this embodiment, the masked attention layer includes a masked multi-head attention sublayer and a feedforward neural network sublayer. The embedded coding vector and the visible matrix are input into the masked multi-head attention sublayer for multi-head attention calculation to obtain multi-head attention features; the embedded coding vector and the multi-head attention features are enhanced by the feedforward neural network sublayer to obtain attention enhancement features.

[0112] Among them, the multi-head attention sublayer can capture contextual information when calculating the attention vector; the feedforward network sublayer is used to connect and enhance the features output by the multi-head attention sublayer to prevent insufficient fitting.

[0113] In this embodiment, the use of masked attention layers can limit the visible area of ​​the self-attention mechanism, thereby capturing the deep bidirectional structure in the description information. The multi-head attention sublayer is composed of multiple mask-self-attention layers stacked together, and adds a visible matrix M on the basis of mask-self-attention to perform attention calculation on the embedded coding vector.

[0114] Assuming the number of mask-self-attention layers is L and the number of heads is A, determine the query vector parameter matrix W for each layer of mask-self-attention q , key vector parameter matrix W k Sum value vector parameter matrix W v , the self-attention calculation formula of each layer of mask-self-attention is as follows:

[0115]

[0116] Among them, Q i =H i-1 W q ; K i =H i-1 W k ; V i =H i-1 W v ;Q i represents the query vector of the i-th layer, K i represents the key vector of the i-th layer, V i Represents the value vector of the i-th layer; H i represents the mask-self-attention output of the i-th layer; d k Represents the dimension of the embedding encoding vector of the input.

[0117] The calculation formula of multi-head attention features is as follows:

[0118] MultiHead=Concat(head1,head2,…,head A )W0;

[0119] Among them, Concat represents the matrix concatenation function; W0 represents the parameter matrix when compressing each self-attention.

[0120] By performing attention calculation through the masked attention layer, it is possible to capture deep features in the description information, enrich the representation capability of the features, and improve the generalization ability of the model.

[0121] In this embodiment, the output layer can be implemented by combining a fully connected layer with a softmax layer. The intermediate representation attention enhancement feature is mapped to the size of the vocabulary through the fully connected layer, a score is generated for each word, and then the score is converted into a probability distribution for output through the softmax function of the softmax layer, and the words with the top K probabilities are selected as candidate synonyms, where K is a natural number greater than 0.

[0122] The synonym generation model generates candidate synonyms based on the knowledge base query results and description information, which can utilize external knowledge, have a stronger ability to understand the context, improve the accuracy of synonym generation, and reduce hallucinations.

[0123] In this embodiment, after generating candidate synonyms for the field to be replaced, the similarity between the candidate synonyms and the standard field is calculated, and the target synonym is obtained according to the similarity.

[0124] In some optional implementations of this embodiment, the step of obtaining the target synonym based on the similarity between the candidate synonym and the standard field includes:

[0125] Convert candidate synonyms and standard fields into corresponding candidate word vectors and standard field word vectors respectively;

[0126] Calculate the semantic similarity between the candidate word vector and the standard field word vector;

[0127] Qualified candidate synonyms are selected as target synonyms based on semantic similarity.

[0128] The candidate synonyms and standard fields are converted into corresponding word vectors, and then the similarity between the vectors is calculated. The word vector conversion can be performed by using the pre-trained text representation BERT model, FastText algorithm, etc. to convert the candidate synonyms into candidate word vectors and the standard fields into standard field word vectors.

[0129] A variety of similarity algorithms are used to calculate semantic similarity, including: Jaccard similarity algorithm, cosine similarity algorithm and edit distance. Specifically, Jaccard similarity algorithm is used to calculate the similarity between fields. The larger the calculated Jaccard coefficient value, the higher the field similarity. Cosine similarity measures the similarity between two vectors by measuring the cosine value of the angle between them. Edit distance refers to the minimum number of editing operations required to convert one string into another, which describes the similarity between two strings. The larger their distance, the more different they are. After calculating these three similarities, the semantic similarity is obtained by weighted summation.

[0130] Candidate synonyms whose semantic similarity is greater than or equal to a preset threshold are qualified candidate synonyms, and candidate synonyms whose semantic similarity is greater than or equal to the preset threshold are screened out as target synonyms.

[0131] Screening out target synonyms through a similarity algorithm can improve screening efficiency and the accuracy of synonyms.

[0132] Step S206: Establish a mapping relationship between the target synonym, the field to be replaced and the standard field, store the mapping relationship in a synonym mapping table, and obtain an updated synonym mapping table.

[0133] Specifically, check whether the target synonym exists in the synonym mapping table; if the target synonym exists in the synonym mapping table, add a mapping relationship between the field to be replaced and the standard field in the synonym mapping table; if the target synonym does not exist in the synonym mapping table, add a mapping relationship between the target synonym, the field to be replaced and the standard field in the synonym mapping table.

[0134] In this embodiment, the target synonyms and the field to be replaced are both used as synonyms of the standard field, and the mapping relationship among the target synonyms, the field to be replaced and the standard field is configured in a preset synonym mapping table.

[0135] By updating the synonym mapping table with the target synonyms and the fields to be replaced, the synonym database can be updated and expanded, which can enrich and improve the synonym database, facilitate subsequent data processing, and thus improve business efficiency. At the same time, it can also reduce the operation and maintenance costs of developers.

[0136] Step S207: replace the to-be-replaced fields with corresponding standard fields according to the updated synonym mapping table to obtain a standard target table.

[0137] In this embodiment, the field to be replaced is replaced with the corresponding standard field according to the mapping relationship between the field to be replaced and the standard field in the updated synonym mapping table. Specifically, a suitable replacement tool may be selected for replacement, such as SQL statements, Python scripts, etc. For example, in SQL, an UPDATE statement and a CASE expression may be used; in Python, a replace method of the pandas library may be used, etc.

[0138] Step S208, processing the table data in the standard target table.

[0139] Specifically, the parsing strategy corresponding to the standard field in the standard target table is obtained; according to the data conversion method in the parsing strategy, the field value corresponding to the standard field is read and converted to obtain the table parsing data; and the table parsing data is imported or exported.

[0140] The parsing strategy includes the data conversion method corresponding to the parsing field. The sample data corresponding to the header field in the standard target table is parsed according to the data conversion method. After the standard target tables with different data formats are parsed by the corresponding data conversion method, the table parsing data with a unified data format can be obtained. For example, text data does not need to be converted; numerical data such as price, sales, and quantity need to be converted according to the corresponding format; date and time data are converted according to the format of standardized date and time.

[0141] Parse the table parsed data of the standard target table and import or export it according to the business request. For example, obtain the corresponding business table data from the business system according to the business request, and export the business table data according to steps S201 to S208.

[0142] By obtaining the data conversion method of the standard fields corresponding to the standard target table through the parsing strategy, the efficiency of parsing the table data can be greatly improved.

[0143] This application ensures the consistency of header fields by identifying table fields that do not match standard fields in the table as fields to be replaced, and replaces the headers according to the synonyms corresponding to the fields to be replaced, thereby allowing the system to process data from different sources and improve data compatibility; reducing data import failures due to header mismatches and improving data processing efficiency; the system can adapt to changing business needs without frequent code modifications and server restarts; ensuring that business processes will not be interrupted due to data format issues, improving business continuity and stability; in addition, ensuring that data can be correctly identified and used, improving data accessibility and availability.

[0144] The table data processing method of the present application can be used in the field of financial technology. According to the business request, the corresponding pending forms are obtained, such as financial statements, sales data tables, inventory management tables, project management tables, order processing tables, supply chain management tables, risk management tables, etc. The pending forms are processed by executing steps S201 to S208.

[0145] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0146] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0147] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0148] Further references Figure 3 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a table data processing device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0149] like Figure 3As shown, the table data processing device 300 described in this embodiment includes: an extraction module 301, a matching module 302, a determination module 303, a first replacement module 304, a synonym generation module 305, a mapping module 306, a second replacement module 307 and a processing module 308. Among them:

[0150] The extraction module 301 is used to receive a table to be processed and extract the header fields of the table to be processed;

[0151] The matching module 302 is used to obtain the standard field corresponding to the header field according to the table type of the table to be processed, and match the header field with the standard field to obtain a matching result;

[0152] The determination module 303 is used to determine the unmatched header field as a field to be replaced according to the matching result, and determine whether the field to be replaced exists in a preset synonym mapping table;

[0153] The first replacement module 304 is used to replace the to-be-replaced field with the corresponding standard field according to the synonym mapping table if it exists, to obtain a standard target table;

[0154] The synonym generation module 305 is used to generate candidate synonyms for the field to be replaced if none exists, and obtain a target synonym based on the similarity between the candidate synonyms and the standard field;

[0155] The mapping module 306 is used to establish a mapping relationship between the target synonym, the field to be replaced and the standard field, store it in the synonym mapping table, and obtain an updated synonym mapping table;

[0156] The second replacement module 307 is used to replace the to-be-replaced field with the corresponding standard field according to the updated synonym mapping table to obtain a standard target table;

[0157] The processing module 308 is used to process the table data in the standard target table.

[0158] It should be emphasized that in order to further ensure the privacy and security of the pending forms, the above-mentioned pending forms can also be stored in a node of a blockchain.

[0159] Based on the above-mentioned table data processing device 300, by identifying the table fields that do not match the standard fields in the table as the fields to be replaced, and replacing the table headers according to the synonyms corresponding to the fields to be replaced, the consistency of the table header fields is ensured, and the system can process data from different sources to improve data compatibility; reduce data import failures due to header mismatches and improve data processing efficiency; the system can adapt to changing business needs without frequent code modifications and server restarts; ensure that business processes will not be interrupted due to data format problems, thereby improving business continuity and stability; in addition, ensure that data can be correctly identified and used, thereby improving data accessibility and availability.

[0160] In some optional implementations, the matching module 302 is further configured to:

[0161] Constructing a first string of the header field and a second string of the standard field;

[0162] constructing a comparison matrix according to the first character string and the second character string;

[0163] Calculating a matching window value according to a first length of the first character string and a second length of the second character string;

[0164] Calculate the number of matching characters and the number of matching character permutations based on the comparison matrix and the matching window value;

[0165] Calculate the string similarity according to the first length, the second length, the number of matching characters, and the number of matching character transpositions;

[0166] Determine a common prefix length of the first character string and the second character string, and calculate a matching degree according to the common prefix length and the character string similarity;

[0167] A matching result is obtained according to the matching degree.

[0168] By comparing the header fields with the standard fields, we can improve the consistency and availability of data, enhance the quality of data analysis, improve the efficiency of comparison, and avoid wasting resources.

[0169] In some optional implementations, the synonym generation module 305 includes:

[0170] An acquisition submodule, used for acquiring description information of the field to be replaced from a preset field dictionary;

[0171] The generation submodule is used to input the description information into the trained synonym generation model to generate candidate synonyms.

[0172] By generating synonyms for the fields to be replaced through the preset field dictionary and synonym generation model, it is possible to enrich vocabulary resources, enhance the relevance of the text, enhance the model's ability to understand the text, and improve the accuracy of generating synonyms.

[0173] In some optional implementations of this embodiment, the synonym generation model includes a knowledge layer, an embedding layer, a view layer, a mask attention layer and an output layer; the generation submodule includes:

[0174] A query injection unit, configured to query an entity relationship corresponding to the description information from a preset knowledge base through the knowledge layer, and inject the entity relationship into the description information to form a sentence tree, and input the sentence tree into the embedding layer and the view layer respectively;

[0175] An embedding unit, used for performing position embedding on the sentence tree through the embedding layer to obtain an embedded coding vector;

[0176] A construction unit, configured to construct a visible matrix of the sentence tree through the view layer;

[0177] An attention unit, configured to input the embedded coding vector and the visible matrix into the masked attention layer for attention calculation, and output an attention enhancement feature;

[0178] An output unit is used to predict the attention enhancement feature through the output layer and output candidate synonyms.

[0179] The synonym generation model generates candidate synonyms based on the knowledge base query results and description information, which can utilize external knowledge, have a stronger ability to understand the context, improve the accuracy of synonym generation, and reduce hallucinations.

[0180] In some optional implementations, the synonym generation module 305 further includes:

[0181] A vector conversion submodule, used to convert the candidate synonyms and the standard fields into corresponding candidate word vectors and standard field word vectors respectively;

[0182] A calculation submodule, used to calculate the semantic similarity between the candidate word vector and the standard field word vector;

[0183] The screening submodule is used to screen out qualified candidate synonyms as target synonyms based on the semantic similarity.

[0184] Screening out target synonyms through a similarity algorithm can improve screening efficiency and the accuracy of synonyms.

[0185] In this embodiment, the mapping module 306 is further used to:

[0186] Querying whether the target synonym exists in the synonym mapping table;

[0187] If the target synonym exists in the synonym mapping table, a mapping relationship between the field to be replaced and the standard field is added to the synonym mapping table;

[0188] If the target synonym does not exist in the synonym mapping table, a mapping relationship between the target synonym, the field to be replaced and the standard field is added to the synonym mapping table.

[0189] By updating the synonym mapping table with the target synonyms and the fields to be replaced, the synonym database can be updated and expanded, which can enrich and improve the synonym database, facilitate subsequent data processing, and thus improve business efficiency. At the same time, it can also reduce the operation and maintenance costs of developers.

[0190] In some optional implementations of this embodiment, the processing module 308 includes:

[0191] A parsing strategy acquisition submodule, used to acquire the parsing strategy corresponding to the standard field in the standard target table;

[0192] A parsing submodule, used for reading and converting the field value corresponding to the standard field according to the data conversion method in the parsing strategy to obtain table parsing data;

[0193] The processing submodule is used to import or export the table parsed data.

[0194] By obtaining the data conversion method of the standard fields corresponding to the standard target table through the parsing strategy, the efficiency of parsing the table data can be greatly improved.

[0195] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0196] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital processors (Digital Signal Processor, DSP), embedded devices, etc.

[0197] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0198] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the table data processing method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0199] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run the computer-readable instructions stored in the memory 41 or process data, such as running the computer-readable instructions of the table data processing method.

[0200] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0201] This embodiment implements the steps of the table data processing method of the above embodiment when the processor executes computer-readable instructions stored in the memory. By identifying the table fields that do not match the standard fields in the table as the fields to be replaced, and replacing the headers according to the synonyms corresponding to the fields to be replaced, the consistency of the header fields is ensured, which allows the system to process data from different sources and improve data compatibility; it reduces data import failures caused by header mismatches and improves data processing efficiency.

[0202] The present application also provides another implementation mode, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor so that the at least one processor executes the steps of the table data processing method as described above, by identifying the table fields that do not match the standard fields in the table as the fields to be replaced, and replacing the table headers according to the synonyms corresponding to the fields to be replaced, thereby ensuring the consistency of the table header fields, allowing the system to process data from different sources, and improving data compatibility; reducing data import failures due to header mismatches, and improving data processing efficiency.

[0203] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0204] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A table data processing method, characterized in that: The steps include: Receiving a form to be processed, and extracting a header field of the form to be processed; Acquire the standard field corresponding to the header field according to the table type of the table to be processed, match the header field with the standard field, and obtain a matching result; Determine the unmatched header field as a field to be replaced according to the matching result, and determine whether the field to be replaced exists in a preset synonym mapping table; If it exists, then replace the to-be-replaced field with the corresponding standard field according to the synonym mapping table to obtain a standard target table; If it does not exist, generating candidate synonyms for the field to be replaced, and obtaining a target synonym based on the similarity between the candidate synonyms and the standard field; Establishing a mapping relationship between the target synonym, the field to be replaced and the standard field, storing the mapping relationship in the synonym mapping table, and obtaining an updated synonym mapping table; Replacing the to-be-replaced fields with corresponding standard fields according to the updated synonym mapping table to obtain a standard target table; The table data in the standard target table is processed.

2. The table data processing method according to claim 1, characterized in that: The step of matching the header field with the standard field to obtain a matching result comprises: Constructing a first string of the header field and a second string of the standard field; constructing a comparison matrix according to the first character string and the second character string; Calculating a matching window value according to a first length of the first character string and a second length of the second character string; Calculate the number of matching characters and the number of matching character permutations based on the comparison matrix and the matching window value; Calculate the string similarity according to the first length, the second length, the number of matching characters, and the number of matching character transpositions; Determine a common prefix length of the first character string and the second character string, and calculate a matching degree according to the common prefix length and the character string similarity; A matching result is obtained according to the matching degree.

3. The table data processing method according to claim 1, characterized in that: The step of generating candidate synonyms for the field to be replaced comprises: Obtaining description information of the field to be replaced from a preset field dictionary; The description information is input into a trained synonym generation model to generate candidate synonyms.

4. The table data processing method according to claim 3, characterized in that: The synonym generation model includes a knowledge layer, an embedding layer, a view layer, a mask attention layer and an output layer; The step of inputting the description information into the trained synonym generation model to generate candidate synonyms includes: Querying the entity relationship corresponding to the description information from the preset knowledge base through the knowledge layer, injecting the entity relationship into the description information to form a sentence tree, and inputting the sentence tree into the embedding layer and the view layer respectively; Positionally embedding the sentence tree through the embedding layer to obtain an embedded coding vector; Constructing a visible matrix of the sentence tree through the view layer; Inputting the embedded coding vector and the visible matrix into the masked attention layer for attention calculation, and outputting attention enhancement features; The attention enhancement feature is predicted through the output layer to output candidate synonyms.

5. The table data processing method according to claim 1, characterized in that: The step of obtaining a target synonym based on the similarity between the candidate synonym and the standard field includes: Convert the candidate synonyms and the standard fields into corresponding candidate word vectors and standard field word vectors respectively; Calculating the semantic similarity between the candidate word vector and the standard field word vector; Qualified candidate synonyms are selected as target synonyms based on the semantic similarity.

6. The table data processing method according to claim 1, characterized in that: The step of establishing a mapping relationship between the target synonym, the field to be replaced and the standard field comprises: Querying whether the target synonym exists in the synonym mapping table; If the target synonym exists in the synonym mapping table, a mapping relationship between the field to be replaced and the standard field is added to the synonym mapping table; If the target synonym does not exist in the synonym mapping table, a mapping relationship between the target synonym, the field to be replaced and the standard field is added to the synonym mapping table.

7. The table data processing method according to claim 1, characterized in that: The step of processing the table data in the standard target table comprises: Obtaining the parsing strategy corresponding to the standard field in the standard target table; According to the data conversion method in the parsing strategy, the field value corresponding to the standard field is read and converted to obtain table parsing data; Import or export the table parsed data.

8. A table data processing device, characterized in that: include: An extraction module, used for receiving a form to be processed and extracting a header field of the form to be processed; A matching module, used for acquiring the standard field corresponding to the header field according to the table type of the table to be processed, matching the header field with the standard field, and obtaining a matching result; A determination module, configured to determine, according to the matching result, an unmatched header field as a field to be replaced, and determine whether the field to be replaced exists in a preset synonym mapping table; A first replacement module, used for replacing the to-be-replaced field with a corresponding standard field according to the synonym mapping table if it exists, to obtain a standard target table; A synonym generation module, used for generating candidate synonyms for the field to be replaced if none exists, and obtaining a target synonym based on the similarity between the candidate synonyms and the standard field; A mapping module, used to establish a mapping relationship between the target synonym, the field to be replaced and the standard field, store the mapping relationship in the synonym mapping table, and obtain an updated synonym mapping table; A second replacement module, used to replace the to-be-replaced field with a corresponding standard field according to the updated synonym mapping table to obtain a standard target table; The processing module is used to process the table data in the standard target table.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the table data processing method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the table data processing method according to any one of claims 1 to 7 are implemented.

Images