A model training method, system and electronic device

Abstract

A model training method, system and electronic device, the method comprising: obtaining standard data and original data, establishing a first association relationship between the standard data and the original data, and obtaining training samples; establishing a second association relationship between the standard data and the real data according to the semantic similarity between the standard data and the real data; obtaining initial graph data according to the first association relationship and the second association relationship; and putting the training samples into the initial graph data; putting the initial graph data into a graph neural network model for N times of training to obtain N loss values; taking the training model corresponding to the minimum loss value in the N loss values as a prediction model; learning and training the initial graph data through the above method to obtain the prediction model; and when matching in an input field, the prediction model can filter out the data element corresponding to the minimum loss value, thereby improving the accuracy of the result of field association.

Description

Technical Field

[0001] This application relates to the field of model training, and more particularly to a method, system, and electronic device for model training. Background Technology

[0002] With the advent of the big data era, data has become an important asset for enterprises. When the same type of data is analyzed in different enterprises, the evaluation criteria differ due to the different nature and needs of the enterprises. This results in different evaluation criteria for the same type of data. For example, for the same type of live-streaming e-commerce data, the evaluation criterion for media companies is whether the ratio of the number of fans who make purchases in the live stream to the number of fans in the live stream meets the expected ratio, while the evaluation criterion for financial companies is whether the revenue from the live stream meets the expected revenue.

[0003] Because a unified evaluation standard cannot be achieved, data from different companies cannot be compared and analyzed. To reduce the complexity of data analysis, a unified evaluation standard is needed. The approach is to benchmark the fields of the company's original data and establish a relationship between the fields and data elements. The fields are the names of the company's original data, and the data elements are the collective names of the fields. This links the company's original data into unified standard data. For example, if the fields of the company's original data are "landlord's ID number", "tenant's ID number", "resident ID card", and "temporary resident ID card", after linking the company's original data into unified standard data, the data element corresponding to the fields of the company's original data will be "ID number".

[0004] Currently, in order to associate fields of enterprise raw data with standard data, existing models are trained based on the semantic similarity between fields and data elements. Fields are the names of data in the enterprise's raw data. Due to the variety of field naming methods, including underscores, spaces, and mixed Chinese and English characters, the association between fields and data elements obtained by the trained model is inaccurate, or the model cannot obtain the data elements corresponding to the input fields. Summary of the Invention

[0005] This application provides a method, system, and electronic device for model training. By learning and training on initial atlas data, a prediction model is constructed. The prediction model stores a large amount of already matched data and potentially matched data. When predicting data elements of a field, the problem of fields not being able to match data elements is avoided. The optimal data elements are selected as the result output, thereby improving the accuracy of field association results.

[0006] Firstly, this application provides a method for model training, the method comprising:

[0007] Obtain standard data and raw data, establish a first association between the standard data and the raw data, and obtain training samples, wherein the raw data is a data table from each enterprise that has not undergone any processing;

[0008] Based on the semantic similarity between the standard data and the real data, a second association relationship is established between the standard data and the real data.

[0009] Based on the first and second association relationships, initial map data is obtained, and the training samples are placed into the initial map data;

[0010] The initial graph data is fed into a graph neural network model for N training iterations to obtain N loss values, where N is a positive integer;

[0011] The training model corresponding to the minimum loss value among the N loss values ​​is used as the prediction model, where the loss value represents the accuracy of the prediction result of the prediction model.

[0012] By using the method described above, multiple training models and their corresponding loss values ​​are obtained through iterative training of the initial map data. The training model with the minimum loss value is then selected as the prediction model. This prediction model is the one with the highest accuracy in the prediction results obtained during the training process. This avoids the problem of users' fields not matching data elements or the data elements matching the fields having weak correlation with the fields, thus improving the accuracy of the prediction results.

[0013] In one possible design, establishing a first association between the standard data and the original data includes:

[0014] Retrieve fields from the data table in the original data;

[0015] Data elements are obtained from the standard data;

[0016] The fields are associated with the data elements according to preset rules;

[0017] Based on the association between the field and the data element, a first association between the standard data and the original data is established.

[0018] In one possible design, a second association between the standard data and the real-world data is established based on the semantic similarity between the standard data and the real-world data, including:

[0019] Retrieve S real-world tables from the real-world data, where each real-world table consists of fields, and S is a positive integer;

[0020] Obtain M standard tables from the standard data, where each standard table consists of data elements and M is a positive integer;

[0021] The association between the reality table and the standard table is established based on the semantic similarity between the reality table and the standard table.

[0022] In one possible design, the association between the reality table and the standard table is established based on the semantic similarity between the reality table and the standard table, including:

[0023] Calculate the semantic similarity between each of the real-world tables and the M standard tables to obtain the values ​​of the M semantic similarity.

[0024] Sort the M semantic similarity values ​​according to the order of smallest to largest, and select the standard table corresponding to the largest semantic similarity value;

[0025] Establish an association between the real-world table and the standard table corresponding to the maximum semantic similarity value.

[0026] In one possible design, the initial graph data is fed into a graph neural network model for N training iterations, including:

[0027] Input the initial graph data into the graph neural network model for training to obtain the relation graph vector of each data node in the training graph data, where the relation graph represents the data node and other data nodes directly associated with the data node;

[0028] The specified relation graph vector is input into the classification model, and the standard data corresponding to the real data is obtained based on the classification model.

[0029] The training model is determined based on the real data and the standard data, and the loss value is determined based on the training model and the training samples.

[0030] The loss value is fed back to the graph neural network model each time and trained on the initial graph data.

[0031] In one possible design, specifying the relational graph vector includes:

[0032] From the vector of the relation graph, select relation graph vectors with Z fields and relation graph vectors with X data elements, where Z and X are positive integers.

[0033] In one possible design, before inputting the specified relation graph vector into the classification model and obtaining the standard data corresponding to the real-world data based on the classification model, the following steps are included:

[0034] The relationship graph vector of each field is combined with the relationship graph vectors of the X data elements to obtain X vector groups;

[0035] Calculate the existence probability value of each vector group to obtain X existence probability values, where the existence probability value is the probability that the field and the data element are related;

[0036] The association between the field and the data element is established based on the probability value of existence.

[0037] In one possible design, standard data corresponding to the real-world data is obtained based on the classification model, including:

[0038] Fields are obtained from the real-world data, and data elements are obtained from the standard data;

[0039] Based on the association between the field and the data element, the standard data corresponding to the real data is determined.

[0040] In one possible design, the association between the field and the data element is established based on the probability value of existence, including:

[0041] Sort the X probability values ​​in ascending order and select the value with the highest probability of existence.

[0042] Establish an association between the data element corresponding to the maximum probability value and the field.

[0043] In one possible design, the training model is determined based on the real-world data and the standard data, including:

[0044] Obtain the data element corresponding to the maximum existence probability value of each field, and obtain the association relationship between the Z fields and the data element;

[0045] The loss value is calculated based on the relationship between the fields and data elements and the difference between the fields and training samples.

[0046] The loss value is fed back to the graph neural network and classification model, and the parameters of the graph neural network and classification model are updated to determine the training model.

[0047] In one possible design, the training model corresponding to the minimum loss value among the N loss values ​​is used as the prediction model, including:

[0048] The initial atlas data is iteratively trained N times to obtain N training models;

[0049] N loss values ​​are obtained based on the N training models and the training samples;

[0050] Sort the N loss values ​​in ascending order and select the training model corresponding to the smallest loss value;

[0051] The training model corresponding to the minimum loss value is used as the prediction model.

[0052] Secondly, this application provides a data processing method, the method comprising:

[0053] Receive user-input data and classify the data;

[0054] If the data is a field, then the field is processed using the first mode;

[0055] If the data is a real table, then the real table is processed using the second mode.

[0056] In one possible design, receiving user input data and classifying the data includes:

[0057] Determine the data type based on the file format; or

[0058] The data type is determined based on whether the storage space occupied by the data exceeds a preset value; or...

[0059] Determine the data type based on whether it contains a table structure and the type of table.

[0060] In one possible design, the data type is determined based on the file format of the data, including:

[0061] Parse the received data to obtain the file format of the data;

[0062] If the file format is a table, then the data will be displayed as a table.

[0063] If the file format is text, then the data will be treated as a field.

[0064] In one possible design, the data type is determined based on whether the storage space occupied by the data exceeds a preset value, including:

[0065] Obtain the value corresponding to the storage space occupied by the data, and determine whether the value corresponding to the storage space exceeds a preset value;

[0066] If the value corresponding to the storage space is higher than the preset value, then the data will be used as the display table;

[0067] If the value corresponding to the storage space is lower than the preset value, then the data will be used as a field.

[0068] In one possible design, the data type is determined based on whether the data contains a table structure and the type of table, including:

[0069] Determine whether a table structure and a table are extracted from the data, wherein the table structure is the distribution of rows and columns;

[0070] If so, then the data will be used as the display table;

[0071] If not, then the data will be treated as a field.

[0072] In one possible design, the field is processed using the first pattern, including:

[0073] Remove special characters from the field and adjust the field format to the preset format;

[0074] Perform data benchmarking on the adjusted fields.

[0075] In one possible design, the reality table is processed using a second pattern, including:

[0076] Extract the text from the table, use the text as a field, and adjust the field to a preset format;

[0077] The adjusted field set is then input into the prediction model for data benchmarking.

[0078] Thirdly, this application provides a data benchmarking method, the method comprising:

[0079] When user input data is received, the data is input into the prediction model;

[0080] The training results obtained by inputting the data into the prediction model are used as data elements for output.

[0081] In one possible design, the training results obtained from the data input prediction model are used as data element outputs, including:

[0082] The data is matched with each data element in the prediction model to obtain the probability value of the existence between the data and each data element;

[0083] Sort the existence probability values ​​in descending order to obtain the data element corresponding to the highest existence probability value;

[0084] The data element is output as the data element corresponding to the data.

[0085] Fourthly, this application provides a system for training a model, the system comprising:

[0086] The acquisition module is used to acquire standard data and raw data, establish a first association between the standard data and the raw data, and obtain training samples;

[0087] A module is established to establish a second association between the standard data and the real data based on the semantic similarity between the standard data and the real data.

[0088] The association module is used to obtain initial map data based on the first association relationship and the second association relationship, and to put the training samples into the initial map data;

[0089] The training module is used to feed the initial atlas data into the graph neural network model for N training iterations to obtain N loss values;

[0090] The generation module is used to take the training model corresponding to the minimum loss value among the N loss values ​​as the prediction model.

[0091] Fifthly, this application provides a data processing system, the system comprising:

[0092] The classification module is used to receive user-input data and classify the data;

[0093] The processing module is configured to process the field using a first mode when the data is a field, and to process the table using a second mode when the data is a real table.

[0094] Sixthly, this application provides a data benchmarking system, the system comprising:

[0095] An input module is used to input data into the prediction model when it receives data input from the user;

[0096] The output module is used to output the training results obtained from the data input prediction model as data elements.

[0097] In a seventh aspect, this application provides an electronic device, comprising:

[0098] Memory, used to store computer programs;

[0099] When a processor executes a computer program stored in the memory, it implements the above-described method steps for model training, data processing, and data benchmarking.

[0100] Eighthly, a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method steps for model training, data processing, and data benchmarking.

[0101] For details on the various aspects from the first to the eighth aspects mentioned above, as well as the technical effects that each aspect may achieve, please refer to the description of the technical effects that various possible solutions for the first, second, and third aspects may achieve. These details will not be repeated here. Attached Figure Description

[0102] Figure 1 A flowchart of the steps of a model training method provided in this application;

[0103] Figure 2 A flowchart of the steps of a data processing method provided in this application;

[0104] Figure 3 A flowchart of the steps of a data benchmarking method provided in this application;

[0105] Figure 4 Standard data charts of the enterprise provided for this application;

[0106] Figure 5 Original data diagrams of the company provided for this application;

[0107] Figure 6 A diagram showing the relationship between the company's standard data and its original data provided for this application;

[0108] Figure 7 A diagram showing the relationship between the company's standard data and its actual data provided for this application;

[0109] Figure 8 A schematic diagram of the structure of a model training system provided in this application;

[0110] Figure 9 A schematic diagram of the structure of a data processing system provided in this application;

[0111] Figure 10 A schematic diagram of the structure of a data benchmarking system provided in this application;

[0112] Figure 11 This is a schematic diagram of the structure of an electronic device provided in this application. Detailed Implementation

[0113] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The specific operational methods in the method embodiments can also be applied to the system embodiments. It should be noted that in the description of this application, "multiple" is understood as "at least two". "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. A connected to B can represent: A and B directly connected, and A and B connected through C. Furthermore, in the description of this application, terms such as "first" and "second" are used only for distinguishing the purpose of description and should not be construed as indicating or implying relative importance or order.

[0114] Currently, when associating raw enterprise data with unified standard data, existing models are trained based on the semantic similarity between field names and data element names. This involves benchmarking the fields of the raw enterprise data against the data elements in the standard data to establish associations. However, due to diverse field naming conventions, including underscores, spaces, and mixed Chinese and English characters, the associations derived from the model are inaccurate, and the model cannot retrieve the corresponding data elements for input fields. This results in low accuracy of the field association results. Furthermore, the benchmarked raw enterprise data is not processed, leading to resource waste. Therefore, the inaccuracy of the associations derived from the model and the inability to retrieve the corresponding data elements for input fields are problems that need to be addressed.

[0115] To address the aforementioned problems, this application provides a model training method to obtain a predictive model, thereby improving the accuracy of field association results and avoiding the problem of fields failing to match corresponding data elements. The methods and systems described in this application are based on the same technical concept. Since the principles by which the methods and systems solve the problems are similar, embodiments of the systems and methods can be referred to interchangeably, and repeated details will not be elaborated further.

[0116] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0117] Example 1

[0118] Reference Figure 1 This application provides a method for model training, which can obtain a prediction model. This prediction model can improve the accuracy of field association results and avoid the problem of fields not matching corresponding data elements. The implementation process of this method is as follows:

[0119] Step S1: Obtain standard data and raw data, establish a first association between the standard data and the raw data, and obtain training samples.

[0120] The specific process of obtaining standard data in this application embodiment is as follows: Since national standards organizations have already formulated standard documents for relevant data elements, and various enterprises have professional databases related to data processing, the standard documents formulated by the national standards organizations and the professional databases of various enterprises are read. Standard tables, standard fields, and data elements specified in national standards and / or the data elements corresponding to fields in the professional databases are extracted from these standard documents and professional databases. Enterprise standard data is generated based on the extracted standard tables, standard fields, and data elements specified in national standards and / or the data elements corresponding to fields in the professional databases. At this time, the enterprise standard data has standard fields, standard tables, and data elements, such as... Figure 4The figure shows the standard data of the enterprise. In this embodiment of the application, the circle in the figure represents a node, and the line segment represents the relationship between two nodes. If the circle contains a data element, then the node is a data element node. The naming of other nodes is the same as that of the data element node, which will not be described one by one here.

[0121] It should be further noted that data elements and qualifiers can form standard fields, and standard fields can form standard tables.

[0122] For example, the data element can be "Resident ID Card", and the standard fields can be "Mainland Resident ID Card", "Temporary Resident ID Card", "Taiwan Resident ID Card", etc. The standard table composed of the standard fields is shown in Table 1:

[0123]

[0124] Table 1

[0125] The specific process for obtaining raw data in this application embodiment is as follows: Unprocessed data tables from various enterprises are obtained, and then the fields in the data tables can be extracted. Based on the obtained fields and data tables, raw enterprise data can be generated. At this point, the raw enterprise data includes fields and data tables. The raw enterprise data is as follows: Figure 5 As shown.

[0126] After obtaining the standard data and the original data, since the standard data and the original data are independent of each other, in order to establish the association between the standard data and the original data, it is necessary to obtain the data elements in the standard data and the fields in the original data. After obtaining the data elements and the fields, the association between the data elements and the fields is established according to the preset rules. The preset rules can be to determine whether the type of the field is consistent with that of the data element. If they are consistent, the association between the field and the data element is established; if they are inconsistent, no processing is performed.

[0127] After establishing the association between fields and data elements through the above steps, the association between standard data and original data is then established. The association between standard data and original data is as follows: Figure 6 As shown, the fields and data elements that establish the association are then placed into the training samples to obtain the training sample data. It should be noted that in the training samples, the fields and their corresponding data elements are stored in the form shown in Table 2:

[0128]

[0129] Table 2

[0130] Step S2: Based on the semantic similarity between the standard data and the real data, establish a second association between the standard data and the real data.

[0131] To obtain the initial map data, the raw data needs to be processed to obtain the actual data. Since the data tables in the raw data do not have a unified standard format, it is necessary to read the fields in the raw data and delete all special symbols in all fields. Special symbols can be one or more of emoticons, mathematical symbols, icon symbols, and underscores. After deleting all special symbols in all fields, the format of all fields is adjusted to a unified standard, and the processed raw data is used as the actual data.

[0132] For example: the original data is "_Xiaoming#'s ID card number ￥", after processing the original data it becomes "Xiaoming's ID card number", and the corresponding real data is "Xiaoming's ID card number".

[0133] After obtaining the real-world data, since the standard data and the real-world data are independent of each other, in order to establish a connection between the standard data and the enterprise's real-world data, it is necessary to obtain the standard tables in the standard data and the data tables in the real-world data, obtain the semantic similarity value between each data table and all standard tables, and establish a connection between the data table and the standard table corresponding to the maximum semantic similarity value, such as... Figure 7 As shown, the second correlation between standard data and real-world data is obtained.

[0134] Step S3: Obtain initial map data based on the first association relationship and the second association relationship, and put the training samples into the initial map data.

[0135] After obtaining the first correlation between standard data and original data and the second correlation between standard data and real data, the initial map data is obtained based on the established first and second correlations, and the training sample is placed into the initial map data. At this time, the initial map data contains the training sample, real data, and the standard data and real data with established correlations.

[0136] Step S4: Input the initial map data into the graph neural network model for N training iterations to obtain N loss values.

[0137] After obtaining the initial graph data, in order to obtain the prediction model, the initial graph data needs to be put into the graph neural network model for N training iterations, where N is a positive integer. In each training iteration of the graph neural network model on the initial graph data, the more layers of the graph neural network the initial graph data passes through, the more associated data will be obtained. In the embodiments of this application, the number of layers of the graph neural network in the graph neural network model can be set according to actual needs.

[0138] When the initial graph data is trained by the graph neural network, training graph data is obtained. Based on the training graph data, the relation graph corresponding to each node in the training graph data can be obtained. The relation graph represents the data node and other nodes directly related to the node. The node can be a data element node, a standard field node, a standard field set node, a data table node, or a field node. The relation graph of each node is converted into a vector and output from the graph neural network. At this time, each node corresponds to a vector.

[0139] After obtaining the vector of each node, extract the vector corresponding to the field node and the vector corresponding to the data element node. Concatenate the vector corresponding to the field node with the vector corresponding to each data element node to form a new vector. The number of new vectors is at least one. Calculate the existence probability value of each vector corresponding to the field based on the classification model and record the existence probability value. The existence probability value represents the probability that the field and the data element are related. In this embodiment, the classification model can be a multilayer perceptron. Extract and record the vector corresponding to the maximum existence probability value of each field. Obtain the data element corresponding to the field based on the vector. One field corresponds to one data element. When a field has multiple identical existence probability values, randomly select a loss value from the multiple maximum probability values ​​and record the association between the field and the loss value.

[0140] Following the above method, obtain the data elements corresponding to each field, and generate a training model based on the association between the record fields and the data elements. After generating the training model, it is necessary to calculate the loss value based on the difference between the data elements corresponding to the fields in the training samples and the training model. The loss value represents the accuracy of the prediction result of the prediction model.

[0141] After obtaining the loss value, the loss value is fed back to the graph neural network and classification model to update the parameters of the graph neural network and classification model. Based on the updated model, the initial graph data is trained again. The above process is repeated until N training cycles are completed, resulting in N trained models and N loss values.

[0142] Step S5: Use the training model corresponding to the minimum loss value among the N loss values ​​as the prediction model.

[0143] After obtaining N training models and N loss values, extract the training model corresponding to the minimum loss value from the N loss values, and save the training model as the prediction model.

[0144] By using the above method, a graph neural network is trained on the initial graph data. The more layers of the graph neural network, the more related data are obtained. At the same time, the graph neural network model is trained N times to obtain a sufficient number of training models and their corresponding accuracy. The prediction model is determined from the training models, ensuring that the obtained prediction model is the most accurate training model obtained during the training process, thereby improving the accuracy of field-related data elements.

[0145] By using the above method, the field is matched with each data element during the training process, ensuring that the field input by the user can match the data element, thus avoiding the problem of the user input field not matching the data element. Furthermore, the minimum loss value is selected from multiple loss values ​​and the data element corresponding to the minimum loss value is output as the result, ensuring that the data element obtained by the user is highly related to the user input field.

[0146] Based on the method provided in this application, standard data and raw data are obtained, a first association relationship between the standard data and the raw data is established, and training samples are obtained. A second association relationship between the standard data and the real-world data is established based on the semantic similarity between the standard data and the real-world data. Initial graph data is obtained based on the first association relationship and the second association relationship, and the training samples are placed into the initial graph data. The initial graph data is then placed into a graph neural network model for N training iterations to obtain N loss values. The training model corresponding to the minimum loss value among the N loss values ​​is used as the prediction model. By learning and training the initial graph data using the above method, a prediction model is obtained. When matching input fields, the data elements corresponding to the minimum loss value can be selected, improving the accuracy of data elements associated with fields.

[0147] Example 2

[0148] Reference Figure 2 This application provides a data processing method that can process an enterprise's raw data, thereby improving the accuracy of field association results and avoiding the problem of fields not matching corresponding data elements. The implementation process of this method is as follows:

[0149] Step S21: Receive user-inputted data and classify the data.

[0150] To distinguish between user input data and the displayed table, the user input data needs to be categorized. The specific categorization method is as follows:

[0151] Method 1: Determine the data type based on the file format.

[0152] After receiving the user input data, the system determines the file format of the data. If the data format is a document format, such as TXT or DOC, the user input data is used as a field. If the data format is a table format, such as xlsx, the user input data is used as a display table.

[0153] Method 2: Determine the data type based on whether the storage space occupied by the data exceeds a preset value.

[0154] After receiving the user input data, it checks whether the storage space occupied by the user input data exceeds a preset value. If the storage space occupied by the user input data exceeds the preset value, the user input data is used as a field. If the storage space occupied by the user input data is less than the preset value, the user input data is used as a display table. Since the preset value can be set according to the actual scenario, it will not be elaborated on here.

[0155] Method 3: Determine the data type based on whether the data contains a table structure and the type of table.

[0156] After receiving user input data, the system checks whether the user input data contains a table structure and a table. The table structure is the distribution of the rows and columns of a table. If a table structure and / or a table is detected in the user input data, the user input data is used as a display table. If no table structure or table is detected in the user input data, the user input data is used as a field.

[0157] In this embodiment of the application, the data input by the user can be classified in one of the methods described above, namely, method one, method two, and method three. Alternatively, method one, method two, and method three can be combined in any way, such as combining method one with method two, or combining method one, method two, and method three. Other combinations are described above and will not be elaborated on here.

[0158] If the user input data is a field, proceed to step S22; if the user input data is a display table, proceed to step S23.

[0159] The methods described above are used to classify user-input data, employing diverse classification approaches to ensure accuracy in data classification.

[0160] Step S22: Process the field using the first mode.

[0161] After the user inputs data as fields, since these fields are the company's original data and contain special characters, it is necessary to detect the special characters in the fields in order to process the text into the format set by the prediction model. The fields are then matched against a preset special character table, and the matched special characters are deleted from the fields. After deleting the special characters, the field format needs to be adjusted to the preset format in the prediction model, such as: field size 3, non-bold font, etc. Since the purpose of the preset format is to make the input fields consistent in format, the preset format will not be elaborated on here.

[0162] After the field format is changed to the preset format, the adjusted field is input into the prediction model for data benchmarking.

[0163] Step S23: Process the reality table using the second mode.

[0164] After the user inputs data into a display table, since the table's results are more complex than the fields, and the display table is composed of fields, it is necessary to process the display table. The text in the display table is extracted in order from top to bottom and from left to right. After obtaining the text in the display table, the text extracted from the display table is used as fields.

[0165] Since the extracted data is text, there are no special characters in this field. We only need to adjust the format of this field to the preset format in the prediction model, obtain the adjusted field, and then input this field into the prediction model for data benchmarking.

[0166] Example 3

[0167] Reference Figure 3 This application provides a data benchmarking method that can benchmark fields into data elements, avoiding the problem of fields not matching corresponding data elements. The implementation process of this method is as follows:

[0168] Step S31: Upon receiving user input data, input the data into the prediction model.

[0169] When user input data is received, since the data has already been categorized into fields and a real-world table, and the real-world table has been transformed into a multi-field format, it is only necessary to input the data into the prediction model.

[0170] Step S32: Input the data into the prediction model and use the training results obtained as data elements as output.

[0171] After data processing, the user inputs the data elements that need to be associated into the prediction model. The data will be matched with each data element in the prediction model to obtain the existence probability value corresponding to each data element. After obtaining the existence probability values, the existence probability values ​​are sorted in ascending order, and the highest existence probability value is selected. In this embodiment, when there is more than one highest existence probability value, one of the data elements corresponding to the highest existence probability value is arbitrarily selected as the data element that needs to be associated with the data, and the data element corresponding to the highest existence probability value is output as the training result. The user can then obtain the data elements associated with the input data through the prediction model, realizing the alignment between the user's input data and the data elements in the prediction model.

[0172] By using the method described above, the processed data is matched with the data elements in the prediction model, and the data elements corresponding to the highest probability values ​​are selected, thus ensuring the accuracy of the input data in matching the target data elements.

[0173] Example 4

[0174] Based on the same inventive concept, this application also provides a model training system, which implements the function of a model training method, as described above. Figure 8 The system includes:

[0175] The acquisition module 801 is used to acquire standard data and raw data, establish a first association between the standard data and the raw data, and obtain training samples;

[0176] A module 802 is established to establish a second association between the standard data and the real data based on the semantic similarity between the standard data and the real data.

[0177] The association module 803 is used to obtain initial map data based on the first association relationship and the second association relationship, and to put the training samples into the initial map data;

[0178] The training module 804 is used to feed the initial map data into the graph neural network model for N training iterations to obtain N loss values.

[0179] The generation module 805 is used to take the training model corresponding to the minimum loss value among the N loss values ​​as the prediction model.

[0180] In one possible design, the acquisition module 801 is specifically used to acquire fields from the data table in the original data, acquire data elements from the standard data, associate the fields with the data elements according to preset rules, and establish a first association relationship between the standard data and the original data based on the association relationship between the fields and the data elements.

[0181] In one possible design, the establishment module 802 is specifically used to obtain S real tables from the real data, obtain M standard tables from the standard data, and establish the association between the real tables and the standard tables based on the semantic similarity between the real tables and the standard tables.

[0182] In one possible design, the establishment module 802 is further configured to acquire S real tables from the real data, acquire M standard tables from the standard data, and establish the association between the real tables and the standard tables based on the semantic similarity between the real tables and the standard tables.

[0183] In one possible design, the training module 804 is specifically used to input initial graph data into a graph neural network model for training, obtain the relation graph vector of each data node in the training graph data, input the specified relation graph vector into a classification model, obtain standard data corresponding to the real data according to the classification model, determine the training model according to the real data and the standard data, determine the loss value according to the training model and the training samples, and feed back the loss value each time to the graph neural network model to train the initial graph data.

[0184] In one possible design, the training module 804 is further configured to combine the relation graph vector of each field with the relation graph vector of the X data elements to obtain X vector groups, calculate the existence probability value of each vector group to obtain X existence probability values, and establish the association relationship between the field and the data elements based on the existence probability values.

[0185] In one possible design, the training module is further configured to obtain the data element corresponding to the maximum existence probability value of each field, obtain the association relationship between the Z fields and the data element, and determine the training model based on the association relationship.

[0186] In one possible design, the training module 804 is further configured to iteratively train the initial atlas data N times to obtain N training models, obtain N loss values ​​based on the N training models and the training samples, sort the N loss values ​​in ascending order, select the training model corresponding to the minimum loss value, and use the training model corresponding to the minimum loss value as the prediction model.

[0187] Example 5

[0188] Based on the same inventive concept, this application also provides a data processing system. This model training system implements the functionality of a data processing method. (Refer to...) Figure 9 The system includes:

[0189] The classification module 901 is used to receive data input by the user and classify the data;

[0190] The processing module 902 is used to process the field using a first mode when the data is a field, and to process the real table using a second mode when the data is a real table.

[0191] In one possible design, the classification module 901 is specifically used to determine the type of data based on the file format of the data, or to determine the type of data based on whether the storage space occupied by the data exceeds a preset value, or to determine the type of data based on whether the data contains a table structure and the table.

[0192] In one possible design, the classification module 901 is further configured to parse the received data, obtain the file format of the data, and if the file format is a table, then the data is used as a display table; if the file format is text, then the data is used as a field.

[0193] In one possible design, the classification module 901 is further configured to obtain the value corresponding to the storage space occupied by the data, determine whether the value corresponding to the storage space exceeds a preset value, and if the value corresponding to the storage space is higher than the preset value, then the data is used as a display table; if the value corresponding to the storage space is lower than the preset value, then the data is used as a field.

[0194] In one possible design, the classification module 901 is further used to determine whether a table structure and a table are extracted from the data. If so, the data is used as a display table; otherwise, the data is used as a field.

[0195] In one possible design, the processing module 902 is specifically used to delete special characters in the field and adjust the format of the field to a preset format, and perform data alignment processing on the adjusted field.

[0196] In one possible design, the processing module 902 is further configured to extract text from the real table, use the text as a field, adjust the field to a preset format, and input the adjusted field set into the prediction model for data benchmarking processing.

[0197] Example 6

[0198] Based on the same inventive concept, this application also provides a data benchmarking system. This model training system implements the functionality of a data benchmarking method. (Refer to...) Figure 10 The system includes:

[0199] The input module 1001 is used to input the data into the prediction model when it receives data input from the user;

[0200] The output module 1002 is used to output the training results obtained from the data input prediction model as data elements.

[0201] In one possible design, the output module 1002 is specifically used to match the data with each data element in the prediction model to obtain the existence probability value between the data and each data element, sort the existence probability values ​​in descending order, obtain the data element corresponding to the maximum existence probability value, and output the data element as the data element corresponding to the data.

[0202] Example 7

[0203] Based on the same inventive concept, this application also provides an electronic device that can realize the functions of the aforementioned model training system, data processing system, and data benchmarking system. (See reference...) Figure 11 The electronic device includes:

[0204] At least one processor 1101 and a memory 1102 connected to at least one processor 1101. In this embodiment, the specific connection medium between the processor 1101 and the memory 1102 is not limited. Figure 11 The example shown is the connection between processor 1101 and memory 1102 via bus 1100. Bus 1100 is... Figure 11 The connections between other components are shown in thick lines only and are not intended to be limiting. Bus 1100 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 11 The term 1101 is represented by a single thick line, but this does not imply that there is only one bus or one type of bus. Alternatively, the processor 1101 may also be referred to as a controller; there is no restriction on the name.

[0205] In this embodiment, memory 1102 stores instructions executable by at least one processor 1101. By executing the instructions stored in memory 1102, at least one processor 1101 can perform the model training method, data processing method, and data benchmarking method described above. Processor 1101 can implement... Figure 8 , Figure 9 as well as Figure 10 The system shown illustrates the functions of each module.

[0206] The processor 1101 is the control center of the system. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 1102 and calling data stored in memory 1102, it can monitor the various functions and data processing of the system as a whole.

[0207] In one possible design, processor 1101 may include one or more processing units. Processor 1101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 1101. In some embodiments, processor 1101 and memory 1102 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0208] Processor 1101 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The method steps for model training, data processing, and data benchmarking disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0209] Memory 1102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 1102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 1102 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 1102 can also be a circuit or any other system capable of implementing storage functions for storing program instructions and / or data.

[0210] By designing and programming the processor 1101, the code corresponding to the model training method, the data processing method, and the data benchmarking method described in the foregoing embodiments can be embedded into the chip, thereby enabling the chip to execute these methods during operation. Figure 1 The illustrated embodiment includes a model training step. Figure 2 The illustrated embodiment includes a data processing method and steps. Figure 3 The illustrated embodiment presents a method for data benchmarking. How to design and program the processor 1101 is a technique well-known to those skilled in the art and will not be described further here.

[0211] Based on the same inventive concept, embodiments of this application also provide a storage medium storing computer instructions that, when executed on a computer, cause the computer to perform a model training method, a data processing method, and a data benchmarking method described above.

[0212] In some possible implementations, various aspects of the model training method, data processing method, and data benchmarking method provided by this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a system, the program code is used to cause the control device to perform the steps of the model training method, data processing method, and data benchmarking method described above according to various exemplary embodiments of this application.

[0213] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A system that specifies functions in one or more boxes.

[0215] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction set implemented in a process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0217] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for model training, the method comprising: The method comprises the following steps: obtaining standard data and original data, establishing a first association relationship between the standard data and the original data, and obtaining a training sample, wherein the original data is a data table in each enterprise without any processing; establishing a second association relationship between the standard data and the real data according to the semantic similarity between the standard data and the real data, wherein the real data is the original data after standardization processing; obtaining initial graph data according to the first association relationship and the second association relationship, and putting the training sample into the initial graph data; putting the initial graph data into a graph neural network model for N times of training to obtain N loss values, wherein N is a positive integer; taking the training model corresponding to the minimum loss value in the N loss values as a prediction model, wherein the loss value reflects the prediction result accuracy of the prediction model.

2. The method of claim 1, wherein, The first association relationship between the standard data and the original data is established, comprising: obtaining a field from a data table in the original data, wherein the data table is composed of fields; obtaining a data element from the standard data, wherein the standard data is composed of data elements and standard tables; associating the field with the data element according to a preset rule; establishing the first association relationship between the standard data and the original data according to the association relationship between the field and the data element.

3. The method of claim 1, wherein, The second association relationship between the standard data and the real data is established according to the semantic similarity between the standard data and the real data, comprising: obtaining S real tables in the real data, wherein the real table is composed of fields, and S is a positive integer; obtaining M standard tables in the standard data, wherein the standard table is composed of data elements, and M is a positive integer; establishing the association between the real table and the standard table according to the semantic similarity between the real table and the standard table.

4. The method of claim 3, wherein, The association between the real table and the standard table is established according to the semantic similarity between the real table and the standard table, comprising: calculating the semantic similarity between each real table and M standard tables to obtain M semantic similarity values; sorting the M semantic similarity values in ascending order, and screening out the standard table corresponding to the maximum semantic similarity value; establishing the association between the real table and the standard table corresponding to the maximum semantic similarity value.

5. The method of claim 1, wherein, The initial graph data is put into the graph neural network model for N times of training, comprising: inputting the initial graph data into the graph neural network model for training to obtain a relationship graph vector of each data node in the training graph data, wherein the relationship graph represents the data node and other data nodes directly associated with the data node; inputting the specified relationship graph vector into a classification model to obtain the standard data corresponding to the real data according to the classification model; determining a training model according to the real data and the standard data, and determining a loss value according to the training model and the training sample; feeding back the loss value of each time to the graph neural network model and training the initial graph data.

6. The method of claim 5, wherein, The relationship graph vector is specified, comprising: Screening a relationship graph vector of Z fields and a relationship graph vector of X data elements from the vector of the relationship graph, wherein Z and X are positive integers.

7. The method of claim 6, wherein, Before inputting the designated relationship graph vector into a classification model, the method further comprises: Combining the relationship graph vector of each field with the relationship graph vector of the X data elements to obtain X vector groups; Calculating a presence probability value of each vector group to obtain X presence probability values, wherein the presence probability value is a probability of a field being associated with a data element; Establishing an association between the field and the data element according to the presence probability value.

8. The method of claim 7, wherein, According to the classification model, the method further comprises: Obtaining a field from the real data and a data element from the standard data; According to the association between the field and the data element, determining the standard data corresponding to the real data.

9. The method of claim 7, wherein, According to the presence probability value, the method further comprises: Sorting the X presence probability values in descending order to screen a maximum presence probability value; Establishing an association between the data element corresponding to the maximum presence probability value and the field.

10. The method of claim 6, wherein, According to the real data and the standard data, the method further comprises: Obtaining the association between the Z fields and the data elements by obtaining the data element corresponding to the maximum presence probability value of each field; According to the difference between the association between the field and the data element and the training sample, calculating a loss value; Feeding the loss value back to the graph neural network and the classification model and updating the parameters of the graph neural network and the classification model to determine the training model.

11. The method of claim 1, wherein, According to the N training models and the training sample, the method further comprises: Iteratively training the initial graph data N times to obtain N training models; According to the N training models and the training sample, the method further comprises: Sorting the N loss values in descending order to screen a training model corresponding to a minimum loss value; Taking the training model corresponding to the minimum loss value as the prediction model.

12. A method of data processing, characterized by, The method further comprises: Receiving user input data and classifying the data; If the data is a field, processing the field in a first mode, inputting the processed field into a prediction model for data benchmarking, wherein the prediction model is a prediction model trained by the method for training a model in any one of claims 1-11; If the data is a real table, processing the real table in a second mode, extracting text from the processed real table as a field, and inputting the adjusted field into a prediction model for data benchmarking, wherein the prediction model is a prediction model trained by the method for training a model in any one of claims 1-11.

13. The method of claim 12, wherein, The method further comprises: Determining the type of data according to the file format of the data; or Determining the type of data according to whether the storage space occupied by the data exceeds a preset value; or Determining the type of data according to whether the data contains a table structure and a table.

14. The method of claim 12, wherein, The method further comprises: Determining the type of data according to the file format of the data. analyzing the received data to obtain a file format of the data; if the file format is a table, regarding the data as a real table; if the file format is text, regarding the data as a field.

15. The method of claim 12, wherein, judging the type of the data according to whether the storage space occupied by the data exceeds a preset value, comprising: obtaining a value corresponding to the storage space occupied by the data, and judging whether the value corresponding to the storage space exceeds a preset value; if the value corresponding to the storage space is higher than the preset value, regarding the data as a real table; if the value corresponding to the storage space is lower than the preset value, regarding the data as a field.

16. The method of claim 12, wherein, judging the type of the data according to whether the data contains a table structure and a table, comprising: judging whether a table structure and a table are extracted from the data, wherein the table structure is the distribution of row and column lines; if yes, regarding the data as a real table; if no, regarding the data as a field.

17. The method of claim 12, wherein, processing the field in a first mode, comprising: deleting special characters in the field and adjusting the format of the field to a preset format; performing data matching processing on the field after adjustment.

18. The method of claim 12, wherein, processing the real table in a second mode, comprising: extracting the text in the real table, regarding the text as a field and adjusting the field to a preset format; inputting the adjusted field set into a prediction model for data matching processing.

19. A method of data targeting, the method comprising: comprising: when receiving data input by a user, inputting the data into the prediction model, wherein the prediction model is a prediction model trained by the method of any one of claims 1-11; regarding the training result obtained by inputting the data into the prediction model as a data element output.

20. The method of claim 19, wherein, regarding the training result obtained by inputting the data into the prediction model as a data element output, comprising: matching the data with each data element in the prediction model to obtain an existence probability value between the data and each data element; sorting the existence probability values in descending order to obtain a data element corresponding to the maximum existence probability value; regarding the data element as the data corresponding data element output.

21. A model training system, comprising: the system comprises: an acquisition module configured to acquire standard data and original data, establish a first association relationship between the standard data and the original data, and obtain a training sample, wherein the original data is a data table in each enterprise that has not been processed in any way; an establishment module configured to establish a second association relationship between the standard data and real data according to the semantic similarity between the standard data and the real data, wherein the real data is the original data after standardization processing; an association module configured to obtain initial graph data according to the first association relationship and the second association relationship, and put the training sample into the initial graph data; a training module configured to put the initial graph data into a graph neural network model for N times of training to obtain N loss values; a generation module configured to regard a training model corresponding to the minimum loss value among the N loss values as a prediction model.

22. A data-to-target system, comprising: the system comprises: An input module, configured to input data input by a user into the prediction model, wherein the prediction model is a prediction model trained by the method of training the model of any one of claims 1-11; An output module, configured to output a training result of the prediction model as a data element.

23. An electronic device, comprising: The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps in the method of any one of claims 1-20. The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps in the method of any one of claims 1-20. The computer program is stored in the computer readable storage medium and is executed by the processor to implement the steps in the method of any one of claims 1-20.

24. A computer-readable storage medium, characterized in that, ​

Images