Table data selection method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a table data selection method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the field of financial science and technology. The method comprises the following steps: acquiring initial table data and a data category label; calculating first loss data of the initial table data based on a first category prediction model; dividing the initial table data based on the first loss data to obtain basic table data and candidate table data, and taking the first loss data of the candidate table data as candidate loss data; calculating second loss data of the candidate table data based on a second category prediction model; performing batch data selection on the candidate table data based on the candidate loss data, a candidate loss difference value of the second loss data difference and the basic table data to obtain batch selection data; and combining the batch selection data and the basic table data to obtain a target data set. According to the embodiment of the invention, the table data selection accuracy and efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, is applicable to the field of financial technology, and in particular to a table data selection method and device, electronic device, and storage medium. Background Art

[0002] Tabular data refers to data organized into rows and columns for easy viewing and processing. It can be numbers, text, or other types of information. By accurately selecting tabular data, that is, removing impurity information in tabular data, such as non-tabular data and meaningless text data, high-quality data can be provided for subsequent data training and analysis. For example, in the FinTech scenario, tabular data can be insurance data, financial transaction data, and object data. By accurately and efficiently selecting tabular data, the accuracy and efficiency of business decisions can be improved.

[0003] At present, related technologies usually use methods such as rule-based tabular data screening, machine learning, or only a certain deep model to select data. However, rule-based tabular data screening has low processing efficiency for flexible and changeable data, and methods such as machine learning or deep models also have shortcomings. For example, when the model used is relatively complex, although the accuracy is high, it is easy to overfit, and it is not easy to accurately remove tabular data containing noise; when the model used is relatively simple, it is easy to underfit, so it is easy to exclude a large amount of correct tabular data, making the selection of tabular data less accurate. Therefore, how to improve the accuracy and efficiency of selecting tabular data has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a table data selection method and device, electronic device, and storage medium, aiming to improve the accuracy and efficiency of selecting table data.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a table data selection method, the method comprising:

[0006] Obtaining initial table data and data category labels of the initial table data;

[0007] Based on the first category prediction model, the loss of the initial table data is calculated to obtain first loss data; wherein the first loss data is used to indicate the degree of difference between the first prediction label and the data category label, and the first prediction label is a label obtained by predicting the initial table data based on the first category prediction model;

[0008] Based on the first loss data, the initial table data is divided into data to obtain basic table data and candidate table data, and the first loss data of the candidate table data is acquired to obtain candidate loss data;

[0009] Performing loss calculation on the candidate table data based on the second category prediction model to obtain second loss data; wherein the second loss data is used to indicate the degree of difference between the second prediction label and the data category label, the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is less than the model complexity of the first category prediction model;

[0010] Calculating the difference between the candidate loss data and the second loss data to obtain a candidate loss difference of the candidate table data;

[0011] Performing batch data selection on the candidate table data based on the candidate loss difference and the basic table data to obtain batch selected data;

[0012] Data is combined based on the batch selected data and the basic table data to obtain a target data set.

[0013] In some embodiments, the step of performing batch data selection on the candidate table data based on the candidate loss difference and the basic table data to obtain batch selected data includes:

[0014] For the first screening of the preset screening batch, data similarity calculation is performed on the candidate table data and the basic table data to obtain a first data similarity;

[0015] selecting first comparison table data from the basic table data based on the first data similarity;

[0016] Perform data detection based on the first comparison table data and the candidate loss difference to obtain a candidate detection score for the candidate table data;

[0017] Performing data selection on the candidate table data based on the candidate detection score and the preset screening batch to obtain the first batch selected sub-data and the first batch remaining sub-data of the first screening;

[0018] Combining the first batch selected sub-data and the basic table data to obtain a first screened data set, wherein the first screened data set includes first selected table data;

[0019] For the second screening of the preset screening batch, data selection is performed on the first batch remaining sub-data based on the preset screening batch, the candidate loss difference and the first selection table data to obtain second batch selected sub-data;

[0020] The batch selection data is determined based on the first batch selection sub-data, the second batch selection sub-data, and the preset screening batch.

[0021] In some embodiments, the data selection of the candidate table data based on the candidate detection score and the preset screening batch to obtain the first batch selected sub-data and the first batch remaining sub-data of the first screening includes:

[0022] Calculate the ratio of the total number of the basic table data to the preset screening batches to obtain the number of batch sampling;

[0023] Performing data sampling on the candidate table data based on the batch sampling quantity and the candidate detection score to obtain candidate sampling data;

[0024] Determining a first screening data amount based on the preset screening batch, the preset filtering ratio and the total amount of the basic table data;

[0025] Performing data selection on the candidate sampling data based on the candidate detection score and the first screening data volume to obtain the first batch selected sub-data of the first screening;

[0026] The first batch selected sub-data are eliminated based on the candidate sampling data to determine the first batch remaining sub-data.

[0027] In some embodiments, the performing data detection based on the first comparison table data and the candidate loss difference to obtain a candidate detection score for the candidate table data includes:

[0028] Combining any two of the first comparison table data to obtain a comparison data combination, wherein the comparison data combination includes the first combination table data and the second combination table data;

[0029] Performing data encoding on the first combined table data to obtain a first data vector;

[0030] Performing data encoding on the second combined table data to obtain a second data vector;

[0031] Calculating similarity between the first data vector and the second data vector to obtain vector similarity;

[0032] The candidate detection score of the candidate table data is determined based on the vector similarity and the candidate loss difference.

[0033] In some embodiments, determining the candidate detection score of the candidate table data based on the vector similarity and the candidate loss difference includes:

[0034] Performing vector similarity mean calculation based on the number of combinations of the comparison data combinations and the vector similarities to obtain a combination vector similarity mean;

[0035] A weighted calculation is performed on the combined vector similarity mean and the candidate loss difference to obtain the candidate detection score of the candidate table data.

[0036] In some embodiments, before performing loss calculation on the initial table data based on the first category prediction model to obtain the first loss data, the method further includes: constructing the first category prediction model, specifically including:

[0037] Acquire an initial training set, wherein the initial training set includes sample table data having preset sample categories;

[0038] Based on the initial neural network model, data category prediction is performed on the sample table data to obtain a category prediction probability, where the category prediction probability is used to represent the probability value that the initial neural network model predicts the category of the sample table data as the preset sample category in one training cycle;

[0039] Performing data selection on the sample table data based on the category prediction probability to obtain screened sample data;

[0040] The initial neural network model is trained based on the screened sample data to obtain the first category prediction model.

[0041] In some embodiments, the step of selecting the sample table data based on the category prediction probability to obtain the screened sample data includes:

[0042] Calculating the mean of the predicted probabilities of the categories for a preset number of training cycles to obtain a predicted probability mean;

[0043] Performing average information entropy calculation based on the preset number of training cycles and the category prediction probability to obtain sample average information entropy;

[0044] Calculate the difference between the predicted probability mean and the sample average information entropy to obtain the probability information difference of the sample table data;

[0045] The sample table data is selected based on the probability information difference to obtain filtered sample data.

[0046] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a table data selection device, the device comprising:

[0047] An acquisition module, used for acquiring initial table data and data category labels of the initial table data;

[0048] A first loss calculation module, configured to perform loss calculation on the initial table data based on a first category prediction model to obtain first loss data; wherein the first loss data is used to indicate the degree of difference between a first prediction label and the data category label, the first prediction label being a label obtained by predicting the initial table data based on the first category prediction model;

[0049] a data partitioning module, configured to partition the initial table data based on the first loss data to obtain basic table data and candidate table data, and to obtain the first loss data of the candidate table data to obtain candidate loss data;

[0050] A second loss calculation module, used to perform loss calculation on the candidate table data based on a second category prediction model to obtain second loss data; wherein the second loss data is used to indicate the degree of difference between a second prediction label and the data category label, the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is less than the model complexity of the first category prediction model;

[0051] A loss difference calculation module, used for performing difference calculation on the candidate loss data and the second loss data to obtain a candidate loss difference of the candidate table data;

[0052] A data selection module, configured to select the candidate table data in batches based on the candidate loss difference and the basic table data to obtain batch selected data;

[0053] The data combination module is used to combine the data based on the batch selected data and the basic table data to obtain a target data set.

[0054] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements a table data selection method described in the first aspect when executing the computer program.

[0055] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the table data selection method described in the first aspect is implemented.

[0056] The present application proposes a table data selection method and device, electronic device, and storage medium, which obtains initial table data and a data category label of the initial table data; and calculates the loss of the initial table data based on a first category prediction model to obtain first loss data; wherein the first loss data is used to indicate the degree of difference between the first prediction label and the data category label, and the first prediction label is a label obtained by predicting the initial table data based on the first category prediction model; further, the initial table data is divided into basic table data and candidate table data based on the first loss data, and the first loss data of the candidate table data is obtained to obtain the candidate loss data; the candidate data is divided into the first prediction label and the candidate table data based on the second category prediction model; Select table data for loss calculation to obtain second loss data; wherein the second loss data is used to indicate the degree of difference between the second prediction label and the data category label, the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is less than the model complexity of the first category prediction model; further, the candidate loss data and the second loss data are calculated to obtain the candidate loss difference of the candidate table data; and the candidate table data is selected in batches based on the candidate loss difference and the basic table data to obtain the batch selected data; further, the data is combined based on the batch selected data and the basic table data to obtain the target data set. Compared with the related art that uses rule-based table data screening, single machine learning or deep model methods to select data, the present application can first screen the table data based on the loss value, and then calculate the difference between the loss values ​​obtained by processing the candidate table data based on different models to determine the difference changes in the prediction of the table data on different models, thereby reducing the data's dependence on a single model. And the candidate table data is selected in batches based on the candidate loss difference and the basic table data, which can increase the diversity and accuracy of data selection. Therefore, the present application processes data by loss difference and batch data selection, which can effectively improve the accuracy and efficiency of selecting table data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 is a flow chart of a table data selection method provided in an embodiment of the present application;

[0058] Figure 2 This is an optional flow chart for constructing a first category prediction model in an embodiment of the present application;

[0059] Figure 3 yes Figure 2 A flowchart of step S230 in FIG.

[0060] Figure 4 yes Figure 1 A flowchart of step S160 in FIG.

[0061] Figure 5 yes Figure 4 A flowchart of step S430 in FIG.

[0062] Figure 6 yes Figure 5 A flowchart of step S550 in FIG.

[0063] Figure 7 yes Figure 4 A flowchart of step S440 in FIG.

[0064] Figure 8 is a structural diagram of a table data selection device provided in an embodiment of the present application;

[0065] Fig. 9 It is a hardware structure diagram of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0067] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0068] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0069] First, some nouns involved in this application are analyzed:

[0070] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0071] Tabular Neural Network (TabNet): is a neural network model specially designed to process tabular data. TabNet is designed to combine the advantages of neural networks and decision trees. It has both the end-to-end learning and representation learning capabilities of neural networks and the interpretability and sparse feature selection capabilities of decision trees.

[0072] Tabular data refers to data organized into rows and columns for easy viewing and processing. It can be numbers, text, or other types of information. By accurately selecting tabular data, that is, removing impurity information in tabular data, such as non-tabular data and meaningless text data, high-quality data can be provided for subsequent data training and analysis. For example, in the FinTech scenario, tabular data can be insurance data, financial transaction data, and object data. By accurately and efficiently selecting tabular data, the accuracy and efficiency of business decisions can be improved.

[0073] At present, related technologies usually use methods such as rule-based tabular data screening, statistical analysis, machine learning (such as AllSH, CleanLab and Conf_Agr) or only a certain deep model to judge the quality of data. However, rule-based tabular data screening has low processing efficiency for flexible and changeable data, and methods such as machine learning or deep models also have shortcomings. For example, when the model used is relatively complex, although the accuracy is high, it is easy to overfit, and the confidence in the classification of noise data is also high, and it is not easy to accurately remove tabular data containing noise; when the model used is relatively simple, it is easy to underfit, and the confidence in the classification of noise data will be low, which makes it easy to exclude a large amount of correct tabular data, making the selection of tabular data less accurate. Therefore, how to improve the accuracy and efficiency of selecting tabular data has become a technical problem that needs to be solved urgently.

[0074] Based on this, the embodiments of the present application provide a table data selection method and device, an electronic device, and a storage medium, which can improve the accuracy and efficiency of selecting table data.

[0075] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0077] The table data selection method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The table data selection method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN) and big data and artificial intelligence platforms and other basic cloud computing services; the software can be an application that implements the table data selection method, etc., but is not limited to the above forms.

[0078] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0079] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as the object's insurance information, the object's identity information, etc., the object's permission or consent will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. In addition, when the embodiment of the present application needs to obtain the sensitive personal information of the object, the object's separate permission or separate consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the object's separate permission or separate consent, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.

[0080] See also Figure 1 , Figure 1 is an optional flow chart of the table data selection method provided in the embodiment of the present application. In some embodiments of the present application, Figure 1 The method may include but is not limited to steps S110 to S170.

[0081] Step S110, obtaining initial table data and data category labels of the initial table data;

[0082] Step S120, performing loss calculation on the initial table data based on the first category prediction model to obtain first loss data;

[0083] Step S130, dividing the initial table data based on the first loss data to obtain basic table data and candidate table data, and acquiring first loss data of the candidate table data to obtain candidate loss data;

[0084] Step S140, performing loss calculation on the candidate table data based on the second category prediction model to obtain second loss data;

[0085] Step S150, performing a difference calculation on the candidate loss data and the second loss data to obtain a candidate loss difference of the candidate table data;

[0086] Step S160, performing batch data selection on the candidate table data based on the candidate loss difference and the basic table data to obtain batch selected data;

[0087] Step S170 , combining the batch selected data and the basic table data to obtain a target data set.

[0088] In step S110 of some embodiments, the initial table data is an instance of raw data stored in a table data format, and the initial table data may be text, images, documents, etc., without specific limitation. The data category label is a real identifier of the category to which the initial table data belongs, and can be used to evaluate the prediction accuracy of the subsequent prediction model for the initial table data. For example, in the insurance field of financial technology, the initial table data may be the asset information of the insured person, and the data category label corresponding to the initial table data is the asset risk category of the insured person. For another example, the initial table data may be a claim certificate document of the insured person, and the data category label corresponding to the initial table data is the certificate document type.

[0089] It should be noted that the initial table data may include impurity data (i.e., noise data), i.e., data irrelevant to classification, such as non-tabular data, meaningless text data, etc., and these data are easily misclassified into a certain category, thus affecting the accurate use of the data under that category. For example, when using the data under that category to make business decisions, if there is a large amount of impurity data in the data under the selected category, it is easy to affect the accuracy and efficiency of the business decision.

[0090] In step S120 of some embodiments, when the initial table data is screened, the present application may first use the first category prediction model to perform loss calculation on each initial table data to obtain the first loss data corresponding to each initial table data. Specifically, the first category prediction model may be used to first perform data category prediction on each initial table data to obtain a first prediction label; based on the first prediction label and the data category label, the loss calculation is performed to obtain the first loss data. Based on this, the first loss data is used to indicate the degree of difference between the first prediction label and the data category label, and the first prediction label is a label obtained by predicting the initial table data based on the first category prediction model.

[0091] It should be noted that the first category prediction model of the present application is equivalent to a reference model, and can be constructed using a model structure with higher model complexity and better prediction effect (and this higher complexity can be equivalent to the second category prediction model, mainly reflecting two prediction models with obvious differences). For example, the first category prediction model can adopt the TabNet model, Mambular model, etc., without limitation.

[0092] It should be noted that loss calculation refers to the use of a loss function to measure the difference between the model's predicted value and the true value. The loss function is a mathematical function that can take the model's predicted value (such as the first predicted label) and the true value (data category label) as input, and output a non-negative real number (such as the first loss data) to represent the loss, that is, the "distance" or difference between the predicted value and the true value, and the function used in this application to calculate the loss data can be a cross entropy loss function, a mean square error loss function, a mean absolute error loss, etc., without limitation. Therefore, the result calculated by the loss function quantifies the degree of difference between the first predicted label and the data category label, and can reflect the prediction accuracy of the first category prediction model.

[0093] See also Figure 2 , Figure 2 It is an optional flow chart for constructing a first category prediction model provided by an embodiment of the present application. In some embodiments of the present application, before step S120, the table data selection method provided by the embodiment of the present application also includes: constructing a first category prediction model, and this step may specifically include but is not limited to steps S210 to S240.

[0094] Step S210, obtaining an initial training set;

[0095] Step S220, predicting the data category of the sample table data based on the initial neural network model to obtain the category prediction probability;

[0096] Step S230, selecting the sample table data based on the category prediction probability to obtain filtered sample data;

[0097] Step S240, training the initial neural network model based on the screened sample data to obtain a first category prediction model.

[0098] In step S210 of some embodiments, since the first category prediction model of the present application can be constructed using a neural network model with higher model complexity and higher effect, a high-quality data set can be selected to train it, thereby providing a more stable reference for the second category prediction model. Based on this, the present application can first obtain an initial training set containing impurities, and the initial training set includes a plurality of sample table data with preset sample categories. The preset sample category is the same as the above-mentioned data category label category, and the specific form of the sample table data is the same as the above-mentioned initial table data, which will not be repeated. In addition, the initial training set and the original data set containing multiple initial table data can be of the same type, but are not in the original data set, so as to train the first category prediction model to different degrees.

[0099] In step S220 of some embodiments, the model structure of the initial neural network model and the first category prediction model is the same, and is the model structure before the first category prediction model is trained. The present application can perform data category prediction on the sample table data based on the initial neural network model to obtain multiple category prediction probabilities, and each category prediction probability is used to characterize the probability value (i.e., confidence) that the initial neural network model predicts the category of the sample table data as a preset sample category in one training cycle (i.e., epoch, an epoch refers to a complete traversal and learning of the entire initial training set by the model).

[0100] In step S230 of some embodiments, the present application may further select sample table data based on the category prediction probability to obtain screened sample data. Such screening can help improve the quality of subsequent training data because these samples provide clearer learning signals.

[0101] See also Figure 3 , Figure 3 It is an optional flow chart of step S230 provided in an embodiment of the present application. In some embodiments of the present application, step S230 may specifically include but is not limited to steps S310 to S340.

[0102] Step S310, calculating the mean of the category prediction probabilities of a preset number of training cycles to obtain the mean of the prediction probabilities;

[0103] Step S320, calculating the average information entropy based on the preset number of training cycles and the category prediction probability to obtain the sample average information entropy;

[0104] Step S330, performing difference calculation on the predicted probability mean and the sample average information entropy to obtain the probability information difference of the sample table data;

[0105] Step S340, selecting the sample table data based on the probability information difference to obtain filtered sample data.

[0106] In step S310 of some embodiments, the present application can implement an automated data screening method based on the difference in confidence and uncertainty, so as to screen k0 high-quality screened sample data with the largest difference in confidence and uncertainty from the initial training set to construct a high-quality data set (High_data). Specifically, the present application can first calculate the mean of the category prediction probabilities of a preset number of training cycles to obtain the mean of the prediction probabilities. The preset number of training cycles refers to the total number of training cycles required to train the initial neural network model, such as 10, 50, etc., without limitation.

[0107] Based on this, the average calculation of the category prediction probabilities obtained for the same sample table data in multiple training cycles can reflect the overall trend or stability of the initial neural network model's prediction of the sample category. In addition, the predicted probability mean can also represent the data confidence of the corresponding sample table data. The higher the data confidence, the better the model can accurately predict the data to the correct type, and it also means that the data can be more easily learned in subsequent practical applications, and the quality is better. Therefore, the specific calculation process of the predicted probability mean of each sample table data can be seen in the following formula 1:

[0108]

[0109] Where x represents the input of the sample table data, P(x) represents the predicted probability mean of the sample table data, E represents the preset number of training cycles, and P e (x) represents the category prediction probability that the initial neural network model predicts the category of the sample table data as the preset sample category in the e-th training cycle.

[0110] In step S320 of some embodiments, further, the present application can calculate the average information entropy based on the preset number of training cycles and the category prediction probability to obtain the sample average information entropy. Information entropy is an indicator to measure the uncertainty of the prediction probability distribution. The higher the information entropy, the more uncertain the model's prediction of the sample category; the lower the information entropy, the more certain the model's prediction of the sample category. In other words, by calculating the average information entropy, the uncertainty of the initial neural network model's prediction of each sample table data can be better quantified. Therefore, the specific calculation process of the sample average information entropy of each sample table data can be found in the following public

[0111] As shown in formula 2:

[0112]

[0113] Among them, x represents the input of sample table data, v al represents the sample average information entropy of the sample table data, E represents the preset number of training cycles, P e (x) represents the category prediction probability that the initial neural network model predicts the category of the sample table data as the preset sample category in the e-th training cycle. In other words, the larger the sample average information entropy, the smaller the effective information brought by the corresponding sample table data; the smaller the sample average information entropy, the larger the effective information brought by the corresponding sample table data, indicating that the corresponding data quality is better.

[0114] In step S330 of some embodiments, further, the present application may perform a difference calculation on the predicted probability mean and sample average information entropy corresponding to the same sample table data to obtain the probability information difference of the sample table data. The present application may define the screened sample data contained in the high-quality data set as data with low uncertainty and high confidence (i.e., the larger the predicted probability mean and the smaller the sample average information entropy), which is manifested as the larger the difference between the predicted probability mean and the sample average information entropy, the better.

[0115] In step S340 of some embodiments, the present application may further arrange the sample table data in descending order based on the probability information difference value to obtain a data sequence, and select k0 sample table data with larger values ​​starting from the data with the largest probability information difference value as the filtered sample data to construct a high-quality data set. Therefore, the process of constructing the filtered sample data can be shown in the following formula 3:

[0116] High_data=(P(x)-v al )istop k0 (Formula 3)

[0117] Among them, High_data represents the constructed high-quality data set, P(x) represents the predicted probability mean of the sample table data, and v al represents the sample average information entropy of the sample table data, istop k0 represents the average information entropy of (P(x)-v al ) is used to sort the maximum values ​​calculated, and k0 data with larger values ​​are selected as the sample data after selection.

[0118] In step S240 of some embodiments, after a high-quality data set is constructed based on the screened sample data, the initial neural network model can be retrained based on the screened sample data to obtain a first category prediction model. In addition, since the first category prediction model uses a model with a more complex model structure, it can have a better prediction confidence.

[0119] In the above embodiment, the present application evaluates the stability and uncertainty of the model's prediction of each sample table data by calculating the mean prediction probability and the average information entropy, and then selects samples based on these evaluation results in order to improve the performance of the model on key samples, so as to optimize the model more effectively.

[0120] In step S130 of some embodiments, after the first category prediction model is trained, the present application may perform loss calculation on the initial table data based on the first category prediction model to obtain first loss data, and use the first loss data to perform data division on the initial table data to obtain basic table data and candidate table data. Specifically, the present application may select a first number of initial table data with larger values ​​in the first loss data as basic table data, and the remaining initial table data as candidate table data. For example, the first loss data is the cross entropy loss value pe, and the first number is b0 (b0 is a positive integer). The present application may select the first b0 initial table data with larger cross entropy loss value pe as basic table data. At the same time, the present application may use the first loss data of the candidate table data as candidate loss data.

[0121] In step S140 of some embodiments, after determining the candidate table data, the present application may further screen the candidate table data to exclude noise data and extract relatively pure data. Specifically, the present application may simultaneously perform loss calculation on the candidate table data based on the second category prediction model to obtain second loss data. The second loss data is used to indicate the degree of difference between the second prediction label and the data category label, and the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is greater than the model complexity of the first category prediction model.

[0122] It should be noted that the second category prediction model of the present application can be trained based on an original data set containing multiple initial table data, and the specific training process is not described in detail. Among them, the first category prediction model can adopt a three-layer linear neural network model, a five-layer linear neural network model, etc., without limitation.

[0123] In step S150 of some embodiments, the present application may further calculate the difference between the loss data of the candidate table data when using the first category prediction model and the second category prediction model, and this difference reflects the impact of the increase in model complexity on the prediction accuracy. Therefore, the specific calculation process of the candidate loss difference of the candidate table data is shown in the following formula 4:

[0124] logits(i)=l r (x i |θ * )-e l (xi |θ) (Formula 4)

[0125] Among them, x i represents the input of the i-th candidate table data, logits(i) represents the candidate loss difference of the i-th candidate table data, l r (x i |θ*) represents the candidate loss data obtained by the i-th candidate table data under the first category prediction model, θ * represents the model parameters of the first category prediction model, l l (x i |θ) represents the second loss data obtained by the i-th candidate table data under the second category prediction model, and θ represents the model parameters of the second category prediction model. Among them, the larger the candidate loss difference, the lower the prediction accuracy of the corresponding candidate table data in the second category prediction model, but the higher the prediction accuracy in the first category prediction model, which means that the candidate table data can improve the corresponding prediction accuracy according to the improvement of the model, and it is not impurity data.

[0126] In the above embodiment, the present application constructs a first category prediction model (equivalent to a learning model) and a second category prediction model (equivalent to a reference model), and filters the data through the characteristic differences of the two models on the data, thereby retaining a large amount of correct data and filtering out noise data, thereby improving the quality of the data selected for actual use.

[0127] It should be noted that in the second category prediction model, since the prediction accuracy of these candidate loss data is low, if it is simply judged according to the accuracy of the loss data (such as the cross entropy value), it is easy to be misjudged as impurity data. In the complex first category prediction model, the accuracy range of these candidate loss data often overlaps with the accuracy range of some impurity data, and it is not easy to exclude impurity data by judging by accuracy. Therefore, the present application can further extract real data that is easily affected by the model from the candidate table data through the difference between the candidate loss data and the second loss data. These data are not impurity data. Compared with the related art that only uses a single model, it is easy to eliminate these and reduce the amount of valid data. The present application can reduce the dependence on the benchmark model by constructing the first category prediction model and the second category prediction model, and screen the data through the difference in the characteristics of the two models to the data, thereby reducing the dependence of the data on a single model.

[0128] In step S160 of some embodiments, after determining the candidate loss data, since the candidate table data also includes some impurity data, the present application can perform batch data selection on the candidate table data based on the candidate loss difference and the basic table data, that is, optimize data selection by batching and iteratively updating the scores, increase data diversity and coverage, and prevent overfitting. Batch selected data refers to data obtained after batch data selection of the candidate table data.

[0129] See also Figure 4 , Figure 4 It is an optional flow chart of step S160 provided in an embodiment of the present application. In some embodiments of the present application, step S160 may specifically include but is not limited to steps S410 to S470.

[0130] Step S410, for the first screening of the preset screening batch, data similarity calculation is performed on the candidate table data and the basic table data to obtain a first data similarity;

[0131] Step S420, selecting first comparison table data from the basic table data based on the first data similarity;

[0132] Step S430, performing data detection based on the first comparison table data and the candidate loss difference to obtain a candidate detection score for the candidate table data;

[0133] Step S440, selecting candidate table data based on candidate detection scores and preset screening batches to obtain first batch selected sub-data and first batch remaining sub-data of the first screening;

[0134] Step S450, combining the first batch selected sub-data and the basic table data to obtain a first screened data set, wherein the first screened data set includes the first selected table data;

[0135] Step S460, for the second screening of the preset screening batch, data selection is performed on the remaining sub-data of the first batch based on the preset screening batch, the candidate loss difference and the first selection table data to obtain the second batch selected sub-data;

[0136] Step S470: determining batch selection data based on the first batch selection sub-data, the second batch selection sub-data and the preset screening batch.

[0137] In step S410 of some embodiments, the present application can be divided into k1 preset screening batches when performing batch data selection, that is, the remaining n (n is a positive integer) candidate table data can be divided into k1 batches for screening. The present application can first calculate the similarity between any two candidate table data and the basic table data, and the data similarity calculation is to measure the similarity between the two sets of data in terms of features or attributes. Commonly used methods include cosine similarity, Euclidean distance, Jaccard similarity coefficient, etc. Specifically, the candidate table data and the basic table data can be vectorized respectively to obtain candidate data vectors and basic data vectors, and then the candidate data vectors and the basic data vectors can be vectorized to obtain the first data similarity.

[0138] In step S420 of some embodiments, because the distribution of impurity data is discrete, for the first screening, b first comparison table data similar to each candidate table data can be selected from the basic table data (i.e., the basic table data selected after arranging in descending order according to the first data similarity).

[0139] In step S430 of some embodiments, the first comparison table data and the candidate loss difference may be used to evaluate the candidate table data to obtain a candidate detection score, which may reflect the uncertainty or importance of the candidate table data in the model prediction and be used for subsequent data selection.

[0140] See also Figure 5 , Figure 5 It is an optional flow chart of step S430 provided in an embodiment of the present application. In some embodiments of the present application, step S430 may specifically include but is not limited to steps S510 to S550.

[0141] Step S510, combining data of any two first comparison table data to obtain a comparison data combination;

[0142] Step S520, performing data encoding on the first combined table data to obtain a first data vector;

[0143] Step S530, performing data encoding on the second combined table data to obtain a second data vector;

[0144] Step S540, performing similarity calculation on the first data vector and the second data vector to obtain vector similarity;

[0145] Step S550, determining the candidate detection score of the candidate table data based on the vector similarity and the candidate loss difference.

[0146] In steps S510 to S550 of some embodiments, if there is no correlation between the b first comparison table data selected by the same candidate table data, the similarity between the b first comparison table data is very low. On the contrary, if the similarity between the b first comparison table data is very high, it means that the corresponding candidate table data is normal data. Specifically, the present application can first perform data combination on any two first comparison table data to obtain multiple comparison data combinations, and each comparison data combination includes first combination table data and second combination table data. Further, the similarity is calculated for the vectorized vectors of the first combination table data and the second combination table data to obtain vector similarity. Further, the similarity mean calculation can be performed on the vector similarities corresponding to the multiple comparison data combinations to obtain the combined similarity mean. And the combined similarity mean and the candidate loss difference are weighted to obtain the candidate detection score of the candidate table data. Therefore, the specific calculation process of the candidate detection score can be shown in the following formula 5:

[0147]

[0148] Among them, logits'(i) represents the candidate detection score of the i-th candidate table data, logits(i) represents the candidate loss difference of the i-th candidate table data, δ is a hyperparameter (the value is 0.1-0.7, which can be flexibly adjusted according to actual needs), b is the number of first comparison table data selected by the i-th candidate table data, C(b,2) represents the number of comparison data combinations formed by pairing two by two from b first comparison table data, cosin(j,z) represents the vector similarity between the j-th first comparison table data and the z-th first comparison table data, and j≠z. In addition, α represents the weight of the candidate loss difference, β represents the weight of the combined similarity mean, and the two weights can be flexibly adjusted according to actual needs.

[0149] See also Figure 6 , Figure 6 It is an optional flow chart of step S550 provided in an embodiment of the present application. In some embodiments of the present application, step S550 may specifically include but is not limited to steps S610 to S620.

[0150] Step S610, calculating the mean of vector similarity based on the number of combinations of the comparison data combinations and the vector similarity, to obtain the mean of the combination vector similarity;

[0151] Step S620, performing weighted calculation on the combined vector similarity mean and the candidate loss difference to obtain the candidate detection score of the candidate table data.

[0152] In step S610 and step S620 of some embodiments, the combined similarity mean This is equivalent to adding a factor to exclude impurity data. The smaller its value, the more likely the corresponding candidate table data is impurity data. In other words, the larger the value of the candidate detection score logits'(i), the more the corresponding candidate table data meets the requirements of data selection.

[0153] In step S440 of some embodiments, for the first screening, the b pieces of first comparison table data are derived from a data set B0 consisting of basic table data.

[0154] See also Figure 7 , Figure 7 It is an optional flow chart of step S440 provided in an embodiment of the present application. In some embodiments of the present application, step S440 may specifically include but is not limited to steps S710 to S750.

[0155] Step S710, calculating the ratio between the total quantity of basic table data and the preset screening batches to obtain the batch sampling quantity;

[0156] Step S720, sampling the candidate table data based on the batch sampling quantity and the candidate detection score to obtain candidate sampling data;

[0157] Step S730, determining the first screening data amount based on the preset screening batch, the preset filtering ratio and the total amount of basic table data;

[0158] Step S740, selecting candidate sampling data based on the candidate detection score and the first screening data volume to obtain the first batch of selected sub-data for the first screening;

[0159] Step S750 , eliminating the first batch selected sub-data based on the candidate sampling data to determine the first batch remaining sub-data.

[0160] In step S710 of some embodiments, at each screening, the present application will set the number of data selected in batches. At this time, the total number of basic table data and the preset screening batch can be first calculated to obtain the number of batch sampling. The number of batch sampling represents the number of data sampled based on the candidate detection score for each screening. For example, the total number of basic table data is n, and the preset screening batch is k, then the number of batch sampling is n / k.

[0161] In step S720 of some embodiments, further, based on the batch sampling quantity and the candidate detection score, a batch sampling quantity of candidate sampling data can be sampled from the candidate table data. Specifically, the present application can compare the candidate detection scores of multiple candidate table data to select a batch sampling quantity of candidate table data with a higher candidate detection score as the candidate sampling data.

[0162] It should be noted that, in order to facilitate rapid score comparison, the present application may also first normalize the candidate detection score of each candidate table data to obtain the detection probability of each candidate table data. At this time, the probability values ​​of the detection probabilities of multiple candidate table data may be compared to select a batch sampling number of candidate table data with a higher detection probability as the candidate sampling data.

[0163] In step S730 of some embodiments, further, the present application may set a preset filtering ratio f, and the preset filtering ratio f indicates how much proportion of poor data the present application is to filter out. Specifically, the first screening data amount may be determined based on the preset screening batch, the preset filtering ratio, and the total amount of basic table data, and the first screening data amount indicates the amount of data with good quality obtained by the first screening. Among them, the first screening data amount = (total amount of basic table data / preset screening batch) * preset filtering ratio = (n / k) * (1-f).

[0164] In step S740 and step S750 of some embodiments, further, the candidate sampling data can be sorted in descending order based on the candidate detection score, and the first screening data amount candidate sampling data with a higher candidate detection score is selected according to the sorting result as the first batch selection sub-data for the first screening, and the remaining data is used as the first batch remaining sub-data. In other words, after obtaining (n / k) subsequent sampling data, the application can select (n / k)*f data with the smallest value as the first batch remaining sub-data for the next screening according to the size of the subsequent detection score, and select (n / k)*(1-f) data with a larger value as the first batch selection sub-data, so as to finally construct the target data set with the basic table data. In this way, the diversity of the data can be increased, and the situation of serious homogeneity of all data can be effectively avoided.

[0165] In step S450 of some embodiments, further, the first batch selected sub-data and the basic table data can be combined to obtain a first screened data set, the first screened data set includes the first selected table data, and the first screened data set can be used as the data basis for the next screening, so as to facilitate the next screening to obtain b comparison table data. That is, the present application can add (n / k)*(1-f) first batch selected sub-data to the data set B0 to obtain a new data set B1.

[0166] In step S460 and step S470 of some embodiments, as in the first screening, in the second screening, the second comparison table data of each first batch of remaining sub-data can be calculated again, as shown in formula 5. At this time, the b second comparison table data corresponding to the second comparison table data formula are derived from the new data set B1. And the data set relied on by each subsequent screening is the data set Bk-1 updated in the previous screening, and the remaining steps are the same as above. In this way, after the data of the preset screening batch is screened, multiple high-quality batch selection data are finally obtained.

[0167] In the above embodiment, the present application uses the data set B0 constructed based on the basic table data as the basic data, gradually expands the data that meets the requirements, and the expanded data is the basis for the next iteration of expanding the data. Compared with the one-time expansion of data in the related art, which will bring about a larger deviation, the present application updates the candidate detection scores in batches and iteratively to optimize data selection, increase data diversity and coverage, and prevent overfitting.

[0168] In step S170 of some embodiments, after obtaining a plurality of batch selected data, the present application may combine the batch selected data with the selected plurality of basic table data to obtain a high-quality target data set. In addition, the target data set may be applied to table data quality screening services, such as data quality screening of insurance data, financial transaction data, and customer information data, to ensure the accuracy and effectiveness of the selected data, thereby improving the accuracy and efficiency of business decisions.

[0169] It should be noted that the non-Company software tools or components appearing in the embodiments of the present application are merely examples and do not represent actual use.

[0170] A method for selecting tabular data provided in an embodiment of the present application utilizes the difference in loss between a first category prediction model and a second category prediction model with different model complexities to evaluate the "learnability" of the data in order to select the data that is most valuable to the model. The present application's method of combining different models for data selection can effectively reduce the data's dependence on a single model. In addition, the present application adopts a dynamic data screening strategy, that is, batching and iteratively updating scores to optimize data selection, increase data diversity and coverage, and prevent overfitting, which can effectively improve the accuracy and efficiency of selecting tabular data and provide a data basis for subsequent data applications.

[0171] See also Figure 8 The present application also provides a table data selection device, which can implement the above table data selection method, and the device includes:

[0172] An acquisition module 810 is used to acquire initial table data and data category labels of the initial table data;

[0173] A first loss calculation module 820 is used to perform loss calculation on the initial table data based on the first category prediction model to obtain first loss data; wherein the first loss data is used to indicate the degree of difference between the first prediction label and the data category label, and the first prediction label is a label obtained by predicting the initial table data based on the first category prediction model;

[0174] A data partitioning module 830 is used to partition the initial table data based on the first loss data to obtain basic table data and candidate table data, and obtain first loss data of the candidate table data to obtain candidate loss data;

[0175] A second loss calculation module 840 is used to perform loss calculation on the candidate table data based on the second category prediction model to obtain second loss data; wherein the second loss data is used to indicate the degree of difference between the second prediction label and the data category label, the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is less than the model complexity of the first category prediction model;

[0176] The loss difference calculation module 850 is used to perform a difference calculation on the candidate loss data and the second loss data to obtain a candidate loss difference of the candidate table data;

[0177] A data selection module 860 is used to select candidate table data in batches based on the candidate loss difference and the basic table data to obtain batch selected data;

[0178] The data combination module 870 is used to combine the data based on the batch selection data and the basic table data to obtain a target data set.

[0179] The specific implementation of the table data selection device is basically the same as the specific implementation of the table data selection method described above, and will not be described in detail here.

[0180] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above table data selection method when executing the computer program. The electronic device can be any smart terminal including a tablet computer, a car computer, etc.

[0181] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0182] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0183] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the table data selection method of the embodiment of this application;

[0184] Input / output interface 930, used to implement information input and output;

[0185] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0186] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0187] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0188] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned table data selection method is implemented.

[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0190] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0191] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0192] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0194] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0195] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0196] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0197] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0198] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0199] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0200] The preferred embodiments of the present application are described above with reference to the accompanying drawings, but the scope of the rights of the present application is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present application should be within the scope of the rights of the present application.

Claims

1. A method for selecting table data, characterized in that: The method comprises: Obtaining initial table data and data category labels of the initial table data; Based on the first category prediction model, the loss of the initial table data is calculated to obtain first loss data; wherein the first loss data is used to indicate the degree of difference between the first prediction label and the data category label, and the first prediction label is a label obtained by predicting the initial table data based on the first category prediction model; Based on the first loss data, the initial table data is divided into data to obtain basic table data and candidate table data, and the first loss data of the candidate table data is acquired to obtain candidate loss data; Performing loss calculation on the candidate table data based on the second category prediction model to obtain second loss data; wherein the second loss data is used to indicate the degree of difference between the second prediction label and the data category label, the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is less than the model complexity of the first category prediction model; Calculating the difference between the candidate loss data and the second loss data to obtain a candidate loss difference of the candidate table data; Performing batch data selection on the candidate table data based on the candidate loss difference and the basic table data to obtain batch selected data; Data is combined based on the batch selected data and the basic table data to obtain a target data set.

2. The method according to claim 1, characterized in that The step of selecting the candidate table data in batches based on the candidate loss difference and the basic table data to obtain batch selected data includes: For the first screening of the preset screening batch, data similarity calculation is performed on the candidate table data and the basic table data to obtain a first data similarity; selecting first comparison table data from the basic table data based on the first data similarity; Perform data detection based on the first comparison table data and the candidate loss difference to obtain a candidate detection score for the candidate table data; Performing data selection on the candidate table data based on the candidate detection score and the preset screening batch to obtain the first batch selected sub-data and the first batch remaining sub-data of the first screening; Combining the first batch selected sub-data and the basic table data to obtain a first screened data set, wherein the first screened data set includes first selected table data; For the second screening of the preset screening batch, data selection is performed on the first batch remaining sub-data based on the preset screening batch, the candidate loss difference and the first selection table data to obtain second batch selected sub-data; The batch selection data is determined based on the first batch selection sub-data, the second batch selection sub-data, and the preset screening batch.

3. The method according to claim 2, characterized in that The step of selecting the candidate table data based on the candidate detection score and the preset screening batch to obtain the first batch selected sub-data and the first batch remaining sub-data for the first screening includes: Calculate the ratio of the total number of the basic table data to the preset screening batches to obtain the number of batch sampling; Performing data sampling on the candidate table data based on the batch sampling quantity and the candidate detection score to obtain candidate sampling data; Determining a first screening data amount based on the preset screening batch, the preset filtering ratio and the total amount of the basic table data; Performing data selection on the candidate sampling data based on the candidate detection score and the first screening data volume to obtain the first batch selected sub-data of the first screening; The first batch selected sub-data are eliminated based on the candidate sampling data to determine the first batch remaining sub-data.

4. The method according to claim 2, characterized in that: The performing data detection based on the first comparison table data and the candidate loss difference to obtain a candidate detection score for the candidate table data includes: Combining any two of the first comparison table data to obtain a comparison data combination, wherein the comparison data combination includes the first combination table data and the second combination table data; Performing data encoding on the first combined table data to obtain a first data vector; Performing data encoding on the second combined table data to obtain a second data vector; Calculating similarity between the first data vector and the second data vector to obtain vector similarity; The candidate detection score of the candidate table data is determined based on the vector similarity and the candidate loss difference.

5. The method according to claim 4, characterized in that The determining the candidate detection score of the candidate table data based on the vector similarity and the candidate loss difference includes: Performing vector similarity mean calculation based on the number of combinations of the comparison data combinations and the vector similarities to obtain a combination vector similarity mean; A weighted calculation is performed on the combined vector similarity mean and the candidate loss difference to obtain the candidate detection score of the candidate table data.

6. The method according to any one of claims 1 to 5, characterized in that: Before performing loss calculation on the initial table data based on the first category prediction model to obtain the first loss data, the method further includes: constructing the first category prediction model, specifically including: Acquire an initial training set, wherein the initial training set includes sample table data having preset sample categories; Based on the initial neural network model, data category prediction is performed on the sample table data to obtain a category prediction probability, where the category prediction probability is used to represent the probability value that the initial neural network model predicts the category of the sample table data as the preset sample category in one training cycle; Performing data selection on the sample table data based on the category prediction probability to obtain screened sample data; The initial neural network model is trained based on the screened sample data to obtain the first category prediction model.

7. The method according to claim 6, characterized in that The step of selecting the sample table data based on the category prediction probability to obtain the screened sample data includes: Calculating the mean of the predicted probabilities of the categories for a preset number of training cycles to obtain a predicted probability mean; Performing average information entropy calculation based on the preset number of training cycles and the category prediction probability to obtain sample average information entropy; Calculate the difference between the predicted probability mean and the sample average information entropy to obtain the probability information difference of the sample table data; The sample table data is selected based on the probability information difference to obtain filtered sample data.

8. A table data selection device, characterized in that: The device comprises: An acquisition module, used for acquiring initial table data and data category labels of the initial table data; A first loss calculation module, configured to perform loss calculation on the initial table data based on a first category prediction model to obtain first loss data; wherein the first loss data is used to indicate the degree of difference between a first prediction label and the data category label, the first prediction label being a label obtained by predicting the initial table data based on the first category prediction model; a data partitioning module, configured to partition the initial table data based on the first loss data to obtain basic table data and candidate table data, and to obtain the first loss data of the candidate table data to obtain candidate loss data; A second loss calculation module, used to perform loss calculation on the candidate table data based on a second category prediction model to obtain second loss data; wherein the second loss data is used to indicate the degree of difference between a second prediction label and the data category label, the second prediction label is a label obtained by predicting the candidate table data based on the second category prediction model, and the model complexity of the second category prediction model is less than the model complexity of the first category prediction model; A loss difference calculation module, used for performing difference calculation on the candidate loss data and the second loss data to obtain a candidate loss difference of the candidate table data; A data selection module, configured to select the candidate table data in batches based on the candidate loss difference and the basic table data to obtain batch selected data; The data combination module is used to combine the data based on the batch selected data and the basic table data to obtain a target data set.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images