A data processing method and device

Abstract

The present invention discloses a data processing method and device, which relates to the field of computer technology. The main purpose of the present invention is to select samples that can effectively improve the model recognition effect from a large amount of text data for annotation, thereby saving annotation costs. The main technical scheme of the present invention is: obtaining a first feature vector and a second feature vector of the same data; determining the confidence of the data according to the first feature vector and the second feature vector; and selecting the data as annotated data according to the confidence.

Description

Technical Field

[0001] The present invention relates to the field of computer science and technology, and in particular to a data processing method and device. Background Art

[0002] Named Entity Recognition (NER) refers to the identification of entities with specific meanings in text, mainly including names of people, places, institutions, proper nouns, etc. In the e-commerce title scenario, named entities refer to product names, commodity categories, commodity attribute words, etc. When conducting e-commerce single-term mining, the main method for named entity recognition is based on the implementation method of neural networks. Commonly used recognition models include the BiLSTM-CRF model.

[0003] The use of the model requires pre-training to achieve a good recognition effect, which requires the use of samples with labeled data, most of which are obtained through manual labeling. In general, the number of samples required for training the model is not linear with the model effect, that is, the performance of the model will not increase infinitely with the increase of training sample data, and it will have a corresponding bottleneck value. Therefore, in order to save the cost of manually labeling training samples, it is necessary to selectively label the sample data so that the model can be trained with fewer samples to approach or reach its bottleneck value. Summary of the invention

[0004] In view of the above problems, the present invention proposes a data processing method and device, the main purpose of which is to select samples that can effectively improve the model recognition effect from a large amount of text data for annotation, thereby saving annotation costs.

[0005] In order to achieve the above object, the present invention mainly provides the following technical solutions:

[0006] In one aspect, the present invention provides a data processing method, which specifically comprises:

[0007] Obtain a first eigenvector and a second eigenvector of the same piece of data;

[0008] Determining the confidence of the data according to the first eigenvector and the second eigenvector;

[0009] The data is selected as labeled data according to the confidence level.

[0010] Preferably, the first eigenvector and the second eigenvector are obtained by:

[0011] Inputting the data into a first embedding layer and a second embedding layer of a recognition model respectively;

[0012] Inputting the content output by the first embedding layer into a first neural network layer to obtain a first feature vector;

[0013] The content output by the second embedding layer is input into the second neural network layer to obtain a second feature vector.

[0014] Preferably, determining the confidence of the data according to the first eigenvector and the second eigenvector includes:

[0015] determining a similarity between the first feature vector and the second feature vector;

[0016] Determine the average information entropy of the data using the first eigenvector and the second eigenvector;

[0017] The difference between the average information entropy and the similarity is used as the confidence of the data.

[0018] Preferably, determining the average information entropy of the data according to the first eigenvector and the second eigenvector includes:

[0019] concatenating the first feature vector and the second feature vector into a combined feature vector;

[0020] Determining the label probability corresponding to each word in the data according to the combined feature vector;

[0021] Calculate the information entropy corresponding to each word segmentation according to the label probability;

[0022] The average information entropy of the data is determined according to the information entropy.

[0023] Preferably, the method further comprises:

[0024] Setting an upper limit value of the similarity;

[0025] If the similarity between the first feature vector and the second feature vector corresponding to the data is higher than the upper limit value, it is suggested to modify the recognition model to improve the differentiation degree of parameter settings.

[0026] Preferably, selecting the data as the labeled data according to the confidence level includes:

[0027] Get confidence of multiple data;

[0028] Sorting in descending order of confidence;

[0029] Data with high confidence is selected from the plurality of data as labeled data according to a preset number and the sorting.

[0030] Preferably, the recognition model is a BiLSTM-CRF model based on a neural network.

[0031] In another aspect, the present invention provides a data processing device, comprising:

[0032] An acquisition unit, used to acquire a first eigenvector and a second eigenvector of the same piece of data;

[0033] a determining unit, configured to determine the confidence of the data according to the first feature vector and the second feature vector obtained by the acquiring unit;

[0034] A selection unit is used to select the data as the labeled data according to the confidence obtained by the determination unit.

[0035] Preferably, the acquisition unit includes:

[0036] An input module, used for inputting the data into a first embedding layer and a second embedding layer of a recognition model respectively;

[0037] An acquisition module, configured to input the content output by the first embedding layer into a first neural network layer to obtain a first feature vector;

[0038] The acquisition module is also used to input the content output by the second embedding layer into the second neural network layer to obtain a second feature vector.

[0039] Preferably, the determining unit includes:

[0040] A first determining module, configured to determine the similarity between the first feature vector and the second feature vector;

[0041] A second determination module, configured to determine an average information entropy of the data using the first eigenvector and the second eigenvector;

[0042] A calculation module is used to use the difference between the average information entropy obtained by the second determination module and the similarity obtained by the first determination module as the confidence of the data.

[0043] Preferably, the second determining module is further used for:

[0044] concatenating the first feature vector and the second feature vector into a combined feature vector;

[0045] Determining the label probability corresponding to each word in the data according to the combined feature vector;

[0046] Calculate the information entropy corresponding to each word segmentation according to the label probability;

[0047] The average information entropy of the data is determined according to the information entropy.

[0048] Preferably, the device further comprises:

[0049] a setting unit, configured to set an upper limit value of the similarity determined by the determining unit;

[0050] The alarm unit is used to prompt to modify the recognition model to improve the differentiation degree of parameter settings if the similarity between the first feature vector and the second feature vector corresponding to the data is higher than the upper limit value set by the setting unit.

[0051] Preferably, the selection unit includes:

[0052] An acquisition module, used for obtaining the confidence of multiple data;

[0053] A sorting module, used for sorting the confidences obtained by the acquisition module in descending order;

[0054] A selection module is used to select data with high confidence from the multiple data as labeled data according to a preset number and the sorting determined by the sorting module.

[0055] Preferably, the recognition model is a BiLSTM-CRF model based on a neural network.

[0056] On the other hand, the present invention provides a processor, which is used to run a program, wherein the program executes the above-mentioned data processing method when running.

[0057] Through the above technical scheme, the present invention provides a data processing method and device, which obtains two corresponding feature vectors by using a feature extraction method based on different parameter settings for the same data. For example, the data is encoded by an encoder with different parameter settings in the recognition model, and the obtained first feature vector and second feature vector are used to comprehensively determine the confidence of the data for training the recognition model. Since the comprehensive processing of different feature vectors can determine the effect of using the data to train the recognition model from more dimensions, and a corresponding quantized value can be given for each data, a smaller number of data can be selected from a large amount of data according to the confidence level for labeling, so as to train the recognition effect of the recognition model to the bottleneck value, thereby reducing the cost of manually labeling training samples.

[0058] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0060] Figure 1 A flow chart of a data processing method proposed in an embodiment of the present invention is shown;

[0061] Figure 2 A flowchart of another data processing method proposed in an embodiment of the present invention is shown;

[0062] Figure 3 A comparison diagram showing the effects of a data processing method proposed in an embodiment of the present invention and other methods in practical applications is shown;

[0063] Figure 4 A block diagram showing another data processing device according to an embodiment of the present invention is shown;

[0064] Figure 5 A flow chart of another data processing method proposed in an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0065] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present invention and to enable the scope of the present invention to be fully communicated to those skilled in the art.

[0066] The data processing method provided by the embodiment of the present invention is to improve the screening of sample data during the model training process, mainly to optimize the selection method of manually labeled samples. The sample data is no longer selected by random sampling, but the confidence calculation is used to measure the value of manual labeling of all sample data, so as to select the best sample data for sample labeling. The specific steps of this method are as follows: Figure 1 As shown, the method includes:

[0067] Step 101: Obtain a first eigenvector and a second eigenvector of the same piece of data.

[0068] Among them, the feature vector in this embodiment is obtained based on the processing of the data by the recognition model. Specifically, the recognition model in this step can be a variety of named entity recognition implementation methods, such as the commonly used BiLSTM-CRF model, or a model built with network structures such as RNN, CNN, Tranformer, etc. When performing named entity recognition, these models all need to convert the input data into vectors, and then extract the vector features therein for named entity recognition, that is, the input data needs to be converted and extracted from the feature vector, that is, the encoder encoder used in the model, the different settings of the encoder parameters will result in different feature vectors, which will affect the efficiency and accuracy of named entity recognition.

[0069] For sample data or data to be labeled, the greater the difference between the feature vectors extracted by the recognition model using different parameter settings (i.e., the first feature vector and the second feature vector), the more difficult it is for the recognition model to identify the sample data or the data to be labeled. Different sample data or data to be labeled have different differences. This difference can also be used as a reference standard for selecting data to improve the diversity of data sources. At the same time, the use of the first feature vector and the second feature vector can also be used to measure whether the recognition model can accurately identify the sample data or the data to be labeled from another dimension through information entropy. It can be seen that by obtaining the first feature vector and the second feature vector of the same data, comparison, combination and other methods can be used to define measurement standards of different dimensions, so that the data has a multi-dimensional measurement method.

[0070] It should be noted that the first eigenvector and the second eigenvector in this step are not used to limit the number of eigenvectors. That is to say, according to different parameter settings, multiple sets of eigenvectors for the same data can be obtained, thereby realizing the definition and setting of more dimensional measurement standards.

[0071] Step 102: Determine the confidence of the data according to the first eigenvector and the second eigenvector.

[0072] According to the measurement standards of different dimensions defined by the combination of different feature vectors described in the previous step, this step is to use the first feature vector and the second feature vector to conduct a comprehensive evaluation of the measurement standards of different dimensions, so as to obtain the confidence of the corresponding data. The confidence is used to measure the recognition effect of the data used to train the recognition model. The confidence will be used as a quantitative standard for selecting data for manual labeling.

[0073] Specifically, the specific measurement criteria contained in the confidence may include the difference between different feature vectors, which can be determined by similarity, and the information entropy of words in the data, which needs to be calculated based on the probability of each word in the data on the required annotation label. The larger the information entropy, the greater the uncertainty of the data, that is, the less accurate the prediction of the recognition model for the sample. The measurement criteria of different dimensions defined by different feature vectors can also be obtained by statistics based on the weighted values ​​corresponding to different feature vectors, and the specific statistical method is not limited to summing, finding the maximum value, finding the average, etc.

[0074] Step 103: Select data as labeled data according to the confidence level.

[0075] In the embodiment of the present invention, the confidence calculation of all data is realized through steps 101 and 102, and this is used as the selection standard. The data with greater confidence is more suitable for labeling. Therefore, this step is to select according to the preset selection quantity combined with the confidence, or to select data higher than the preset value of the confidence as the labeled data according to the preset value of the confidence, so as to further perform manual labeling.

[0076] Through the description of the above embodiments, it can be known that a data processing method provided by an embodiment of the present invention is to perform quantitative calculation of the confidence of all data, use this as the standard for selecting data, and select data that is more helpful for training the recognition model for labeling. The calculation of confidence is achieved by multiple sets of encoders realized by different settings of parameters by the recognition model, which encode the same data differently, determine multiple sets of feature vectors, and use these different feature vectors to measure the accuracy of the recognition model in predicting the data from multiple dimensions. It can be seen that the embodiment of the present invention comprehensively quantifies and scores the data from multiple dimensions, and selects the required data based on the score, avoiding omissions caused by random sampling in the existing selection method, and also avoiding the problem of lack of diversity in the selected data through multi-dimensional evaluation.

[0077] Furthermore, for the above Figure 1 The data processing method described in the embodiment of the present invention will describe in detail the process of determining the confidence by two feature vectors in named entity recognition and performing data screening processing based on them, and the specific steps include:

[0078] Step 201: Based on different settings of parameters of the recognition model, obtain a first feature vector and a second feature vector of the same piece of data.

[0079] Specifically, the recognition model in this step is a BiLSTM-CRF model based on a neural network. The BiLSTM-CRF model is a popular model for processing sequence labeling problems and is widely used in named entity recognition tasks. The specific structure and principle of the model will not be described in detail in this embodiment.

[0080] The parameters set for the BiLSTM-CRF model in this step include the number of dimensions and initialization weights, etc. These parameters mainly affect the embedding layer (embedding layer) and the Bi-LSTM layer (neural network layer) of the BiLSTM-CRF model, wherein the embedding layer is used to vectorize the sentence input into the model, and the Bi-LSTM layer is used to extract the feature vector from the vector representation of the sentence. Specifically in the embodiment of the present invention, it is necessary to determine two sets of model parameters for the BiLSTM-CRF model, and extract the feature vectors of the data respectively by using the recognition model configured by the two sets of model parameters. For a piece of data, the corresponding first feature vector and second feature vector can be obtained, and the same piece of data is respectively input into the embedding layer corresponding to the two sets of model parameters of the recognition model, that is, the first embedding layer and the second embedding layer, and then the content output by the first embedding layer is input into the first Bi-LSTM layer, that is, the first neural network layer, to obtain the first feature vector, and the content output by the second embedding layer is input into the second Bi-LSTM layer, that is, the second neural network layer, to obtain the second feature vector. For example, for the same piece of data, the vector representations converted using a 100-dimensional vector and a 300-dimensional vector are different. The feature vector is then extracted based on the vector representation. This process is related to the initialization vector weight of the model. The feature vectors extracted with different weights are also different. It can be seen that the corresponding feature vectors extracted with different parameter settings are also different.

[0081] When other network structures are used, the parameters for feature vector conversion and extraction need to be grouped and set accordingly to obtain multiple groups of feature vectors. The specific parameters to be set are related to the corresponding network structure and will not be described in detail here.

[0082] Step 202: Determine the similarity between the first feature vector and the second feature vector.

[0083] The similarity between the first eigenvector and the second eigenvector can indicate the degree of divergence when the BiLSTM-CRF model with different parameter settings is used to classify the same data. The greater the divergence, the more difficult it is for the model to correctly predict the data, that is, to correctly identify the named entities therein.

[0084] The similarity in this step is the similarity between two feature vectors, and commonly used similarity calculation methods include cosine similarity, Euclidean distance, Pearson correlation coefficient, etc. The embodiment of the present invention does not specifically limit the specific algorithm of similarity.

[0085] Furthermore, since the similarity between the first eigenvector and the second eigenvector can be used to measure the ability of the model to recognize the data, the higher the similarity, the stronger the corresponding ability, indicating that the data does not need to be selected for manual labeling. The reason affecting the size of the similarity may be the recognition ability of the model itself, or the model parameter setting. For the latter, the embodiment of the present invention can also pre-set the upper limit of the similarity. If the similarity between the first eigenvector and the second eigenvector corresponding to all data is higher than the upper limit, an alarm prompt is issued. The alarm prompt is used to prompt the modification of the different settings of the parameters of the recognition model, that is, to reset the two sets of model parameters to execute steps 201 and 202, and increase the difference between the eigenvectors corresponding to the same data by increasing the degree of differentiation of the parameter settings.

[0086] Step 203: Determine the average information entropy of the data using the first eigenvector and the second eigenvector.

[0087] Information entropy is an indicator in machine learning, often used as a quantitative indicator of the information content of a system. The lower the probability of an event, the greater the amount of information it can provide when it occurs. In this embodiment, the greater the information entropy, the greater the uncertainty, that is, the lower the accuracy of the model in identifying the data.

[0088] Specifically in this embodiment, the process of using the BiLSTM-CRF model to determine the average information entropy of data is as follows:

[0089] First, the first feature vector and the second feature vector are concatenated into a combined feature vector, and the concatenation gives the sentence corresponding to the data richer semantic information.

[0090] Afterwards, the BiLSTM-CRF model is used to process the combined feature vector to determine the label probability corresponding to each word in the data.

[0091] For the BiLSTM-CRF model, the Bi-LSTM layer will output the label probability corresponding to each word in the input sentence. In this step, the combined feature vector is input into the fully connected layer and processed by the softmax function to obtain a processed vector. Based on the vector, the labels of each word in the sentence and the label probability corresponding to the label are further determined.

[0092] Third, the information entropy corresponding to each word segment is calculated based on the label probability.

[0093] According to the definition of information entropy, it is calculated based on event probability. Therefore, in this embodiment, according to the label probability calculated above, the information entropy corresponding to each word segment can be obtained using the definition formula of information entropy.

[0094] Finally, the average information entropy of the data is determined based on the information entropy. That is, the average information entropy corresponding to the entire sentence is calculated based on the information entropy corresponding to each word segment. Most existing sample annotations use the average information entropy to measure the uncertainty of a sample data, and use this as a sampling criterion for selecting annotations. In this embodiment, the average information entropy is used as an indicator of a dimension to participate in the assessment of whether the data can be selected for manual annotation.

[0095] It should be noted that there is no logical execution sequence relationship between this step and step 202.

[0096] Step 204: The difference between the average information entropy and the similarity is used as the confidence of the data.

[0097] In this example, the definition of confidence is the difference between the average information entropy and the similarity. The larger the average information entropy, the more the corresponding data needs to be manually labeled; and the smaller the similarity, the more the corresponding data needs to be manually labeled. Therefore, the difference between the two is proportional to the degree of need for the data to be selected.

[0098] Step 205: Select data as labeled data according to the confidence level.

[0099] The specific selection process in this step is to first obtain the confidence of multiple data, then sort these data in order of confidence from high to low, and then select data with high confidence from these multiple data according to the preset number and sorting as labeled data, that is, data that needs to be manually labeled.

[0100] Through the steps of the above-mentioned embodiment and the corresponding diagrams, it can be seen that the embodiment of the present invention can perform a multi-dimensional quantitative evaluation on all the data in the data set, that is, to measure whether manual labeling is required by determining the confidence of each data. Compared with the conventional methods of selecting data for labeling, such as random sampling, uncertainty sampling and other methods, the embodiment of the present invention performs a multi-dimensional quantitative evaluation on all data samples. The diversity of the selected data can be ensured by comparing the similarity of different feature vectors, and the uncertainty of the selected data can be ensured by calculating the information entropy. It is realized that a high recognition effect can also be achieved by selecting a small amount of data for manual labeling and then training the model, thereby reducing the workload of manual labeling and improving labeling efficiency.

[0101] The present invention also demonstrates through experiments conducted on e-commerce title data that the present invention uses fewer labeled samples than the random sampling method and the uncertainty sampling method. The comparative effect is as follows: Figure 3 As shown in the figure, the x-axis represents the amount of data selected from an unlabeled data pool, 2k represents 2000 sentences, and the y-axis represents the performance of the e-commerce NER model trained using sentences selected using different methods. Figure 3 The line segment A in the figure is a line segment formed by the confidence method adopted in the embodiment of the present invention, the line segment B is a line segment formed by the uncertainty sampling method, and the line segment C is a line segment formed by the random sampling method. It can be seen from the figure that in each round of sentence selection, the maximum confidence criterion shows better performance. Compared with line segments B and C, when 16k sentences need to be annotated, line segment A using the confidence method only needs to annotate 14k sentences to achieve better performance than random sampling or uncertainty sampling of 16k sentences, which can save about 12.5% ​​of the annotation cost.

[0102] Based on the contents of the above embodiments, the data processing method proposed in the embodiments of the present invention can be used in various application scenarios that require model training, such as e-commerce platforms, translation software or platforms, human-computer interaction services, etc. The present invention can quickly and accurately screen out training samples for the model, and achieve the goal of using a small number of training samples to obtain a model with the same performance, thereby reducing the cost of model training or maintenance for operating users or enterprises.

[0103] Furthermore, as a response to the above Figure 1 , 2 In order to realize the method shown in the figure, the embodiment of the present invention provides a data processing device, the main purpose of which is to select samples that can effectively improve the model recognition effect from a large amount of text data for annotation, thereby saving the annotation cost. For the sake of ease of reading, this device embodiment will no longer repeat the details of the aforementioned method embodiment one by one, but it should be clear that the device in this embodiment can correspond to all the contents of the aforementioned method embodiment. The device is as follows Figure 4 As shown, specifically including:

[0104] An acquisition unit 31 is used to acquire a first feature vector and a second feature vector of the same piece of data;

[0105] A determining unit 32, configured to determine the confidence of the data according to the first feature vector and the second feature vector obtained by the acquiring unit 31;

[0106] The selection unit 33 is used to select the data as the labeled data according to the confidence obtained by the determination unit 32.

[0107] Further, such as Figure 5As shown, the acquisition unit 31 includes:

[0108] An input module 311, used to input the data into a first embedding layer and a second embedding layer of a recognition model respectively;

[0109] An acquisition module 312 is used to input the content output by the input module 311 after inputting the content into the first embedding layer into the first neural network layer to obtain a first feature vector;

[0110] The acquisition module 312 is further used to input the content output by the input module 311 into the second embedding layer into the second neural network layer to obtain a second feature vector.

[0111] Further, such as Figure 5 As shown, the determining unit 32 includes:

[0112] A first determination module 321, configured to determine the similarity between the first feature vector and the second feature vector;

[0113] A second determination module 322, configured to determine the average information entropy of the data using the first eigenvector and the second eigenvector;

[0114] The calculation module 323 is used to use the difference between the average information entropy obtained by the second determination module 322 and the similarity obtained by the first determination module 321 as the confidence of the data.

[0115] Furthermore, the second determining module 322 is further configured to:

[0116] concatenating the first feature vector and the second feature vector into a combined feature vector;

[0117] Determining the label probability corresponding to each word in the data according to the combined feature vector;

[0118] Calculate the information entropy corresponding to each word segmentation according to the label probability;

[0119] The average information entropy of the data is determined according to the information entropy.

[0120] Further, such as Figure 5 As shown, the device also includes:

[0121] A setting unit 34, configured to set an upper limit value of the similarity determined by the determining unit 32;

[0122] The alarm unit 35 is used to prompt to modify the recognition model to improve the differentiation degree of parameter settings if the similarity between the first feature vector and the second feature vector corresponding to the data is higher than the upper limit value set by the setting unit 34.

[0123] Further, such as Figure 5 As shown, the selection unit 33 includes:

[0124] An acquisition module 331 is used to obtain the confidence of multiple data;

[0125] A sorting module 332, used for sorting the confidences obtained by the acquisition module 331 in descending order;

[0126] The selection module 333 is used to select data with high confidence from the multiple data as labeled data according to a preset number and the order determined by the ordering module 332.

[0127] Furthermore, the recognition model is a BiLSTM-CRF model based on a neural network.

[0128] In addition, an embodiment of the present invention further provides a processor, which is used to run a program, wherein the program executes the data processing method provided by any one of the above embodiments when running.

[0129] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0130] It is understandable that the related features in the above methods and devices can be referenced to each other. In addition, the "first", "second" and the like in the above embodiments are used to distinguish the embodiments, and do not represent the advantages and disadvantages of the embodiments.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0132] The algorithm and display provided herein are not inherently related to any particular computer, virtual system or other device. Various general purpose systems can also be used together with the teachings based on this. According to the above description, it is obvious that the structure required for constructing such systems. In addition, the present invention is not directed to any specific programming language either. It should be understood that various programming languages ​​can be utilized to realize the content of the present invention described herein, and the description of the above specific languages ​​is for disclosing the preferred embodiment of the present invention.

[0133] In addition, the memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash memory.

[0134] (flash RAM), the memory includes at least one memory chip.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0136] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0137] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0139] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0140] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0141] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0142] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0143] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0144] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data processing method, the method comprising: Obtaining a first feature vector and a second feature vector of the same piece of data, wherein the first feature vector and the second feature vector are obtained by extracting the same piece of data using a recognition model, and the recognition model uses a feature extraction method with different parameter settings; Determining the confidence of the data according to the first eigenvector and the second eigenvector; The data is selected as labeled data according to the confidence level.

2. The method according to claim 1, characterized in that The first eigenvector and the second eigenvector are obtained by: Inputting the data into a first embedding layer and a second embedding layer of the recognition model respectively; Inputting the content output by the first embedding layer into a first neural network layer to obtain a first feature vector; The content output by the second embedding layer is input into the second neural network layer to obtain a second feature vector.

3. The method according to claim 2, characterized in that Determining the confidence of the data according to the first feature vector and the second feature vector includes: determining a similarity between the first feature vector and the second feature vector; Determine the average information entropy of the data using the first eigenvector and the second eigenvector; The difference between the average information entropy and the similarity is used as the confidence of the data.

4. The method according to claim 3, characterized in that The determining the average information entropy of the data according to the first eigenvector and the second eigenvector includes: concatenating the first feature vector and the second feature vector into a combined feature vector; Determining the label probability corresponding to each word in the data according to the combined feature vector; Calculate the information entropy corresponding to each word segmentation according to the label probability; The average information entropy of the data is determined according to the information entropy.

5. The method according to claim 3, characterized in that: The method further comprises: Setting an upper limit value of the similarity; If the similarity between the first feature vector and the second feature vector corresponding to the data is higher than the upper limit, It is suggested to modify the recognition model to improve the degree of differentiation of parameter settings.

6. The method according to claim 1, characterized in that The selecting the data as the labeled data according to the confidence level includes: Get confidence of multiple data; Sorting in descending order of confidence; Data with high confidence is selected from the plurality of data as labeled data according to a preset number and the sorting.

7. The method according to any one of claims 2 to 6, characterized in that: The recognition model is a BiLSTM-CRF model based on a neural network.

8. A data processing device, comprising: an acquisition unit, configured to acquire a first feature vector and a second feature vector of the same piece of data, wherein the first feature vector and the second feature vector are obtained by extracting the same piece of data using a recognition model, and the recognition model adopts a feature extraction method with different parameter settings; a determining unit, configured to determine the confidence of the data according to the first feature vector and the second feature vector obtained by the acquiring unit; A selection unit is used to select the data as the labeled data according to the confidence obtained by the determination unit.

9. The device according to claim 8, characterized in that The acquisition unit comprises: An input module, used for inputting the data into a first embedding layer and a second embedding layer of a recognition model respectively; An acquisition module, configured to input the content output by the first embedding layer into a first neural network layer to obtain a first feature vector; The acquisition module is also used to input the content output by the second embedding layer into the second neural network layer to obtain a second feature vector.

10. The device according to claim 9, characterized in that The determining unit comprises: A first determining module, used to determine the similarity between the first feature vector and the second feature vector; A second determination module, configured to determine an average information entropy of the data using the first eigenvector and the second eigenvector; A calculation module is used to use the difference between the average information entropy obtained by the second determination module and the similarity obtained by the first determination module as the confidence of the data.

11. The device according to claim 10, characterized in that The second determining module is further used for: concatenating the first feature vector and the second feature vector into a combined feature vector; Determining the label probability corresponding to each word in the data according to the combined feature vector; Calculate the information entropy corresponding to each word segmentation according to the label probability; The average information entropy of the data is determined according to the information entropy.

12. The device according to claim 10, characterized in that The device also includes: a setting unit, configured to set an upper limit value of the similarity determined by the determining unit; The alarm unit is used to prompt to modify the recognition model to improve the differentiation degree of parameter settings if the similarity between the first feature vector and the second feature vector corresponding to the data is higher than the upper limit value set by the setting unit.

13. The device according to claim 8, characterized in that The selection unit comprises: An acquisition module, used for obtaining the confidence of multiple data; A sorting module, used for sorting the confidences obtained by the acquisition module in descending order; A selection module is used to select data with high confidence from the multiple data as labeled data according to a preset number and the sorting determined by the sorting module.

14. The device according to any one of claims 9 to 13, characterized in that The recognition model is a BiLSTM-CRF model based on a neural network.

15. A processor, characterized in that: The processor is used to run a program, wherein the program executes the data processing method according to any one of claims 1 to 7 when running.

Images