A method, apparatus, and storage medium for processing unlabeled data

Abstract

This invention discloses a method for processing unlabeled data. The method includes: comparing the semantic vector of the unlabeled data with the semantic vector of each labeled data to obtain a similarity score corresponding to each labeled data; selecting a preset number of similarities according to a preset selection rule; using a labeling model trained on the labeled data to predict the unlabeled data to obtain a prediction result for the unlabeled data; and classifying the unlabeled data into unlabeled data that can be labeled by a device or unlabeled data that cannot be labeled by a device based on the preset number of similarities and the prediction result. This invention also discloses a processing device and a storage medium for unlabeled data.

Description

Technical Field

[0001] This invention relates to computer application technology, and more particularly to a method, apparatus, and storage medium for processing unlabeled data. Background Technology

[0002] In question-answering systems, it's necessary to establish the relationship between user queries and knowledge points. This relationship is then used to train a semantic similarity model, which can improve question-answering performance. However, this model requires labeled data in the format "<user query>\t<knowledge point>", meaning manual annotation of user queries is essential to establish the relationship between them. This labeled data is used to train supervised models, but accumulating such data requires significant manual resources. Furthermore, related technologies have limited accuracy in distinguishing between unlabeled data that can be labeled by devices and unlabeled data that cannot. Summary of the Invention

[0003] To overcome the problems existing in related technologies, the present invention provides a method, apparatus and storage medium for processing unlabeled data.

[0004] According to a first aspect of the present invention, a method for processing unlabeled data is provided, including...

[0005] The semantic vectors of unlabeled data are compared with the semantic vectors of each labeled data to obtain the similarity score corresponding to each labeled data.

[0006] According to preset selection rules, a preset number of similarities are selected;

[0007] A labeled model trained on labeled data is used to predict unlabeled data, thus obtaining the prediction results for the unlabeled data.

[0008] Based on the preset number of similarities and the prediction results, the unlabeled data is divided into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device.

[0009] In one embodiment, selecting a preset number of similarities according to a preset selection rule includes:

[0010] Based on the similarity from high to low, a preset number of similarities are selected from the similarities corresponding to each labeled data.

[0011] In one embodiment, classifying the unlabeled data into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device based on the preset number of similarities and the prediction results includes:

[0012] When the statistical value of the similarity of the preset number reaches the preset similarity threshold,

[0013] and,

[0014] When the total number of labeled data corresponding to all similarities in the preset number of similarities is the same as the prediction result, and reaches the preset threshold of the same number, the unlabeled data is determined as unlabeled data that can be labeled by the device.

[0015] Otherwise, the unlabeled data will be determined as unlabeled data that cannot be labeled by the device.

[0016] In one embodiment, when the unlabeled data is determined to be unlabeled data that can be labeled by the device, the method further includes:

[0017] The unlabeled data is labeled using the labeling results of the labeled data corresponding to the highest similarity among the preset number of similarities.

[0018] or,

[0019] Determine the first annotation results of the first labeled data corresponding to the preset number of similarities;

[0020] Determine the total number of the first labeled results that are identical to the predicted results;

[0021] When the total number reaches a preset threshold of the same number, the unlabeled data is labeled using the prediction results.

[0022] In one embodiment, the preset number of statistical values ​​of the similarity include:

[0023] The highest similarity among the preset number of similarities, and / or the average similarity among the preset number of similarities.

[0024] In one embodiment, comparing the similarity between the semantic vector of the unlabeled data and the semantic vector of each labeled data includes:

[0025] The average word vector of the unlabeled data is compared with the average word vector of each labeled data.

[0026] or,

[0027] The prediction results of the Long Short-Term Memory (LSTM) network for the unlabeled data are compared with the prediction results of the LSTM network for each labeled data.

[0028] In one embodiment, obtaining the similarity to each labeled data point includes:

[0029] The cosine similarity is obtained for each of the labeled data.

[0030] or,

[0031] The Euclidean similarity to each of the labeled data is obtained.

[0032] According to a second aspect of the present invention, an apparatus for processing unlabeled data is provided, the apparatus comprising: a comparison module, a selection module, a prediction module, and a determination module; wherein,

[0033] The comparison module is used to compare the similarity of the semantic vector of the unlabeled data with the semantic vector of each labeled data, and to obtain the similarity with each labeled data.

[0034] The selection module is used to select a preset number of similarities according to preset selection rules;

[0035] The prediction module is used to use a labeled model trained on labeled data to predict unlabeled data and obtain the prediction result of the unlabeled data.

[0036] The determining module is used to classify the unlabeled data into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device, based on the preset number of similarities and the prediction results.

[0037] In one embodiment, the selection module includes:

[0038] The prediction submodule is used to select a preset number of similarities from the obtained similarities corresponding to each labeled data, sorted from high to low similarity.

[0039] In one embodiment, determining the module includes:

[0040] The determination submodule is used to determine when the statistical value of the similarity of the preset number of samples reaches a preset similarity threshold.

[0041] and,

[0042] When the total number of labeled data corresponding to all similarities within the preset number of similarities that are identical to the predicted result reaches a preset threshold,

[0043] The unlabeled data is determined to be unlabeled data that can be labeled by the device; otherwise, the unlabeled data is determined to be unlabeled data that cannot be labeled by the device.

[0044] In one embodiment, the apparatus further includes:

[0045] The annotation module is used to identify unannotated data that can be annotated by the device when the unannotated data is determined to be unannotated data.

[0046] The unlabeled data is labeled using the labeling results of the labeled data corresponding to the highest similarity among the preset number of similarities.

[0047] or,

[0048] Determine the first annotation results of the first labeled data corresponding to the preset number of similarities;

[0049] Determine the total number of the first labeled results that are identical to the predicted results;

[0050] When the total number reaches a preset threshold of the same number, the unlabeled data is labeled using the prediction results.

[0051] In one embodiment, the preset number of statistical values ​​of the similarity include:

[0052] The highest similarity among the preset number of similarities, and / or the average similarity among the preset number of similarities.

[0053] In one embodiment, the comparison module includes:

[0054] The comparison submodule may be used to compare the average word vector of the unlabeled data with the average word vector of each of the labeled data.

[0055] or,

[0056] The prediction results of the Long Short-Term Memory (LSTM) network for the unlabeled data are compared with the prediction results of the LSTM network for each labeled data.

[0057] In one embodiment, the similarity to each labeled data point includes:

[0058] The cosine similarity to each of the labeled data points.

[0059] or,

[0060] The Euclidean similarity to each of the labeled data.

[0061] According to a third aspect of the present invention, an apparatus for processing unlabeled data is provided, comprising a processor, a memory, and an executable program stored in the memory and executable by the processor, wherein the processor executes the steps of the unlabeled data processing method as described in the first aspect when executing the executable program.

[0062] According to a fourth aspect of the present invention, a storage medium is provided storing an executable program thereon, characterized in that the executable program, when executed by a processor, implements the steps of the unlabeled data processing method as described in the first aspect. The present invention discloses a method, apparatus, and storage medium for processing unlabeled data; comparing the semantic vector of unlabeled data with the semantic vector of each labeled data to obtain a similarity score corresponding to each labeled data; selecting a preset number of similarities according to a preset selection rule; using a labeling model trained on labeled data to predict the unlabeled data to obtain a prediction result for the unlabeled data; and classifying the unlabeled data into unlabeled data that can be labeled by a device or unlabeled data that cannot be labeled by a device based on the preset number of similarities and the prediction result. By establishing a direct link, i.e., similarity, between labeled data and unlabeled data, and combining the prediction result to distinguish whether unlabeled data can be labeled by a device, the accuracy of the distinction is improved, thereby reducing labeling errors caused by unlabelable data being labeled by a device.

[0063] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0064] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0065] Figure 1 This is a flowchart illustrating a method for processing unlabeled data according to an exemplary embodiment;

[0066] Figure 2 This is a detailed flowchart illustrating a method for processing unlabeled data according to an exemplary embodiment;

[0067] Figure 3 This is a block diagram illustrating an apparatus for processing unlabeled data according to an exemplary embodiment;

[0068] Figure 4 This is a block diagram illustrating another unlabeled data processing apparatus according to an exemplary embodiment;

[0069] Figure 5 This is a block diagram illustrating yet another unlabeled data processing apparatus according to an exemplary embodiment;

[0070] Figure 6 This is a block diagram illustrating another unlabeled data processing apparatus according to an exemplary embodiment;

[0071] Figure 7This is a block diagram illustrating another unlabeled data processing apparatus according to an exemplary embodiment;

[0072] Figure 8 This is a block diagram illustrating another unlabeled data processing apparatus according to an exemplary embodiment. Detailed Implementation

[0073] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0074] The terminology used in this embodiment of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the embodiments of the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of the invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0075] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of the present invention, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of embodiments of the present invention, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0076] Figure 1 This is a flowchart illustrating a method for processing unlabeled data according to an exemplary embodiment, such as... Figure 1 As shown, the method includes the following specific steps:

[0077] Step 101: Compare the similarity between the semantic vector of the unlabeled data and the semantic vector of each labeled data to obtain the similarity with each labeled data.

[0078] Step 102: Select a preset number of similarities according to preset selection rules;

[0079] Step 103: Use the labeled model trained on the labeled data to predict the unlabeled data and obtain the prediction results for the unlabeled data;

[0080] Step 104: Based on the preset number of similarities and the prediction results, the unlabeled data is divided into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device.

[0081] Here, labeled data can be statements that have been labeled with knowledge points in scenarios such as intelligent voice dialogue or text dialogue; unlabeled data can be statements that have not been labeled with knowledge points in scenarios such as intelligent voice dialogue or text dialogue; for example, in a question-and-answer system scenario, labeled data can be labeled user queries, unlabeled data can be unlabeled user queries, and the labeling result can be the knowledge points corresponding to the user query.

[0082] The semantic vectors of unlabeled data can be compared with the semantic vectors of multiple existing labeled data to obtain multiple similarity scores corresponding to the labeled data.

[0083] The preset selection rules can be determined based on the similarity level. Selecting a preset number of similarities from the above multiple similarities can improve processing efficiency in subsequent processing.

[0084] A supervised learning model trained on labeled data can be used to predict the prediction results of unlabeled data.

[0085] The preset number of similarities and prediction results are used as a reference to measure the feasibility of using the annotation results of labeled data to annotate unlabeled data, thereby improving the accuracy of distinguishing between unlabeled data that can be labeled by the device and unlabeled data that cannot be labeled by the device.

[0086] Before calculating semantic vectors, both labeled and unlabeled data can be deduplicated to reduce the amount of data and improve data processing speed and efficiency.

[0087] In one embodiment, selecting a preset number of similarities according to a preset selection rule includes:

[0088] Based on the similarity from high to low, a preset number of similarities are selected from the similarities corresponding to each labeled data.

[0089] Here, selecting similarity from high to low allows us to filter out labeled data that is more relevant to unlabeled data, providing more accurate reference data for distinguishing between unlabeled data that can be labeled by the device and unlabeled data that cannot be labeled by the device.

[0090] In one embodiment, classifying the unlabeled data into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device based on the preset number of similarities and the prediction results includes:

[0091] When the statistical value of the preset number of similarities reaches a preset similarity threshold, and when the total number of annotation results of the labeled data corresponding to all similarities in the preset number of similarities is the same as the prediction result, the unlabeled data is determined to be unlabeled data that can be labeled by the device; otherwise, the unlabeled data is determined to be unlabeled data that cannot be labeled by the device.

[0092] Each of the preset number of similarities corresponds to a labeling result of an already labeled data, and the similarity can be used as the confidence level of the corresponding labeling result.

[0093] The statistical value can be the maximum, minimum, or arithmetic mean of a preset number of similarities. By comparing the statistical value of the preset number of similarities with a preset similarity threshold, the overall confidence level or highest confidence level of each annotation result corresponding to the preset number of similarities can be determined.

[0094] The statistical values ​​can be compared with a preset similarity threshold to determine the overall confidence or highest confidence of the annotation results corresponding to a preset number of similarities.

[0095] By comparing the annotation results corresponding to all similarities in the preset number of similarities with the prediction results, we can obtain the number of annotation results that are the same as the prediction results for all similarities. For example, by comparing the annotation results corresponding to 10 similarities with the prediction results, we can obtain 5 annotation results that are the same as the prediction results.

[0096] Based on the two comparisons, the reliability of the unlabeled data labeled with equipment can be determined.

[0097] If the statistical value reaches a preset similarity threshold, and the total number of data that is the same as the predicted result reaches a preset same number threshold, then the reliability of the unlabeled data labeled by the device is considered to be high, and the unlabeled data can be modified by device labeling.

[0098] If the statistical value does not reach the preset similarity threshold, and / or the total number of the same as the predicted result does not reach the preset same number threshold, then the reliability of the unlabeled data labeled by the device is considered to be low, and the unlabeled data cannot be labeled by the device; it needs to be labeled manually.

[0099] By combining similarity and the number of identical labels and predictions, the degree of association between labeled and unlabeled data can be determined. Furthermore, the confidence level of unlabeled data labels can be judged by the confidence levels of multiple labeling results, thereby improving the accuracy of distinguishing unlabeled data labels.

[0100] For example, if there are 10 similarities, the average similarity is used as the statistical value, and the preset similarity threshold is 0.9. The preset threshold for the number of identical items is 5.

[0101] If the average similarity of the preset number is 0.95, which is greater than the preset similarity threshold, and 6 of the 10 similarity values ​​correspond to the same annotation result as the prediction result, which is greater than the preset same number threshold, then the unlabeled data is considered to be unlabeled data that can be guaranteed by the device.

[0102] If the average similarity of the preset number is 0.89, which is less than the preset similarity threshold, or if 4 out of 10 similarity values ​​correspond to the same prediction result, which is less than the preset same number threshold, then the unlabeled data is considered to be unlabeled data that cannot be guaranteed by the device.

[0103] In practical applications, determining the corresponding labeled data based on similarity can be achieved by building an index. First, using supervised information, i.e., labeled data, in the dialogue system, data in the form of "<user query>\t<knowledge point>" is organized. After deduplication, a user query vector is calculated using an unsupervised method. Here, the user query is the labeled data, and the knowledge point is the labeled result of the labeled data. Two indexes are built using the query user identifier index (query id index) corresponding to the user query, along with the semantic vector of the user query and the corresponding knowledge point information. The first type of index is the labeled result index, with the key being the query id index and the index value being the labeled result corresponding to the user query. The second type of index is the semantic index, with the key being the query id index and the index value being the semantic vector of the user query. This results in a labeled result index library and a semantic index library.

[0104] In the unlabeled data, an unsupervised method is also used to calculate the user query vector. For each user query vector in the unlabeled data, the semantic index is queried to obtain the K most similar user query ID indices. Then, the corresponding labeled results are queried from the labeled result index using the K most similar user query IDs. Each labeled result corresponds to a similarity, resulting in the K most similar labeled results. Here, K is the preset number, which can be determined based on the actual situation such as the total number of labeled data samples, and can be 3 to 20, such as 3, 5, or 10.

[0105] Finally, based on the K annotation results and their corresponding confidence scores (i.e., the corresponding similarity scores), it is determined whether to use a device to annotate the unannotated data or to use a manual annotation of the unannotated data. The confidence score threshold can be configured according to the actual situation.

[0106] In one embodiment, when the unlabeled data is determined to be unlabeled data that can be labeled by the device, the method further includes: labeling the unlabeled data using the labeling result of the labeled data corresponding to the highest similarity among the preset number of similarities, or determining the first labeling result of the first labeled data corresponding to the preset number of similarities respectively; determining the total number of the first labeling results that are the same as the prediction result; when the total number reaches a preset threshold of the same number, labeling the unlabeled data using the prediction result.

[0107] Here, the unlabeled data can be labeled using a device. The labeling result with the highest confidence, i.e., the labeling result corresponding to the highest similarity, can be used to label the unlabeled data; thus, using the labeling result with the highest confidence to label the unlabeled data can improve the labeling accuracy.

[0108] Alternatively, among the labeled data corresponding to all similarities in the preset number of similarities, select the labeled data that is the same as the predicted data. If the total number of labeled data that is the same as the predicted data reaches the preset threshold, then use the labeled data that is the same as the predicted data to label the unlabeled data. Using the labeled data with a higher probability to label the unlabeled data can also improve the labeling accuracy.

[0109] In one embodiment, the preset number of statistical values ​​of the similarity include:

[0110] The highest similarity among the preset number of similarities, and / or the average similarity among the preset number of similarities.

[0111] Here, the similarity between the semantic vectors of labeled data and unlabeled data can be determined from multiple aspects using the statistical values ​​of the similarity. For example, the labeled data that is semantically closest to the unlabeled data can be determined by the highest similarity; the average similarity can determine the overall semantic similarity between the labeled data and the unlabeled data corresponding to the preset number of similarities, thus providing a standard for selecting the labeling results based on the unlabeled data.

[0112] The preset similarity threshold for comparison with the highest similarity can be 0.95, and the preset similarity threshold for comparison with the average similarity can be 0.90.

[0113] In one embodiment, comparing the similarity between the semantic vectors of the unlabeled data and the semantic vectors of each labeled data includes: comparing the average word vector of the unlabeled data with the average word vector of each labeled data, or comparing the LSTM prediction results of the unlabeled data with the LSTM prediction results of each labeled data.

[0114] To calculate the average word vector of a user query (either the labeled data or the unlabeled data), we can first calculate the vector representation of each word in the user query. Assuming the dictionary size is V and the word vector dimension is d, the word vector of the i-th word is... Then calculate the average word vector. Assuming there are n words in the labeled or unlabeled data, the average word vector is...

[0115] The LSTM cell structure in the LSTM method can be defined, and the corresponding preset LSTM results of labeled or unlabeled data can be taken. For example, the prediction result of the last word in the user query can be used as the semantic index result.

[0116] In one embodiment, obtaining the similarity to each labeled data includes: obtaining the cosine similarity to each labeled data, or obtaining the Euclidean similarity to each labeled data.

[0117] Specifically, when comparing the semantic vectors of unlabeled data with the semantic vectors of each labeled data, cosine similarity and Euclidean similarity can be used for comparison. The calculation of cosine similarity can be expressed by expression (1):

[0118]

[0119] The calculation of Euclidean similarity can be expressed by expression (2):

[0120]

[0121] In expression (1) or expression (2), v i and v j This represents two semantic vectors being compared, where d represents the word vector dimension.

[0122] The following is a specific example in conjunction with the above embodiments:

[0123] Establishing a direct link between labeled and unlabeled data, and utilizing active learning methods to improve labeling efficiency and accuracy, the entire process is as follows: Figure 2 As shown.

[0124] Semantic indexing establishes connections between labeled and unlabeled data. Unlike traditional word-based matching methods such as TF-IDF or BM25, it uses semantic vectors as index elements, capturing semantic relevance during the query process. For example, if the index storage contains the query "What to do if my phone is broken?", and the user query is "My phone is malfunctioning", TF-IDF and BM25 can only establish a relationship between the user query and index A through "phone". However, semantic indexing can establish a connection between "broken" and "malfunctioning", retrieving information from index A.

[0125] First, using supervised information, i.e. labeled data, we organize data in the form of "<user query>\t<knowledge point>" in the dialogue system. After deduplication, we combine unsupervised methods to calculate the semantic vector of the user query. There are many methods to calculate the semantic vector of the user query. In the experiment, we can use word vector averaging and LSTM.

[0126] For the labeled data, two indexes are created: the query id index and the corresponding knowledge point information. The first type of index is the labeling result index, with the key value being the query id index and the index value being the labeling result corresponding to the user's query. A labeling result index library is created. The second type of index is the semantic index, with the key value being the query id index and the index value being the semantic vector of the user's query. A semantic index library is created.

[0127] For unlabeled data, the user query vector is still calculated using an unsupervised method, resulting in a semantic vector for the user query of each unlabeled data point.

[0128] The semantic vectors of user queries from unlabeled data are queried in the semantic index to obtain the top K similar user query IDs with the highest similarity. The similarity measurement methods can include cosine similarity, Euclidean similarity, etc.

[0129] Then, using the query IDs of the top K similar users, the corresponding annotation results are retrieved from the annotation result index. Each annotation result corresponds to a similarity score. Based on the confidence level, i.e., the similarity score, it is determined whether to annotate the unannotated data. The confidence level criterion is configurable.

[0130] In the process of building the semantic index, word vector averaging and LSTM methods are used respectively. The word vector method first calculates the vector representation of each word in the user query. Assuming the dictionary size is V and the word vector dimension is d, the word vector of the i-th word is... Then calculate the average word vector. Assuming there are n words in the labeled or unlabeled data, the average word vector is...

[0131] The LSTM method defines an LSTM cell structure, which takes the prediction result of the last word in the user query as the semantic index result.

[0132] Specifically, when comparing the semantic vectors of unlabeled data with the semantic vectors of each labeled data, cosine similarity and Euclidean similarity can be used for comparison. The calculation of cosine similarity can be expressed by expression (1), and the calculation of Euclidean similarity can be expressed by expression (2):

[0133] After selecting the annotation results corresponding to the Top K similarities, the average of the highest similarity and the Top K similarities is used to filter the samples to be annotated. If the average of the highest similarity and the Top K similarities is lower than a certain threshold, it is added to the set of samples to be manually annotated.

[0134] Here, five methods—random strategy, word vector average + cosine similarity, LSTM + cosine similarity, word vector average + Euclidean distance, and LSTM + Euclidean distance—are used to select an equal number of unlabeled samples from two datasets to be labeled. Through manual labeling, the differences between the manually labeled data (those that cannot be labeled using a device) and the device-labeled data (those that can be labeled using a device) are statistically analyzed. A larger difference indicates a better distinction between data that can and cannot be labeled using a device. Assuming there are N equal samples to be labeled, and M samples whose labeling results differ from the semantic index retrieval results, the difference is defined as... The specific results are shown in Table 1. As can be seen from Table 1, the difference achieved by the method of the present invention is better than that achieved by the random strategy.

[0135] Table 1

[0136] Random Policy 0.37 0.42 Word vector average + cosine similarity 0.51 0.49 LSTM+cosine similarity 0.48 0.47 Word vector average + Euclidean distance 0.43 0.47 LSTM+Euclidean Distance 0.48 0.5

[0137] Figure 3 This is a block diagram of an apparatus 100 for processing unlabeled data according to an exemplary embodiment. (Refer to...) Figure 2 The device includes: a comparison module 110, a selection module 120, a prediction module 130, and a determination module 140; wherein,

[0138] The comparison module 110 is used to compare the similarity of the semantic vector of the unlabeled data with the semantic vector of each labeled data, and obtain the similarity with each labeled data.

[0139] The selection module 120 is used to select a preset number of similarities according to preset selection rules;

[0140] The prediction module 130 is used to use a labeled model trained on labeled data to predict unlabeled data and obtain the prediction result of the unlabeled data.

[0141] The determining module 140 is used to classify the unlabeled data into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device, based on the preset number of similarities and the prediction results.

[0142] In one embodiment, such as Figure 4 As shown, the selection module 120 includes:

[0143] The prediction submodule 121 is used to select a preset number of similarities from the obtained similarities corresponding to each labeled data, sorted from high to low similarity.

[0144] In one embodiment, such as Figure 5 As shown, the determining module 140 includes:

[0145] The determining submodule 141 is used to determine when the statistical value of the similarity of the preset number reaches a preset similarity threshold.

[0146] and,

[0147] When the total number of labeled data corresponding to all similarities within the preset number of similarities that are identical to the predicted result reaches a preset threshold,

[0148] The unlabeled data is determined to be unlabeled data that can be labeled by the device; otherwise, the unlabeled data is determined to be unlabeled data that cannot be labeled by the device.

[0149] In one embodiment, such as Figure 6 As shown, the device 100 further includes:

[0150] The annotation module 150 is used to, when the unannotated data is determined to be unannotated data that can be annotated by the device,

[0151] The unlabeled data is labeled using the labeling results of the labeled data corresponding to the highest similarity among the preset number of similarities.

[0152] or,

[0153] Determine the first annotation results of the first labeled data corresponding to the preset number of similarities;

[0154] Determine the total number of the first labeled results that are identical to the predicted results;

[0155] When the total number reaches a preset threshold of the same number, the unlabeled data is labeled using the prediction results.

[0156] In one embodiment, the preset number of statistical values ​​of the similarity include:

[0157] The highest similarity among the preset number of similarities, and / or the average similarity among the preset number of similarities.

[0158] In one embodiment, such as Figure 7 As shown, the comparison module 110 includes:

[0159] The comparison submodule 111 is used to compare the average word vector of the unlabeled data with the average word vector of each of the labeled data.

[0160] or,

[0161] The prediction results of the Long Short-Term Memory (LSTM) network for the unlabeled data are compared with the prediction results of the LSTM network for each labeled data.

[0162] In one embodiment, the similarity to each labeled data point includes:

[0163] The cosine similarity to each of the labeled data points.

[0164] or,

[0165] The Euclidean similarity to each of the labeled data.

[0166] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0167] Figure 8 This is a block diagram illustrating an apparatus 800 for processing unlabeled data according to an exemplary embodiment. For example, apparatus 800 may be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, etc.

[0168] Reference Figure 8 The device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0169] Processing component 802 typically controls the overall operation of device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 802 may include one or more processors 820 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0170] Memory 804 is configured to store various types of data to support the operation of device 800. Examples of this data include instructions for any application or method operating on device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0171] Power supply component 806 provides power to various components of device 800. Power supply component 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to device 800.

[0172] Multimedia component 808 includes a screen that provides an output interface between the device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0173] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0174] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0175] Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of device 800. For example, sensor assembly 814 may detect the on / off state of device 800, the relative positioning of components such as the display and keypad of device 800, changes in the position of device 800 or a component of device 800, the presence or absence of user contact with device 800, the orientation or acceleration / deceleration of device 800, and temperature changes of device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0176] Communication component 816 is configured to facilitate wired or wireless communication between device 800 and other devices. Device 800 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0177] In an exemplary embodiment, the apparatus 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0178] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 804 including instructions, which can be executed by a processor 820 of the device 800 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0179] A non-transitory computer-readable storage medium, when instructions in the storage medium are executed by a processor of a mobile terminal, enables the mobile terminal to perform a method for processing unlabeled data, the method comprising:

[0180] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0181] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A method for processing unlabeled data, characterized in that, include: The semantic vectors of unlabeled data are compared with the semantic vectors of each labeled data to obtain the similarity score corresponding to each labeled data. According to preset selection rules, a preset number of similarities are selected; A labeled model trained on labeled data is used to predict unlabeled data, thus obtaining the prediction results for the unlabeled data. Based on the preset number of similarities and the prediction results, the unlabeled data is divided into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device. The step of classifying the unlabeled data into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device based on the preset number of similarities and the prediction results includes: When the statistical value of the similarity of the preset number reaches the preset similarity threshold, and, When the total number of labeled data corresponding to all similarities in the preset number of similarities is the same as the prediction result, and reaches the preset threshold of the same number, the unlabeled data is determined as unlabeled data that can be labeled by the device. Otherwise, the unlabeled data will be determined as unlabeled data that cannot be labeled by the device; The step of selecting a preset number of similarities according to preset selection rules includes: Based on the similarity scores from high to low, a preset number of similarity scores are selected from the similarity scores corresponding to each labeled data. The preset number of statistical values ​​of the similarity include: The highest similarity among the preset number of similarities, and / or the average similarity among the preset number of similarities; When the unlabeled data is determined to be unlabeled data that can be labeled by the device, the method further includes: Determine the first annotation results of the first labeled data corresponding to the preset number of similarities; Determine the total number of the first labeled results that are identical to the predicted results; When the total number reaches a preset threshold of the same number, the unlabeled data is labeled using the prediction results.

2. The method according to claim 1, characterized in that, When the unlabeled data is determined to be unlabeled data that can be labeled by the device, the method further includes: The unlabeled data is labeled using the labeling results of the labeled data corresponding to the highest similarity among the preset number of similarities.

3. The method according to any one of claims 1 or 2, characterized in that, The step of comparing the similarity between the semantic vector of the unlabeled data and the semantic vector of each labeled data includes: The average word vector of the unlabeled data is compared with the average word vector of each labeled data. or, The prediction results of the Long Short-Term Memory (LSTM) network for the unlabeled data are compared with the prediction results of the LSTM network for each labeled data.

4. The method according to claim 3, characterized in that, The process of obtaining the similarity to each labeled data point includes: The cosine similarity is obtained for each of the labeled data. or, The Euclidean similarity to each of the labeled data is obtained.

5. A processing apparatus for unlabeled data, characterized in that, The device includes: a comparison module, a selection module, a prediction module, and a determination module; wherein, The comparison module is used to compare the similarity of the semantic vector of the unlabeled data with the semantic vector of each labeled data, and to obtain the similarity with each labeled data. The selection module is used to select a preset number of similarities according to preset selection rules; The prediction module is used to use a labeled model trained on labeled data to predict unlabeled data and obtain the prediction result of the unlabeled data. The determining module is used to classify the unlabeled data into unlabeled data that can be labeled by the device or unlabeled data that cannot be labeled by the device, based on the preset number of similarities and the prediction results. The determining module includes: The determination submodule is used to determine when the statistical value of the similarity of the preset number of samples reaches a preset similarity threshold. and, When the total number of labeled data corresponding to all similarities within the preset number of similarities that are identical to the predicted result reaches a preset threshold, The unlabeled data is determined to be unlabeled data that can be labeled by the device; otherwise, the unlabeled data is determined to be unlabeled data that cannot be labeled by the device. The selection module includes: The prediction submodule is used to select a preset number of similarities from the obtained similarities corresponding to each labeled data, sorted by similarity from high to low. The preset number of statistical values ​​of the similarity include: The highest similarity among the preset number of similarities, and / or the average similarity among the preset number of similarities; The device further includes a labeling module, used to determine the first labeling result of the first labeled data corresponding to the preset number of similarities; Determine the total number of the first labeled results that are identical to the predicted results; When the total number reaches a preset threshold of the same number, the unlabeled data is labeled using the prediction results.

6. The apparatus according to claim 5, characterized in that, The device further includes: The annotation module is used to identify unannotated data that can be annotated by the device when the unannotated data is determined to be unannotated data. The unlabeled data is labeled using the labeling results of the labeled data corresponding to the highest similarity among the preset number of similarities.

7. The apparatus according to any one of claims 5 or 6, characterized in that, The comparison module includes: The comparison submodule may be used to compare the average word vector of the unlabeled data with the average word vector of each of the labeled data. or, The prediction results of the Long Short-Term Memory (LSTM) network for the unlabeled data are compared with the prediction results of the LSTM network for each labeled data.

8. The apparatus according to claim 7, characterized in that, The similarity to each of the labeled data includes: The cosine similarity to each of the labeled data points. or, The Euclidean similarity to each of the labeled data.

9. An apparatus for processing unlabeled data, comprising a processor, a memory, and an executable program stored in the memory and capable of being executed by the processor, characterized in that, When the processor runs the executable program, it performs the steps of the method for processing unlabeled data as described in any one of claims 1 to 4.

10. A storage medium storing an executable program thereon, characterized in that, When the executable program is executed by the processor, it implements the steps of the method for processing unlabeled data as described in any one of claims 1 to 4.

Images