Data classification method for filtering outlier text data

Abstract

To provide a data classifying method.SOLUTION: A data classification method includes: a step for obtaining multiple text samples from a dataset; a step for converting the text samples into multiple text embeddings in a semantic space; a step for generating an outlier-inlier ranking of the text samples based on an outlier detection algorithm according to distances between the text embeddings in the semantic space; a step for selecting multiple partial samples from the text samples according to the outlier-inlier ranking; a step for receiving a manual input command to assign manual-input labels on the partial samples; a step for generating a prompt message according to the partial samples with the manual-input labels and with unlabeled samples of the text samples; and a step for providing the prompt message to a generative pre-trained transformer model for generating multiple inlier-outlier prediction labels relating to the unlabeled samples.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to classification methods, and in particular to classification methods for identifying inliers or outliers in unlabeled data. [Background technology]

[0002] In machine learning technology, outlier detection is a common operational method used to identify whether there are data instances in a data set that deviate significantly from normal distribution. Outlier detection is important in a variety of application areas, including medical prevention, fraud detection, network security, quality control, and anomaly detection in medical health or industrial processes. Summary of the Invention

[0003] The present disclosure provides a data classification method, including obtaining a plurality of character samples from a dataset; converting the plurality of character samples into a plurality of character embeddings in a semantic space; generating an outlier-inlier order of the plurality of character samples based on an outlier detection algorithm based on a distance between the character embeddings in the semantic space; selecting a plurality of sub-samples from the character samples based on the outlier-inlier order; receiving a manual input instruction to specify manual input labels for the sub-samples; generating a prompt message including task instructions, unlabeled data, and anchor data generated based on the partial samples with the manually input labels and a plurality of unlabeled samples in the character samples; and providing the prompt message to a generative pre-training Transformer model to generate a plurality of inlier-outlier predicted labels associated with the plurality of unlabeled samples.

[0004] The present disclosure provides another method for classifying data, the method comprising: obtaining a plurality of character samples from a dataset; converting the plurality of character samples into a plurality of character embeddings in a semantic space; generating an outlier-inlier order of the character samples based on a distance between the character embeddings in the semantic space based on an outlier detection algorithm; selecting a plurality of sub-samples from the character samples based on the outlier-inlier order; receiving manual input instructions to specify a plurality of manual input labels for the sub-samples; generating a first prompt message including the sub-samples with the manual input labels and feature engineering task instructions; providing the first prompt message to a generative pre-training Transformer model to generate a plurality of distinguishable features; generating a second prompt message including the distinguishable features, the character samples, and feature scoring task instructions; providing the second prompt message to a generative pre-training Transformer model to generate a plurality of feature predictions for the distinguishable features of the character samples; and running a classification algorithm based on the feature predictions of the sub-samples in the character samples and the unlabeled samples to generate a plurality of inlier-outlier predicted labels for the unlabeled samples.

[0005] It should be noted that the present application is intended to further interpret the invention which has been set forth in the above general description and the following detailed description by way of examples and which are claimed. [Brief description of the drawings]

[0006] [Figure 1] FIG. 1 is a block diagram of an electronic device according to an embodiment of the present disclosure. [Diagram 2] 1 is a flowchart of a data classification method according to an embodiment of the present disclosure. [Figure 3A] FIG. 2 is a schematic diagram of character samples taken from a dataset according to one embodiment of the present disclosure. [Figure 3B]FIG. 2 is a schematic diagram of character embeddings in semantic space for character samples taken from a dataset according to one embodiment of the present disclosure. [Figure 3C] FIG. 13 is a schematic diagram of an outlier-inlier order of character samples according to an embodiment of the present disclosure. [Figure 3D] FIG. 13 is a schematic diagram of a partial sample and a manually entered label associated with the partial sample according to one embodiment of the present disclosure. [Figure 3E] FIG. 13 illustrates how to generate a prompt message based on a partial sample of a manually input label and an unlabeled sample of a character sample according to one embodiment of the present disclosure. [Figure 4] FIG. 2 is a schematic diagram of a prompt message according to an embodiment of the present disclosure. [Diagram 5] 1 is a flowchart of a data classification method according to another embodiment of the present disclosure. [Figure 6] FIG. 13 is a schematic diagram of a first prompt message according to an embodiment of the present disclosure. [Figure 7] FIG. 13 is a schematic diagram of a second prompt message according to an embodiment of the present disclosure. [Figure 8] FIG. 2 illustrates how a classification algorithm generates inlier-outlier predicted labels according to one embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Embodiments of the present disclosure will now be described with reference to the associated drawings, in which like reference numbers represent the same or similar elements or method steps.

[0008] When developing a chat robot or a large-scale language model, it is very important to collect a training dataset with input characters and corresponding output characters. For example, when developing a chat robot, the input characters may be example questions of potential users, and the output characters may be answers that the chat robot should provide. The training dataset is not only used for model development (e.g., building a chat robot), but also for model evaluation (e.g., evaluating the accuracy or effectiveness of the chat robot). In known techniques, the training dataset is manually labeled by experts, but hiring such a team is very expensive. Using an automatic generator to automatically generate the training dataset (automatically generate corresponding training character samples) instead of the above-mentioned human labelers can reduce the cost and time required to build the training dataset. The training character samples generated by the automatic generator have unstable quality, so the generated character samples need to be verified.

[0009] The above validation is used to remove low-quality samples (e.g., erroneously generated characters) from the training data set, and use the validated character samples in the training data set during model development and model evaluation. Since low-quality samples have a serious impact on the effectiveness of the model, removing low-quality samples during the validation process can contribute to improving the model quality when building a subsequent model.

[0010] In some cases, the automatically generated dataset contains more than 10,000 sample characters that need to be verified. One solution is to hire professional labelers to manually inspect the quality of all the sample characters. However, the cost of manual inspection is very high, and the manual inspection process is very slow.

[0011] Please refer to FIG. 1 and FIG. 2 together. FIG. 1 is a block diagram of an electronic device 100 according to an embodiment of the present disclosure. FIG. 2 is a flow chart of a data classification method 200 according to an embodiment of the present disclosure. The data classification method 200 is used to generate inlier-outlier predictions associated with text samples in a dataset DB. The dataset DB may be obtained from a text content server. For example, the dataset DB may be collected from an encyclopedia website, a news website, a question and answer database, a forum, a novel storage server, a journal database, or any similar text content storage center.

[0012] As shown in FIG. 1 , the electronic device 100 includes a processing unit 110, an input interface 120, a storage unit 130, a display 140, and a communication circuit 150. In some embodiments, the electronic device 100 may be a computer, a smartphone, a tablet computer, an image processing server, a data server, or any image processing device with similar functionality. The input interface 120 is used to receive the dataset DB and to manually input instructions. In some embodiments, the electronic device 100 is used to classify character samples in the dataset DB, generate inlier-outlier predictions for the character samples, and remove the outlier samples and keep the inlier samples based on the inlier-outlier predictions.

[0013] The input interface 120 may include a data transmission interface, a wireless communication circuit, a keyboard, a mouse, a microphone, or any input device having similar functionality. The processing unit 110 is coupled to the input interface 120, the storage unit 130, the display 140, and the communication circuit 150. The storage unit 130 is used to store program code. The program code stored in the storage unit 130 is used to instruct the processing unit 110 to execute the data classification method 200 shown in FIG. 2. In some embodiments, the processing unit 110 may be any of a processor, a graphics processor, an application specific integrated circuit, or a processing circuit having similar functionality. The communication circuit 150 may be a network transceiver (e.g., a wireless area network transceiver, a telecommunications transceiver, or an Ethernet transceiver). The communication circuit 150 is used to contact and communicate with the generative pre-training transformer model 190. In some embodiments, the generative pre-training transformer model 190 may be an independently operating separate model that is operated on an external server other than the electronic device 100.

[0014] Most ideally, a machine learning model is trained based on a training dataset that is free of outliers. When a training dataset contains outliers, it can affect the effectiveness and behavior of the machine learning model in various ways. For example, outliers in a training dataset can cause problems such as model deviation, increased model complexity, overfitting, reduced robustness, and difficulty in anomaly detection.

[0015] In a large-scale language model application program, the unlabeled text paragraphs in the training dataset DB may include different types of text content, such as news reports, popular novels, literary works, bank statements, chat records, questions and answers, research papers, or program codes. The different text contents have different applications in different fields. These text contents need to be filtered to remove noise data and extract clean data, such as valid text samples, to train a machine learning model for a specific purpose.

[0016] For example, in one embodiment, if the training goal of a machine learning model is to be used to answer medical questions, text content related to medical notes, health discussions, diagnoses, symptoms, and / or medical history should be considered as normal text samples, while other text content related to global warming, financial crises, and / or baseball games should be considered as outlying text samples.

[0017] Manually labeling the in-between and out-of-bench samples in a dataset DB can be time-consuming and costly, especially when dealing with large datasets. In such cases, experts need to manually review and identify the in-between and out-of-bench samples, which may not be practical for dealing with large datasets with a large number of character samples.

[0018] In some embodiments, the electronic device 100 and data classification method 200 provide a time- and cost-beneficial scheme to generate inlier-outlier predictions for each of a plurality of character samples in a dataset DB.

[0019] As shown in FIG. 1 and FIG. 2, the processing unit 110 executes step S210 of acquiring character samples from the dataset DB. Please further refer to FIG. 3A. FIG. 3A is a schematic diagram of character samples TS1, TS2, TS3...TS14 acquired from the dataset DB according to one embodiment of the present disclosure. The character samples TS1-TS14 may be character paragraphs (e.g., sentences, paragraphs, or paragraphs) or combinations of questions and corresponding answers. The number of character samples TS1-TS14 shown in FIG. 3A is a simplified example for display purposes, and the present disclosure is not limited thereto. In practice, the dataset DB may include hundreds, thousands, or more character samples. In some embodiments, the character samples TS1-TS14 may be acquired from the dataset DB by a character segmentation algorithm executed by the processing unit 110.

[0020] For example, the character sample TS1 may be "Question: How can I get rid of my headache? Answer: Try putting a heating pad on my neck", the character sample TS2 may be "Question: How can I reduce abdominal fat? Answer: Moderate exercise and a diet plan may help", and the character sample TS3 may be "Question: What is the weather like today? Answer: It's sunny outside." The character samples TS1-TS14 may include normal samples related to medical topics (e.g., character samples TS1 and TS2) and outlier samples not related to medical topics (e.g., character sample TS3). As shown in FIG. 3A, assume that the character samples TS3, TS6, TS7, TS11, and TS13 are noise data (i.e., outlier samples not related to medical topics), and the character samples TS1, TS2, TS4, TS5, TS8, TS9, TS10, TS12, and TS14 are clean data (i.e., normal samples related to medical topics). In this embodiment, the character samples TS1-TS14 are not initially labeled. The electronic device 100 and data classification method 200 are used to label the character samples TS1-TS14 and separate the noisy data from the clean data.

[0021] 1 and 2, the processing unit 110 performs step S220 of converting the character samples TS1-TS14 into character embeddings in a semantic space. Please refer to Fig. 3B for further details. Fig. 3B is a schematic diagram of character embeddings eTS1-eTS14 in a semantic space SS for the character samples TS1-TS14 obtained from the dataset DB according to an embodiment of the present disclosure.

[0022] The character embedding eTS1 is a projection result of the character sample TS1 into a high-dimensional latent space. The character embedding eTS1 in the semantic space SS is a vector or a series of digit strings. For convenience of illustration, the semantic space SS shown in FIG. 3B is described as a two-dimensional distribution, and in practice, the semantic space SS may have more dimensions, such as 768 dimensions or 1536 dimensions. If two of the character samples TS1-TS14 have similar meanings, the corresponding two of the character embeddings eTS1-eTS14 will be close to each other in the semantic space SS. On the other hand, if two of the character samples TS1-TS14 have different meanings, the corresponding two of the character embeddings eTS1-eTS14 will be far away from each other in the semantic space SS.

[0023] 1 and 2, the processing unit 110 performs step S230 of generating an outlier-inlier order of the character samples TS1-TS14 in the semantic space SS based on the distances between the character embeddings eTS1-eTS14 based on an outlier detection algorithm. Please further refer to FIG. 3C, which is a schematic diagram of an outlier-inlier order RK of the character samples TS1-TS14 according to one embodiment of the present disclosure.

[0024] In some implementations, the outlier detection algorithm may be implemented by a random sampling consistency-NN algorithm. In some other implementations, the outlier detection algorithm may be an isolation forest algorithm or a local outlier factor algorithm.

[0025] The processing unit 110 is used to perform an outlier detection algorithm (e.g., a random sampling consistency-NN algorithm) on the character samples TS1-TS14 based on the distance between the character embeddings eTS1-eTS14 in the semantic space SS, and the processing unit 110 can obtain an "inlier score" for each sample. A sample that is adjacent to other character embeddings (having a short distance to other characters) in the character embeddings eTS1-eTS14 has a high "inlier score", while a sample that is far away from other character embeddings (having a long distance to other characters) in the character embeddings eTS1-eTS14 has a low "inlier score". The closer the "inlier score" is to 1, the higher the quality of this character sample is indicated. The lower the "inlier score" is, the lower the quality of this character sample is indicated. Since each of the character samples TS1-TS14 has its own "inlier score", an outlier-inlier order RK for the character samples TS1-TS14 can be generated.

[0026] Based on the outlier-inlier order RK shown in Fig. 3C, the character samples TS11, TS6, TS9, TS13 and TS5 (having low inlier scores) tend to belong to the outlier samples. Also, as shown in Fig. 3C, the character samples TS11, TS6, TS9, TS13 and TS5 are all arranged at the top of the outlier-inlier order RK.

[0027] Based on the outlier-inlier order RK shown in Fig. 3C, the character samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2 and TS7 (having high inlier scores) tend to belong to the inlier samples. Also, as shown in Fig. 3C, the character samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2 and TS7 are arranged at the bottom of the outlier-inlier order RK.

[0028] 1, 2 and 3C, the processing unit 110 executes step S240 of selecting partial samples PS from the character samples TS1-TS14 based on the outlier-inlier order RK. As shown in FIG. 3C, based on the outlier-inlier order RK, the character samples TS11, TS6, TS9, TS13 and TS5 that tend to belong to the outlier samples are selected as the partial samples PS. The partial samples PS are transmitted from the processing unit 110 to the display 140.

[0029] In some embodiments, the label creator can view / verify the partial sample PS on the display 140, and then the label creator can input a manually input label MLB associated with the partial sample PS via the input interface 120. See further Fig. 3D, which is a schematic diagram of a partial sample PS and a manually input label MLB associated with the partial sample PS according to one embodiment of the present disclosure.

[0030] As shown in Figure 3D, the text content of character samples TS11, TS6, TS9, TS13, and TS5 included in the text content of sub-sample PS may be displayed on display 140 shown in Figure 1. A labeller may read the text content associated with character samples TS11, TS6, TS9, TS13, and TS5, and then enter a manual input label MLB associated with sub-sample PS via input interface 120. In some embodiments, the labeller may choose to remove character samples TS11, TS6, and TS13 (which are not related to medical topics) and retain character samples TS9 and TS5 (which are related to medical topics).

[0031] 1, 2 and 3D, the input interface 120 executes step S250, which receives a manual input command (related to a manual input label MLB) and uses it to assign the manual input label MLB to the partial sample. As shown in FIG. 3D, the manual input label MLB includes the retained labels in the character samples TS5 and TS9, so that the character samples TS5 and TS9 are assigned to the retained samples TS in the partial sample PS. KP On the other hand, as shown in Fig. 3D, since the manually input label MLB includes removal labels in the character samples TS6, TS11, and TS13, the character samples TS6, TS11, and TS13 may be regarded as removal samples TSRMV in the partial samples.

[0032] As shown in FIG. 3C and FIG. 3D, partial samples PS that tend to belong to outlier samples (evaluated by an outlier detection algorithm based on the outlier-inlier order RK) may be selected from the entire character samples TS1-TS14. In such a case, the labeler does not need to manually label each character sample TS1-TS14. The partial samples PS that tend to belong to outlier samples are presented to the labeler. Other character samples that are not included in the partial samples PS (e.g., character samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2, and TS7 in the embodiment shown in FIG. 3C) are regarded as unlabeled samples ULS. The number of partial samples PS is less than the number of unlabeled samples ULS. In practical operation, the dataset DB may include 10,000 character samples, so the number of partial samples PS will obviously be less than the number of unlabeled samples ULS, and the partial samples PS will be a small portion of the entire character samples compared with all the character samples in the dataset DB.

[0033] In the above embodiment, the process of collecting manual labels in one round is described. In some other embodiments, for unlabeled samples ULS (for example, character samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2 and TS7 in the embodiment shown in FIG. 3C), step S230 (generate outlier-inlier order of another unlabeled sample ULS), step S240 (select another set of partial samples), and step S250 (receive another set of manual input labels MLB) may be performed. In some embodiments, steps S230, S240 and S250 may be repeated a certain number of times.

[0034] As shown in FIGS. 1 and 2, the processing unit 110 executes a step S260 of generating a prompt message based on the partial samples PS with the manually input labels MLB and the unlabeled samples ULS of the character samples TS.

[0035] Further refer to Fig. 3E, which is a diagram illustrating how to generate a prompt message PM based on a partial sample PS with a manually input label MLB and an unlabeled sample ULS of a character sample TS according to one embodiment of the present disclosure.

[0036] In some embodiments, the prompt message PM includes anchor data D AN As shown in Figures 2 and 3E, in some embodiments, step S260 may include the anchor data D of the prompt message PM. AN The processing unit 110 further includes steps S261, S262, and S263 to generate the retained sample TS. KP Calculate the cluster distribution of the retained sample TS KP Retain samples based on the cluster distribution of TS KP First anchor sample TS KAN In some embodiments, the retained sample TS is selected in step S261. KP The retained sample TS adjacent to the cluster center KP is the first anchor sample TSKAN The first anchor sample TS is selected as KAN Other retained samples not selected as TS KP is the remaining retained sample TS Kleft It is considered that.

[0037] The processing unit 110 processes the removed sample TS RMV Calculate the cluster distribution of , and remove the sample TS RMV Remove samples based on the cluster distribution of TS RMV From the second anchor sample TS RAN In some embodiments, the removal sample TS is selected in step S262. RMV The removal sample TS adjacent to the cluster center RMV is the second anchor sample TS RAN The first anchor sample TS is selected as KAN Other retained samples not selected as TS KP is the remaining retained sample TS Kleft The processing unit 110 regards the first anchor sample TS KAN and the second anchor sample TS RAN and the anchor data D in the prompt message PM are merged. AN Step S263 is executed to form

[0038] In some embodiments, the prompt message PM includes unlabeled data D UL and unlabeled data D UL may be a mixture of unlabeled samples ULS and some calibration samples. As shown in FIG. 2 and FIG. 3E, in some embodiments, step S260 includes the unlabeled data D of the prompt message PM. UL The processing unit 110 further includes steps S264, S265, and S266 to generate the remaining retained samples TS. Kleft (For example, the first anchor sample TS KAN Retention samples not selected as TS KP) and the unlabeled sample ULS, and calculate the similarity between the first calibration sample TS Kcal The remaining retained samples TS Kleft In some embodiments, a number of remaining retained samples TSKleft similar to the unlabeled samples ULS are selected from the first calibration sample TSS. Kcal is selected as.

[0039] The processing unit 110 processes the remaining removed sample TS Rleft (For example, the second anchor sample TS RAN Removal samples not selected as TS RMV ) and the unlabeled sample ULS, and calculate the similarity between the second calibration sample TS Rcal Remove remaining sample TS Rleft In some embodiments, the unlabeled samples ULS may be similar to the remaining removed samples Ts. Rleft is the second calibration sample TS Rcal is selected as.

[0040] The processing unit 110 receives a first calibration sample TS Kcal and the second calibration sample TS Rcal and are mixed into the unlabeled sample ULS to generate the unlabeled data D in the prompt message PM. UL Step S266 is executed to form

[0041] Please further refer to FIG. 4. FIG. 4 is a schematic diagram of a prompt message PM in step S260 (including steps S261-S266 shown in FIG. 3E) according to an embodiment of the present disclosure. As shown in FIG. 4, the prompt message PM includes a task instruction INST, anchor data D generated based on a partial sample PS, and AN , anchor label data LB AN and unlabeled data D UL Includes.

[0042] As shown in FIG. 4, the task instruction INST in the prompt message PM is used to inform the generative pre-training Transformer model 190 to identify outliers in a particular data set. As shown in FIG. 4, the task instruction INST informs the generative pre-training Transformer model 190 to act as a data analyst and generate anchor data D AN and anchor label data LB AN Based on the prompt, unlabeled data D UL We require predicting inlier / outlier labels for character samples in

[0043] In the embodiment shown in FIG. 3D and FIG. 4, the removed sample TS RMV Character samples TS11 and TS6, and retention sample TS KP The character sample TS9 is the anchor data D in the prompt message PM. AN The manually input labels associated with the character samples TS11, TS6 and TS9 are used to form the anchor label data LB AN is used to form

[0044] In the embodiment shown in FIG. 3D and FIG. 4, the retention sample TS KP The character sample TS5 is the first proofread sample TS Kcal , and the removed sample TS RMV The character sample TS13 is the second proofread sample TS Rcal The first calibration sample TS is used to form Kcal , 2nd calibration sample TS Rcal is mixed with other unlabeled samples ULS to generate unlabeled data D in the prompt message PM. UL Form.

[0045] As shown in FIGS. 1, 2 and 4, the processing unit 110 and the communication circuitry 150 provide a prompt message PM to the generative pre-training Transformer model 190 to generate unlabeled data D UL(Unlabeled sample ULS, first calibration sample TS Kcal and the second calibration sample TS Rcal (including the inlier-outlier prediction label LB PRED In some embodiments, the generative pre-training Transformer model 190 performs a step S270 of generating anchor data D AN and anchor label data LB AN Based on the prompt, unlabeled data D UL Inlier-outlier prediction label LB associated with PRED Generate the inlier-outlier prediction label LB PRED is each unlabeled data D UL It is possible to predict whether a value is a normal value or an outlier.

[0046] In some embodiments, the manual input labels MLB associated with the character samples TS5 and TS13 are not added to the content of the prompt message PM. Kcal and the second calibration sample TS Rcal is the inlier-outlier prediction label LB generated by the generative pre-training transformer model 190. PRED In other words, the first calibration sample TSKcal and the second calibration sample TSRcal are used to verify the confidence level associated with the inlier-outlier prediction label LB generated by the generative pre-training transformer model 190. PRED may be used to assess the accuracy of

[0047] Inlier-outlier prediction labels LB generated by generative pre-training transformer model 190 PRED is the calibration sample (e.g., the first calibration sample TS Kcal and the second calibration sample TS Rcal ) Calibrated predicted label LB cal The processing unit 110 calculates the manually input labels MLB and the calibrated predicted labels LB associated with the character samples TS5 and TS13. calThe more the manually input label MLB matches with the calibrated predicted label LBcal, the higher the confidence level. The less the manually input label MLB matches with the calibrated predicted label LBcal, the lower the confidence level.

[0048] In some embodiments, the electronic device 100 generates the inlier-outlier prediction label LB by the generative pre-training Transformer model 190. PRED Then, the processing unit 110 is used to remove character samples with outlier labels from the dataset DB, so that all remaining character samples in the dataset DB are clean data (e.g., inlier data) that are free of noise data.

[0049] Based on the above embodiment, some partial samples PS are extracted from the whole character sample dataset DB. A small number of partial samples PS need to be manually labeled. The partial samples PS are labeled with anchor data D in the prompt message PM. AN and calibration data (e.g., the first calibration sample TS Kcal and the second calibration sample TS Rcal ) The prompt message PM is processed to provide the inlier-outlier predicted label LB associated with a large number of unlabeled samples ULS. PRED The generative pre-training Transformer model 190 is used to trigger the generative pre-training Transformer model 190 to generate a set of character samples, which can be used to avoid the heavy burden of labelers manually labeling a large number of character samples. The labelers need to label only a small fraction of the character samples, and most of the character samples may be automatically processed by the data classification method 200.

[0050] Please further refer to Fig. 5, which is a flowchart of a data classification method 500 according to another embodiment of the present disclosure. The data classification method 500 may be performed by the electronic device 100 shown in Fig. 1. Similar to the data classification method 200 shown in Fig. 2, the data classification method 500 is used to generate inlier-outlier predictions associated with character samples in a dataset DB.

[0051] As shown in Fig. 5, the data classification method 500 includes steps S510, S520, S530, S540, S550, S560, S565, S570, S575 and S580. Steps S510, S520, S530, S540 and S550 of the data classification method 500 in Fig. 5 are similar to steps S210, S220, S230, S240 and S250 of the data classification method 200 in Fig. 2, and these steps have been fully disclosed in the embodiment shown in Fig. 2, so the details of steps S510, S520, S530, S540 and S550 will not be repeated here.

[0052] As shown in FIG. 3D and FIG. 5, in step S550, since the manually input label MLB includes the retained labels in the character samples TS5 and TS9, the character samples TS5 and TS9 are included in the retained samples TS in the partial sample PS. KP On the other hand, since the manually input label MLB includes the removed labels in the character samples TS6, TS11, and TS13, the character samples TS6, TS11, and TS13 are regarded as the removed samples TS RMV may be regarded as such.

[0053] As shown in FIGS. 1 and 5, the processing unit 110 executes step S560 of generating a first prompt message including a partial sample with a manually input label and a feature engineering task instruction.

[0054] Please further refer to Fig. 6. Fig. 6 is a schematic diagram of the first prompt message PM1 generated in step S560 according to an embodiment of the present disclosure. As shown in Fig. 6, the first prompt message PM1 includes a feature engineering task instruction INST1, a retained sample TS KP and removal sample TS RMV Includes.

[0055] Retention Sample TS KP and removal sample TS RMV is a partial sample PS with manually input labels MLB, which is used to provide multiple prompts to the generative pre-training Transformer model 190 to achieve the feature engineering task.

[0056] The feature engineering task instruction INST1 in the first prompt message PM1 is a removal sample TS RMV and retention sample TS KP The vector DF is used to trigger the generative pre-training Transformer model 190 to generate distinguishable features DF that can distinguish both vector DF and DF.

[0057] As shown in Figures 1, 5 and 6, the processing unit 110 and the communication circuit 150 perform a step S565 of providing a first prompt message PM1 to the generative pre-training Transformer model 190 to generate distinguishable features DF. As shown in FIG. 6, the generative pre-training Transformer model 190 generates a feature set of 10 distinguishable features DF based on the first prompt message PM1.

[0058] As shown in FIGS. 1 and 5, the processing unit 110 executes step S570 of generating a second prompt message including the distinguishable features DF, the character samples, and feature scoring task instructions.

[0059] Please further refer to Fig. 7. Fig. 7 is a schematic diagram of the second prompt message PM2 generated in step S570 according to an embodiment of the present disclosure. As shown in Fig. 7, the second prompt message PM2 includes feature scoring task instructions INST2, character samples TS and distinguishable features DF.

[0060] In some embodiments, the feature scoring task instruction INST2 in the second prompt message PM2 is used to trigger the generative pre-training Transformer model to recognize whether the character sample TS has attributes that match the possible options of the distinguishing features DF.

[0061] As shown in FIGS. 1, 5 and 7, the processing unit 110 and the communication circuitry 150 provide a second prompt message PM2 to the generative pre-training Transformer model 190 to generate feature predictions F for the distinguishable features DF of the character sample TS. PRED As shown in FIG. 7, the feature prediction F PRED indicates that the text sample TS1 is an English text paragraph related to a health topic, and indicates that the text sample TS2 is a Chinese text paragraph related to a health topic. In some embodiments, the distinguishable features DF are human-recognizable features (e.g., language, topic, time, length), as shown in Figures 6 and 7. In other embodiments, the distinguishable features DF may include human-invisible features, such as latent vectors.

[0062] As shown in FIGS. 1 and 5, the processing unit 110 performs step S580 of running a classification algorithm to generate predicted inlier-outlier labels associated with the unlabeled samples.

[0063] Please refer to Fig. 8. Fig. 8 shows how the inlier-outlier predicted label LB is generated by the classification algorithm in step S580 according to an embodiment of the present disclosure. PRED FIG.

[0064] As shown in FIG. 8, the classification algorithm in step S580 calculates feature predictions F for the subsamples P S . PRED1 and the feature prediction F for the unlabeled sample ULS PRED2 Based on this, the inlier-outlier prediction label LB associated with the unlabeled sample ULS is calculated. PRED In one embodiment, the character sample TS includes the partial sample PS and the unlabeled sample ULS, so that the feature prediction F of the partial sample PS in FIG. PRED1 and the feature prediction F for the unlabeled sample ULS PRED2 Both of these are the feature predictions for the character sample TS in Figure 7. PRED is obtained from

[0065] As shown in FIG. 8, the classification algorithm in step S580 calculates feature predictions F for the subsamples P S . PREDl It is performed based on the training data TD including the manually input labels MLB of the partial samples PS, and then the feature prediction F of the unlabeled samples ULS is performed. PRED2 Based on the unlabeled sample ULS, we predict the inlier-outlier labels LB PRED Generate.

[0066] For example, one unlabeled sample ULS predicts the feature prediction F of character sample TS11. PRED1 A set of feature predictions F PRED2 If one unlabeled sample ULS has a feature prediction F of character sample TS9, the classification algorithm can classify this unlabeled sample ULS as an outlier. PRED1 A set of feature predictions F PRED2 If the classification algorithm has the following, the classification algorithm can classify the unlabeled sample ULS as an inlier. In some embodiments, the classification algorithm in step S580 may be implemented by an extreme gradient boosting algorithm, a classification boosting algorithm, or a random forest algorithm.

[0067] In some implementations, the extreme gradient boosting algorithm can generate a prediction score for each character sample. If the prediction score of a character sample is greater than 0.5 (e.g., prediction score = 0.7), the character sample may be labeled as an inlier and its confidence level may be equal to its score (e.g., confidence level = 0.7). If the prediction score of a character sample is less than 0.5 (e.g., prediction score = 0.2), the character sample may be labeled as an outlier and its confidence level will be equal to the complement of its score (e.g., confidence level = 1 - 0.2 = 0.8).

[0068] Based on the above embodiment, some partial samples PS are extracted from the whole character sample dataset DB. A small amount of partial samples PS need to be manually labeled. The partial samples PS are processed to generate a first prompt message PM1 and a second prompt message PM2. The first prompt message PM1 and the second prompt message PM2 are used to trigger the generative pre-training Transformer model 190 to perform feature engineering and feature scoring tasks. Based on the feedback from the generative pre-training Transformer model 190, inlier-outlier prediction labels LB for a large amount of unlabeled samples ULS are generated. PRED can be quickly generated. In this way, the heavy burden of manually labeling a large number of character samples by an expert can be avoided. The labeler only needs to label a small portion of the character samples, and most of the character samples may be automatically processed by the data classification method 500.

[0069] In the specification and the patent application, specific terms are used to refer to specific elements. However, those skilled in the art should understand that the same elements may be referred to by different nouns. The specification and the patent application do not use name differences as a way to distinguish elements, but rather use differences in the functions of elements as a basis for distinction. The term "including" referred to in the specification and the patent application is an open term and should be interpreted as "including, but not limited to."

[0070] Furthermore, unless otherwise specified herein, any singular term includes the plural reference as well.

[0071] The above are merely preferred embodiments of the present disclosure, and various modifications and equivalent changes can be made to the present disclosure without departing from the scope or spirit of the present disclosure. Therefore, all modifications and equivalent changes made to the present disclosure within the scope of the following claims are intended to be covered by the present disclosure. [Explanation of symbols]

[0072] 100:Electronic equipment 110: Processing unit 120: Input interface 130: Storage unit 140: Display 150: Communication circuit 190: Generative Pre-Training Transformer Models DB: Dataset MLB:Manual Input Label PS: Partial sample 200, 500: Data classification method S210, S220, S230, S240, S250, S260, S261, S262, S263, S264, S265, S266, S270, S510, S520, S530, S540, S550, S560, S565, S570, S575, S580: Step TS1-TS14: Character samples SS: Semantic Space eTS1-TS14: Character embedding ULS: Unlabeled Samples RK: Outlier-normal order TS RMV :Removal sample TS KP : Retention samples TS KAN :First anchor sample TS Kcal : First calibration sample TS RAN :Second anchor sample TS Rcal : 2nd calibration sample D UL :Unlabeled data PM: Prompt message D AN :Anchor Data LB AN :Anchor label data INST: Task instruction LB PRED : Inlier-outlier predicted labels LB cal : Calibrated predicted label PM1: 1st prompt message PM2: Second prompt message INST1: Feature Engineering Task Instructions DF: Distinguishing features INST2: Feature scoring task instructions F PRED , F PRED1 , F PRED2 :Feature prediction TS: Character sample TD: Training data

Claims

1. obtaining a number of character samples from a dataset; converting the character samples into character embeddings in a semantic space; generating an outlier-inlier ordering of the plurality of character samples based on an outlier detection algorithm based on distances between the plurality of character embeddings in the semantic space; selecting a plurality of sub-samples from the plurality of character samples based on the outlier-inlier ordering; receiving a manual input instruction to specify a plurality of manual input labels for the plurality of sub-samples; generating a prompt message based on the plurality of sub-samples having the plurality of manually-input labels and a plurality of unlabeled samples in the plurality of character samples, the prompt message including task instructions, unlabeled data, and anchor data generated based on the plurality of sub-samples of the plurality of manually-input labels; providing the prompt message to a generative pre-training Transformer model to generate a plurality of inlier-outlier predicted labels associated with the plurality of unlabeled samples; Data classification methods, including:

2. 2. The data classification method according to claim 1, further comprising the steps of: selecting, as the plurality of partial samples, at least one of the plurality of character samples that tends to belong to an outlier sample based on the outlier-inlier order; and wherein the number of the plurality of partial samples is less than the number of the plurality of unlabeled samples.

3. The manual input instructions are used to assign a plurality of retention labels to a plurality of retention samples within the plurality of sub-samples and assign a plurality of removal labels to a plurality of removal samples within the plurality of sub-samples, and the method of generating a prompt message includes: selecting a plurality of first anchor samples from the plurality of retained samples based on a cluster distribution of the plurality of retained samples; selecting a plurality of second anchor samples from the plurality of removed samples based on a cluster distribution of the plurality of removed samples; Merging the first anchor samples and the second anchor samples to form the anchor data in a prompt message; The method of claim 1 , comprising:

4. The method for generating a prompt message includes: selecting a plurality of first calibration samples from the plurality of retained samples not selected as the plurality of first anchor samples; selecting a plurality of second calibration samples from the plurality of removed samples not selected as the plurality of second anchor samples; forming the unlabeled data in the prompt message based on a mixture of the plurality of unlabeled samples, the plurality of first calibration samples, and the plurality of second calibration samples; The method of claim 3 further comprising:

5. 5. The method of claim 4, wherein the first calibration samples and the second calibration samples are used to verify a confidence level of the plurality of inlier-outlier predicted labels generated by the generative pre-training Transformer model.

6. The method of selecting the plurality of first calibration samples comprises: comparing a plurality of similarities between the plurality of retained samples not selected as the plurality of first anchor samples and the plurality of unlabeled samples; selecting a plurality of first calibration samples based on the plurality of similarity measures; 5. The method of claim 4, comprising:

7. 2. The method of claim 1, wherein the task instructions in the prompt message are used to inform the generative pre-training Transformer model to identify outliers in a particular data set.

8. The method of claim 1, wherein the outlier detection algorithm is realized by a random sampling consistency-NN algorithm, an isolation forest algorithm, or a local outlier factor algorithm.

9. 2. The method of claim 1, wherein each of the character samples comprises a text paragraph or a question and answer combination.

10. obtaining a number of character samples from a dataset; converting the character samples into character embeddings in a semantic space; generating an outlier-inlier ordering of the plurality of character samples based on an outlier detection algorithm based on distances between the plurality of character embeddings in the semantic space; selecting a plurality of sub-samples from the plurality of character samples based on the outlier-inlier ordering; receiving a manual input instruction to specify a plurality of manual input labels for the plurality of sub-samples; generating a first prompt message including the plurality of sub-samples having the plurality of manually input labels and a feature engineering task instruction; providing the first prompt message to a generative pre-training Transformer model to generate a plurality of distinguishable features; generating a second prompt message including the plurality of distinguishable features, the plurality of character samples, and feature scoring task instructions; providing the second prompt message to the generative pre-training Transformer model to generate feature predictions for the distinguishable features of the character samples; running a classification algorithm based on the feature predictions of the sub-samples in the character samples and a plurality of unlabeled samples to generate a plurality of inlier-outlier predicted labels for the plurality of unlabeled samples; Data classification methods, including:

11. 11. The method of claim 10, wherein at least one of the plurality of character samples that tends to belong to an outlier sample based on the outlier-inlier order is selected as one of the plurality of partial samples, and the number of the plurality of partial samples is less than the number of the plurality of unlabeled samples.

12. 11. The method of claim 10, wherein the manual input instructions are used to assign a plurality of retention labels to a plurality of retention samples within the plurality of sub-samples and to assign a plurality of removal labels to a plurality of removal samples within the plurality of sub-samples.

13. 13. The method of claim 12, wherein the feature engineering task instructions in the first prompt message are used to trigger the generative pre-training Transformer model to generate the plurality of distinguishable features for separating the plurality of removed samples and the plurality of retained samples.

14. 11. The method of claim 10, wherein the feature scoring task instruction in the second prompt message is used to trigger the generative pre-training Transformer model to distinguish whether the plurality of character samples have a plurality of attributes of the plurality of distinguishable features or not.

15. 11. The method of claim 10, wherein the classification algorithm is performed based on training data, the training data including the plurality of feature predictions for the plurality of sub-samples and the plurality of manually entered labels for the plurality of sub-samples, thereby generating the plurality of predicted inlier-outlier labels associated with the plurality of unlabeled samples based on the plurality of feature predictions for the plurality of unlabeled samples.

16. The method of claim 10, wherein the classification algorithm is implemented by an extreme gradient boosting algorithm, a classification boosting algorithm, or a random forest algorithm.

17. The method of claim 10, wherein the outlier detection algorithm is realized by a Random Sampling Consistency-NN algorithm, an Isolation Forest algorithm, or a Local Outlier Factor algorithm.

18. 11. The method of claim 10, wherein each of the text samples comprises a text paragraph or a question and answer combination.

Images