Data classification method for screening outlier literal data

Abstract

A data classification method includes the following steps. A plurality of text samples are obtained from a data set. And converting the character sample into a plurality of character embedding in a semantic space. An outlier detection algorithm is used to generate an outlier-ingroup ranking of the character samples according to the distance between character embedding in the semantic space. And selecting a plurality of partial samples from the character samples according to the sorting. A manual input instruction is received to specify a manual input flag for a portion of the sample. And generating prompt information according to the partial sample with the manual input mark and the unmarked sample of the text sample. The cue information is provided to the generative pre-trained converter model to generate a plurality of intra-group-outlier predictive flags. Based on the data classification method, a large number of intra-group value-outlier prediction marks can be quickly generated, and the burden caused by manual marking of a large number of samples by marking personnel is avoided. Marking personnel only need to mark a small part of samples, and most of the samples can be automatically processed by a data classification method.

Description

Technical Field

[0001] The present invention relates to a classification method, and more particularly to a classification method for identifying inliers or outliers in unlabeled data. Background Art

[0002] In machine learning techniques, outlier detection is a common operation that is used to identify whether there are data instances that significantly deviate from the normal distribution within the data set. Detecting outliers is critical in various application areas, including medical prevention, fraud detection, network security, quality control, and anomaly detection in healthcare or industrial processes. Summary of the invention

[0003] The present disclosure provides a data classification method, comprising the following steps. Obtain multiple text samples from a data set. Convert the multiple text samples into multiple text embeddings in a semantic space. Generate an outlier-ingroup value ranking of the multiple text samples based on an outlier detection algorithm according to the distance between the text embeddings in the semantic space. Select multiple partial samples from the text samples according to the aforementioned outlier-ingroup value ranking. Receive a manual input instruction to specify a manual input label of the partial sample. Generate prompt information based on the partial sample with the manual input label and multiple unlabeled samples in the text sample, wherein the prompt information includes a task instruction, unlabeled data, and anchor point data generated based on the partial sample with the manual input label. Provide the prompt information to a generative pre-trained converter model to generate multiple ingroup value-outlier prediction labels for the multiple unlabeled samples.

[0004] In some embodiments, the data classification method selects at least one of the multiple text samples that tends to be an outlier sample as the multiple partial samples according to the outlier value-ingroup value sorting, and the number of the multiple partial samples is less than the number of the multiple unlabeled samples.

[0005] In some embodiments, the manual input instruction is used to assign multiple retention marks to multiple retained samples among the multiple partial samples, and assign multiple removal marks to multiple removed samples among the multiple partial samples. The method for generating the prompt information includes: selecting multiple first anchor point samples from the multiple retained samples based on the cluster distribution of the multiple retained samples; selecting multiple second anchor point samples from the multiple removed samples based on the cluster distribution of the multiple removed samples; and merging the multiple first anchor point samples and the multiple second anchor point samples to form the anchor point data in the prompt information.

[0006] In some embodiments, the method of generating the prompt information by the data classification method further includes: selecting a plurality of first calibration samples from the plurality of retained samples that have not been selected as the plurality of first anchor point samples; selecting a plurality of second calibration samples from the plurality of removed samples that have not been selected as the plurality of second anchor point samples; and forming the unlabeled data in the prompt information based on a mixture of the plurality of unlabeled samples, the plurality of first calibration samples, and the plurality of second calibration samples.

[0007] In some embodiments, the plurality of first calibration samples and the plurality of second calibration samples are used to verify the confidence of the plurality of inlier-outlier prediction labels generated by the generative pre-trained transformer model.

[0008] In some embodiments, the method of selecting the plurality of first calibration samples in the data classification method includes: comparing a plurality of similarities between the plurality of retained samples that are not selected as the plurality of first anchor point samples and the plurality of unlabeled samples; and selecting a plurality of first calibration samples according to the plurality of similarities.

[0009] In some embodiments, the task instruction in the prompt message is used to inform the generative pre-trained transformer model to identify multiple outliers in a given data set.

[0010] In some embodiments, the outlier detection algorithm is implemented by a random sampling consensus-NN algorithm, an isolation forest algorithm, or a regional outlier factor algorithm.

[0011] In some embodiments, each of the text samples includes a text paragraph or a combination of a question and an answer.

[0012] The present disclosure provides another data classification method comprising the following steps. Obtain multiple text samples from a data set. Convert the multiple text samples into multiple text embeddings in a semantic space. Generate an outlier-ingroup value ranking of the text samples based on the distance between the text embeddings in the semantic space based on an outlier detection algorithm. Select multiple partial samples from the text samples based on the outlier-ingroup value ranking. Receive a manual input instruction to specify multiple manual input tags of the partial samples. Generate a first prompt message, the first prompt message includes a partial sample with a manual input tag and a feature engineering task instruction. Provide the first prompt message to a generative pre-trained converter model to generate multiple distinguishable features. Generate a second prompt message, the second prompt message includes distinguishable features, text samples and feature scoring task instructions. Provide the second prompt message to a generative pre-trained converter model to generate multiple feature predictions of the text samples relative to the distinguishable features. Execute a classification algorithm based on the feature predictions of the partial samples and unlabeled samples among the text samples, thereby generating multiple ingroup-outlier prediction tags for the unlabeled samples.

[0013] In some embodiments, the data classification method selects at least one of the multiple text samples that tends to belong to outlier samples according to the outlier value-ingroup value sorting as one of the multiple partial samples, and the number of the multiple partial samples is less than the number of the multiple unlabeled samples.

[0014] In some embodiments, the manual input command is used to assign a plurality of retention tags to a plurality of retention samples among the plurality of partial samples, and assign a plurality of removal tags to a plurality of removal samples among the plurality of partial samples.

[0015] In some embodiments, the data classification method is characterized in that the feature engineering task instruction in the first prompt information is used to trigger the generative pre-trained converter model to generate the multiple distinguishable features used to separate the multiple removed samples from the multiple retained samples.

[0016] In some embodiments, the feature scoring task instruction in the second prompt information is used to trigger the generative pre-trained converter model to distinguish whether the multiple text samples have multiple attributes of the multiple distinguishable features.

[0017] In some embodiments, the classification algorithm is executed based on a training data, and the training data includes the multiple feature predictions of the multiple partial samples and the multiple manually input labels of the multiple partial samples, so as to generate the multiple inlier-outlier prediction labels of the multiple unlabeled samples according to the multiple feature predictions of the multiple unlabeled samples.

[0018] In some embodiments, the classification algorithm is implemented by an extreme gradient boosting algorithm, a classification boosting algorithm, or a random forest algorithm.

[0019] In some embodiments, the outlier detection algorithm is implemented by a random sampling consensus-nearest neighbor algorithm, an isolation forest algorithm, or a regional outlier factor algorithm.

[0020] In some embodiments, each of the text samples includes a text paragraph or a combination of a question and an answer.

[0021] It is worth noting that the above general description and the following detailed description are provided through embodiments, and are intended to further explain the present invention to be protected. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 FIG. 4 is a block diagram of an electronic device according to an embodiment of the present disclosure.

[0023] Figure 2 Flow chart of a data classification method according to an embodiment of the present disclosure.

[0024] Figure 3A FIG. 1 is a schematic diagram of text samples obtained from a data set according to an embodiment of the present disclosure.

[0025] Figure 3B FIG. 1 is a schematic diagram of text embedding in a semantic space relative to text samples obtained from a data set according to an embodiment of the present disclosure.

[0026] Figure 3C FIG. 4 is a schematic diagram of outlier-ingroup value sorting of text samples according to an embodiment of the present disclosure.

[0027] Figure 3D A schematic diagram of some samples and manually input marks about some samples according to an embodiment of the present disclosure.

[0028] Figure 3E A schematic diagram of how to generate prompt information based on manually input marked partial samples and unlabeled samples of text samples in one embodiment of the present disclosure.

[0029] Figure 4 It is a schematic diagram of prompt information according to an embodiment of the present disclosure.

[0030] Figure 5 FIG. 4 is a flow chart of a data classification method according to another embodiment of the present disclosure.

[0031] Figure 6 It is a schematic diagram of the first prompt information according to an embodiment of the present disclosure.

[0032] Figure 7 Schematic diagram of second prompt information according to an embodiment of the present disclosure.

[0033] Figure 8 Schematic diagram of how to generate in-group value-outlier prediction labels through a classification algorithm according to an embodiment of the present disclosure.

[0034] Explanation of symbols:

[0035] 100: Electronic devices

[0036] 110: Processing unit

[0037] 120: Input interface

[0038] 130: Storage unit

[0039] 140: Display

[0040] 150: Communication circuit

[0041] 190: Generative Pre-trained Transformer Model

[0042] DB: Dataset

[0043] MLB: Manually enter tags

[0044] PS: Some samples

[0045] 200, 500: Data classification method

[0046] S210, S220, S230, S240, S250, S260, S261, S262, S263, S264, S265, S266, S270, S510, S520, S530, S540, S550, S560, S565, S570, S575, S580: Steps

[0047] TS1-TS14: Text samples

[0048] SS: Semantic Space

[0049] eTS1-TS14: Text Embedding

[0050] ULS: Unlabeled Samples

[0051] RK: Outlier-Ingroup Sorting

[0052] TS RMV : Remove samples

[0053] TS KP : Reserved samples

[0054] TS KAN : The first anchor point sample

[0055] TS Kcal : First calibration sample

[0056] TS RAN : Second anchor point sample

[0057] TS Rcal : Second calibration sample

[0058] D UL : Unlabeled data

[0059] PM: Prompt information

[0060] D AN : Anchor information

[0061] LB AN : Anchor mark data

[0062] INST: Mission Instructions

[0063] LB PRED : In-group value-outlier prediction marker

[0064] LB cal : Calibrate prediction markers

[0065] PM1: First prompt information

[0066] PM2: Second prompt message

[0067] INST1: Feature Engineering Task Instructions

[0068] DF: Distinguishing Features

[0069] INST2: Feature Scoring Task Instructions

[0070] F PRED 、F PRED1 、F PRED2 : Feature prediction

[0071] TS: Text Sample

[0072] TD: Training Data DETAILED DESCRIPTION

[0073] The following will illustrate the embodiments of the present disclosure with reference to the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same or similar elements or method flows.

[0074] When developing a chatbot or large language models (LLMs), it is very important to collect a training dataset with input text and corresponding output text. For example, when developing a chatbot, the input text can be a sample question of a potential user, and the output text can be the answer that the chatbot should provide. The aforementioned training dataset is used for both model development (e.g., building a chatbot) and model evaluation (e.g., evaluating the accuracy or effectiveness of the chatbot). In conventional technology, the training dataset is manually labeled by a group of professionals, but hiring such a team is very expensive. If an automatic generator is used instead of the above-mentioned labeling personnel to automatically produce a training dataset (automatically generate corresponding training text samples), the cost and time required to establish a training dataset can be reduced. Since the quality of the training text samples generated by the automatic generator is unstable, the generated text samples need to be verified.

[0075] The aforementioned validation system is used to remove low-quality samples (e.g., incorrectly generated text) from the training data set, and use the validated text samples in the training data set during model development and model evaluation. Since low-quality samples can seriously affect model performance, removing low-quality samples during the validation process can help improve model quality during subsequent model building.

[0076] In some cases, the automatically generated dataset contains more than 10,000 sample texts that need to be verified. One solution is to hire professional labelers to manually check the quality of all sample texts. However, the cost of manual inspection is very high and the process of manual inspection is very slow.

[0077] Please refer to Figure 1 and Figure 2 . Figure 1 FIG. 1 is a block diagram of an electronic device 100 according to an embodiment of the present disclosure. Figure 2 2 is a flow chart of a data classification method 200 according to an embodiment of the present disclosure. The data classification method 200 is used to generate inlier-outlier predictions, and the inlier-outlier predictions are related to text samples in a data set DB. The data set DB can be obtained from a text-content server. For example, the data set DB can be collected from an encyclopedia website, a news website, a question-and-answer database, a forum, a novel storage server, a journal database, or any similar text content storage center.

[0078] like Figure 1 As shown, the electronic device 100 includes a processing unit 110, an input interface 120, a storage unit 130, a display 140, and a communication circuit 150. In some embodiments, the electronic device 100 can be a computer, a smart phone, a tablet computer, an image processing server, a data server, or any image processing device with similar functions. The input interface 120 is used to receive a data set DB and input instructions manually. In some embodiments, the electronic device 100 is used to classify text samples in the data set DB, generate ingroup value-outlier value predictions for the text samples, remove outlier samples based on the ingroup value-outlier value predictions, and retain ingroup samples.

[0079] The input interface 120 may include a data transmission interface, a wireless communication circuit, a keyboard, a mouse, a microphone, or any input device with similar functions. The processing unit 110 is coupled to the input interface 120, the storage unit 130, the display 140, and the communication circuit 150. The storage unit 130 is used to store program codes. The program codes stored in the storage unit 130 are used to instruct the processing unit 110 to execute Figure 2Data classification method 200 shown. In some embodiments, processing unit 110 may be a processor, a graphics processor, an application specific integrated circuit (ASIC) or any processing circuit having similar functions. Communication circuit 150 may be a network transceiver (e.g., a WiFi transceiver, a telecommunications transceiver or an Ethernet transceiver). Communication circuit 150 is used to communicate with generative pre-trained transformer (GPT) model 190. In some embodiments, generative pre-trained transformer model 190 may be an independently operated separate model running on an external server outside electronic device 100.

[0080] Ideally, machine learning models are trained on training data sets that do not contain outliers. If the training data set contains outliers, it may have a variety of effects on the performance and behavior of machine learning models. For example, outliers in the training data set may cause model bias, increased model complexity, overfitting, reduced robustness, and difficulty in anomaly detection.

[0081] In large language model (LLM) applications, the unlabeled text paragraphs of the training dataset DB can contain different types of text content, such as news reports, popular novels, literary works, bank statements, chat records, questions and answers, research papers, or program codes. The aforementioned different text contents have different uses in different fields. These text contents must be filtered to remove noisy data and extract clean data, such as valid text samples, to train machine learning models for specific purposes.

[0082] For example, in one embodiment, when the training goal of the machine learning model is to answer medical questions, text content related to medical records, health discussions, diagnoses, symptoms and / or medical history should be considered as in-group text samples; while other text content related to global warming, financial crises and / or baseball games should be considered as outlier text samples.

[0083] Manually labeling in-group and outlier samples in a dataset DB can be time-consuming and costly, especially when dealing with large datasets. In this case, experts are required to manually review and identify in-group and outlier samples, which may be impractical for large datasets with a large number of text samples.

[0084] In some embodiments, the electronic device 100 and the data classification method 200 provide a time-efficient and cost-effective way to generate inlier-outlier predictions for each of a plurality of text samples in a data set DB.

[0085] like Figure 1 and Figure 2 As shown, the processing unit 110 executes step S210 to obtain a text sample from the data set DB. Figure 3A , Figure 3A Schematic diagram of text samples TS1, TS2, TS3, ..., TS14 obtained from a data set DB according to an embodiment of the present disclosure. The text samples TS1-TS14 may be text paragraphs (eg, sentences, a paragraph, or an article) or a combination of questions and corresponding answers. Figure 3A The number of text samples TS1-TS14 shown is a simplified example for demonstration, and the present disclosure is not limited thereto. In actual situations, the data set DB may include hundreds, thousands, or more text samples. In some embodiments, the text samples TS1-TS14 may be obtained from the data set DB by a text fragmenting algorithm executed by the processing unit 110.

[0086] For example, text sample TS1 may be “Q: How to get rid of headaches? A: Try putting a heating pad on your neck”; text sample TS2 may be “Q: How to reduce belly fat? A: Moderate exercise and diet plan may help”; text sample TS3 may be “Q: What is the weather like today? A: It is sunny outside.” Text samples TS1-TS14 may include in-group samples related to medical topics (e.g., text samples TS1 and TS2) and outlier samples not related to medical topics (e.g., text sample TS3). Figure 3A As shown, it is assumed that text samples TS3, TS6, TS7, TS11, and TS13 are noise data (i.e., outlier samples irrelevant to the medical topic), and text samples TS1, TS2, TS4, TS5, TS8, TS9, TS10, TS12, and TS14 are clean data (i.e., in-group samples related to the medical topic). In this embodiment, text samples TS1-TS14 are initially unlabeled. The electronic device 100 and the data classification method 200 are used to label the text samples TS1-TS14 to separate the noise data from the clean data.

[0087] like Figure 1 and Figure 2 As shown, the processing unit 110 executes step S220 to convert the text samples TS1-TS14 into text embeddings in the semantic space. Figure 3B , Figure 3BSchematic diagram of text embedding eTS1 - eTS14 in a semantic space SS relative to text samples TS1 - TS14 obtained from a data set DB according to an embodiment of the present disclosure.

[0088] The text embedding eTS1 is the projection result of the text sample TS1 to the high-dimensional latent space. The text embedding eTS1 in the semantic space SS is a vector or a long sequence of numbers. For the convenience of illustration, Figure 3B The semantic space SS shown is illustrated as a two-dimensional distribution. In actual situations, the semantic space SS may have more dimensions, such as 768 dimensions or 1536 dimensions. If two of the text samples TS1-TS14 have similar semantics, the corresponding two of the text embeddings eTS1-eTS14 will be close to each other in the semantic space SS. On the other hand, if two of the text samples TS1-TS14 have different semantics, the corresponding two of the text embeddings eTS1-eTS14 will be far away from each other in the semantic space SS.

[0089] like Figure 1 and Figure 2 As shown, the processing unit 110 executes step S230 to generate an outlier-ingroup ranking of the text samples TS1-TS14 in the semantic space SS based on the outlier detection algorithm according to the distance between the text embeddings eTS1-eTS14. Figure 3C , Figure 3C FIG. 4 is a schematic diagram of an outlier-ingroup ranking RK of text samples TS1 - TS14 according to an embodiment of the present disclosure.

[0090] In some implementations, the outlier detection algorithm may be implemented by a RANSAC-NN algorithm. In some other embodiments, the outlier detection algorithm may be an Isolation Forest algorithm or a Local Outlier Factor algorithm.

[0091] The processing unit 110 is used to perform an outlier detection algorithm (e.g., random sampling consistency-NN algorithm) on the text samples TS1-TS14 according to the distance between the text embeddings eTS1-eTS14 in the semantic space SS, and the processing unit 110 can obtain the "inlier score" of each sample. Samples that are adjacent to other text embeddings (shorter distance from other texts) in the text embeddings eTS1-eTS14 will have a higher "inlier score", while samples that are far away from other text embeddings (farther distance from other texts) in the text embeddings eTS1-eTS14 will have a lower "inlier score". The closer the "inlier score" is to 1, the higher the quality of the text sample. The lower the "inlier score", the lower the quality of the text sample. Since each of the text samples TS1-TS14 has its own "inlier score", the outlier-inlier ranking RK of the text samples TS1-TS14 can be generated.

[0092] according to Figure 3C As shown in the outlier-ingroup ranking RK, text samples TS11, TS6, TS9, TS13, and TS5 (with relatively low ingroup scores) tend to be outlier samples. Figure 3C As shown, text samples TS11, TS6, TS9, TS13 and TS5 are arranged together at the top of the outlier-ingroup ranking RK.

[0093] according to Figure 3C As shown in the outlier-ingroup ranking RK, text samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2, and TS7 (with relatively high ingroup value scores) tend to belong to the ingroup samples. Figure 3C As shown, text samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2 and TS7 are arranged at the bottom of the outlier-ingroup value ranking RK.

[0094] like Figure 1 , Figure 2 and Figure 3C As shown, the processing unit 110 executes step S240 to select a portion of samples PS from the text samples TS1 to TS14 according to the outlier-ingroup value ranking RK. Figure 3C As shown, according to the outlier-ingroup value ranking RK, the text samples TS11, TS6, TS9, TS13 and TS5 that tend to be outlier samples are selected as partial samples PS. The partial samples PS are transmitted from the processing unit 110 to the display 140.

[0095] In some embodiments, the marking staff can view / verify the partial sample PS on the display 140, and then the marking staff can input the manual input mark MLB about the partial sample PS through the input interface 120. Figure 3D , Figure 3D Schematic diagram of a portion of a sample PS and a manually input mark MLB related to the portion of the sample PS according to an embodiment of the present disclosure.

[0096] like Figure 3D As shown, the text content of the text samples TS11, TS6, TS9, TS13 and TS5 included in the text content of the partial sample PS can be found in Figure 1 The display 140 is shown. The marking staff can read the text content of the text samples TS11, TS6, TS9, TS13 and TS5, and then input the manual input mark MLB of the partial sample PS through the input interface 120. In some embodiments, the marking staff can choose to remove the text samples TS11, TS6 and TS13 (irrelevant to medical topics) and keep the text samples TS9 and TS5 (related to medical topics).

[0097] like Figure 1 , Figure 2 and Figure 3D As shown, the input interface 120 executes step S250 to receive a manual input instruction (related to the manual input mark MLB) and is used to specify the manual input mark MLB on some samples. Figure 3D As shown, the manually input mark MLB contains the keep mark on the text samples TS5 and TS9, which makes the text samples TS5 and TS9 be regarded as the keep samples TS in the partial sample PS. KP On the other hand, Figure 3D As shown, the manually input mark MLB includes removal marks on the text samples TS6, TS11 and TS13, which makes the text samples TS6, TS11 and TS13 be regarded as the removal samples (remove sample) TS in the partial sample. RMV .

[0098] like Figure 3C and Figure 3D As shown, partial samples PS that tend to be outlier samples (evaluated by an outlier detection algorithm based on outlier-ingroup value ranking RK) can be selected from the entire text samples TS1-TS14. In this case, the labeler does not need to manually label each text sample TS1-TS14. The partial samples PS that tend to be outlier samples will be presented to the labeler. Other text samples not included in the partial samples PS (for example, Figure 3CThe text samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2 and TS7 in the illustrated embodiment will be regarded as unlabeled samples ULS. The number of partial samples PS is less than the number of unlabeled samples ULS. In actual operation, the data set DB may contain 10,000 text samples, which makes the number of partial samples PS significantly less than the number of unlabeled samples ULS, and the partial samples PS will also be a small part of the overall text samples compared to all the text samples in the data set DB.

[0099] The above embodiment describes a single round of collection of manual labels. In some other embodiments, unlabeled sample ULS (e.g., Figure 3C The text samples TS14, TS1, TS12, TS4, TS10, TS8, TS3, TS2, and TS7 in the illustrated embodiment may undergo step S230 (generating an outlier-ingroup ranking of another unlabeled sample ULS), step S240 (selecting another group of partial samples), and step S250 (receiving another group of manually input labels MLB) again. In some embodiments, steps S230, S240, and S250 may be repeated for a certain number of times.

[0100] like Figure 1 and Figure 2 As shown, the processing unit 110 executes step S260 to generate prompt information according to the partial sample PS with the manual input mark MLB and the unlabeled sample ULS of the text sample TS.

[0101] Further references Figure 3E , Figure 3E It is a schematic diagram of how to generate prompt information PM according to a partial sample PS with a manually input mark MLB and an unlabeled sample ULS of a text sample TS in an embodiment of the present disclosure.

[0102] In some embodiments, the prompt information PM may include anchor point data D AN .like Figure 2 and Figure 3E As shown, in some embodiments, step S260 further includes steps S261, S262 and S263 to generate anchor data D of the prompt information PM. AN The processing unit 110 executes step S261 to calculate the reserved sample TS KP The clustering distribution of the retained sample TS KP The cluster distribution of the retained sample TS KP Select the first anchor point sample TS KAN In some embodiments, the sample TS is retained KPThe reserved samples TS adjacent to the clustering centroid KP Selected as the first anchor sample TS KAN TS is not selected as the first anchor point sample KAN Other reserved samples TS KP is considered as the remaining retained sample TS Kleft .

[0103] The processing unit 110 executes step S262 to calculate the removed sample TS RMV The cluster distribution of TS is removed according to RMV The cluster distribution of removed samples TS RMV Select the second anchor point sample TS RAN In some embodiments, removing the sample TS RMV Remove samples TS adjacent to the cluster center RMV Selected as the second anchor sample TS RAN TS is not selected as the first anchor point sample KAN Other reserved samples TS KP is considered as the remaining retained sample TS Kleft The processing unit 110 executes step S263 to convert the first anchor point sample TS KAN With the second anchor sample TS RAN Merge to form anchor data D in prompt information PM AN .

[0104] In some embodiments, the prompt information PM may include unmarked data D UL , and no data D is marked UL It can be a mixture of unlabeled sample ULS and some calibration samples. Figure 2 and Figure 3E As shown, in some embodiments, step S260 further includes steps S264, S265 and S266 to generate the unlabeled data D of the prompt information PM. UL The processing unit 110 executes step S264 to calculate the remaining reserved samples TS Kleft (For example, not selected as the first anchor sample TS KAN The reserved sample TS KP ) and the unlabeled sample ULS, and select the first calibration sample TS Kcal From the remaining retained samples TS Kleft In some embodiments, some remaining reserved samples TS similar to the unlabeled samples ULS Kleft The first calibration sample TS is selected Kcal .

[0105] The processing unit 110 executes step S265 to calculate the remaining removed samples TS Rleft (For example, not selected as the second anchor sample TS RAN Remove sample TS RMV ) and the unlabeled sample ULS, and select the second calibration sample TS Rcal From the remaining removed samples TS Rleft In some embodiments, some remaining removed samples TS similar to the unlabeled samples ULS Rleft The second calibration sample TS is selected Rcal .

[0106] The processing unit 110 executes step S266 to convert the first calibration sample TS Kcal With the second calibration sample TS Rcal Mixed into the unlabeled sample ULS to form the unlabeled data D in the prompt information PM UL .

[0107] Please refer to further Figure 4 , Figure 4 According to step S260 (including Figure 3E The schematic diagram of the prompt information PM of steps S261-S266) is shown in FIG. Figure 4 As shown, the prompt information PM includes the task instruction INST, the anchor point data D generated based on the partial sample PS AN , anchor mark data LB AN and unlabeled data D UL .

[0108] like Figure 4 As shown, the task instruction INST in the prompt message PM is used to notify the generative pre-trained transformer model 190 to identify outliers in a given data set. Figure 4 As shown, the task instruction INST requires the generative pre-trained transformer model 190 to act as a data analyst based on the anchor data D AN and anchor mark data LB AN Tips to predict unlabeled data D UL Ingroup / outlier labels for text samples within .

[0109] exist Figure 3D and Figure 4 In the embodiment shown, the sample TS is removed RMV Text samples TS11 and TS6 and reserved samples TS KP The text sample TS9 is used to form the anchor data D in the prompt information PM AN The manually input marks of the text samples TS11, TS6 and TS9 are used to form the anchor mark data LB in the prompt information PMAN .

[0110] exist Figure 3D and Figure 4 In the embodiment shown, the sample TS is retained KP The text sample TS5 is used to form the first calibration sample TS Kcal , and remove the sample TS RMV The text sample TS13 is used to form the second calibration sample TS Rcal The first calibration sample TS Kcal , the second calibration sample TS Rcal Mixed with other unlabeled samples ULS to form the unlabeled data D in the prompt information PM UL .

[0111] like Figure 1 , Figure 2 and Figure 4 As shown, the processing unit 110 and the communication circuit 150 execute step S270 to provide the prompt information PM to the generative pre-trained transformer model 190 to generate the relevant unlabeled data D UL (including unlabeled sample ULS, first calibration sample TS Kcal and the second calibration sample TS Rcal ) of the ingroup value-outlier prediction marker LB PRED In some embodiments, the generative pre-trained transformer model 190 will be based on the anchor data D in the prompt information PM AN and anchor mark data LB AN to generate information about the unlabeled data D UL The ingroup value-outlier prediction marker LB PRED LB PRED It can be predicted that each unlabeled data D UL is an ingroup value or an outlier.

[0112] In some embodiments, the manually input mark MLB for the text samples TS5 and TS13 is not added to the content of the prompt message PM. Kcal and the second calibration sample TS Rcal Used to verify the in-group value-outlier prediction label LB generated by the generative pre-trained transformer model 190 PRED In other words, the first calibration sample TS Kcal and the second calibration sample TS Rcal Can be used to evaluate the ingroup value-outlier prediction label LB produced by the generative pre-trained transformer model 190 PRED accuracy.

[0113] Ingroup value-outlier prediction label LB generated by the generative pre-trained transformer model 190 PRED Including calibration samples (eg, first calibration sample TS Kcal and the second calibration sample TS Rcal ) on the calibration prediction marker LB cal The processing unit 110 is used to compare the manually input mark MLB of the text samples TS5 and TS13 with the calibrated predicted mark LB cal Manual input marker MLB vs. calibrated predicted marker LB cal The closer the match, the higher the confidence. Manual input marker MLB and calibration prediction marker LB cal The greater the mismatch, the lower the confidence.

[0114] In some embodiments, the electronic device 100 is used to generate a pre-trained transformer model 190 to collect inliers-outliers prediction labels LB PRED Then, the processing unit 110 is used to remove the text samples with outlier labels from the data set DB, so that the remaining text samples in the data set DB are all clean data (for example, in-group value data) without noise data.

[0115] Based on the above embodiment, some of the partial samples PS are extracted from the entire text sample data set DB. A small number of partial samples PS need to be manually marked. The partial samples PS are processed to provide anchor data D in the prompt information PM. AN and calibration data (e.g., the first calibration sample TS Kcal and the second calibration sample TS Rcal ). The prompt information PM is used to trigger the generative pre-trained transformer model 190 to generate in-group value-outlier prediction labels LB for a large number of unlabeled samples ULS PRED This can avoid the heavy burden of labelers manually labeling a large number of text samples. Labelers only need to label a small part of the text samples, while most of the text samples can be automatically processed by the data classification method 200.

[0116] Further references Figure 5 , Figure 5 FIG. 5 is a flow chart of a data classification method 500 according to another embodiment of the present disclosure. The data classification method 500 may be Figure 1 The electronic device 100 shown executes Figure 2 Similar to the data classification method 200 shown, the data classification method 500 is used to generate inlier-outlier predictions for text samples in a data set DB.

[0117] like Figure 5As shown, the data classification method 500 includes steps S510, S520, S530, S540, S550, S560, S565, S570, S575 and S580. Figure 5 Steps S510, S520, S530, S540 and S550 of the data classification method 500 are similar to Figure 2 Steps S210, S220, S230, S240 and S250 of the data classification method 200 are described in detail. Figure 2 The illustrated embodiment is fully disclosed, and therefore details of steps S510, S520, S530, S540 and S550 are not described in detail herein.

[0118] like Figure 3D and Figure 5 As shown, in step S550, the manually input mark MLB includes the reserved marks on the text samples TS5 and TS9, which makes the text samples TS5 and TS9 be regarded as the reserved samples TS in the partial sample PS. KP On the other hand, the manually input mark MLB includes removal marks on text samples TS6, TS11, and TS13, which makes the text samples TS6, TS11, and TS13 be regarded as the removal samples TS in the partial sample PS. RMV .

[0119] like Figure 1 and Figure 5 As shown, the processing unit 110 executes step S560 to generate a first prompt message, and the first prompt message includes a portion of samples with manually input marks and feature engineering task instructions.

[0120] Please refer to further Figure 6 , Figure 6 FIG. 1 is a schematic diagram of the first prompt information PM1 generated in step S560 according to an embodiment of the present disclosure. Figure 6 As shown, the first prompt information PM1 includes feature engineering task instruction INST1, retained sample TS KP and remove sample TS RMV .

[0121] Keep sample TS KP and remove sample TS RMV is a partial sample PS with manually input labels MLB, which is used to provide multiple hints for the generative pre-trained transformer model 190 to achieve feature engineering tasks.

[0122] The feature engineering task instruction INST1 in the first prompt information PM1 is used to trigger the generative pre-trained converter model 190 to generate a TS that can distinguish the removed samples RMV With the reserved sample TS KPTwo distinguishable features DF.

[0123] like Figure 1 , Figure 5 and Figure 6 As shown, the processing unit 110 and the communication circuit 150 execute step S565 to provide the first prompt information PM1 to the generative pre-trained transformer model 190 to generate the distinguishable feature DF.

[0124] like Figure 6 As shown, the generative pre-trained transformer model 190 generates a feature set of 10 distinguishable features DF according to the first prompt information PM1.

[0125] like Figure 1 and Figure 5 As shown, the processing unit 110 executes step S570 to generate second prompt information, and the second prompt information includes a distinguishable feature DF, a text sample and a feature scoring task instruction.

[0126] Further references Figure 7 , Figure 7 FIG. 1 is a schematic diagram of the second prompt information PM2 generated in step S570 according to an embodiment of the present disclosure. Figure 7 As shown, the second prompt information PM2 includes feature scoring task instruction INST2, text sample TS and distinguishable feature DF.

[0127] In some embodiments, the feature scoring task instruction INST2 in the second prompt message PM2 is used to trigger the generative pre-trained transformer model to identify whether the text sample TS has attributes that match the possible options of the distinguishable features DF.

[0128] like Figure 1 , Figure 5 and Figure 7 As shown, the processing unit 110 and the communication circuit 150 execute step S575 to provide the second prompt information PM2 to the generative pre-trained transformer model 190 for feature prediction F of the text sample TS relative to the distinguishable feature DF. PRED .like Figure 7 As shown, the feature prediction F PRED The labeled text sample TS1 is an English text paragraph on the health topic, and the labeled text sample TS2 is a Chinese text paragraph on the health topic. In some embodiments, the distinguishable feature DF is a human-recognizable feature (e.g., language, topic, time, length), such as Figure 6 and Figure 7In other embodiments, the distinguishable feature DF may include a feature that is not recognizable to humans, such as a latent vector.

[0129] like Figure 1 and Figure 5 As shown, the processing unit 110 executes step S580 to execute a classification algorithm to generate an inlier-outlier prediction label for the unlabeled sample.

[0130] Further references Figure 8 , Figure 8 How to generate the inlier-outlier prediction label LB by the classification algorithm in step S580 according to an embodiment of the present disclosure PRED Schematic diagram of .

[0131] like Figure 8 As shown, the classification algorithm in step S580 is based on the feature prediction F of the partial sample PS. PRED1 And the feature prediction F of the unlabeled sample ULS PRED2 Execute, thus generating the ingroup value-outlier prediction label LB for the unlabeled sample ULS PRED In one embodiment, since the text sample TS includes a partial sample PS and an unlabeled sample ULS, Figure 8 The feature prediction F of some samples PS in PRED1 And the feature prediction F of the unlabeled sample ULS PRED2 The two are Figure 7 Feature prediction F of text sample TS in PRED Obtain.

[0132] like Figure 8 As shown, the classification algorithm in step S580 is based on the feature prediction F including the partial sample PS. PREDl The training data TD of MLB with manual input of some samples PS is used to predict F based on the features of unlabeled samples ULS. PRED2 Generate ingroup value-outlier prediction label LB for unlabeled samples ULS PRED .

[0133] For example, if one of the unlabeled samples ULS has the same feature prediction F as the text sample TS11 PRED1 A similar set of features predicts F PRED2 , then the classification algorithm can classify the unlabeled sample ULS as an outlier. If one of the unlabeled samples ULS has the same feature prediction F as the text sample TS9 PRED1 A similar set of features predicts F PRED2, the classification algorithm can classify the unlabeled sample ULS as an in-group value. In some embodiments, the classification algorithm in step S580 can be implemented by an extreme gradient boosting (XGBoost) algorithm, a classification boosting (CatBoost) algorithm, or a random forest (RandomForest) algorithm.

[0134] In some embodiments, the extreme gradient boosting algorithm may generate a prediction score for each text sample. When the prediction score of a text sample is greater than 0.5 (e.g., prediction score = 0.7), the text sample may be marked as an in-group value, and its confidence level may be equal to its score (e.g., confidence level = 0.7). When the prediction score of a text sample is less than 0.5 (e.g., prediction score = 0.2), the text sample may be marked as an outlier, and its confidence level is equal to the complement of its score (e.g., confidence level = 1-0.2 = 0.8).

[0135] Based on the above embodiment, some partial samples PS are extracted from the entire text sample data set DB. A small number of partial samples PS need to be manually labeled. The partial samples PS are processed to generate first prompt information PM1 and second prompt information PM2. The first prompt information PM1 and the second prompt information PM2 are used to trigger the generative pre-trained converter model 190 to perform feature engineering tasks and feature scoring tasks. Based on the feedback from the generative pre-trained converter model 190, a large number of in-group value-outlier prediction labels LB of unlabeled samples ULS can be quickly generated. PRED This can avoid the heavy burden of professionals manually marking a large number of text samples. The labeler only needs to mark a small part of the text samples, while the majority of the text samples can be automatically processed by the data classification method 500.

[0136] Certain words are used in the specification and claims to refer to specific components. However, those with ordinary knowledge in the art should understand that the same component may be referred to by different terms. The specification and claims do not distinguish between components based on differences in name, but rather on differences in function. The term "including" mentioned in the specification and claims is an open-ended term and should be interpreted as "including but not limited to".

[0137] In addition, unless otherwise specified in the specification, any singular term also includes the plural term.

[0138] The above are only preferred embodiments of the present disclosure. Various modifications and equivalent changes may be made to the present disclosure without departing from the scope or spirit of the present disclosure. In summary, all modifications and equivalent changes made to the present disclosure within the scope of the following claims are within the scope of the present disclosure.

Claims

1. A data classification method, characterized in that: Include: Obtain multiple text samples from a data set; Converting the plurality of text samples into a plurality of text embeddings in a semantic space; Generating an outlier-ingroup ranking of the plurality of text samples based on an outlier detection algorithm according to the distances between the plurality of text embeddings in the semantic space; Selecting a plurality of partial samples from the plurality of text samples according to the outlier value-ingroup value sorting; receiving a manual input instruction for specifying a plurality of manual input tags of the plurality of partial samples; generating a prompt message according to the plurality of partial samples with the plurality of manual input labels and a plurality of unlabeled samples among the plurality of text samples, wherein the prompt message comprises a task instruction generated according to the plurality of partial samples with the plurality of manual input labels, an unlabeled data, and an anchor point data; and The prompt information is provided to a generative pre-trained transformer model to generate a plurality of inlier-outlier prediction labels associated with the plurality of unlabeled samples.

2. The data classification method according to claim 1, characterized in that: At least one of the plurality of text samples that tends to be an outlier sample is selected as the plurality of partial samples according to the outlier value-ingroup value sorting, and the number of the plurality of partial samples is less than the number of the plurality of unlabeled samples.

3. The data classification method according to claim 1, characterized in that: The manual input instruction is used to assign multiple retention marks to multiple retention samples among the multiple partial samples, and assign multiple removal marks to multiple removal samples among the multiple partial samples. The method for generating the prompt information includes: Selecting a plurality of first anchor point samples from the plurality of retained samples according to the cluster distribution of the plurality of retained samples; Selecting a plurality of second anchor point samples from the plurality of removed samples according to the cluster distribution of the plurality of removed samples; as well as The plurality of first anchor point samples and the plurality of second anchor point samples are combined to form the anchor point data in the prompt information.

4. The data classification method according to claim 3, characterized in that: The method for generating the prompt information further comprises: selecting a plurality of first calibration samples from the plurality of retained samples that are not selected as the plurality of first anchor point samples; selecting a plurality of second calibration samples from the plurality of removed samples that are not selected as the plurality of second anchor point samples; as well as The unlabeled data in the prompt information is formed according to a mixture of the plurality of unlabeled samples, the plurality of first calibration samples, and the plurality of second calibration samples.

5. The data classification method according to claim 4, characterized in that: The plurality of first calibration samples and the plurality of second calibration samples are used to verify the confidence of the plurality of inliers-outliers prediction labels generated by the generative pre-trained transformer model.

6. The data classification method according to claim 4, characterized in that: The method of selecting the plurality of first calibration samples comprises: comparing a plurality of similarities between the plurality of retained samples that are not selected as the plurality of first anchor point samples and the plurality of unlabeled samples; and A plurality of first calibration samples are selected according to the plurality of similarities.

7. The data classification method according to claim 1, characterized in that: The task instructions in the prompt information are used to inform the generative pre-trained transformer model to identify multiple outliers in a given data set.

8. The data classification method according to claim 1, wherein: The outlier detection algorithm is implemented by random sampling consensus-NN, isolation forest algorithm or regional outlier factor algorithm.

9. The data classification method according to claim 1, wherein: Each of the text samples includes a text paragraph or a combination of a question and an answer.

10. A data classification method, characterized in that: Include: Obtain multiple text samples from a data set; Convert the multiple text samples into multiple text embeddings in a semantic space; Generating an outlier-ingroup ranking of the plurality of text samples based on an outlier detection algorithm according to the distances between the plurality of text embeddings in the semantic space; Selecting a plurality of partial samples from the plurality of text samples according to the outlier value-ingroup value sorting; receiving a manual input instruction for specifying a plurality of manual input tags of the plurality of partial samples; generating a first prompt message, wherein the first prompt message includes the plurality of partial samples having the plurality of manually input marks and a feature engineering task instruction; providing the first prompt information to a generative pre-trained transformer model to generate a plurality of distinguishable features; generating a second prompt message, wherein the second prompt message includes the plurality of distinguishable features, the plurality of text samples, and a feature scoring task instruction; providing the second prompt information to the generative pre-trained transformer model to generate a plurality of feature predictions for the plurality of text samples relative to the plurality of distinguishable features; as well as A classification algorithm is executed based on the plurality of feature predictions of the plurality of partial samples and a plurality of unlabeled samples among the plurality of text samples, thereby generating a plurality of inlier-outlier prediction labels of the plurality of unlabeled samples.

11. The data classification method according to claim 10, characterized in that: At least one of the plurality of text samples that tends to be an outlier sample according to the outlier value-ingroup value ranking is selected as one of the plurality of partial samples, and the number of the plurality of partial samples is less than the number of the plurality of unlabeled samples.

12. The data classification method according to claim 10, wherein: The manual input instruction is used to assign a plurality of retention marks to a plurality of retention samples among the plurality of partial samples, and to assign a plurality of removal marks to a plurality of removal samples among the plurality of partial samples.

13. The data classification method according to claim 12, characterized in that: The feature engineering task instruction in the first prompt information is used to trigger the generative pre-trained transformer model to generate the plurality of distinguishable features for separating the plurality of removed samples from the plurality of retained samples.

14. The data classification method according to claim 10, wherein: The feature scoring task instruction in the second prompt information is used to trigger the generative pre-trained converter model to distinguish whether the multiple text samples have multiple attributes of the multiple distinguishable features.

15. The data classification method according to claim 10, wherein: The classification algorithm is executed based on a training data, and the training data includes the multiple feature predictions of the multiple partial samples and the multiple manually input labels of the multiple partial samples, so as to generate the multiple inlier-outlier prediction labels related to the multiple unlabeled samples according to the multiple feature predictions of the multiple unlabeled samples.

16. The data classification method according to claim 10, wherein: The classification algorithm is implemented by an extreme gradient boosting algorithm, a classification boosting algorithm or a random forest algorithm.

17. The data classification method according to claim 10, wherein: The outlier detection algorithm is implemented by a random sampling consensus-NN algorithm, an isolation forest algorithm or a regional outlier factor algorithm.

18. The data classification method according to claim 10, wherein: Each of the text samples includes a text paragraph or a combination of a question and an answer.

Images