Classification model training method and device and data labeling method and device

Abstract

The invention discloses a classification model training method and a data annotation method and device.The classification model training method comprises the steps that first to-be-annotated text data is classified based on an initial classification model, and a first classification result of the first to-be-annotated text data is obtained; under the condition that the first classification result does not meet the preset condition, expanding attribute features of the first to-be-labeled text data to obtain second to-be-labeled text data; based on the initial classification model, classifying the second to-be-labeled text data to obtain a second classification result of the second to-be-labeled text data; under the condition that the second classification result meets a preset condition, determining the second classification result as a target classification label of the first to-be-labeled text data; and based on the first to-be-labeled text data and the target classification label thereof, updating the initial classification model to obtain a target classification model. The data classification accuracy of the classification model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of data classification, and specifically to a classification model training method, a data labeling method and a device. Background Art

[0002] The terms and concepts of data classification and grading are relatively complex, requiring in-depth professional knowledge and expertise in data classification and grading to correctly understand. Neural networks are usually used for grading and classification. If the accuracy of the model classification is low, it will lead to low data classification accuracy. Summary of the invention

[0003] The embodiments of the present application provide a classification model training method, a data labeling method and a device, which can improve the accuracy of data classification.

[0004] In a first aspect, the training method of the classification model provided in the present application includes:

[0005] Based on the initial classification model, classify the first text data to be annotated to obtain a first classification result of the first text data to be annotated;

[0006] When the first classification result does not meet the preset condition, expanding the attribute features of the first text data to be annotated to obtain second text data to be annotated;

[0007] Based on the initial classification model, classify the second text data to be annotated to obtain a second classification result of the second text data to be annotated;

[0008] In the case where the second classification result meets the preset condition, determining the second classification result as the target classification label of the first text data to be annotated;

[0009] Based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain a target classification model.

[0010] In the second aspect, the data annotation method provided by the present application includes:

[0011] Determining fourth text data to be annotated;

[0012] Based on the target classification model, classify the fourth text data to be annotated to obtain a fourth classification result of the fourth text data to be annotated, wherein the target classification model is a target classification model obtained by the training method of the classification model described in any one of the first aspects;

[0013] Based on the fourth classification result, a classification label of the fourth text data to be annotated is determined, and the fourth text data to be annotated is annotated based on the classification label.

[0014] In a third aspect, the present application provides a training device for a classification model, comprising:

[0015] A first classification module, used for classifying the first text data to be annotated based on the initial classification model to obtain a first classification result of the first text data to be annotated;

[0016] An expansion module, used for expanding the attribute features of the first text data to be annotated to obtain second text data to be annotated when the first classification result does not meet a preset condition;

[0017] A second classification module, used for classifying the second text data to be annotated based on the initial classification model to obtain a second classification result of the second text data to be annotated;

[0018] A determination module, configured to determine the second classification result as a target classification label of the first text data to be annotated if the second classification result meets the preset condition;

[0019] An updating module is used to update the initial classification model based on the first text data to be annotated and its target classification label to obtain a target classification model.

[0020] In a fourth aspect, the data annotation device provided by the present application includes:

[0021] A determination module, used for determining fourth text data to be annotated;

[0022] A classification module, used for classifying the fourth text data to be annotated based on a target classification model to obtain a fourth classification result of the fourth text data to be annotated;

[0023] The labeling module is used to determine the classification label of the fourth text data to be labeled based on the fourth classification result, and label the fourth text data to be labeled based on the classification label.

[0024] In a fifth aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the training method of the classification model provided in the present application or the steps in the data labeling method.

[0025] In a sixth aspect, the computer-readable storage medium provided in the present application stores a plurality of instructions, which are suitable for loading by a processor to implement the steps in the training method of the classification model provided in the present application or the steps in the data labeling method.

[0026] In the seventh aspect, the computer program product provided in the present application includes a computer program or instructions, which, when executed by a processor, implements the steps in the training method of the classification model or the steps in the data labeling method provided in the present application.

[0027] In the present application, compared with the related art, based on the initial classification model, the first text data to be annotated is classified to obtain the first classification result of the first text data to be annotated; when the first classification result does not meet the preset conditions, the attribute characteristics of the first text data to be annotated are expanded to obtain the second text data to be annotated; based on the initial classification model, the second text data to be annotated is classified to obtain the second classification result of the second text data to be annotated; when the second classification result meets the preset conditions, the second classification result is determined as the target classification label of the first text data to be annotated; based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain the target classification model. On the basis that the classification result of the first data to be annotated does not meet the preset conditions, the present application adopts the method of expanding the attribute characteristics of the first data to be annotated to obtain the second data to be annotated with rich features, and then continues to classify the second data to be annotated so that the classification result can meet the preset conditions, and the classification result of the second data to be annotated is used as the classification label of the first data to be annotated to update the initial classification model to obtain the target classification model. Since the second data to be annotated has richer features, the classification result of the second data to be annotated is used as the classification label of the first data to be annotated to update the initial classification model, which can improve the data classification accuracy of the classification model. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0029] Figure 1 is a scenario diagram of a data annotation system provided in an embodiment of the present application;

[0030] Figure 2 It is a flowchart of an embodiment of a training method for a classification model provided in an embodiment of the present application;

[0031] Figure 3 It is a flowchart of another embodiment of the training method of the classification model provided in the embodiment of the present application;

[0032] Figure 4 This is a schematic diagram of information flow of another embodiment of the training method of the classification model provided in the embodiment of the present application;

[0033] Figure 5 This is a schematic diagram of an embodiment of the data labeling method provided in the embodiment of the present application;

[0034] Figure 6 is a schematic diagram of the application of a target classification model in one embodiment of a data labeling method provided in an embodiment of the present application;

[0035] Figure 7 It is a structural diagram of an embodiment of a training device for a classification model provided in an embodiment of the present application;

[0036] Figure 8 It is a structural diagram of an embodiment of a data labeling device provided in an embodiment of the present application;

[0037] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0039] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0040] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0041] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0042] The terms and concepts of data classification and grading in the financial field are relatively complex, and require in-depth financial expertise and data classification and grading expertise to correctly understand, and neural networks are needed for grading and classification. The data categories in the financial field are not limited to sensitive data, but also include a large number of other categories, resulting in low data classification accuracy. Based on the fact that the classification result of the first data to be labeled does not meet the preset conditions, the present application adopts a method of expanding the attribute characteristics of the first data to be labeled to obtain a second data to be labeled with rich features, and then continues to classify the second data to be labeled so that the classification result can meet the preset conditions, and uses the classification result of the second data to be labeled as the classification label of the first data to be labeled to update the initial classification model to obtain the target classification model. Since the second data to be labeled has richer features, the classification result of the second data to be labeled is used as the classification label of the first data to be labeled to update the initial classification model, which can improve the data classification accuracy of the classification model.

[0043] In order to improve the effect of data classification, the embodiments of the present application provide a classification model training method, a classification model training device, a data labeling method, a data labeling device, an electronic device, a computer-readable storage medium, and a computer program product. Among them, the classification model training method can be executed by the classification model training device, or by an electronic device integrated with the classification model training device. Among them, the data labeling method can be executed by the data labeling device, or by an electronic device integrated with the data labeling device.

[0044] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0045] Please refer to Figure 1 The present application also provides a data annotation system, which includes an electronic device. The electronic device is integrated with a training device and / or a data annotation device of the classification model provided by the present application.

[0046] In order to better understand the training method, data labeling method, device, electronic device and storage medium of a classification model provided in an embodiment of the present application, the application environment applicable to the embodiment of the present application is first described below.

[0047] See also Figure 1 , Figure 1A schematic diagram of an application environment of the training method of the classification model provided in an embodiment of the present application is shown. As an implementation mode, the training method of the classification model provided in the embodiment of the present application can be applied to the same electronic device. The electronic device can be Figure 1 The server 110 shown in the figure can be connected to the terminal device 120 through a network. The network is used to provide a medium for a communication link between the server 110 and the terminal device 120. The network can include various connection types, such as a wired communication link, a wireless communication link, etc., which are not limited in the embodiments of the present application. Optionally, in other embodiments, the electronic device can also be a smart phone, a laptop computer, etc.

[0048] It should be understood that Figure 1 The server 110, network and terminal device 120 are merely illustrative. Depending on the implementation requirements, there may be any number of servers, networks and terminal devices. For example, the server 110 may be a physical server or a server cluster composed of multiple servers, and the terminal device 120 may be a mobile phone, a tablet, a desktop computer, a laptop computer, and the like. It is understood that the embodiments of the present application may also allow multiple terminal devices 120 to access the server 110 at the same time.

[0049] In addition, the data annotation system may also include a memory for storing original data, intermediate data, and result data.

[0050] In the embodiment of the present application, the memory may be a cloud memory. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as the storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network through application software or application interfaces to work together and provide external data storage and business access functions.

[0051] At present, the storage method of the storage system is: create a logical volume, and when creating a logical volume, allocate physical storage space for each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, stores the data on the file system. The file system divides the data into many parts, each of which is an object. The object contains not only data but also additional information such as data identification (ID entity, ID). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0052] The process of the storage system allocating physical storage space to a logical volume is as follows: based on the estimated capacity of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of the Redundant Array of Independent Disks (RAID), the physical storage space is divided into stripes in advance. A logical volume can be understood as a stripe, thereby allocating physical storage space to the logical volume.

[0053] It should be noted that Figure 1 The scenario diagram of the data annotation system shown is merely an example. The data annotation system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the data annotation system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0054] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0055] Please refer to Figure 2 , Figure 2 FIG. 1 is a flow chart of an embodiment of a training method for a classification model provided in an embodiment of the present application. Figure 2 As shown, the process of the training method of the classification model provided in this application is as follows:

[0056] 201. Classify first text data to be annotated based on an initial classification model to obtain a first classification result of the first text data to be annotated.

[0057] In the embodiment of the present application, the first text data to be annotated can be data in the fields of finance, law, etc., which can be set according to specific circumstances. Among them, the first text data to be annotated can include multiple attribute features. Taking the first text data to be annotated as financial data as an example, the attribute features can be date, transaction quantity, transaction location, etc., which can be set according to specific circumstances.

[0058] In the embodiment of the present application, the first classification result includes the probability that the first text data to be annotated is predicted to be each classification label. For example, the first text data to be annotated is data A, and the probability that data A is predicted to be category 1 level 1 is 90%, the probability that data A is predicted to be category 1 level 2 is 80%, and the probability that data A is predicted to be category 2 level 2 is 70%.

[0059] In an embodiment of the present application, the first classification model can be a pre-trained model or a fine-tuned pre-trained model. For example, the first classification model can be a GPT (Generative Pre-Trained) model, a Llama-3 model, a Llama-2 model, etc., which can be set according to the specific situation, and the present application does not limit this. The GPT model is an Internet-based, data-trained, text-generated deep learning model. Llama-3 has two parameter amounts of 8 billion and 70 billion. Model types cover basic pre-training and instruction fine-tuning, and models with more than 400 billion parameters are under development. Llama-3 uses advanced technologies such as grouped query attention and masking to maximize performance while reducing energy consumption.

[0060] It should be noted that artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0061] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in the fields of machine vision and natural speech processing.

[0062] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is very helpful for interpreting data such as text, images, and sound. Using deep learning technology and corresponding sample sets, network models that can achieve different functions can be trained.

[0063] Furthermore, in order to reduce the amount of calculation, the probability of the first text data to be annotated being predicted as each classification label is sorted from large to small, and the third preset number of classification labels of the first text data to be annotated that are ranked higher are determined as the first classification result. The third preset number is less than the total number of classification labels. For example, if there are 6 classification labels, the third preset number is half of the total number of classification labels, that is, 3. Since the categories with lower probabilities have less effect, they can be directly eliminated to reduce the amount of calculation.

[0064] 202. When the first classification result does not satisfy a preset condition, expand the attribute features of the first text data to be annotated to obtain second text data to be annotated.

[0065] The number of attribute features of the second text data to be annotated is greater than the number of attribute features of the first text data to be annotated.

[0066] In a specific embodiment, the preset condition includes that the difference between the maximum value and the second largest value of the probabilities of the M classification labels is greater than a preset value, wherein the preset value may be 0, 1% or other values.

[0067] Specifically, the maximum and second largest probabilities of the M classification labels in the first classification result are determined; if the difference between the maximum and second largest probabilities is not greater than a preset value, it is determined that the first classification result does not meet the preset condition. If the difference between the maximum and second largest probabilities is greater than a preset value, it is determined that the first classification result meets the preset condition.

[0068] Furthermore, when the first classification result meets the preset conditions, it indicates that the model can accurately determine the category, and the first classification result is determined as the target classification label of the first text data to be annotated.

[0069] Specifically, the second classification result includes the probabilities of M classification labels. The maximum and second largest probabilities among the M classification labels are determined based on the second classification result; if the difference between the maximum and the second largest probabilities is greater than a preset value, it is determined that the second classification result meets the preset conditions.

[0070] In a specific embodiment, the plurality of first text data to be annotated are data A and data B, and the first classification results of data A and data B are shown in the following table:

[0071]

[0072] For example, the preset value is 0. The first text data to be annotated is data B, and the first classification result is: the probability that data B is predicted as category 1 level 1 is 95%, the probability that data B is predicted as category 1 level 2 is 95%, and the probability that data B is predicted as category 2 level 2 is 70%. The maximum value of the probability of each classification label in the first classification result is 95%, and the second largest value of the probability of each classification label in the first classification result is 95%. The difference between the maximum value and the second largest value is 0, which is not greater than the preset value. The first classification result does not meet the preset conditions, indicating that the maximum probability recognition result is not unique, and the model cannot label the first text data to be annotated, then the attribute features of the first text data to be annotated are expanded to obtain the second text data to be annotated.

[0073] For another example, the preset value is 0. The first text data to be annotated is data A, and the first classification result is: the probability that data A is predicted to be category 1 level 1 is 90%, the probability that data A is predicted to be category 1 level 2 is 80%, and the probability that data A is predicted to be category 2 level 2 is 70%. The maximum value of the probability of each classification label in the first classification result is 90%, and the second largest value of the probability of each classification label in the first classification result is 80%. The difference between the maximum value and the second largest value is 10%, and 10% is greater than the preset value, indicating that the maximum probability recognition result is unique, and the model can label the first text data to be annotated, and can directly label it.

[0074] In the embodiment of the present application, when the first classification result does not meet the preset condition, the attribute features of the first text data to be annotated are expanded to obtain the second text data to be annotated.

[0075] In a specific embodiment, expanding the attribute characteristics of first text data to be annotated to obtain second text data to be annotated includes: obtaining the attribute characteristics of a first target quantity, wherein the attribute characteristics of the first target quantity include at least one of metadata, data source, data management entity, data storage time and data size; adding the attribute characteristics of the first target quantity to the first text data to be annotated to obtain the second text data to be annotated.

[0076] The newly added attribute features may be metadata, data source, and data management entity, etc. The attribute features of the first text data to be annotated may include: date, transaction amount, transaction location, and the expanded attribute features of the second text data to be annotated may include: date, transaction amount, transaction location, metadata, data source, etc. The data source may be the department or individual of the output data, and the data management entity may be the person in charge or department responsible for managing the data.

[0077] Metadata, also known as intermediary data or relay data, is data about data. It mainly describes the properties of data and is used to support functions such as indicating storage location, historical data, resource search, and file records.

[0078] The data source refers to the source of the obtained data. In this application, it is the source of each text data in the first text data to be annotated. Text data with the same source are more likely to belong to the same category. Combining the source of the text data is conducive to more accurate classification of the text data.

[0079] Data management entities are used to represent physical devices that manage data. Data management is the process of effectively collecting, storing, processing and applying data using computer hardware and software technologies. Its purpose is to fully and effectively play the role of data. The key to achieving effective data management is data organization.

[0080] Since some attribute features have little effect on the classification result, if all attribute features are added, the amount of model calculation will increase. In order to reduce the amount of model calculation and improve the model training efficiency, further, obtaining the attribute features of the first target number includes: obtaining the attribute features of the second target number; sorting the attribute features of the second target number based on the feature importance of the attribute features; based on the sorting, obtaining the attribute features of the first target number from the attribute features of the second target number, and the first target number is less than the second target number. Specifically, based on the sorting, obtaining the attribute features of the first target number that are ranked higher in terms of feature importance from the attribute features of the second target number.

[0081] Among them, feature importance is used to indicate the degree of influence of the feature on the first text data to be annotated, that is, the importance of the attribute feature in the classification model. By screening out attribute features with higher feature importance, it is helpful to improve the accuracy and reliability of the classification model in classifying the text data to be annotated.

[0082] Optionally, there are many methods for judging the importance of features, one of which is based on a decision tree. In a decision tree, the weight of the gain obtained by a feature at each decision node is not the same, that is, the gain obtained by a feature at a decision node should be weighted according to the proportion of samples that the decision node helps to distinguish. Based on this intuition, a formula for calculating the importance of each feature in a decision tree can be derived. In addition, random forest, as an integrated learning algorithm, is also commonly used for feature selection. Random forest uses decision trees as the base learner to improve prediction accuracy and stability by constructing multiple decision trees. In a random forest, the importance of a feature can be evaluated by observing the frequency and depth of the feature being used, thereby determining which features have a greater impact on the prediction results of the model. In an embodiment of the present application, the importance of attribute features can be determined by the above method, so that the attribute features are sorted to select the first target number of attribute features.

[0083] Based on the fact that the classification result of the first text data to be annotated does not meet the preset conditions, the present application adopts a method of expanding the attribute characteristics of the first text data to be annotated to obtain second text data to be annotated with rich features, and then continues to classify the second text data to be annotated so that the classification result can meet the preset conditions, thereby using the classification result of the second text data to be annotated as the classification label of the first text data to be annotated. Since the second text data to be annotated has richer features, the accuracy of data classification can be improved.

[0084] 203. Classify the second text data to be annotated based on the initial classification model to obtain a second classification result of the second text data to be annotated.

[0085] 204. When the second classification result meets a preset condition, determine the second classification result as a target classification label for the first text data to be annotated.

[0086] Specifically, the classification label of the second text data to be annotated in the second classification result is determined as the target classification label of the corresponding first text data to be annotated.

[0087] 205. Based on the first text data to be annotated and its target classification label, update the initial classification model to obtain a target classification model.

[0088] In a specific embodiment, based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain the target classification model, including: when the number of the first text data to be annotated with the target classification label reaches the first preset number, the annotation accuracy is determined based on the first preset number of the first text data to be annotated and its target classification label; when the annotation accuracy is not greater than the preset accuracy, the initial classification model and the target classification label of the first text data to be annotated are updated to obtain the target classification model. The first preset number can be set according to the specific situation, for example, the first preset number is 1000. The preset accuracy can be 95%, which can be set according to the specific situation.

[0089] The annotation accuracy rate indicates the proportion of the first text data to be annotated that are correctly annotated to the total number.

[0090] In a specific embodiment, first text data to be annotated and a corresponding target classification label are sent out, and an expert judges the first text data to be annotated and the corresponding target classification label, and obtains a second judgment result of manual judgment. If the first text data to be annotated and the corresponding target classification label are judged to match according to the second judgment result, it is determined that the first text data to be annotated is correctly annotated, and the annotation accuracy is obtained. Most of the existing technologies rely on manual judgment. For example, given data A, the classification label of data A is manually determined, and it is necessary to manually match data A with the rules of each classification label, which is a large workload. However, the present application provides a small number of options through the model, and humans only need to judge whether the classification label with the highest probability matches data A, which greatly reduces the workload.

[0091] In another specific embodiment, the annotation accuracy is determined based on a first preset number of first text data to be annotated and its target classification label, including: obtaining a classification matching condition corresponding to the target classification label of the first text data to be annotated, wherein different classification labels correspond to different classification matching conditions; determining the first text data to be annotated that meets the classification matching condition as correctly annotated, and determining the ratio of the number of correctly annotated first text data to be annotated among the first preset number of first text data to be annotated to the first preset number as the annotation accuracy.

[0092] Among them, the classification matching conditions corresponding to each classification label are pre-set. The classification matching conditions may include the range of each attribute feature. For example, the classification matching conditions corresponding to category 1 level 1 may be: the date belongs to the preset date range, the transaction quantity belongs to the preset data range, etc. Although setting the classification matching conditions to directly predict data is less efficient than model prediction, after the model predicts the result, the classification matching conditions can be used to verify whether the model prediction result is correct. Compared with manual verification, the verification efficiency can be improved, thereby quickly calculating the annotation accuracy and improving the annotation efficiency.

[0093] Taking the first text data to be annotated as financial data as an example, the classification matching conditions corresponding to each classification label are determined according to the classification and grading standards of existing financial data, the list of categories and levels, and the determination strategy of categories and levels. Among them, the classification and grading standards include laws, administrative regulations, departmental rules, national standards, industry standards, etc. The list of categories and levels includes: category list and definition, level list and definition, and the relationship between categories and levels. The determination strategy of categories and levels includes determination process and rules.

[0094] Among them, the classification matching condition can be expressed by a regular expression. Regular expression, also known as regular expression (Regular Expression, often abbreviated as regex, regexp or RE in the code), is a text pattern, including common characters and special characters, and is a concept in computer science. Regular expression uses a single string to describe and match a series of strings that match a certain syntactic rule, and is usually used to retrieve and replace text that meets a certain pattern. By expressing the classification matching condition through a regular expression, the first text data to be annotated that meets the classification matching condition can be quickly screened out.

[0095] Further, if the annotation accuracy is calculated based on the first preset number of first text data to be annotated, if the first preset number is large, the amount of calculation is large, resulting in low efficiency. In order to improve the efficiency of calculating the annotation accuracy, the annotation accuracy is determined based on the first preset number of first text data to be annotated and its target classification label, including: randomly sampling the first preset number of first text data to be annotated to obtain a second preset number of text data to be annotated, wherein the first preset number is greater than the second preset number; and determining the annotation accuracy based on the second preset number of first text data to be annotated and its target classification label. For example, the second preset number is one-fourth of the first preset number, which can be set according to the specific situation, and this application does not limit this.

[0096] Specifically, the first text data to be annotated that meets the classification matching condition is determined to be correctly annotated, and the ratio of the number of correctly annotated first text data to be annotated in the second preset number of first text data to be annotated to the second preset number is determined as the annotation accuracy rate.

[0097] In the embodiment of the present application, based on the initial classification model, the first text data to be annotated is classified to obtain a first classification result of the first text data to be annotated, including:

[0098] (1) Based on the first classification model, classify the third text data to be annotated to obtain a third classification result of the third text data to be annotated.

[0099] In the embodiment of the present application, the third text data to be annotated can be text data in the fields of finance, law, etc., which can be set according to specific circumstances. Among them, the third text data to be annotated can include multiple attribute features. Taking the first text data to be annotated as financial data as an example, the attribute features can be date, transaction amount, transaction location, etc., which can be set according to specific circumstances.

[0100] The third classification result includes the probability that the third text data to be annotated is predicted to be each classification label.

[0101] The classification label can be preset, and the classification label can include two sub-labels: category and level. For example, the classification label can be: category 1 level 1, category 1 level 2, category 2 level 1, category 2 level 2, category 3 level 1, category 3 level 2.

[0102] For example, the third text data to be annotated is data A, and the probability that data A is predicted to be category 1 level 1 is 90%, the probability that data A is predicted to be category 1 level 2 is 80%, and the probability that data A is predicted to be category 2 level 2 is 70%.

[0103] In an embodiment of the present application, the first classification model can be a pre-trained model or a fine-tuned pre-trained model. For example, the first classification model can be a GPT (Generative Pre-Trained) model, a Llama-3 model, a Llama-2 model, etc., which can be set according to the specific situation, and the present application does not limit this. The GPT model is an Internet-based, data-trained, text-generated deep learning model. Llama-3 has two parameter amounts of 8 billion and 70 billion. Model types cover basic pre-training and instruction fine-tuning, and models with more than 400 billion parameters are under development. Llama-3 uses advanced technologies such as grouped query attention and masking to maximize performance while reducing energy consumption.

[0104] Furthermore, in order to improve the classification accuracy of the model, the training method of the classification model also includes:

[0105] First, obtain the initial annotated text data and the second classification model.

[0106] The amount of the initial annotated text data is less than the amount of the third text data to be annotated, and the initial annotated text data includes classification labels. The initial annotated text data includes multiple attribute features and classification labels. The classification labels of the initial annotated text data are manually annotated by experts and have high accuracy. The amount of the initial annotated text data is less than the amount of the third text data to be annotated, which can reduce the workload of manual annotation.

[0107] Among them, the second classification model can be a GPT (Generative Pre-Trained) model, a Llama-3 model, a Llama-2 model, etc., which can be set according to the specific situation, and this application does not limit this.

[0108] After obtaining the initial annotated text data and the second classification model, the attribute features of the initial annotated text data are expanded to obtain the target annotated text data.

[0109] The number of attribute features of the target annotated text data is greater than the number of attribute features of the initial annotated text data.

[0110] In an embodiment of the present application, the newly added attribute features may be a data management entity, etc. The attribute features of the initial annotated text data may include: date, transaction quantity, transaction location, and the attribute features of the expanded target annotated text data may include: date, transaction quantity, transaction location, data management entity, etc.

[0111] After expanding the attribute features of the initial annotated text data to obtain the target annotated text data, the second classification model is fine-tuned based on the target annotated text data to obtain the first classification model.

[0112] Fine-tuning improves accuracy by using a pre-trained model obtained on big data to initialize the weights of your own model. This requires that the quality of the pre-trained model must be guaranteed. Fine-tuning is usually faster and more accurate. Of course, you can also use your own trained model as a pre-trained model, and then train it on your own dataset to make the model suitable for your own scenarios and tasks.

[0113] Specifically, after fine-tuning the second classification model, determine whether the prediction accuracy is greater than the preset accuracy. If the prediction accuracy is not greater than the preset accuracy, fine-tune and update the second classification model. If the prediction accuracy is greater than the preset accuracy, output the second classification model to obtain the first classification model.

[0114] (2) issuing a third classification result of the third text data to be annotated, and obtaining a first judgment result returned based on the third classification result.

[0115] Among them, the first judgment result includes a judgment value for judging whether each classification label in the third classification result is a correct item. The first judgment result can be obtained by manual or machine judgment. Optionally, the first judgment result includes a judgment value for judging whether each classification label of a third preset number whose probability is ranked top from large to small in the third classification result is a correct item. The third preset number is less than the total number of classification labels. For example, the total number of classification labels that can be obtained by the first classification model is 6, and the third preset number is 3. Only judging each classification label of the third preset number whose probability is ranked top from large to small by manual or machine can reduce the amount of judgment and improve training efficiency.

[0116] In a specific embodiment, the plurality of third text data to be annotated are data A, data B, and data C. The third classification results and the first judgment results of data A, data B, and data C are shown in the following table:

[0117]

[0118]

[0119] (3) When it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is the correct item, the classification label with the highest probability is used as the classification label of the third text data to be annotated, and the first annotated text data is obtained and put into the target training set.

[0120] In an embodiment of the present application, when the classification label with the highest probability in the third classification result is determined to be the correct item based on the first judgment result, it indicates that the model prediction is correct, and the classification label with the highest probability is used as the classification label of the third text data to be annotated, and the first annotated text data is obtained and put into the target training set.

[0121] For example, the third text data to be annotated is data A, and the third classification result is that the probability of data A being predicted as category 1 level 1 is 90%, the probability of data A being predicted as category 1 level 2 is 80%, and the probability of data A being predicted as category 2 level 2 is 70%. The first judgment result is that data A is predicted to be category 1 level 1, which is correct. At this time, based on the first judgment result, the classification label with the highest probability in the third classification result is determined to be the correct item, and the classification label with the highest probability is used as the classification label of the third text data to be annotated, and the first annotated text data is obtained and put into the target training set.

[0122] Furthermore, in order to improve the classification accuracy of the model, the training method of the classification model also includes: when it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is not the correct item, generating a fine-tuning instruction, wherein the fine-tuning instruction is used to instruct the first classification model to output a preset classification result, and the classification label with the highest probability in the preset classification result is the correct item in the first judgment result; and fine-tuning the first classification model based on the fine-tuning instruction.

[0123] Instructional Fine-tuning is a natural language processing (NLP) technique, especially in the application of large pre-trained language models (such as GPT, etc.). In instructional fine-tuning, the model is further trained to better understand and follow instructions or commands given by humans. This method can improve the performance of the model on a specific task, enabling it to generate more accurate results based on the given instructions.

[0124] Specifically, the fine-tuning instruction is generated according to the first judgment result. For example, the fine-tuning instruction is "update the model so that the model predicts the classification label corresponding to the correct item in the first judgment result as the classification label with the highest probability".

[0125] For example, the third text data to be annotated is data B, and the third classification result is that the probability of data B being predicted as category 1 level 1 is 95%, which is not the correct item; the probability of data B being predicted as category 1 level 2 is 85%, which is the correct item; the probability of data B being predicted as category 2 level 2 is 70%, which is not the correct item. At this time, based on the first judgment result, it is determined that the classification label with the highest probability in the third classification result is not the correct item, and a fine-tuning instruction is generated to correct the classification label of the third text data to be annotated; the first classification model is fine-tuned based on the fine-tuning instruction. For example, the fine-tuning instruction is "update the model so that the model predicts category 2 level 2 as the classification label with the highest probability."

[0126] Furthermore, in order to improve the classification accuracy of the model, the training method of the classification model also includes: when it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is not the correct item, correcting the classification label of the third text data to be annotated based on the first judgment result to obtain second annotated text data; and putting the second annotated text data into the target training set.

[0127] Further, when it is determined based on the first judgment result that each classification label in the third classification result is not a correct item, the third text data to be annotated is re-input into the first classification model for classification.

[0128] (4) Fine-tune the first classification model based on the target training set to obtain the initial classification model.

[0129] Please refer to Figure 3 and Figure 4 , Figure 3 is another embodiment flow chart of the training method of the classification model provided in the embodiment of the present application, Figure 4 is an information flow diagram of another embodiment of the training method of the classification model provided in the embodiment of the present application, combined with Figure 3 and Figure 4 The process of the training method of the classification model provided in this application is as follows:

[0130] 301. Obtain initial annotated text data and a second classification model.

[0131] The amount of the initial annotated text data is less than the amount of the third text data to be annotated, and the initial annotated text data includes classification labels. The initial annotated text data includes multiple attribute features and classification labels. The classification labels of the initial annotated text data are manually annotated by experts and have high accuracy. The amount of the initial annotated text data is less than the amount of the third text data to be annotated, which can reduce the workload of manual annotation.

[0132] like Figure 4 As shown in the figure, taking the initial annotated text data as financial data as an example, the classification matching conditions corresponding to each classification label are determined according to the existing classification and grading standards of financial data, the list of categories and levels, and the determination strategy of categories and levels, and the experts perform annotation to obtain the initial annotated text data. Among them, the classification and grading standards include laws, administrative regulations, departmental rules, national standards, industry standards, etc. The list of categories and levels includes: category list and definition, level list and definition, and the relationship between categories and levels. The determination strategy of categories and levels includes determination process and rules.

[0133] Among them, the second classification model can be a GPT (Generative Pre-Trained) model, a Llama-3 model, a Llama-2 model, etc., which can be set according to the specific situation, and this application does not limit this.

[0134] 302. Expand the attribute features of the initial annotated text data to obtain target annotated text data.

[0135] The number of attribute features of the target annotated text data is greater than the number of attribute features of the initial annotated text data. In the embodiment of the present application, the newly added attribute features may be data management entities, etc. The attribute features of the initial annotated text data may include: date, transaction amount, transaction location, and the expanded attribute features of the target annotated text data may include: date, transaction amount, transaction location, data management entity, etc. Figure 4 As shown, the target annotation text data is obtained by expanding the attributes.

[0136] Most of the existing technologies are directly trained based on sample data. This application expands the attributes of the sample data. After the attributes are expanded, the ability of the large model can be used to summarize more features by itself to help it determine the data category and level, which can improve the accuracy of classification and grading.

[0137] 303. Fine-tune the second classification model based on the target annotated text data to obtain the first classification model.

[0138] like Figure 4 As shown, after fine-tuning the second classification model, determine whether the prediction accuracy is greater than the preset accuracy. If the prediction accuracy is not greater than the preset accuracy, fine-tune and update the second classification model. If the prediction accuracy is greater than the preset accuracy, output the second classification model to obtain the first classification model.

[0139] 304. Classify the third text data to be annotated based on the first classification model to obtain a third classification result of the third text data to be annotated.

[0140] In the embodiment of the present application, the amount of the third text data to be annotated is greater than the amount of the target annotated text data. The first classification model needs to be fine-tuned again with a larger amount of the third text data to be annotated when the target annotated text data is smaller.

[0141] The third classification result includes the probability that the third text data to be annotated is predicted to be each classification label.

[0142] 305 . Issue a third classification result of the third text data to be annotated, and obtain a first judgment result returned based on the third classification result.

[0143] The first judgment result includes a judgment value for judging whether each classification label in the third classification result is a correct item. Optionally, the first judgment result includes a judgment value for judging whether each classification label of a third preset number arranged from large to small probability in the third classification result is a correct item.

[0144] 306. When it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is the correct item, the classification label with the highest probability is used as the classification label of the third text data to be annotated, and the first annotated text data is obtained and put into the target training set.

[0145] In an embodiment of the present application, when the classification label with the highest probability in the third classification result is determined to be the correct item based on the first judgment result, it indicates that the model prediction is correct, and the third text data to be annotated and the corresponding classification label with the highest probability are put into the target training set.

[0146] For example, the first text data to be annotated is data A, and the third classification result is that the probability of data A being predicted as category 1 level 1 is 90%, the probability of data A being predicted as category 1 level 2 is 80%, and the probability of data A being predicted as category 2 level 2 is 70%. The first judgment result is that data A is predicted to be category 1 level 1, which is correct. At this time, based on the first judgment result, the classification label with the highest probability in the third classification result is determined to be the correct item, and the third text data to be annotated and the corresponding classification label with the highest probability are put into the target training set.

[0147] 307. When it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is not a correct item, the classification label of the third text data to be annotated is corrected based on the first judgment result to obtain second annotated text data; and the second annotated text data is put into the target training set.

[0148] Furthermore, when it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is not the correct item, a fine-tuning instruction is generated, wherein the fine-tuning instruction is used to instruct the model to output a preset classification result, and the classification label with the highest probability in the preset classification result is the correct item in the first judgment result; the first classification model is fine-tuned based on the fine-tuning instruction.

[0149] This application uses a large model to improve a large model. Currently, most of the training data is manually labeled by labelers or labeled using rule matching models. The former has high labor costs, and the latter labels data with single features, which is not conducive to the large model's adaptation to category recognition in complex scenarios. This application can correct the model by fine-tuning the instructions, and can also use the large model's own capabilities to quickly produce more sample data. At the same time, the expert correction link changes from manual identification to manual judgment, and the daily output of samples by a single person is greatly increased, which greatly reduces labor costs and work difficulty.

[0150] 308. When it is determined based on the first judgment result that each classification label in the third classification result is not a correct item, the third text data to be annotated is re-input into the first classification model for classification.

[0151] 309. Fine-tune the first classification model based on the target training set to obtain an initial classification model.

[0152] 310. Classify the first text data to be annotated based on the initial classification model to obtain a first classification result of the first text data to be annotated.

[0153] In the embodiment of the present application, the first text data to be annotated can be data in the fields of finance, law, etc., which can be set according to specific circumstances. Among them, the first text data to be annotated can include multiple attribute features. Taking the first text data to be annotated as financial data as an example, the attribute features can be date, transaction quantity, transaction location, etc., which can be set according to specific circumstances.

[0154] In the embodiment of the present application, the amount of the third text data to be annotated is less than the amount of the first text data to be annotated. The initial classification model first classifies the smaller amount of the third text data to be annotated, and then annotates the larger amount of the first text data to be annotated.

[0155] 311. When the first classification result meets a preset condition, determine the first classification result as a target classification label for the first text data to be annotated.

[0156] In a specific embodiment, the preset condition includes: the difference between the maximum value and the second largest value of the probability of each classification label in the first classification result is greater than a preset value, wherein the preset value may be 0, 1% or other values.

[0157] 312. When the first classification result does not satisfy a preset condition, expand the attribute features of the first text data to be annotated to obtain second text data to be annotated.

[0158] In an embodiment of the present application, when the first classification result does not meet the preset conditions, the attribute features of the first text data to be annotated are expanded to obtain the second text data to be annotated. The newly added attribute features may be metadata, data source, etc. The attribute features of the first text data to be annotated may include: date, transaction quantity, transaction location, and the attribute features of the expanded second text data to be annotated may include: date, transaction quantity, transaction location, metadata, data source, etc. Since the initial classification model is fine-tuned on a smaller number of third text data to be annotated, at this time, the first text data to be annotated has more attribute features that are not used when fine-tuning the initial classification model. Therefore, the second text data to be annotated obtained by expanding the first text data to be annotated has more attribute features, so that the model can predict more accurately.

[0159] 313. Based on the initial classification model, classify the second text data to be annotated to obtain a second classification result of the second text data to be annotated; when the second classification result meets a preset condition, determine the second classification result as the target classification label of the first text data to be annotated.

[0160] 314. When the number of first text data to be annotated having the target classification label reaches a first preset number, randomly sample the first preset number of first text data to be annotated to obtain a second preset number of text data to be annotated.

[0161] When the number of first text data to be annotated with a target classification label reaches a first preset number, the first preset number of first text data to be annotated is randomly sampled to obtain a second preset number of first text data to be annotated, wherein the first preset number is greater than the second preset number; and the annotation accuracy is determined based on the second preset number of first text data to be annotated and the corresponding target classification label.

[0162] 315. Determine whether the marking accuracy is greater than the preset accuracy.

[0163] The first preset number can be set according to specific circumstances, for example, the first preset number is 1000. The preset accuracy rate can be 95%, which can be set according to specific circumstances.

[0164] 316. Based on the first text data to be annotated and its target classification label, update the initial classification model to obtain a target classification model.

[0165] When the annotation accuracy is not greater than the preset accuracy, it means that the model accuracy is not enough. Based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain the target classification model. This application only uses positive data for training, and continuously instructs fine-tuning through sampling results to improve the accuracy of the model.

[0166] 317. Output the target classification model.

[0167] When the labeling accuracy is greater than the preset accuracy, it means that the model accuracy is sufficient, and the target classification model is output.

[0168] Please refer to Figure 5 , Figure 5 FIG. 1 is a flow chart of an embodiment of a data labeling method provided in an embodiment of the present application. Figure 5 As shown, the process of the data annotation method provided by this application is as follows:

[0169] 401. Determine fourth text data to be annotated.

[0170] 402. Classify the fourth text data to be annotated based on the target classification model to obtain a fourth classification result of the fourth text data to be annotated.

[0171] The target classification model is a target classification model obtained by the training method of any of the above classification models;

[0172] 403. Determine a classification label of fourth text data to be annotated based on the fourth classification result, and annotate the fourth text data to be annotated based on the classification label.

[0173] For further information, see Figure 6 , Figure 6FIG. 1 is a schematic diagram of an application of a target classification model in an embodiment of a data labeling method provided in an embodiment of the present application. Figure 6 As shown,

[0174] In the embodiment of the present application, the classification matching conditions corresponding to each classification label are determined according to the classification and grading standards of existing financial data, the list of categories and levels, and the determination strategy of categories and levels, and the experts perform annotation to obtain the initial annotated text data. The initial annotated text data is expanded to obtain the target annotated text data, and the target annotated text data, the third text data to be annotated, and the first text data to be annotated are used to train and fine-tune the target classification model. Deploy the target classification model. Determine whether to use rules or models. If rules are used, the classification matching conditions of each classification label are used to determine the scanning results. The scanning results are manually confirmed and the warehousing approval is performed after manual confirmation. If a model is used, determine whether it is automatically confirmed. If not automatically confirmed, the target classification model is used to determine the scanning results. The scanning results are scanned for warehousing approval after manual confirmation. If automatically confirmed, the target classification model is used to determine the scanning results, and the warehousing approval scan is performed directly. Determine whether the approval is passed. If the approval is passed, the external empowerment interface is enabled. If the approval is not passed, it is discarded.

[0175] In order to facilitate better implementation of the classification model training method provided in the embodiment of the present application, the embodiment of the present application also provides a classification model training device based on the above classification model training method. The meanings of the terms are the same as those in the above classification model training method. For specific implementation details, please refer to the description in the above method embodiment.

[0176] Please refer to Figure 7 , Figure 7 1 is a schematic diagram of a structure of an embodiment of a training device for a classification model provided in an embodiment of the present application. The training device 700 for a classification model may include:

[0177] A first classification module 701 is used to classify the first text data to be annotated based on the initial classification model to obtain a first classification result of the first text data to be annotated;

[0178] An expansion module 702, configured to expand the attribute features of the first text data to be annotated to obtain second text data to be annotated when the first classification result does not meet a preset condition;

[0179] A second classification module 703 is used to classify the second text data to be annotated based on the initial classification model to obtain a second classification result of the second text data to be annotated;

[0180] A determination module 704 is used to determine the second classification result as a target classification label of the first text data to be annotated when the second classification result meets the preset condition;

[0181] The updating module 705 is used to update the initial classification model based on the first text data to be annotated and its target classification label to obtain a target classification model.

[0182] Please refer to Figure 8 , Figure 8 800 is a schematic diagram of a structure of an embodiment of a data labeling device provided in an embodiment of the present application. The data labeling device 800 may include:

[0183] A determination module 801 is used to determine fourth text data to be annotated;

[0184] A classification module 802 is used to classify the fourth text data to be annotated based on the target classification model to obtain a fourth classification result of the fourth text data to be annotated;

[0185] The labeling module 803 is used to determine the classification label of the fourth text data to be labeled based on the fourth classification result, and label the fourth text data to be labeled based on the classification label.

[0186] The specific implementation of each of the above modules can be found in the previous embodiments and will not be described in detail here.

[0187] In the present application, compared with the related art, based on the initial classification model, the first text data to be annotated is classified to obtain the first classification result of the first text data to be annotated; when the first classification result does not meet the preset conditions, the attribute characteristics of the first text data to be annotated are expanded to obtain the second text data to be annotated; based on the initial classification model, the second text data to be annotated is classified to obtain the second classification result of the second text data to be annotated; when the second classification result meets the preset conditions, the second classification result is determined as the target classification label of the first text data to be annotated; based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain the target classification model. On the basis that the classification result of the first data to be annotated does not meet the preset conditions, the present application adopts the method of expanding the attribute characteristics of the first data to be annotated to obtain the second data to be annotated with rich features, and then continues to classify the second data to be annotated so that the classification result can meet the preset conditions, and the classification result of the second data to be annotated is used as the classification label of the first data to be annotated to update the initial classification model to obtain the target classification model. Since the second data to be annotated has richer features, the classification result of the second data to be annotated is used as the classification label of the first data to be annotated to update the initial classification model, which can improve the data classification accuracy of the classification model.

[0188] Please refer to Fig. 9 , Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0189] The electronic device 100 may include one or more processors 101 of processing cores, one or more computer-readable storage media memories 102, a power supply 103, an input unit 104 and other components. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. Among them:

[0190] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0191] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0192] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0193] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0194] Although not shown, the electronic device may also include a display unit, an image acquisition element, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the training method of the classification model provided in this application and / or the steps in the data labeling method, such as:

[0195] Based on the initial classification model, the first text data to be annotated is classified to obtain a first classification result of the first text data to be annotated; when the first classification result does not meet the preset conditions, the attribute characteristics of the first text data to be annotated are expanded to obtain the second text data to be annotated; based on the initial classification model, the second text data to be annotated is classified to obtain a second classification result of the second text data to be annotated; when the second classification result meets the preset conditions, the second classification result is determined as the target classification label of the first text data to be annotated; based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain the target classification model;

[0196] And / or, determine fourth text data to be annotated; classify the fourth text data to be annotated based on the target classification model to obtain a fourth classification result of the fourth text data to be annotated; determine a classification label of the fourth text data to be annotated based on the fourth classification result, and annotate the fourth text data to be annotated based on the classification label.

[0197] It should be noted that the electronic device provided in the embodiment of the present application belongs to the same concept as the training method and / or data labeling method of the classification model in the above embodiments. The specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0198] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on the processor of the electronic device provided in the embodiment of the present application, the processor of the electronic device executes the steps in the training method of the classification model provided in the present application and / or the steps in the data labeling method. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0199] The present application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes various optional implementations of the above-mentioned classification model training method and / or data labeling method.

[0200] The above is a detailed introduction to the training method, data labeling method and device of a classification model provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0201] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data need to comply with the relevant laws, regulations and standards of the relevant region.

Claims

1. A classification model training method, characterized in that: The method comprises: Based on the initial classification model, classify the first text data to be annotated to obtain a first classification result of the first text data to be annotated; When the first classification result does not meet the preset condition, expanding the attribute features of the first text data to be annotated to obtain second text data to be annotated; Based on the initial classification model, classify the second text data to be annotated to obtain a second classification result of the second text data to be annotated; In the case where the second classification result meets the preset condition, determining the second classification result as the target classification label of the first text data to be annotated; Based on the first text data to be annotated and its target classification label, the initial classification model is updated to obtain a target classification model.

2. The training method of the classification model according to claim 1, characterized in that: The step of expanding the attribute features of the first text data to be annotated to obtain the second text data to be annotated includes: Acquire attribute characteristics of a first target quantity, wherein the attribute characteristics of the first target quantity include at least one of metadata, data source, and data management entity; The first target number of attribute features is added to the first text data to be annotated to obtain the second text data to be annotated.

3. The training method of the classification model according to claim 2, characterized in that: The obtaining of the attribute characteristics of the first target quantity includes: Obtaining attribute characteristics of a second target quantity; sorting the attribute features of the second target number based on the feature importance of the attribute features; Based on the sorting, the attribute features of the first target quantity are obtained from the attribute features of the second target quantity, and the first target quantity is smaller than the second target quantity.

4. The data classification method according to claim 1, characterized in that: The preset condition is that the difference between the maximum value and the second largest value of the probabilities of the M classification labels is greater than a preset value, and the method further includes: Determine the maximum and second maximum values ​​of the probabilities of the M classification labels based on the second classification result; If the difference between the maximum value and the second largest value is greater than a preset value, it is determined that the second classification result meets the preset condition.

5. The training method of the classification model according to claim 1, characterized in that: The initial classification model is trained by the following steps: Based on the first classification model, classify the third text data to be annotated to obtain a third classification result of the third text data to be annotated, wherein the third classification result includes the probability that the third text data to be annotated is predicted to be M classification labels; Sending a third classification result of the third text data to be annotated, and obtaining a first judgment result returned based on the third classification result, wherein the first judgment result includes a judgment value for judging whether the M classification labels in the third classification result are correct items; In the case where the classification label with the highest probability in the third classification result is determined to be the correct item based on the first judgment result, the classification label with the highest probability is used as the classification label of the third text data to be annotated, and the first annotated text data is obtained and put into the target training set; The first classification model is fine-tuned based on the target training set to obtain an initial classification model.

6. The training method of the classification model according to claim 5, characterized in that: The method further comprises: In the case where it is determined based on the first judgment result that the classification label with the highest probability in the third classification result is not a correct item, a fine-tuning instruction is generated, wherein the fine-tuning instruction is used to instruct the first classification model to output a preset classification result, and the classification label with the highest probability in the preset classification result is the correct item in the first judgment result; The first classification model is fine-tuned based on the fine-tuning instruction.

7. A data annotation method, characterized in that: include: Determining fourth text data to be annotated; Based on the target classification model, classifying the fourth text data to be annotated to obtain a fourth classification result of the fourth text data to be annotated, wherein the target classification model is a target classification model obtained by the training method of the classification model according to any one of claims 1 to 6; Based on the fourth classification result, a classification label of the fourth text data to be annotated is determined, and the fourth text data to be annotated is annotated based on the classification label.

8. A training device for a classification model, characterized in that: The training device of the classification model comprises: A first classification module, used for classifying the first text data to be annotated based on the initial classification model to obtain a first classification result of the first text data to be annotated; An expansion module, used for expanding the attribute characteristics of the first text data to be annotated to obtain second text data to be annotated when the first classification result does not meet a preset condition; A second classification module, used for classifying the second text data to be annotated based on the initial classification model to obtain a second classification result of the second text data to be annotated; A determination module, configured to determine the second classification result as a target classification label of the first text data to be annotated if the second classification result meets the preset condition; An updating module is used to update the initial classification model based on the first text data to be annotated and its target classification label to obtain a target classification model.

9. A data labeling device, characterized in that: The data labeling device comprises: A determination module, used for determining fourth text data to be annotated; A classification module, used for classifying the fourth text data to be annotated based on a target classification model to obtain a fourth classification result of the fourth text data to be annotated, wherein the target classification model is a target classification model obtained by the training method of the classification model according to any one of claims 1 to 6; The labeling module is used to determine the classification label of the fourth text data to be labeled based on the fourth classification result, and label the fourth text data to be labeled based on the classification label.

10. An electronic device, characterized in that: It comprises a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the training method of the classification model described in any one of claims 1 to 6 or the steps in the data labeling method described in claim 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, which are suitable for being loaded by a processor to execute the steps in the training method for a classification model described in any one of claims 1 to 6 or the steps in the data labeling method described in claim 7.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the training method of the classification model described in any one of claims 1 to 6 or the steps in the data labeling method described in claim 7 are implemented.

Images