Data classification and labeling method and device, electronic equipment and storage medium
By identifying and annotating the to-process data in Taichung, combined with feature extraction and classification models, automated data classification and annotation are realized, solving the problems of low data labeling efficiency and high error rate in the existing technology, and improving the efficiency and accuracy of data processing and analysis.
Patent Information
- Application Number
- CN202411947862.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-03-21
AI Technical Summary
In the prior art, data labeling is inefficient and prone to errors, which cannot meet the needs of fast data processing and accurate analysis in the big data era.
By obtaining the data to be processed in Taichung in the data, identifying its data format category and performing preliminary annotation, and then analyzing and performing business category annotation based on feature extraction and pre-trained classification models to achieve automated data classification and annotation.
It improves the efficiency and accuracy of data annotation, reduces the dependence of manual annotation, and can quickly process and accurately analyze data in the era of big data.
Smart Images
Figure CN119669863A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data management technology, and in particular to a data classification and labeling method and device, electronic equipment, and storage medium. Background Art
[0002] With the development of big data technology, data is becoming more and more important to enterprises. However, the scale of enterprise data is huge and the types are diverse. Therefore, in order to facilitate the management and use of these data, the stored enterprise data is usually labeled.
[0003] The current way of labeling enterprise data is mainly for staff to review and analyze the stored enterprise data, and then label the business data through labeling operations based on the results of their analysis, or to have relevant personnel label it before storage, and then upload it to management evaluation and store it together with the data.
[0004] However, due to the huge amount of data that needs to be labeled and the large types of data, that is, there are many types of annotations, manual labeling is not only inefficient but also prone to errors, and cannot meet the needs of fast data processing and accurate analysis in the big data era. Summary of the invention
[0005] Based on the above-mentioned deficiencies of the prior art, the present application provides a data classification and labeling method and device, electronic device, and storage medium to solve the problem that the prior art is low in efficiency and prone to errors.
[0006] In order to achieve the above objectives, this application provides the following technical solutions:
[0007] The first aspect of the present application provides a data classification and labeling method, including:
[0008] Get the data to be processed in the data center;
[0009] Identifying the data format category to which the data to be processed belongs, and marking the data to be processed according to the data format category to which the data to be processed belongs;
[0010] Extracting features of the data to be processed according to the data format category to which the data to be processed belongs, to obtain feature data of the data to be processed;
[0011] Based on the characteristic data of the data to be processed, the business category to which the data to be processed belongs is analyzed by a pre-trained classification model; wherein the classification model is pre-trained using a large amount of sample data labeled with the business category to which it belongs;
[0012] The data to be processed are marked according to the business category to which the data to be processed belongs.
[0013] Optionally, in the above data classification and labeling method, the data format category to which the data to be processed belongs is text data, and the step of extracting features of the data to be processed according to the data format category to which the data to be processed belongs to obtain feature data of the data to be processed includes:
[0014] Extracting features of the data to be processed by a deep learning algorithm to obtain basic feature data of the data to be processed;
[0015] Using natural language processing technology to perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed to obtain lexical, syntactic and semantic information of the data to be processed;
[0016] The basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed are combined into feature data of the data to be processed.
[0017] Optionally, in the above data classification and labeling method, the classification model training method includes:
[0018] Acquire multiple sample data belonging to each data format category respectively;
[0019] According to the data format category of each sample data, feature extraction is performed on each sample data to obtain feature data of each sample data;
[0020] Labeling each of the sample data with the business category to which it belongs, to obtain labels for the sample data;
[0021] The classification model is iteratively trained using the feature data of the sample data and the annotations of the sample data through a stochastic gradient descent optimization algorithm until the error of the classification model converges, thereby obtaining the trained classification model.
[0022] Optionally, in the above data classification and labeling method, it also includes:
[0023] In response to a sampling operation triggered by a user on the data platform, a plurality of labeled data are selected as sampling data;
[0024] Feedback each of the sampling inspection data and its annotation to the user;
[0025] In response to the user's modification operation on any one of the sampled data, the annotation of the sampled data is modified.
[0026] Optionally, in the above data classification and labeling method, it also includes:
[0027] Obtain the data and annotations that have been modified within the current preset time period as the current optimization data;
[0028] The classification model is optimized using the optimization data.
[0029] Optionally, in the above data classification and labeling method, it also includes:
[0030] In response to a data query operation triggered by a user on the data platform, a current query annotation input by the user is obtained; wherein the current query annotation includes an annotation of a data format category and / or an annotation of a business category;
[0031] The data marked as the current query mark is searched from the data stored in the data center, and fed back to the user.
[0032] The second aspect of the present application provides a data classification and labeling device, including:
[0033] A data acquisition unit, used to acquire the data to be processed in the data center;
[0034] a coarse-grained identification unit, used to identify the data format category to which the data to be processed belongs, and to mark the data to be processed according to the data format category to which the data to be processed belongs;
[0035] A first feature extraction unit, configured to extract features of the data to be processed according to the data format category to which the data to be processed belongs, so as to obtain feature data of the data to be processed;
[0036] A fine-grained identification unit, used to analyze the business category of the data to be processed based on the feature data of the data to be processed by using a pre-trained classification model; wherein the classification model is pre-trained using a large amount of sample data labeled with the business category to which it belongs;
[0037] The marking unit is used to mark the data to be processed according to the business category to which the data to be processed belongs.
[0038] Optionally, in the above-mentioned data classification and labeling device, the data format category to which the to-be-processed data belongs is text data, and the first feature extraction unit includes:
[0039] A basic feature extraction unit, used to extract features from the data to be processed by a deep learning algorithm to obtain basic feature data of the data to be processed;
[0040] A text analysis unit, used to perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed by using natural language processing technology to obtain lexical, syntactic and semantic information of the data to be processed;
[0041] The combining unit is used to combine the basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed into feature data of the data to be processed.
[0042] Optionally, in the above-mentioned data classification and labeling device, it also includes:
[0043] A sample acquisition unit, used to acquire a plurality of sample data belonging to each data format category;
[0044] A sample feature extraction unit, used for extracting features from each sample data according to the data format category of each sample data, to obtain feature data of each sample data;
[0045] A sample labeling unit, used to label each sample data with its corresponding business category, so as to obtain a label for the sample data;
[0046] The training unit is used to utilize the feature data of the sample data and the annotations of the sample data to iteratively train the classification model through a stochastic gradient descent optimization algorithm until the error of the classification model converges to obtain the trained classification model.
[0047] Optionally, in the above-mentioned data classification and labeling device, it also includes:
[0048] A sampling unit, configured to respond to a sampling operation triggered by a user on the data platform and select a plurality of marked data as sampling data;
[0049] A sampling data feedback unit, which feeds back each sampling data and its annotation to the user;
[0050] The modification unit is used to modify the annotation of the sampling data in response to the user's modification operation on any one of the sampling data.
[0051] Optionally, in the above-mentioned data classification and labeling device, it also includes:
[0052] The optimization data acquisition unit is used to acquire the data and annotations that have been modified within a preset time period as current optimization data;
[0053] A model optimization unit is used to optimize the classification model using the optimization data.
[0054] Optionally, in the above-mentioned data classification and labeling device, it also includes:
[0055] An input acquisition unit, used to respond to a data query operation triggered by a user on the data platform, and acquire a current query annotation input by the user; wherein the current query annotation includes an annotation of a data format category and / or an annotation of a business category;
[0056] The query unit is used to query the data marked as the current query mark from the data stored in the data center and feed it back to the user.
[0057] A third aspect of the present application provides an electronic device, including:
[0058] Memory and processor;
[0059] Wherein, the memory is used to store programs;
[0060] The processor is used to execute the program, and when the program is executed, it is specifically used to implement the data classification and labeling method as described in any one of the above items.
[0061] A fourth aspect of the present application provides a computer storage medium for storing a computer program, which, when executed by a processor, is used to implement the data classification and labeling method as described in any one of the above.
[0062] The present application provides a data classification and labeling method, which obtains the data to be processed in the data center, then identifies the data format category to which the data to be processed belongs, and labels the data to be processed according to the data format category to which the data to be processed belongs, so that the data can be managed from a coarse granularity based on the data format category, and it is also convenient for subsequent targeted fine-grained category analysis of the data to be processed, thereby improving the accuracy of the fine-grained category analysis. Then, according to the data format category to which the data to be processed belongs, feature extraction is performed on the data to be processed to obtain feature data of the data to be processed, thereby extracting the corresponding feature data in a targeted manner. Finally, based on the feature data of the data to be processed, the business category to which the data to be processed belongs is analyzed through a pre-trained classification model, and the data to be processed is labeled according to the business category to which the data to be processed belongs. Among them, the classification model is pre-trained using a large amount of sample data of the business category to which the annotation belongs, so that the data is automatically classified and labeled in a fine-grained manner through the classification model, without the need for manual labeling, so that an efficient and accurate way of labeling data is achieved. BRIEF DESCRIPTION OF THE DRAWINGS
[0063] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.
[0064] Figure 1 A flowchart of a data classification and labeling method provided in an embodiment of the present application;
[0065] Figure 2 A flowchart of a method for extracting features from data to be processed provided in an embodiment of the present application;
[0066] Figure 3 A flowchart of a classification model training method provided in an embodiment of the present application;
[0067] Figure 4 A flowchart of a data query method provided in an embodiment of the present application;
[0068] Figure 5 A schematic diagram of the architecture of a data classification and labeling device provided in an embodiment of the present application;
[0069] Figure 6 A schematic diagram of the architecture of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0071] In this application, relational terms such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.
[0072] The present application embodiment provides a data classification and labeling method, such as Figure 1 As shown, the following steps are included:
[0073] S101, obtaining the data to be processed in the data center.
[0074] It should be noted that in the embodiment of the present application, the data is uniformly managed by the data center, so the data will be uniformly uploaded to the data center for storage and management. Therefore, the data that needs to be labeled in the data center is obtained as the data to be processed. Specifically, when the data is uploaded to the data center, it is processed as the data to be processed for labeling. It can also be extracted from the data in the data center that has been stored.
[0075] S102: Identify the data format category to which the data to be processed belongs, and mark the data to be processed according to the data format category to which the data to be processed belongs.
[0076] It should be noted that in order to be able to label the data to be processed at a coarser granularity and to facilitate subsequent more detailed labeling, in the embodiment of the present application, the data to be processed will first be classified based on the data format.
[0077] Among them, the data format category may include text, image, audio, character, etc., which can be set as needed. Therefore, optionally, the data format category to which the data to be processed belongs can be identified by identifying the format, structure or data model of the data to be processed. Specifically, the format or data model of the data to be processed can be identified by matching the template corresponding to each data format category with the data to be processed, thereby determining the data format category to which the data to be processed belongs. Then, the data to be processed is annotated according to the data format category to which the data to be processed belongs, and the coarse-grained annotation of the data to be processed is set to the data format category to which the data to be processed belongs.
[0078] S103 . Extract features of the data to be processed according to the data format category to which the data to be processed belongs, to obtain feature data of the data to be processed.
[0079] In order to carry out fine-grained labeling of the data to be processed from the business perspective, it is necessary to extract the features of the data to be processed so as to analyze the business category to which it belongs based on its features.
[0080] Specifically, since the formats and structures of the data to be processed belonging to different data format categories are different, the features to be extracted and the methods of extracting the features are different. Therefore, it is necessary to call the corresponding method to extract features from the data to be processed according to the data format category to which the data to be processed belongs, so as to extract various feature data of the corresponding type.
[0081] Optionally, in another embodiment of the present application, when the data format category of the data to be processed is text data, a specific implementation of step S103 is as follows: Figure 2 As shown, the following steps are included:
[0082] S201. Extract features of the data to be processed by a deep learning algorithm to obtain basic feature data of the data to be processed.
[0083] Specifically, for each data format category, the data to be processed can be subjected to feature extraction through deep learning algorithms, so that the deep-level basic feature data of the data to be processed can be extracted. Therefore, for text data, feature extraction is also performed through deep learning algorithms to obtain the basic feature data of the data to be processed.
[0084] S202: Perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed using natural language processing technology to obtain lexical, syntactic and semantic information of the data to be processed.
[0085] Among them, lexical structure refers to the grammar of each word in the data to be processed, while syntax refers to the grammar of each sentence in the data to be processed.
[0086] Since the syntax and semantics of text data reflect the information it expresses, and the expressed content can reflect which businesses it is related to, that is, it can be classified into the business to which it belongs, in an embodiment of the present application, in order to accurately annotate the text data, natural language processing technology is also used to perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed to obtain the lexical, syntactic and semantic information of the data to be processed.
[0087] S203 , combining the basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed into feature data of the data to be processed.
[0088] In order to enable the classification model to analyze the business category to which the data to be processed belongs based on the basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed, the two are combined to form the feature data of the data to be processed and provided to the classification model for use.
[0089] S104: Based on the feature data of the data to be processed, the business category to which the data to be processed belongs is analyzed by a pre-trained classification model.
[0090] The classification model is pre-trained using a large amount of sample data labeled with the business category to which it belongs.
[0091] In order to objectively and accurately classify the business category to which the data belongs, in an embodiment of the present application, a multi-layer classification model is constructed and trained. Through this classification model, the business category to which the data to be processed belongs is shared based on the feature data of the data to be processed, without relying on manual analysis.
[0092] Optionally, another embodiment of the present application provides a training method for a classification model, such as Figure 3 As shown, the following steps are included:
[0093] S301: Acquire a plurality of sample data belonging to different data format categories.
[0094] In order to enable the classification model to accurately analyze the business category of data in each data format category, it is necessary to obtain multiple sample data in each data format category when obtaining sample data.
[0095] Optionally, the data format categories of a large amount of data may be analyzed, and then the same amount of data of each data format category may be selected as sample data.
[0096] S302 . Extract features of each sample data according to the data format category of each sample data to obtain feature data of each sample data.
[0097] It should be noted that the method of extracting features from sample data is the same as the method of extracting features from the data to be processed, so it will not be repeated here.
[0098] S303: Label each sample data with its corresponding business category to obtain labels for the sample data.
[0099] In order to analyze the errors in the classification results of the classification model, it is necessary to first mark the actual business category of each sample data.
[0100] S304: Using the feature data of the sample data and the annotations of the sample data, the classification model is iteratively trained through a stochastic gradient descent optimization algorithm until the error of the classification model converges, thereby obtaining a trained classification model.
[0101] In order to quickly and completely train, in the embodiment of the present application, the classification model is iteratively trained by a stochastic gradient descent optimization algorithm. Of course, other optimization algorithms can also be used for training.
[0102] S105: Mark the data to be processed according to the business category to which the data to be processed belongs.
[0103] Specifically, a label corresponding to the business category to which the data to be processed belongs is generated for the data to be processed, thereby completing the labeling of the data to be processed.
[0104] Optionally, considering that the analysis results of the classification model may be erroneous, especially when the type of data changes, in order to ensure the accuracy of the annotation of each data, in another embodiment of the present application, the annotation of the data may be further checked, specifically including:
[0105] In response to the sampling operation triggered by the user on the data platform, multiple labeled data are selected as sampling data, and each sampling data and its annotation are fed back to the user. In response to the user's modification operation on any sampling data, the annotation of the sampling data is modified.
[0106] It should be noted that in the embodiment of the present application, the verification is performed by rechecking. Optionally, when the user needs to perform a random check, the random check operation can be triggered, and the data center randomly selects multiple labeled data from the data labeled in the recent period as random check data, and feeds back each random check data and its annotation to the user for verification. Of course, the user can also select them one by one for verification.
[0107] When the user finds that there is an error in the labeling of a certain data, he or she can modify the labeling of the data through modification operation.
[0108] Accordingly, after sampling the data, in order to continuously optimize the classification model and adapt it to the changing data so as to consistently maintain accurate classification, in the embodiment of the present application, the classification model may be further optimized, specifically including:
[0109] The data and its annotations that have been modified within the current preset time period are obtained as the current optimization data, and the classification model is optimized using the optimization data.
[0110] Optionally, after the data is annotated, the data can be queried according to the data annotation. Therefore, in another embodiment of the present application, the data can be further queried. Figure 4 As shown, a data query method provided in an embodiment of the present application includes the following steps:
[0111] S401, responding to a data query operation triggered by a user on the data platform, and obtaining a current query annotation input by the user.
[0112] The current query annotation includes an annotation of a data format category and / or an annotation of a business category.
[0113] It should be noted that in the embodiment of the present application, the user can enter only the data format category annotation or the business category annotation according to his or her query needs, or can enter both the data format category annotation and the business category annotation for query.
[0114] S402: Query the data marked as the current query mark from the data stored in the data center, and feed it back to the user.
[0115] It should be noted that if the user only inputs one annotation, the data with the annotation will be searched and fed back to the user. If the user inputs two annotations, the data with both annotations will be searched and fed back first, followed by the data with one of the annotations.
[0116] The embodiment of the present application provides a data classification and labeling method, which obtains the data to be processed in the data center, then identifies the data format category to which the data to be processed belongs, and labels the data to be processed according to the data format category to which the data to be processed belongs, so that the data can be managed from a coarse granularity based on the data format category, and it is also convenient for the subsequent targeted fine-grained category analysis of the data to be processed, thereby improving the accuracy of the fine-grained category analysis. Then, according to the data format category to which the data to be processed belongs, feature extraction is performed on the data to be processed to obtain feature data of the data to be processed, thereby extracting the corresponding feature data in a targeted manner. Finally, based on the feature data of the data to be processed, the business category to which the data to be processed belongs is analyzed through a pre-trained classification model, and the data to be processed is labeled according to the business category to which the data to be processed belongs. Among them, the classification model is pre-trained using a large amount of sample data of the business category to which the label belongs, so that the data is automatically classified and labeled in a fine-grained manner through the classification model, without the need for manual labeling, so that an efficient and accurate way of labeling data is achieved.
[0117] Another embodiment of the present application provides a data classification and labeling device, such as Figure 5 As shown, including:
[0118] The data acquisition unit 501 is used to acquire the data to be processed in the data center.
[0119] The coarse-grained identification unit 502 is used to identify the data format category to which the data to be processed belongs, and to label the data to be processed according to the data format category to which the data to be processed belongs.
[0120] The first feature extraction unit 503 is used to extract features from the data to be processed according to the data format category to which the data to be processed belongs, so as to obtain feature data of the data to be processed.
[0121] The fine-grained identification unit 504 is used to analyze the business category of the data to be processed based on the feature data of the data to be processed by using a pre-trained classification model, wherein the classification model is pre-trained using a large amount of sample data labeled with the business category to which it belongs.
[0122] The marking unit 505 is used to mark the data to be processed according to the business category to which the data to be processed belongs.
[0123] Optionally, in a data classification and labeling device provided in another embodiment of the present application, the data format category to which the data to be processed belongs is text data, and the first feature extraction unit includes:
[0124] The basic feature extraction unit is used to extract features from the data to be processed through a deep learning algorithm to obtain basic feature data of the data to be processed.
[0125] The text analysis unit is used to perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed using natural language processing technology to obtain the lexical, syntactic and semantic information of the data to be processed.
[0126] The combining unit is used to combine the basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed into the feature data of the data to be processed.
[0127] Optionally, in another embodiment of the present application, the data classification and labeling device further includes:
[0128] The sample acquisition unit is used to acquire a plurality of sample data belonging to each data format category.
[0129] The sample feature extraction unit is used to extract features from each sample data according to the data format category of each sample data to obtain feature data of each sample data.
[0130] The sample labeling unit is used to label each sample data according to its business category to obtain the label of the sample data.
[0131] The training unit is used to utilize the characteristic data of the sample data and the annotation of the sample data to iteratively train the classification model through a stochastic gradient descent optimization algorithm until the error of the classification model converges to obtain a trained classification model.
[0132] Optionally, in another embodiment of the present application, the data classification and labeling device further includes:
[0133] The sampling unit is used to respond to the sampling operation triggered by the user on the data platform and select multiple labeled data as sampling data.
[0134] The sampling data feedback unit feeds back each sampling data and its annotation to the user.
[0135] The modification unit is used to respond to the user's modification operation on any sampled data and modify the annotation of the sampled data.
[0136] Optionally, in another embodiment of the present application, the data classification and labeling device further includes:
[0137] The optimization data acquisition unit is used to acquire the data and its annotations that have been modified within a current preset time period as current optimization data.
[0138] The model optimization unit is used to optimize the classification model using the optimization data.
[0139] Optionally, in another embodiment of the present application, the data classification and labeling device further includes:
[0140] The input acquisition unit is used to respond to the data query operation triggered by the user on the data platform and acquire the current query annotation input by the user. The current query annotation includes the annotation of the data format category and / or the annotation of the business category.
[0141] The query unit is used to query the data marked as the current query mark from the data stored in the data center and feed it back to the user.
[0142] It should be noted that the specific working process of each unit provided in the above embodiments of the present application can refer to the implementation process of the corresponding steps in the above method embodiments, and will not be repeated here.
[0143] Another embodiment of the present application provides an electronic device, such as Figure 6 As shown, including:
[0144] Memory 601 and processor 602 .
[0145] The memory 601 is used to store programs.
[0146] The processor 602 is used to execute the program stored in the memory 601. When the program is executed, it is specifically used to implement the data classification and labeling method provided in any one of the above embodiments.
[0147] Another embodiment of the present application provides a computer storage medium for storing a computer program. When the computer program is executed by a processor, it is used to implement the data classification and labeling method provided in any of the above embodiments.
[0148] Computer storage media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.
[0149] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.
[0150] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A data classification and labeling method, characterized in that: include: Get the data to be processed in the data center; Identifying the data format category to which the data to be processed belongs, and marking the data to be processed according to the data format category to which the data to be processed belongs; Extracting features of the data to be processed according to the data format category to which the data to be processed belongs, to obtain feature data of the data to be processed; Based on the characteristic data of the data to be processed, the business category to which the data to be processed belongs is analyzed by a pre-trained classification model; wherein the classification model is pre-trained using a large amount of sample data labeled with the business category to which it belongs; The data to be processed are marked according to the business category to which the data to be processed belongs.
2. The method according to claim 1, characterized in that The data format category to which the data to be processed belongs is text data, and the extracting features of the data to be processed according to the data format category to which the data to be processed belongs to obtain feature data of the data to be processed, including: Extracting features of the data to be processed by a deep learning algorithm to obtain basic feature data of the data to be processed; Using natural language processing technology to perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed to obtain lexical, syntactic and semantic information of the data to be processed; The basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed are combined into feature data of the data to be processed.
3. The method according to claim 1, characterized in that The training method of the classification model comprises: Acquire multiple sample data belonging to each data format category respectively; According to the data format category of each sample data, feature extraction is performed on each sample data to obtain feature data of each sample data; Labeling each of the sample data with the business category to which it belongs, to obtain labels for the sample data; The classification model is iteratively trained using the feature data of the sample data and the annotations of the sample data through a stochastic gradient descent optimization algorithm until the error of the classification model converges, thereby obtaining the trained classification model.
4. The method according to claim 1, characterized in that: Also includes: In response to a sampling operation triggered by a user on the data platform, a plurality of labeled data are selected as sampling data; Feedback each of the sampling inspection data and its annotation to the user; In response to the user's modification operation on any one of the sampled data, the annotation of the sampled data is modified.
5. The method according to claim 4, characterized in that Also includes: Obtain the data and annotations that have been modified within the current preset time period as the current optimization data; The classification model is optimized using the optimization data.
6. The method according to claim 1, characterized in that Also includes: In response to a data query operation triggered by a user on the data platform, a current query annotation input by the user is obtained; wherein the current query annotation includes an annotation of a data format category and / or an annotation of a business category; The data marked as the current query mark is searched from the data stored in the data center, and fed back to the user.
7. A data classification and labeling device, characterized in that: include: A data acquisition unit, used to acquire the data to be processed in the data center; a coarse-grained identification unit, used to identify the data format category to which the data to be processed belongs, and to mark the data to be processed according to the data format category to which the data to be processed belongs; A first feature extraction unit, configured to extract features of the data to be processed according to the data format category to which the data to be processed belongs, so as to obtain feature data of the data to be processed; A fine-grained identification unit, used to analyze the business category to which the data to be processed belongs through a pre-trained classification model based on the feature data of the data to be processed; wherein the classification model is pre-trained using a large amount of sample data labeled with the business category to which it belongs; The marking unit is used to mark the data to be processed according to the business category to which the data to be processed belongs.
8. The device according to claim 7, characterized in that The data format category of the data to be processed is text data, and the first feature extraction unit includes: A basic feature extraction unit, used to extract features from the data to be processed by a deep learning algorithm to obtain basic feature data of the data to be processed; A text analysis unit, used to perform lexical analysis, syntactic analysis and semantic analysis on the data to be processed by using natural language processing technology to obtain lexical, syntactic and semantic information of the data to be processed; The combining unit is used to combine the basic feature data of the data to be processed and the lexical, syntactic and semantic information of the data to be processed into feature data of the data to be processed.
9. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to execute the program, and when the program is executed, it is specifically used to implement the data classification and labeling method as described in any one of claims 1 to 6.
10. A computer storage medium, characterized in that: Used to store a computer program, which, when executed by a processor, is used to implement the data classification and labeling method as described in any one of claims 1 to 6.