Label data processing method and device, computer equipment and storage medium

Abstract

The invention relates to an annotation data processing method and device, computer equipment, a storage medium and a computer program product. The method comprises the steps that a data annotation request sent by a terminal is received, the data annotation request carries sample identification data and annotation algorithm identification data, according to preset annotation algorithm attribute configuration data, algorithm input parameter attributes corresponding to the annotation algorithm identification data are determined, a sample set corresponding to the sample identification data is obtained, and a data annotation result is obtained; and matching attribute feature data of the target attribute from the sample set to obtain a to-be-labeled data set, and pushing the to-be-labeled data set to a terminal. By the adoption of the method, the unified annotation data obtaining capacity can be provided for different annotation algorithms and sample sets, the to-be-annotated data can be rapidly and accurately obtained only by configuring the data and determining the attributes needed by the algorithms according to the preset annotation algorithm attributes for the different annotation algorithms, and then the data annotation efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for processing annotated data. Background Art

[0002] With the rapid development of artificial intelligence, artificial intelligence and machine learning have been more and more widely used. Data labeling is an important foundation for artificial intelligence and machine learning. In order to enable the algorithm to handle more and more complex scenarios, it is usually necessary to collect massive amounts of labeled data to continuously train and tune machine learning and artificial intelligence related algorithm models to improve the accuracy of the algorithm model.

[0003] At present, data are usually labeled by manual labeling, and there is also a data labeling method that combines manual labeling and intelligent automatic labeling.

[0004] However, when it comes to data labeling in the battery field, multiple battery data labeling algorithms are involved. One battery sample data can be applied to multiple battery data labeling algorithms at the same time. Each battery data labeling algorithm requires multi-dimensional battery sample data. When the battery data to be labeled is multi-dimensional data and the quantity is very large, it is difficult to quickly and accurately obtain the corresponding battery sample data to be labeled for different battery data labeling algorithms, which affects the efficiency of data labeling. Summary of the invention

[0005] Based on this, it is necessary to provide a labeled data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve data labeling efficiency in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for processing annotated data. The method comprises:

[0007] A data labeling request sent by a receiving terminal carries sample identification data and labeling algorithm identification data;

[0008] According to the preset annotation algorithm attribute configuration data, determine the algorithm input parameter attribute corresponding to the annotation algorithm identification data;

[0009] Obtain a sample set corresponding to the sample identification data, and match the attribute feature data of the target attribute from the sample set to obtain a data set to be labeled;

[0010] Push the dataset to be labeled to the terminal.

[0011] In the technical solution of the embodiment of the present application, by pre-configuring the attributes required by the labeling algorithm, that is, pre-setting the labeling algorithm attribute configuration data, it is possible to quickly determine the target attributes required by the labeling algorithm according to the preset labeling algorithm attribute configuration data when the labeling algorithm identification data is known, and then accurately match the attribute feature data of the target attribute from the sample set according to the algorithm input parameter attributes to obtain an accurate data set to be labeled, and then push the data set to be labeled to the terminal so that the labeling personnel can label the data set to be labeled. The entire solution can provide a unified labeling data acquisition capability for different labeling algorithms and sample sets. For different labeling algorithms, it is only necessary to determine the attributes required by the algorithm according to the preset labeling algorithm attribute configuration data, so as to quickly and accurately obtain the data to be labeled, thereby improving the efficiency of data labeling.

[0012] Furthermore, since the annotation algorithm attribute configuration data is configurable, the attributes required by the extended algorithm can be updated at any time to meet different data annotation requirements.

[0013] In some embodiments, the method further comprises:

[0014] When a historical annotation algorithm associated with the target annotation algorithm is found, the historical annotation algorithm is called to annotate the dataset to be annotated to obtain an initial data annotation result. The target annotation algorithm is the annotation algorithm corresponding to the annotation algorithm identification data.

[0015] Push the initial data annotation results and the data set to be annotated to the terminal;

[0016] The target data labeling result fed back by the receiving terminal includes the revised initial data labeling result.

[0017] In the technical solution of the embodiment of the present application, on the one hand, by calling the historical annotation algorithm to automatically annotate the data set to be annotated, the efficiency of data annotation can be greatly improved; on the other hand, through manual secondary annotation, the accuracy of the data annotation results can be improved, and the target data annotation results after secondary annotation can be used to optimize the annotation algorithm.

[0018] In some embodiments, after receiving the target data annotation result fed back by the terminal, the method further includes:

[0019] Call the target labeling algorithm to predict the labeling results of the labeled data set and obtain the predicted labeling results;

[0020] According to the target data labeling results and the predicted labeling results, the algorithm quality evaluation indicators of the target labeling algorithm are determined.

[0021] In the technical solution of the embodiment of the present application, by comparing the target data labeling result and the predicted labeling result output by the target labeling algorithm, the algorithm quality evaluation index of the target labeling algorithm is determined, so that the algorithm quality of the target labeling algorithm can be evaluated and the target labeling algorithm can be adjusted and optimized.

[0022] In some embodiments, before pushing the to-be-annotated data set to the terminal, the method further includes:

[0023] According to the algorithm input parameter attributes, the dataset to be labeled is grouped and encapsulated;

[0024] Pushing the dataset to be labeled to the terminal includes:

[0025] The grouped and encapsulated datasets to be labeled are pushed to the terminal so that the terminal can display the datasets to be labeled in groups according to the target attributes.

[0026] In the technical solution of the embodiment of the present application, by grouping and encapsulating the data to be labeled, the data to be labeled can be grouped according to certain rules, which is convenient for labeling personnel to perform batch processing, thereby improving labeling efficiency.

[0027] In some embodiments, the sample set is obtained based on the following method:

[0028] According to the preset annotation algorithm attribute configuration data, the algorithm input parameter attributes of the existing annotation algorithm are determined;

[0029] Obtain the original attribute values ​​of the algorithm input attributes at different times to obtain the original attribute value data set;

[0030] Processing the original attribute value data set to obtain the target attribute value data set;

[0031] Construct a sample set based on the target attribute value dataset.

[0032] In the technical solution of the embodiment of the present application, by processing the original attribute value data set, the processed target attribute value data set can be made more standardized and regulated, reducing the abnormal values ​​and missing values ​​in the data, and improving the accuracy and reliability of the data. By constructing a sample set through the processed target attribute value data set, a representative data sample can be extracted.

[0033] In some embodiments, processing the original attribute value dataset to obtain the target attribute value dataset includes at least one of the following methods:

[0034] The first item is, when the original attribute value in the original attribute value data set is an array, the original attribute value is converted into row structure data to obtain the target attribute value data set;

[0035] The second item is to calculate the characteristic values ​​of the original attribute values ​​in the original attribute value data set to obtain the characteristic values ​​of the original attribute values, establish the association relationship between the original attribute values ​​and the characteristic values, and aggregate the original attribute values ​​and the characteristic values ​​to obtain the target attribute value data set.

[0036] In the technical solution of the embodiment of the present application, by converting array data into row structured data, the readability of the data, the flexibility of processing operations, and the convenience of storage and transmission can be improved. By calculating the eigenvalues ​​of the original attribute value data, the characteristics of the data can be highlighted, reflecting the main information and differences of the data, which is conducive to labeling personnel to discover abnormal data in time, and is also conducive to data analysis and modeling, improving the performance and efficiency of the algorithm.

[0037] In some embodiments, the data annotation request carries data screening conditions;

[0038] Before pushing the dataset to be labeled to the terminal, the method further includes:

[0039] Filtering target attribute feature data that meets the data screening conditions from the data set to be labeled to obtain the target data set to be labeled;

[0040] Pushing the dataset to be labeled to the terminal includes:

[0041] Push the target dataset to be labeled to the terminal.

[0042] In the technical solution of the embodiment of the present application, the data to be labeled is filtered through data screening conditions, which can filter out useless information, filter out the data to be labeled that meets the usage conditions, and improve the efficiency of data labeling.

[0043] In some embodiments, after pushing the dataset to be annotated to the terminal, the method further includes:

[0044] Receive and cache the data labeling results with labeling tags fed back by the terminal.

[0045] In the technical solution of the embodiment of the present application, by caching the data annotation results carrying the annotation tags fed back by the terminal, the data annotation results can be saved in time, reducing the possibility of repeated annotation of data.

[0046] In some embodiments, after receiving and caching the data annotation results carrying the annotation tags fed back by the terminal, the method further includes:

[0047] A labeling result reloading request sent by a receiving terminal, wherein the labeling result reloading request carries labeling task identification data;

[0048] Obtain cached data annotation results corresponding to the annotation task identification data;

[0049] Push the cached data annotation results to the terminal so that the terminal reloads the cached data annotation results.

[0050] In the technical solution of the embodiment of the present application, by responding to the terminal's annotation result reloading request, the cached data annotation results are pushed to the terminal, so that when the annotator enters the annotation work page again, the cached data annotation results can be automatically loaded. The annotator can continue the previous annotation work without starting over, thereby improving data annotation efficiency.

[0051] In a second aspect, the present application provides a labeling data processing device, characterized in that the device comprises:

[0052] A data receiving module is used to receive a data labeling request sent by a terminal, where the data labeling request carries sample identification data and labeling algorithm identification data;

[0053] An attribute determination module, used to determine the algorithm input parameter attribute corresponding to the annotation algorithm identification data according to the preset annotation algorithm attribute configuration data;

[0054] The data matching module is used to obtain a sample set corresponding to the sample identification data, and match the attribute feature data of the target attribute from the sample set to obtain a data set to be labeled;

[0055] The data push module is used to push the data set to be labeled to the terminal.

[0056] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps in the above-mentioned embodiments of the method for processing annotated data are implemented.

[0057] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned embodiments of the method for processing labeled data are implemented.

[0058] In a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned embodiments of the method for processing labeled data are implemented.

[0059] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Moreover, the same reference numerals are used throughout the drawings to represent the same components. In the drawings:

[0061] Figure 1 An application environment diagram for marking data processing methods in some embodiments of the present application;

[0062] Figure 2 A flowchart of a method for marking data processing in some embodiments of the present application;

[0063] Figure 3 A flowchart of a method for marking data processing in other embodiments of the present application;

[0064] Figure 4 A flowchart of a method for marking data processing in some embodiments of the present application;

[0065] Figure 5 A detailed flowchart of a method for processing data in some embodiments of the present application is provided;

[0066] Figure 6 A schematic diagram of the result of converting array data into row structure data in some embodiments of the present application;

[0067] Figure 7 A flowchart of a method for marking data processing in some embodiments of the present application;

[0068] Figure 8 A detailed flowchart of a data processing method in some other embodiments of the present application is provided;

[0069] Fig. 9 A detailed flowchart of a data processing method in some embodiments of the present application is provided;

[0070] Fig.10 A structural block diagram of a data processing device in some embodiments of the present application;

[0071] Fig.11 A structural block diagram of a data processing device in some other embodiments of the present application;

[0072] Fig.12 This is a diagram of the internal structure of a computer device in some embodiments of the present application. DETAILED DESCRIPTION

[0073] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0074] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0075] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0076] In the description of the embodiments of the present application, the term "multiple" refers to more than two (including two). Similarly, "multiple groups" refers to more than two groups (including two groups), and "multiple pieces" refers to more than two pieces (including two pieces).

[0077] With the rapid development of artificial intelligence, artificial intelligence and machine learning have been more and more widely used. Data labeling is an important foundation for artificial intelligence and machine learning. In order to enable algorithm models to handle more and more complex scenarios, it is usually necessary to collect massive amounts of labeled data to continuously train and tune machine learning and artificial intelligence related algorithm models to improve the accuracy of algorithm models.

[0078] At present, in the process of data labeling, manual labeling is usually used to label data, and there are also proposals for data labeling methods that combine manual labeling and intelligent automatic labeling. However, when it comes to data labeling in the battery field, since battery data is usually collected in real time, the battery data obtained is massive and involves multiple battery data labeling algorithms. One battery sample data can be applied to multiple battery data labeling algorithms at the same time, and each battery data labeling algorithm requires multi-dimensional battery sample data. When the battery data to be labeled is multi-dimensional data and the number is very large, it is difficult to quickly and accurately obtain the corresponding battery sample data to be labeled for the battery data labeling algorithm, which in turn affects the efficiency of data labeling.

[0079] Specifically, in the actual battery data labeling process, due to the complex relationship between the battery data labeling algorithm and the attribute feature data, different battery data labeling algorithms usually require attribute feature data of different dimensions. For example, for the battery temperature over-high data labeling algorithm, the required attribute feature data is battery temperature. The same attribute feature data can be applied to a variety of different battery data labeling algorithms at the same time, which increases the difficulty of obtaining the data to be labeled. In order to improve the labeling efficiency, it can be proposed from the perspective of providing a unified data acquisition solution to improve the acquisition efficiency of the data to be labeled, thereby improving the data labeling efficiency.

[0080] Based on the above considerations, in order to alleviate the problem of low labeling efficiency in the data labeling scheme, a labeling data processing method is proposed. Specifically, it can be for the labeling algorithm, pre-configure the attributes required by the algorithm, that is, preset the labeling algorithm attribute configuration data, after receiving the data labeling request sent by the terminal carrying sample identification data and labeling algorithm identification data, it can be known that the labeling algorithm identification data, according to the preset labeling algorithm attribute configuration data, quickly determine the target attribute required by the labeling algorithm, and then obtain the sample set according to the sample identification data, according to the algorithm input parameter attribute, the attribute feature data of the target attribute can be accurately matched from the sample set, and an accurate data set to be labeled is obtained, and then the data set to be labeled is pushed to the terminal so that the labeling personnel can label the data set to be labeled. The whole scheme can provide a unified labeling data acquisition capability for different labeling algorithms and sample sets. For different labeling algorithms, it only needs to configure the preset labeling algorithm attribute data to determine the attributes required by the algorithm, so that the data to be labeled can be quickly and accurately obtained, thereby improving the efficiency of data labeling.

[0081] The labeling data processing method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Specifically, when the labeler needs to label the data, he can create a data labeling task on the visual interface of the terminal 102, select the sample set to be labeled, and the data labeling algorithm required for this data labeling task, and then send a data labeling request carrying the sample identification data and the labeling algorithm identification data to the server 104. The server 104 receives the data labeling request, and then determines the algorithm input parameter attribute corresponding to the labeling algorithm identification data according to the preset labeling algorithm attribute configuration data, then obtains the sample set corresponding to the sample identification data, and matches the attribute feature data of the target attribute from the sample set to obtain the data set to be labeled, and finally, pushes the data set to be labeled to the terminal 102, so that the labeler can obtain the data to be labeled in a shorter time, and then label the data set to be labeled.

[0082] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0083] In some embodiments, Figure 2 As shown, a method for processing labeled data is provided, which is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0084] S200, receiving a data labeling request sent by a terminal, where the data labeling request carries sample identification data and labeling algorithm identification data.

[0085] Sample identification data refers to data that can identify a sample set, including but not limited to sample numbers such as sample ID (Identity), sample names, and sample specific identifiers. Labeling algorithm identification data refers to data that can identify a data labeling algorithm (hereinafter referred to as labeling algorithm), including but not limited to labeling algorithm numbers such as labeling algorithm ID (hereinafter referred to as algorithm ID), labeling algorithm names, and labeling algorithm version numbers. A labeling algorithm is an algorithm that labels target objects, including but not limited to machine learning algorithms and data labeling models for labeling targets.

[0086] In the specific implementation, the battery data is used as an example for explanation. The labeler can create a battery data labeling task on the visual interface of the terminal, and select the battery sample set to be labeled and the battery labeling algorithm ID to be used according to the data labeling requirements, and then submit the data labeling task. At this time, the terminal generates and sends a data labeling request to the server, and the server receives the data labeling request.

[0087] S400, determining the algorithm input parameter attributes corresponding to the labeling algorithm identification data according to the preset labeling algorithm attribute configuration data.

[0088] The annotation algorithm attribute configuration data refers to the data related to the attributes that need to be annotated or used by the configuration annotation algorithm. Specifically, the annotation algorithm attribute configuration data may include annotation algorithm identification data such as the annotation algorithm ID, algorithm name, algorithm version information, algorithm input parameter attributes, attribute data types, and parameters that characterize whether the algorithm input parameter attributes are necessary, etc. It also includes the mapping relationship between the names of the algorithm input parameter attributes and the names of the actual data fields stored in the database. Through this mapping relationship, the actual data fields in the database and the attribute feature data of the data fields can be found according to the corresponding attributes required by the algorithm.

[0089] In practical applications, the algorithm input parameter attributes can be configured for each labeling algorithm to obtain the labeling algorithm attribute configuration data corresponding to each labeling algorithm. For example, for the battery temperature over-high labeling algorithm, the corresponding input parameter attributes can be configured to include the cell temperature, battery pack ID, and device ID. In this embodiment, each algorithm input parameter attribute and related configuration parameters can be configured through the OSL (Operator Specific Language) algorithm.

[0090] After receiving the data annotation request, the server can extract the annotation algorithm identification data and sample identification data. Then, according to the algorithm identification data, the algorithm input parameter attributes corresponding to the algorithm identification data are matched in the preset annotation algorithm attribute configuration data. Taking the annotation of battery data as an example, the algorithm input parameter attributes corresponding to the battery annotation algorithm include but are not limited to voltage, temperature, insulation resistance, current, etc.

[0091] S600, obtaining a sample set corresponding to the sample identification data, and matching the attribute feature data of the target attribute from the sample set to obtain a data set to be labeled.

[0092] Attribute feature data includes attribute values ​​and feature data, wherein the feature data includes but is not limited to mean value, variance, standard deviation, etc.

[0093] Continuing with the above content, after obtaining the sample identification data, the sample set corresponding to the sample ID can be obtained based on the sample identification data such as the sample ID. Then, according to the algorithm input parameter attributes, the attribute feature data corresponding to the algorithm input parameter attributes are matched in the sample set to obtain the data set to be labeled. For example, if it is necessary to mark the abnormal data of excessive temperature in the battery sample set of the previous day based on the battery temperature over-high labeling algorithm, the algorithm input parameter attributes can be determined to include temperature attributes based on the algorithm ID of the battery temperature over-high labeling algorithm, and then the battery sample set of the previous day can be found based on the sample ID, and all temperature values ​​and temperature-related feature values, such as average temperature value, temperature standard deviation and other data, can be found from the battery sample set of the previous day. Then, the attribute feature data related to temperature found are collected to obtain the data set to be labeled.

[0094] S800: Push the dataset to be labeled to the terminal.

[0095] Continuing from the previous embodiment, after the data set to be labeled is obtained through query, the data set to be labeled can be pushed to the terminal, and the terminal can call a visualization tool to visualize the data to be labeled so that the labeler can label the data through the visual operation interface of the terminal.

[0096] In the technical solution of the embodiment of the present application, by pre-configuring the attributes required by the labeling algorithm, that is, pre-setting the labeling algorithm attribute configuration data, it is possible to quickly determine the target attributes required by the labeling algorithm according to the preset labeling algorithm attribute configuration data when the labeling algorithm identification data is known, and then accurately match the attribute feature data of the target attribute from the sample set according to the algorithm input parameter attributes to obtain an accurate data set to be labeled, and then push the data set to be labeled to the terminal so that the labeling personnel can label the data set to be labeled. The entire solution can provide a unified labeling data acquisition capability for different labeling algorithms and sample sets. For different labeling algorithms, it is only necessary to determine the attributes required by the algorithm according to the preset labeling algorithm attribute configuration data, so as to quickly and accurately obtain the data to be labeled, thereby improving the efficiency of data labeling.

[0097] like Figure 3 As shown, in some embodiments, before S800, the method further includes: S720, grouping and encapsulating the dataset to be labeled according to the algorithm input parameter attributes.

[0098] S800 includes: S820, pushing the grouped and encapsulated data set to be labeled to the terminal, so that the terminal displays the data set to be labeled in groups according to target attributes.

[0099] In this embodiment, before pushing the data set to be labeled to the terminal, the data set to be labeled can also be grouped and packaged. Specifically, the data set to be labeled can be grouped and packaged according to the algorithm input parameter attributes. Taking the data to be labeled as battery data as an example, the data to be labeled can be grouped and packaged according to the dimensions of device, battery pack, battery cell, etc. The data packaging structure can be MAP, where the data part is MAP, Key is the device ID or battery pack ID or battery cell ID, and VALUE is a list structure, including sequence time points and corresponding attributes or feature values.

[0100] Then, the grouped and encapsulated data set to be labeled is pushed to the terminal. At this time, after receiving the grouped and encapsulated data set to be labeled, the terminal can display the data set to be labeled according to the grouped attributes. For example, the terminal can display the relevant attribute values ​​of a device, a battery pack, or a battery cell according to the group selection needs.

[0101] Specifically, the terminal may call a visualization tool or a visualization chart tool to display the data to be annotated in the form of a chart. The annotator can select one or more attributes and corresponding attribute feature data to be displayed in the chart on the annotation work page. For example, the annotator wants to display the voltage in the battery data to be annotated in the form of a chart. On the annotation work page, select the "voltage" dimension field to be displayed, and the terminal can call the visualization chart tool to display the voltage data of different devices or different battery packs at different times in the form of a chart. The annotator can view the voltage data of a certain device or battery pack at different times.

[0102] In the technical solution of the embodiment of the present application, by grouping and encapsulating the data to be labeled, the data to be labeled can be grouped according to certain rules, which is convenient for labeling personnel to perform batch processing, thereby improving labeling efficiency.

[0103] like Figure 4 As shown, in some optional embodiments of the present application, the method further includes:

[0104] S740, when a historical annotation algorithm associated with the target annotation algorithm is found, the historical annotation algorithm is called to annotate the data set to be annotated to obtain an initial data annotation result, and the target annotation algorithm is the annotation algorithm corresponding to the annotation algorithm identification data.

[0105] S840: Push the initial data labeling result and the data set to be labeled to the terminal.

[0106] S940, receiving the target data labeling result fed back by the terminal, where the target data labeling result includes the revised initial data labeling result.

[0107] In this embodiment, the target annotation algorithm refers to the data annotation algorithm that is required for this data annotation task and corresponds to the annotation algorithm identification data. The target annotation algorithm may be a newly developed algorithm that has not been verified and optimized. The historical annotation algorithm refers to an annotation algorithm that has the same or similar data annotation function as the target annotation algorithm, and may specifically be an annotation algorithm of the historical version of the target annotation algorithm.

[0108] In a specific implementation, after extracting the labeling algorithm identification data, the server may search for the corresponding labeling algorithm and the algorithm label data corresponding to the labeling algorithm according to the labeling algorithm identification data, wherein the algorithm label data may be determined based on the algorithm function. Then, it is searched whether there is a historical labeling algorithm corresponding to the algorithm identification data under the algorithm label data.

[0109] In some other optional embodiments of the present application, the algorithm identification data may be directly associated with the historical annotation algorithm, and then, based on the algorithm identification data, it may be directly searched whether there is a historical annotation algorithm associated with the annotation algorithm identification data.

[0110] If a historical annotation algorithm is found, the historical annotation algorithm is called to perform initial annotation on the dataset to be annotated to obtain the initial data annotation result. Then, the initial data annotation result and the dataset to be annotated are pushed to the terminal. At this time, the terminal displays the initial data annotation result and the dataset to be annotated. The annotator can check and annotate the initial data annotation result based on the data to be annotated. If it is found that there are missing unannotated data or wrong annotated data in the initial data annotation result, the annotation label is added accordingly, or the wrong annotated data is modified or deleted to obtain the final corrected target data annotation result. In the process of manual data annotation, the annotation can be based on the time range or the unique key. Taking battery data annotation as an example, the unique key can include but is not limited to sample ID, cell ID, battery pack ID, device ID and time. In addition, the annotator can judge whether the data within a certain time range is abnormal data based on the displayed dataset to be annotated and combined with his own professional knowledge, and then mark the abnormal data and annotate the reasons for the data anomaly. Then, submit the corrected target data annotation result and send the target data annotation result to the server through the terminal. After receiving the submitted target data annotation results, the server organizes and stores the target data annotation results according to the preset data structure of the submitted data. Among them, the data structure of the target annotation result data can be order_id represents the annotation task instance ID, annotation_type represents the annotation method, including two annotation methods based on time range or based on unique key, device_key is any prefix of the string after the unique key is concatenated (such as any prefix of the string after the concatenation of "device ID-battery pack ID-battery cell ID"), annotation_note is the annotation remark information, and annotation_info is the time range of the annotated data or the key value of the unique key. The annotation results can be stored directly in Mysql or Clickhouse.

[0111] In the technical solution of the embodiment of the present application, on the one hand, by calling the historical annotation algorithm to automatically annotate the data set to be annotated, the efficiency of data annotation can be greatly improved; on the other hand, through manual secondary annotation, the accuracy of the data annotation results can be improved, and the target data annotation results after secondary annotation can be used to optimize the annotation algorithm.

[0112] like Figure 5 As shown, in some embodiments, after S940, the method further includes:

[0113] S960, calling the target labeling algorithm, predicting the labeling results of the labeled data set, obtaining the predicted labeling results, and determining the algorithm quality evaluation index of the target labeling algorithm according to the target data labeling results and the predicted labeling results.

[0114] Algorithm quality evaluation indicators refer to indicators that can be used to evaluate the quality of target labeling algorithms, including but not limited to recall rate, precision rate, accuracy rate, F-mesure and other algorithm quality evaluation indicators.

[0115] Continuing from the previous embodiment, since the dataset to be labeled has completed a labeling process through the historical labeling algorithm, at this time, in terms of whether the labeling process has been performed, the status of the dataset to be labeled is "labeled". Therefore, in order to avoid ambiguity, in this embodiment, the dataset to be labeled that has been labeled is named "labeled dataset". In essence, the data contained in the dataset to be labeled is the same as the data contained in the labeled dataset. For example, if the dataset to be labeled is dataset A, after the historical labeling algorithm has labeled dataset A, at this time, in terms of whether the data has been labeled, dataset A can be called a labeled dataset, but dataset A itself has not changed and is essentially still dataset A.

[0116] In the specific implementation, after the target data annotation result is obtained by combining the historical annotation algorithm and manual annotation, the target annotation algorithm may be called to predict the annotation result of the data set A that has undergone data annotation (i.e., the annotated data set) to obtain the predicted annotation result output by the target annotation algorithm in order to verify and evaluate the algorithm quality of the target annotation algorithm. Since the target annotation result is the data that has been doubly annotated, its annotation accuracy is relatively high, and it can be used as a reference result to evaluate the accuracy of the predicted annotation result. Specifically, the target data annotation result and the predicted annotation result may be compared to calculate the recall rate, precision rate, accuracy rate, and F-mesure of the target annotation algorithm. Then, the algorithm quality of the target annotation algorithm is evaluated according to the algorithm quality evaluation index. Further, the algorithm developer can adjust and optimize the target annotation algorithm according to the algorithm quality evaluation index to obtain a target annotation algorithm whose annotation accuracy meets the accuracy requirement. At this time, the historical annotation algorithm under the historical version associated with the target annotation algorithm can also be replaced with the optimized target annotation algorithm, and the historical annotation algorithm can be deleted.

[0117] In the technical solution of the embodiment of the present application, by comparing the target data labeling result and the predicted labeling result output by the target labeling algorithm, the algorithm quality evaluation index of the target labeling algorithm is determined, so that the algorithm quality of the target labeling algorithm can be evaluated and the target labeling algorithm can be adjusted and optimized.

[0118] In some embodiments, the sample set is obtained based on the following method: according to preset labeling algorithm attribute configuration data, the algorithm input parameter attributes of the existing labeling algorithm are determined, the original attribute values ​​of the algorithm input parameter attributes at different times are obtained, and the original attribute value data set is obtained; the original attribute value data set is processed to obtain the target attribute value data set; and based on the target attribute value data set, a sample set is constructed.

[0119] This embodiment is a specific description of constructing a sample set by processing the original data based on the original data. In the specific implementation, it can be for all existing annotation algorithms, according to the preset annotation algorithm attribute configuration data, query and determine the algorithm input parameter attributes required for each annotation algorithm. Then, the algorithm input parameter attributes required for each annotation algorithm are collected to obtain the algorithm input parameter attribute set. Extract the original attribute value corresponding to each algorithm input parameter attribute in the algorithm input parameter attribute set from a data warehouse such as a data warehouse of a big data platform to obtain an original attribute value data set. Then, the original attribute value data set is processed, including data cleaning, outlier removal, and statistical data processing to obtain a target attribute value data set. Then, multiple sample sets can be constructed based on the processed target attribute value data set. Specifically, it can be based on time as the division basis to construct sample sets in different time periods. In addition, the target attribute value data set can also be connected to the sample through a unique key. Among them, the data extraction method can be based on the mapping relationship between the pre-established algorithm input parameter attributes and the attributes of the actual source data, and the original attribute value corresponding to the algorithm input parameter attribute is extracted from the data warehouse. For example, if the attributes A, B, and C required by each algorithm are configured, A, B, and C are configured to correspond to the A1, B1, and C1 attributes in the original data respectively. When extracting attribute value data, A1, B1, and C1 can be mapped to A, B, and C through SQL (Structured Query Language) AS to extract the original attribute values ​​corresponding to A1, B1, and C1.

[0120] For example, the original battery data of different devices from multiple data sources can be imported into the data warehouse, and then, according to the preset annotation algorithm attribute configuration data, the algorithm input parameter attributes required by the existing battery data annotation algorithms are determined, and then the original attribute values ​​corresponding to the algorithm input parameter attributes are extracted from the data warehouse, including device ID, battery pack ID, battery cell voltage array, battery cell temperature array, current, insulation resistance and sample ID, etc., to obtain the original attribute value data set. Then, the original attribute value data is processed, including data cleaning, outlier removal or statistical analysis, to obtain the target attribute value data set, and then the target attribute value data set is stored in the data warehouse. Then, based on the data annotation requirements, multiple sample sets can be constructed according to the different devices and battery packs, as well as the data collection time. In addition, the data at each time point can be stored in Hive or Clickhouse using the sample ID, device number ID, battery pack number ID, battery cell number ID, and time as the unique key. The data can be indexed to the data at a certain time point or time range corresponding to a certain device or battery pack or battery cell dimension by the unique key.

[0121] In the technical solution of the embodiment of the present application, by processing the original attribute value data set, the processed target attribute value data set can be made more standardized and regulated, reducing the abnormal values ​​and missing values ​​in the data, and improving the accuracy and reliability of the data. By constructing a sample set through the processed target attribute value data set, a representative data sample can be extracted.

[0122] In some embodiments, processing the original attribute value dataset to obtain the target attribute value dataset includes at least one of the following methods:

[0123] The first item is, when the original attribute value in the original attribute value data set is an array, the original attribute value is converted into row structure data to obtain the target attribute value data set;

[0124] The second item is to calculate the characteristic values ​​of the original attribute values ​​in the original attribute value data set to obtain the characteristic values ​​of the original attribute values, establish the association relationship between the original attribute values ​​and the characteristic values, and aggregate the original attribute values ​​and the characteristic values ​​to obtain the target attribute value data set.

[0125] In actual applications, since row-structured data is more flexible and can be easily expanded and changed, columns can be added, deleted, and modified to meet different needs, in this embodiment, the target attribute value data set can be stored in the form of a row structure. Continuing from the previous embodiment, data processing of the original attribute value data set may include: converting the original attribute values ​​of the array type in the original attribute value data set into row-structured data. Specifically, array-type data can be converted into row-structured data through Hive's exploded function, namely, poseExplod function, or using UDF (User Defined Function). For the specific conversion process, see Figure 6 . After converting the original attribute value data set of array type into row structure data, the characteristic value calculation can be performed on the converted original attribute value data set. For example, taking the attribute field as the dimension, the characteristic values ​​such as the mean, variance, standard deviation of the attribute value of each attribute field at different times are calculated, and then the association relationship between the original attribute value and the characteristic value is established, and the original attribute value and the characteristic value are aggregated to obtain the target attribute value data set. The original attribute value data set can also be calculated according to the corresponding battery analysis requirements. For example, data A and data B are processed in a certain data processing method to obtain data C, and the association relationship between data A, B and C is established. It can be understood that in other embodiments, the data processing method can also be a method other than the methods listed above. In practical applications, any one of the above-mentioned data processing methods can be selected or multiple data processing methods can be combined in any way.

[0126] In the technical solution of the embodiment of the present application, by converting array data into row structured data, the readability of the data, the flexibility of processing operations, and the convenience of storage and transmission can be improved. By calculating the eigenvalues ​​of the original attribute value data, the characteristics of the data can be highlighted, reflecting the main information and differences of the data, which is conducive to labeling personnel to discover abnormal data in time, and is also conducive to data analysis and modeling, improving the performance and efficiency of the algorithm.

[0127] like Figure 7 As shown, in some optional embodiments, the data annotation request carries data screening conditions;

[0128] Before S800, the method further includes: S710, filtering out target attribute feature data that meets the data filtering condition from the data set to be labeled, to obtain the target data set to be labeled.

[0129] S800 includes: S820, pushing the target dataset to be labeled to the terminal.

[0130] Data screening conditions can also be called data filtering conditions. In practical applications, since the sample set contains a lot of data, it usually includes data that is not related to the labeling work. Therefore, in order to facilitate labelers to carry out data labeling work, configurable data screening conditions can be provided in the visual operation interface. When creating a data labeling task, the labeler can select the sample set to be labeled and the labeling algorithm identification data, and configure the data screening conditions according to the needs, and then submit a data labeling request carrying the labeling algorithm identification data, the sample set identification data and the data screening conditions to the server.

[0131] During specific implementation, the server finds the attribute characteristic data corresponding to the algorithm input parameter attribute in the sample set. After obtaining the data to be labeled, it can filter the data to be labeled according to the data screening conditions, filter the target attribute characteristic data that meets the data screening conditions, and obtain the target data set to be labeled. Then, push the target data set to be labeled to the terminal. Specifically, the data screening conditions may include configurable conditions such as the attribute field to be filtered, the time range, and the attribute value interval. That is, the attribute characteristic data of a specific device, battery pack, and battery cell can be filtered out through the data screening conditions, and the attribute characteristic data within a certain attribute value interval or a specific time period can be filtered out. For example, the data screening conditions can be used to filter the data of a certain device through the device ID, further filter the data of a certain battery pack according to the battery pack ID, and further filter the data of a certain battery cell through the battery cell ID.

[0132] In the technical solution of the embodiment of the present application, the data to be labeled is filtered through data screening conditions, which can filter out useless information, filter out the data to be labeled that meets the usage conditions, and improve the efficiency of data labeling.

[0133] like Figure 8 As shown, in some embodiments, after S820, the method further includes: S900, receiving and caching the data annotation results carrying the annotation tags fed back by the terminal.

[0134] The annotation labels include category labels, time labels, and regional annotation data. In specific implementation, after the target data set to be annotated is pushed to the terminal, the terminal displays the target data set to be annotated on the annotation work page. The annotator can annotate the target data to be annotated displayed on the annotation work page. If the data within a certain time range is found to be abnormal data, the corresponding time range can be circled with the mouse. After the circle is completed, enter the basis for judging the data abnormality in the input box and temporarily store the data annotation results. At this time, the terminal will send a data save request to the server, and the server will respond to the data save request and store the data annotation results in Redis or Mysql to avoid the loss of annotation results. In other embodiments, the server can also cache the annotation results entered by the annotator in the annotation work area of ​​the terminal in real time or periodically, and save the data annotation results in time.

[0135] For example, if the data to be annotated is battery data, if it is necessary to annotate 60 voltage data reported by a certain battery cell within one minute, the terminal can call a visual chart tool to draw these 60 voltage data into a line chart in chronological order, where the horizontal axis of the line chart is time and the vertical axis is voltage value. If the voltage suddenly increases at a certain time point, the voltage at this time point will be much higher than the voltage value at other time points in the line chart. The annotator can find the abnormal point through the above line chart, and judge whether the voltage data at this time point is abnormal based on professional knowledge. If the voltage at this time point is determined to be an abnormal point, circle the time point and enter the basis for judging the voltage abnormality in the input box as the annotation remark information. After the circle is completed, enter the basis for judging the voltage abnormality in the input box and temporarily store the data annotation result. At this time, the terminal will send a data save request to the server. The server responds to the data save request and can store the data annotation result in Redis or Mysql.

[0136] In the technical solution of the embodiment of the present application, by caching the data annotation results carrying the annotation tags fed back by the terminal, the data annotation results can be saved in time, reducing the possibility of repeated annotation of data.

[0137] like Figure 8 As shown, in some embodiments, after S900, the method also includes: S920, receiving a labeling result reloading request sent by the terminal, the labeling result reloading request carries labeling task identification data, obtaining the cached data labeling results corresponding to the labeling task identification data, and pushing the cached data labeling results to the terminal so that the terminal reloads the cached data labeling results.

[0138] The labeling task identification data may include but is not limited to the labeling task instance ID or the labeling task name and other data. In actual applications, a unique labeling task instance is created for each data labeling task, and the data related to the labeling task include the data to be labeled, the data labeling result data and the corresponding sample set data, which are all associated with the labeling task instance ID, that is, the associated data labeling results and other related data can be found through the labeling task instance ID. Since in the actual manual labeling process, the labeling personnel may be affected by other factors and switch the currently displayed labeling work page to other pages, when the labeling personnel opens the labeling work page again, the terminal will send a labeling result reload request carrying the labeling task instance ID to the server. The server can query the cached data labeling results corresponding to the labeling task instance ID in Redis or Mysql through the labeling task instance ID, and then push the cached data labeling results to the terminal, so that the terminal reloads the cached data labeling results, and renders the cached data labeling results on the labeling work page, so that the labeling personnel automatically loads the cached data labeling results to the labeling work page when re-entering the labeling work surface.

[0139] In the technical solution of the embodiment of the present application, by responding to the terminal's annotation result reloading request, the cached data annotation results are pushed to the terminal, so that when the annotator enters the annotation work page again, the cached data annotation results can be automatically loaded. The annotator can continue the previous annotation work without starting over, thereby improving data annotation efficiency.

[0140] In order to more clearly illustrate the method for processing labeled data provided by the present application, the following is a specific embodiment and the attached Fig. 9 To illustrate, this specific embodiment includes the following steps:

[0141] S220, receiving a data labeling request sent by the terminal, where the data labeling request carries sample identification data, labeling algorithm identification data, and data screening conditions.

[0142] S400, determining the algorithm input parameter attributes corresponding to the labeling algorithm identification data according to the preset labeling algorithm attribute configuration data.

[0143] Specifically, the labeling algorithm attribute configuration data may include labeling algorithm identification data such as labeling algorithm ID, algorithm name, algorithm version information, algorithm input parameter attributes, attribute data types, and parameters that characterize whether the algorithm input parameter attributes are necessary, etc. It also includes a mapping relationship between the names of the algorithm input parameter attributes and the names of the actual data fields stored in the database. Through this mapping relationship, the actual data fields in the database and the attribute feature data of the data fields can be found according to the attributes required by the algorithm.

[0144] S600, obtaining a sample set corresponding to the sample identification data, and matching the attribute feature data of the target attribute from the sample set to obtain a data set to be labeled.

[0145] Attribute feature data includes attribute values ​​and feature data, wherein the feature data includes but is not limited to mean value, variance, standard deviation, etc.

[0146] S710, filter out target attribute feature data that meets the data screening conditions from the data set to be labeled, and obtain the target data set to be labeled.

[0147] Before pushing the target data set to be labeled to the terminal, it is possible to search whether there is an associated historical labeling algorithm based on the labeling algorithm identification data. If there is a historical labeling algorithm, execute the operations in S712-S960; if there is no historical labeling algorithm, execute the operations in S722-S900.

[0148] S722: Group and encapsulate the target dataset to be labeled according to the algorithm input parameter attributes.

[0149] S822, pushing the grouped and encapsulated target data sets to be labeled to the terminal, so that the terminal displays the data sets to be labeled in groups according to target attributes.

[0150] S900, receiving and caching the data labeling result with the labeling tag fed back by the terminal.

[0151] S712, when a historical annotation algorithm associated with the target annotation algorithm is found, the historical annotation algorithm is called to annotate the data set to be annotated to obtain an initial data annotation result, and the target annotation algorithm is the annotation algorithm corresponding to the annotation algorithm identification data.

[0152] S722: Group and encapsulate the target dataset to be labeled according to the algorithm input parameter attributes.

[0153] S842, push the initial data labeling results and the target data set to be labeled to the terminal.

[0154] S940, receiving and caching the target data labeling result fed back by the terminal, where the target data labeling result includes the revised initial data labeling result.

[0155] S960, calling the target labeling algorithm, predicting the labeling results of the labeled data set, obtaining the predicted labeling results, and determining the algorithm quality evaluation index of the target labeling algorithm according to the target data labeling results and the predicted labeling results.

[0156] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0157] Based on the same inventive concept, the embodiment of the present application also provides a labeling data processing device for implementing the labeling data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more embodiments of the labeling data processing device provided below can refer to the limitations of the labeling data processing method above, and will not be repeated here.

[0158] In some embodiments, Fig.10 As shown, a labeled data processing device 900 is provided, comprising: a data receiving module 910, an attribute determination module 920, a data matching module 930 and a data pushing module 940, wherein:

[0159] The data receiving module 910 is used to receive a data labeling request sent by a terminal, where the data labeling request carries sample identification data and labeling algorithm identification data.

[0160] The attribute determination module 920 is used to determine the algorithm input parameter attributes corresponding to the labeling algorithm identification data according to the preset labeling algorithm attribute configuration data.

[0161] The data matching module 930 is used to obtain a sample set corresponding to the sample identification data, and match the attribute feature data of the target attribute from the sample set to obtain a data set to be labeled.

[0162] The data push module 940 is used to push the data set to be labeled to the terminal.

[0163] In the technical solution of the embodiment of the present application, by pre-configuring the attributes required by the labeling algorithm, that is, pre-setting the labeling algorithm attribute configuration data, it is possible to quickly determine the target attributes required by the labeling algorithm according to the preset labeling algorithm attribute configuration data when the labeling algorithm identification data is known, and then accurately match the attribute feature data of the target attribute from the sample set according to the algorithm input parameter attributes to obtain an accurate data set to be labeled, and then push the data set to be labeled to the terminal so that the labeling personnel can label the data set to be labeled. The entire device can provide a unified labeling data acquisition capability for different labeling algorithms and sample sets. For different labeling algorithms, it only needs to determine the attributes required by the algorithm according to the preset labeling algorithm attribute configuration data, so as to quickly and accurately obtain the data to be labeled, thereby improving the efficiency of data labeling.

[0164] In some embodiments, the data annotation request carries data screening conditions;

[0165] like Fig.11 As shown, the device also includes a data screening module 932, which is used to screen out target attribute feature data that meets the data screening conditions from the data set to be labeled, and obtain the target data set to be labeled.

[0166] The data push module 940 is also used to push the target dataset to be labeled to the terminal.

[0167] like Fig.11 As shown, in some embodiments, the device also includes an intelligent labeling module 934, which is used to call the historical labeling algorithm to label the data set to be labeled when a historical labeling algorithm associated with the target labeling algorithm is found, and obtain an initial data labeling result. The target labeling algorithm is a labeling algorithm corresponding to the labeling algorithm identification data.

[0168] The data push module 940 is also used to push the initial data annotation results and the data set to be annotated to the terminal.

[0169] The data receiving module 910 is further used to receive the target data labeling result fed back by the terminal, where the target data labeling result includes the revised initial data labeling result.

[0170] In some embodiments, the intelligent labeling module 934 is further used to call a target labeling algorithm to predict the labeling result of the labeled data set to obtain a predicted labeling result;

[0171] like Fig.11 As shown, the device also includes an algorithm quality evaluation module 950, which is used to determine the algorithm quality evaluation index of the target labeling algorithm according to the target data labeling result and the predicted labeling result.

[0172] In some embodiments, Fig.11As shown, the device also includes a data grouping and encapsulation module 936, which is used to group and encapsulate the data set to be labeled according to the algorithm input parameter attributes.

[0173] The data push module 940 is further used to push the grouped and encapsulated data sets to be labeled to the terminal, so that the terminal can display the data sets to be labeled in groups according to target attributes.

[0174] like Fig.11 As shown, in some embodiments, the device also includes a data processing module 902, which is used to determine the algorithm input parameter attributes of the existing labeling algorithm according to the preset labeling algorithm attribute configuration data, obtain the original attribute values ​​of the algorithm input parameter attributes at different times, obtain the original attribute value data set, perform data processing on the original attribute value data set, obtain the target attribute value data set, and construct a sample set based on the target attribute value data set.

[0175] In some embodiments, the data processing module 902 is further configured to perform at least one of the following methods:

[0176] The first item is, when the original attribute value in the original attribute value data set is an array, the original attribute value is converted into row structure data to obtain the target attribute value data set.

[0177] The second item is to calculate the characteristic values ​​of the original attribute values ​​in the original attribute value data set to obtain the characteristic values ​​of the original attribute values, establish the association relationship between the original attribute values ​​and the characteristic values, and aggregate the original attribute values ​​and the characteristic values ​​to obtain the target attribute value data set.

[0178] In some embodiments, the data receiving module 910 is further used to receive and cache the data annotation results carrying the annotation tags fed back by the terminal.

[0179] like Fig.11 As shown, in some embodiments, the device also includes a labeling result reloading module 950, which is used to receive a labeling result reloading request sent by the terminal, the labeling result reloading request carries labeling task identification data, obtains the cached data labeling results corresponding to the labeling task identification data, and pushes the cached data labeling results to the terminal so that the terminal reloads the cached data labeling results.

[0180] Each module in the above-mentioned annotation data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0181] In some embodiments, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data such as sample data, annotation algorithm attribute configuration data and data annotation results. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for processing annotated data is implemented.

[0182] Those skilled in the art will understand that Fig.12 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0183] In some embodiments, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the steps in the above-mentioned embodiments of the annotation data processing method are implemented.

[0184] In some embodiments, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned embodiments of the annotation data processing method are implemented.

[0185] In some embodiments, a computer program product is provided, including a computer program, which implements the steps in the above-mentioned embodiments of the annotation data processing method when executed by a processor.

[0186] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A method for processing labeled data, characterized in that: The method comprises: A data labeling request sent by a receiving terminal is carried by the data labeling request and the labeling algorithm identification data; Determine the algorithm input parameter attribute corresponding to the labeling algorithm identification data according to the preset labeling algorithm attribute configuration data; Acquire a sample set corresponding to the sample identification data, and match attribute feature data of the algorithm input parameter attributes from the sample set to obtain a data set to be labeled; Push the data set to be labeled to the terminal.

2. The method according to claim 1, characterized in that The method further comprises: In the case of finding a historical annotation algorithm associated with the target annotation algorithm, calling the historical annotation algorithm to annotate the to-be-annotated data set to obtain an initial data annotation result, wherein the target annotation algorithm is an annotation algorithm corresponding to the annotation algorithm identification data; Pushing the initial data annotation result and the data set to be annotated to the terminal; Receive the target data labeling result fed back by the terminal, where the target data labeling result includes the revised initial data labeling result.

3. The method according to claim 2, characterized in that After receiving the target data annotation result fed back by the terminal, the method further includes: Calling the target labeling algorithm to predict the labeling result of the labeled data set to obtain the predicted labeling result; An algorithm quality evaluation index of the target labeling algorithm is determined according to the target data labeling result and the predicted labeling result.

4. The method according to any one of claims 1 to 3, characterized in that: Before pushing the to-be-annotated data set to the terminal, the method further includes: According to the target attribute, the to-be-annotated data set is grouped and packaged; The pushing the to-be-annotated data set to the terminal comprises: The grouped and encapsulated data set to be labeled is pushed to the terminal, so that the terminal displays the data set to be labeled in groups according to the target attributes, and the attribute feature data of the data set to be labeled at different times.

5. The method according to any one of claims 1 to 3, characterized in that: The sample set is obtained based on the following method: According to the preset annotation algorithm attribute configuration data, the algorithm input parameter attributes of the existing annotation algorithm are determined; Obtaining original attribute values ​​of the algorithm input attributes at different times to obtain an original attribute value data set; Processing the original attribute value data set to obtain a target attribute value data set; Based on the target attribute value data set, a sample set is constructed.

6. The method according to claim 5, characterized in that The processing of the original attribute value data set to obtain the target attribute value data set includes at least one of the following methods: The first item is, when the original attribute value in the original attribute value data set is an array, converting the original attribute value into row structure data to obtain a target attribute value data set; The second item is to calculate the characteristic values ​​of the original attribute values ​​in the original attribute value data set to obtain the characteristic values ​​of the original attribute values, establish the association relationship between the original attribute values ​​and the characteristic values, and aggregate the original attribute values ​​and the characteristic values ​​to obtain the target attribute value data set.

7. The method according to any one of claims 1 to 3, characterized in that: The data annotation request carries data screening conditions; Before pushing the to-be-annotated data set to the terminal, the method further includes: Filtering target attribute feature data that meets the data screening conditions from the data set to be labeled to obtain the target data set to be labeled; The pushing the to-be-annotated data set to the terminal comprises: Push the target dataset to be labeled to the terminal.

8. The method according to any one of claims 1 to 3, characterized in that: After pushing the to-be-annotated data set to the terminal, the method further includes: Receive and cache the data labeling results carrying the labeling tags fed back by the terminal.

9. The method according to claim 8, characterized in that After receiving and caching the data annotation result carrying the annotation tag fed back by the terminal, the method further includes: receiving a labeling result reloading request sent by the terminal, wherein the labeling result reloading request carries labeling task identification data; Obtaining cached data annotation results corresponding to the annotation task identification data; Pushing the cached data annotation results to the terminal so that the terminal reloads the cached data annotation results.

10. A labeling data processing device, characterized in that: The device comprises: A data receiving module, used to receive a data labeling request sent by a terminal, wherein the data labeling request carries sample identification data and labeling algorithm identification data; An attribute determination module, used to determine the algorithm input parameter attribute corresponding to the annotation algorithm identification data according to the preset annotation algorithm attribute configuration data; A data matching module is used to obtain a sample set corresponding to the sample identification data, and match the attribute feature data of the target attribute from the sample set to obtain a data set to be labeled; The data push module is used to push the to-be-annotated data set to the terminal.

11. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

13. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images