Sample data labeling system, method, and related devices

Abstract

The application discloses a sample data labeling system, a labeling method applied to the sample data labeling system and related devices. The sample data labeling system comprises an edge node and a center node. The edge node acquires key features of sample data, judges whether the sample data is unknown sample data according to the key features, performs labeling processing on the sample data when the sample data is unknown sample data, obtains a first labeling result, and sends the first labeling result to the center node. The center node receives the first labeling result, performs consistency processing on the first labeling result when the first labeling result indicates that the unknown sample data is successfully labeled, obtains a second labeling result, and performs labeling processing on the unknown sample data when the first labeling result indicates that the unknown sample data fails to be labeled, and obtains a third labeling result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and in particular, to a sample data labeling system and method and related equipment. BACKGROUND

[0002] Recently, artificial intelligence (AI) has been applied in more and more fields. Most of the existing AI technologies, including supervised learning, deep learning (DL), etc., are based on a large number of labeled data sets. Labeling refers to generating and / or adding labels that can represent the classification of sample data. A common labeling method is to label sample data at a central node: each edge client node (referred to as edge node) uploads sample features (including sample data or key features of sample data) that need to be labeled to the central node for labeling. Another labeling method is to label sample data at edge nodes, that is, each edge node directly labels locally after collecting sample features that need to be labeled. Both of these two labeling methods have certain defects. The former is limited by the need for privacy protection, such as in the field of federated learning (FL), sample data of edge nodes (such as personal photos stored on personal mobile terminals, patient data of hospitals, etc.) cannot be uploaded to the central node for labeling. On the other hand, uploading all sample features of edge nodes to the central node requires a large amount of data communication, causing high communication load. In addition, since all labeling needs to be completed at the central node, the central node is also prone to bottlenecks. The latter is limited by the low computing power of edge nodes, and the labeling efficiency is not high. In addition, since the labeling method is relatively simple compared to central labeling, it may cause certain sample features to be unable to be labeled or labeled incorrectly.

[0003] Therefore, how to improve the efficiency and quality of sample data labeling has become a research problem in the field of AI. SUMMARY

[0004] The present application provides a sample data labeling system, method and related equipment for improving the labeling efficiency and quality of AI sample data.

[0005] In a first aspect, the present application provides a sample data labeling system, comprising an edge node and a center node, the center node being connected to the edge node; the edge node is configured to: obtain a key feature of sample data; determine whether the sample data is unknown sample data according to the key feature; when the sample data is unknown sample data, perform labeling processing on the sample data to obtain a first labeling result; and upload the first labeling result to the center node. The center node is configured to: receive the first labeling result sent by the edge node; perform consistency processing on the first labeling result indicating that the unknown sample is labeled successfully to obtain a second labeling result; and perform labeling processing on the first labeling result indicating that the unknown sample is labeled unsuccessfully to obtain a third labeling result.

[0006] In the present application, the sample data is labeled by the edge node, which can improve the efficiency of labeling. In addition, due to the low computing power of the edge node and other objective conditions, the first labeling result obtained by the edge node may have labeling errors or labeling failures. After the first labeling result is sent to the center node, the center node can correct or re-label the first labeling result, which can improve the quality of labeling.

[0007] In an optional implementation, the edge node determines whether the sample data is unknown sample data according to an unknown sample model and the key feature obtained from the sample data.

[0008] In an optional implementation, the unknown sample model is generated according to a plurality of key features obtained from a known labeling result set. The known labeling result set includes labeling results obtained by the center node before generating the unknown sample model. The labeling results include the second labeling result generated by the center node through consistency processing and / or the third labeling result obtained by the center node after labeling.

[0009] In an optional implementation, the unknown sample model can also include a multi-dimensional coordinate space generated by the center node according to the key features of the known labeling result set. According to the key feature of the sample data or the mapping feature of the key feature, a coordinate distance is obtained, and according to the correlation represented by the coordinate distance, it is determined whether the sample data is unknown sample data. If the coordinate distance is larger, it means that the correlation is lower. When the correlation is higher than a preset threshold, the sample data is determined as unknown sample data. If the coordinate distance is lower, it means that the correlation is lower. When the correlation is lower than a preset threshold, the sample data is determined as unknown sample data.

[0010] In an optional implementation, the unknown sample model can also include a neural network model. Inference is performed on the sample data by the neural network model, and if a correct inference result cannot be obtained, the sample data is considered as unknown sample data.

[0011] In an optional implementation, when the first labeling result indicates that the unknown sample data is successfully labeled, the first labeling result includes: a sample identifier of the sample data, a sample feature of the sample data, and a label determined by the edge node for the sample data; when the first labeling result indicates that the unknown sample data fails to be labeled, the first labeling result includes: a sample identifier of the sample data and a sample feature of the sample data. In an optional implementation, the sample feature of the first labeling result includes the sample data and / or the key feature.

[0012] In an optional implementation, when the consistency processing is performed on the first labeling result, the center node clusters a plurality of first labeling results including the first labeling result by similarity division to obtain a group corresponding to the first labeling result. Further, in an optional implementation, the center node clusters a plurality of first labeling results including the first labeling result in a non-supervised manner, and the non-supervised manner includes one or more of K-MEANS and KNN.

[0013] In an optional implementation, when the labels of the labeling results in the group are inconsistent, the center node performs an integrated decision on the labels of all the labeling results in the group to obtain a group label, and obtains the second labeling result according to the group label. In an optional implementation, the integrated decision includes a voting method or a weighted voting method.

[0014] In an optional implementation, the center node generates a new unknown sample model or updates the unknown sample model according to the second labeling result and / or the third labeling result.

[0015] In a second aspect, the present application discloses a sample data labeling method applied to an edge node of a sample labeling system. The method includes: obtaining a key feature of sample data; determining whether the sample data is unknown sample data according to the key feature; performing labeling processing on the unknown sample data when the sample data is unknown sample data to obtain a first labeling result; and sending the first labeling result to a center node.

[0016] Through the method, the edge node only labels the sample data that needs to be labeled, which can reduce the number of sample data that needs to be labeled, and the first labeling result after labeling is sent to the center node for further processing, which can improve the labeling quality.

[0017] In an optional implementation, the edge node determines whether the sample data is unknown sample data according to the unknown sample model and the key features.

[0018] In an optional implementation, the unknown sample model is generated according to a plurality of key features obtained from a known labeled result set. The known labeled result set includes labeled results successfully labeled by the center node before the unknown sample model is generated.

[0019] In an optional implementation, the unknown sample model can further include a multi-dimensional coordinate space generated by the center node of the sample data labeling system according to key features of a known labeled result set. According to the key features of the sample data or the mapped features of the key features, a coordinate distance is obtained, and according to the correlation represented by the coordinate distance, it is determined whether the sample data is unknown sample data. If the coordinate distance is larger, it means that the correlation is lower. When the correlation is higher than a preset threshold, the sample data is determined as unknown sample data. If the coordinate distance is lower, it means that the correlation is lower. When the correlation is lower than a preset threshold, the sample data is determined as unknown sample data.

[0020] In an optional implementation, the unknown sample model can further include a neural network model. Through the neural network model, an inference action is performed on the sample data, and if a correct inference result cannot be obtained, the sample data is considered as unknown sample data.

[0021] In an optional implementation, when the first labeled result indicates that the unknown sample data is successfully labeled, the first labeled result includes: a sample identifier of the sample data, sample features of the sample data, and a label determined by the edge node for the sample data. When the first labeled result indicates that the unknown sample data is unsuccessfully labeled, the first labeled result includes: a sample identifier of the sample data and sample features of the sample data. In an optional implementation, the sample features of the first labeled result include the sample data and / or the key features.

[0022] In a third aspect, the present application discloses a sample data labeling method, which is used for a center node of a sample labeling system. The method comprises the following steps: receiving a first labeling result sent by an edge node in the sample labeling system, the first labeling result being obtained by labeling and processing unknown sample data by the edge node; when the first labeling result indicates that the unknown sample data is successfully labeled, performing consistency processing on the first labeling result to obtain a second labeling result; and when the first labeling result indicates that the unknown sample data is unsuccessfully labeled, performing labeling processing on the unknown sample data to obtain a third labeling result.

[0023] By the method, the center node performs secondary processing on the first labeling result sent by the edge node, thereby improving the labeling quality of the sample data.

[0024] In an optional implementation, when the consistency processing is performed on the first labeling result, the center node performs clustering on a plurality of first labeling results including the first labeling result by similarity division to obtain a group corresponding to the first labeling result. Further, in an optional implementation, the center node performs clustering on a plurality of first labeling results including the first labeling result by an unsupervised manner, the unsupervised manner comprising one or more of K-MEANS and KNN.

[0025] In an optional implementation, when the labels of the labeling results in the group are inconsistent, the center node performs integrated decision on the labels of all the labeling results in the group to obtain a group label, and obtains the second labeling result according to the group label. In an optional implementation, the integrated decision comprises a voting method or a weighted voting method.

[0026] In an optional implementation, the center node generates a new unknown sample model or updates the unknown sample data model according to the second labeling result and / or the third labeling result.

[0027] In a fourth aspect, the present application discloses a node comprising a functional module for executing the sample labeling method provided in the second aspect or any possible design of the second aspect. The present application does not limit the division of the functional module, which can be divided according to the process steps of the sample labeling method in the second aspect, or can be divided according to the specific implementation needs.

[0028] In a fifth aspect, the present application discloses a node comprising a functional module for executing the sample labeling method provided in the third aspect or any possible design of the third aspect. The present application does not limit the division of the functional module, which can be divided according to the process steps of the sample labeling method in the second aspect, or can be divided according to the specific implementation needs.

[0029] The embodiments of the different aspects of the present application can be combined or cited with each other without conflict.

[0030] In a sixth aspect, the present application discloses a computer program code, which comprises instructions that can make a computer implement the sample data labeling method in the second aspect or any possible implementation manner of the second aspect when the instructions are executed by the computer.

[0031] In a seventh aspect, the present application discloses a computer program code, which comprises instructions that can make a computer implement the sample data labeling method in the third aspect or any possible implementation manner of the third aspect when the instructions are executed by the computer.

[0032] In an eighth aspect, the present application discloses a computer readable storage medium, which stores computer program instructions, when the computer program instructions are run on a computer, the computer is caused to execute the sample data labeling method in the second aspect or any possible implementation manner of the second aspect.

[0033] In a ninth aspect, the present application discloses a computer readable storage medium, which stores computer program instructions, when the computer program instructions are run on a computer, the computer is caused to execute the sample data labeling method in the third aspect or any possible implementation manner of the third aspect. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application.

[0035] Figure 1 A structural schematic diagram of a sample data labeling system provided by the embodiments of the present application;

[0036] Figure 2 A structural schematic diagram of a center node or an edge node of a sample data labeling system provided by the embodiments of the present application;

[0037] Figure 3 A flow schematic diagram of a sample data labeling method provided by the embodiments of the present application;

[0038] Figure 4 A format schematic diagram of a labeling result provided by the embodiments of the present application;

[0039] Figure 5A A module schematic diagram of an edge node provided by the embodiments of the present application;

[0040] Figure 5B A module schematic diagram of a center node provided for an embodiment of the present application is shown in the following figure.

[0041] Figure 6 A multi-dimensional coordinate space schematic diagram provided for an embodiment of the present application is shown in the following figure. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments.

[0043] In the embodiments of the present application, the words such as “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design solution described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design solutions. In fact, the use of the words such as “exemplary” or “for example” is intended to present the relevant concept in a specific manner. In the embodiments of the present application, the meaning of “plurality” is two or two or more, unless otherwise specified. For example, a plurality of nodes means two or two or more nodes. “At least one” means any number, for example, one, two or two or more. “A and / or B” can be only A, only B, or include A and B. “At least one of A, B and C” can be only A, only B, only C, or include A and B, include B and C, include A and C, or include A, B and C. The terms “first”, “second” and the like in the present application are only used to distinguish different objects, and are not used to indicate priority or importance of the objects.

[0044] In order to improve the efficiency and quality of sample data labeling, in one embodiment, as shown in the following figure, a sample data labeling system 100 is provided, which includes a center node 110 and a plurality of edge nodes 120, the center node 110 is connected with each edge node 120A, 120B, 120C, 120D through a bus 130, and the network interface can be an interface connecting an internal network or an interface connecting the Internet. That is, through the bus 130, the labeling system can be deployed in an internal network (campus network, VPN, Cloud, etc.), or on a public network. Figure 1

[0045] The center node 110 or the edge node 120 can be deployed in, for example, Figure 2 ​The at least one processor platform 200 shown can be, for example, a server, a personal computer, a mobile device (e.g., a phone, a smart phone, a tablet such as an iPad), a personal digital assistant (PDA), an Internet appliance, an embedded device, or any other type of computing device. Further, the processor platform 200 includes a memory 210, a processor 250, a bus 260, an interface device 270, and an I / O interface. The memory 210 is used to store program logic 220 (e.g., software) and data 230, where the program logic 220 includes various logical modules for the annotation system 100 to perform consistency processing, annotation, etc. The data 230 can include sample data, annotation results, unknown sample data models, etc. The memory 210 can also include an operating system 240. The processor 250 can include multiple processor cores or can be a single very large scale integrated circuit, and the processor 250 can execute the program logic 220 and process the data 230 using the operating system 240. The bus 260 is used for communication between the memory 210, the processor 250, the interface device 270, and the I / O interface 280. A user can interact with the processor platform 200 through the interface device 270, such as a keyboard, a mouse, etc. The processor platform 200 can also communicate with one or more other processor platforms through the I / O interface, such as a network, a modem, etc. The various components of the processor platform 200 provided by the embodiments of the present application are merely exemplary, and a person skilled in the art can add or reduce components or split the functions of a component into multiple components as needed.

[0046] The sample data annotation system 100 can start one or more sample data annotation tasks, and each sample data annotation task can be started at a specified time or at a specified period or according to an external trigger condition (e.g., when the original sample data reaches a threshold, etc.) or according to an instruction. The following describes a process of the sample data annotation system 100 to perform sample data annotation according to the following Figure 3 The process described below is applicable to any annotation task of the sample annotation system 100, unless otherwise specified.

[0047] As shown in Figure 3 the sample data annotation method provided by the embodiments of the present application includes steps S311-S314 and S321-S323. In a specific implementation, the steps in Figure 3 may be modified or deleted as needed, and the order of any step in Figure 3 may be adjusted as needed.

[0048] Steps S311-S314 disclose a sample data labeling method applied in the edge node 310. In step S311, the edge node 310 acquires a key feature of the sample data. The key feature of the sample data can be extracted by the edge node 310 from the sample data after the edge node 310 acquires the sample data, or can be the key feature of the sample data received by the edge node 310 from other devices. Step 311 can be triggered when the edge node 310 receives the sample data, or can be triggered at a fixed time. The key feature is a relevant attribute of the sample data, such as one or more of the total length of the sample data, the average interval of the sample data packets, the uplink / downlink direction, the acquisition start time of the sample data, and / or the end period of the sample data.

[0049] In step S312, the edge node 310 determines whether the sample data is unknown sample data according to the acquired key feature. In one implementation, the edge node 310 maps the key feature acquired in step S311 to an unknown sample data model of the edge node to determine whether the sample data is unknown sample data. In one implementation, the unknown sample data model is generated by the center node 320 according to a plurality of key features acquired from a known labeling result set. The known labeling result set includes a plurality of labeling results labeled successfully by the center node 320 before the unknown sample data model is generated, each labeling result including a sample identifier of sample data, a sample feature of the sample data, and a label of the sample data, which can be labeled by the center node 320 or labeled by the edge node 310 and obtained after consistency processing by the center node. In another implementation, when the sample data labeling system 100 performs labeling for the first time, the center node 320 generates a known labeling result set according to an empty labeling result set or an externally loaded labeling result set. In one implementation, the unknown sample data model includes a multi-dimensional coordinate space, which is generated by the center node 320 according to the key features of the known labeling result set. As shown in FIG. 6, the multi-dimensional coordinate space 640 is a three-dimensional coordinate space composed of key features 601-603, which can be key features of sample data or features mapped by key features of sample data, and can be one or more. In one example, the sample data labeling system 100 can map ten key features of sample data into a three-dimensional coordinate space, or can map ten key features of sample data into another ten-dimensional coordinate space. Figure 6

[0050] ​In the sample data labeling system 100, the correlation between sample data can be represented by the distance of coordinate distance, and when the correlation is lower than a preset threshold, the sample data is determined as unknown sample data. The edge node 310 can pre-set one or more thresholds, and if the greater the coordinate distance, the lower the correlation, the specific implementation of step S312 can be, for example, when the coordinate distance of the key features of the sample data obtained in step S310 or the mapping features based on the key features in the multi-dimensional coordinate space is greater than or equal to the preset threshold, the sample data is determined as unknown sample data. If the smaller the coordinate distance, the lower the correlation, the specific implementation of step S312 can be, for example, when the coordinate distance of the key features of the sample data obtained in step S310 or the mapping features based on the key features in the multi-dimensional coordinate space is smaller than or equal to the preset threshold, the sample data is determined as unknown sample data. As shown in Figure 6 611-613 are known labels, which are the labels of the sample data of the known labeling result set. When the key features of the sample data to be labeled (621-622, 631-634) are mapped to the multi-dimensional coordinate space 640, the coordinate distance of the known labels 611-613 is generated according to the multi-dimensional coordinates, and Figure 6 In the implementation, the greater the coordinate distance, the lower the correlation, the coordinate distance of 621 and 622 is lower than the preset threshold, and 621 and 622 are determined as known sample data, and the coordinate distance of 631-634 is higher than the preset threshold, and 631-634 are determined as unknown sample data.

[0051] In one implementation, the unknown sample data model further includes a neural network model. The key features obtained in step S311 can be inferred by the neural network model, and when the neural network model cannot obtain correct inference results, the sample data is considered as unknown sample data.

[0052] When the sample data is determined as unknown sample data in step S312, the edge node 310 labels the sample data to obtain the first labeling result in step S313. In one implementation, the format of the labeling result in the present application is as follows Figure 4As shown, the annotation result includes a sample identifier 401 and / or sample features 402. In some embodiments, the annotation result also includes a label 403. The sample identifier 401 is obtained based on the identifier of the sample data and can be the same as the identifier of the sample data or another identifier generated from the identifier of the sample data. The sample features 402 can include one or more key features of the sample data, and / or the sample data itself. The label 403 is a label obtained by the edge node 310 after annotating the unknown sample data. There can be one or more labels 403. In one implementation, the sample data annotation system 100 uses the label value of the first annotation result to indicate whether the unknown sample data was successfully annotated in step S313. When the edge node 310 successfully annotates the unknown sample data, the first annotation result includes the annotated label. When the annotation in step S313 is unsuccessful, the first annotation result does not include the label or the label value is empty. Furthermore, the sample data annotation system 100 can also use a dedicated parameter to indicate whether the unknown sample data was successfully annotated in step S313. For example, when the value of the parameter is True, it indicates that the unknown sample data was successfully annotated; when the value is False, it indicates that the unknown sample data was not annotated. In step S314, the edge node 310 sends the first annotation result to the center node 320.

[0053] Steps S321 to S323 disclose a sample data annotation method applied to the central node 320, which receives data from each edge node (such as...). Figure 1 After the first annotation results sent by nodes 120A, 120B, 120C, and 120D are received, step S321 determines whether the received first annotation results indicate successful annotation. In one implementation, step S321 can be triggered after the central node 320 receives a certain number of first annotation results from each edge node, or it can be triggered by a timer or by any other arbitrary method. Further, in one implementation, step S321 will determine all the first annotation results received by the central node 320, and divide all the first annotation results into two categories: those indicating successful annotation and those indicating failed annotation. Then, steps S322 and S323 will be performed respectively. Further, when step S321 determines that all the received first annotation results indicate successful annotation, the central node 320 will not execute step S323; or when step S321 determines that all the received first annotation results indicate failed annotation, the central node 320 will not execute step S322.

[0054] In another implementation, the center node 320 can also process the received first annotation results in batches, i.e., divide the first annotation results into several batches, and after the first annotation results in the first batch are executed through steps S321-S323, the first annotation results in the second batch are executed again through steps S321-S323, and so on until all the received first annotation results are processed.

[0055] When step S321 indicates that the first annotation result is successfully annotated, step S322 is entered. In step S322, the center node 320 performs consistency processing on the first annotation result indicated as successfully annotated to obtain a second annotation result. The format of the second annotation result is shown in Figure 4 Through similarity division, the center node 320 clusters the plurality of first annotation results including the first annotation result, i.e., divides the successfully annotated plurality of first annotation results into different groups through a clustering algorithm, the first annotation results belonging to the same group contain the same or similar sample features, and the first annotation results belonging to different groups have obviously different sample features. In an implementation, clustering can be performed in a non-supervised manner, and the non-supervised manner includes a K-MEANS (k-means clustering algorithm), a KNN (k-nearest neighbors algorithm), etc. Taking the first annotation result A as an example, when the labels of the first annotation results in the group corresponding to the first annotation result A are inconsistent, the center node 320 performs consistency processing on all the annotation results in the group, i.e., determines the group label of the group through an integrated decision-making manner, and determines the group label as the new label of each first annotation result in the group to obtain the second annotation result corresponding to each first annotation result. When the group label is the same as the label of a first annotation result, the center node 320 maintains the first annotation result, and in this case, it can be considered that the second annotation result corresponding to the first annotation result is the same as the first annotation result. When the group label is different from the label of a first annotation result, the center node 320 modifies the label of the first annotation result to the group label, and in this case, the second annotation result corresponding to the first annotation result is different from the first annotation result. In an implementation, the integrated decision-making includes a voting method or a weighted voting method. Assuming that the labels to be determined are {C1, C2,..., C k}, and the pre-annotation results of the sample data x of the T edge nodes 120 of the sample data annotation system 100 are (h1(x), h2(x),..., h T (x)). Through a relative majority voting method, i.e., a commonly said majority over minority, the most numerous label C iThe final label is the group label obtained by the ensemble decision. If more than one label gets the highest votes, a label is randomly selected as the group label. The voting method can also be an absolute majority voting method, that is, the number of votes needs to be more than half. On the basis of the relative majority voting method, in addition to the number of votes needing to be the highest, the number of votes also needs to be more than half. The weighted voting method is that the label votes of the T edge nodes 120 are multiplied by a weight, and the weighted votes of the same label are summed, and the label corresponding to the maximum value is the final label, that is, the group label obtained by the ensemble decision.

[0056] When step S321 indicates that the first labeling result fails to be labeled, step S323 is entered. In step S323, the center node 320 labels the first labeling result indicated as a labeling failure to obtain a third labeling result. The format of the third labeling result is as shown in Figure 4 In an implementation, the edge nodes and / or the center node of the sample data labeling system 100 label the sample data by rule matching, rule association, or manual labeling. Rule matching is to set a certain rule, and label the sample data matching the rule according to the sample data characteristics. For example, if a keyword YouTube is found in the domain name system (DNS) of a network flow, it can be determined that the network flow is a video flow. Rule association is to determine the same label for a plurality of sample data associated with each other. For example, if an unknown network flow always appears in association with a known network flow, it is determined that the unknown network flow is a network flow of the same category as the known network flow. Manual labeling is to observe sample characteristics and label results by manual observation.

[0057] Further, in an implementation, after the center node 320 obtains the second and third labeling results, a new unknown sample model can be generated or the original unknown sample model can be updated according to the second and third labeling results. The newly generated or updated unknown sample model is sent to the edge node 310. After receiving the unknown sample model, the edge node 310 updates the existing local unknown sample model.

[0058] Further, in an implementation, the edge node 310 or the center node 320 can perform encryption processing on the transmitted data and decryption processing on the received data. The encrypted / decrypted data includes the first labeling result or the unknown sample model.

[0059] The sample data annotation method provided in the embodiments of this application has been described above from a system perspective. It is understood that the edge nodes or central nodes in the embodiments of this application include corresponding hardware structures and / or software modules for performing each function in order to achieve the above-mentioned functions. Those skilled in the art should readily recognize that the functions and steps of the examples described in the embodiments disclosed in this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions, but such implementation should not be considered beyond the scope of this application.

[0060] The structure of the nodes in this application will be described from different perspectives below. To implement this application... Figure 3 The method shown in this application embodiment provides two types of nodes, 500 and 510. For example... Figure 5A As shown, node 500 is used as an edge node in the sample data annotation system 100. Node 500 includes an acquisition unit 501, an identification unit 502, an annotation unit 503, and a communication unit 504. In this embodiment, the acquisition unit 501 is used to perform step S311, the identification unit 502 is used to perform step S312, the annotation unit 503 is used to perform step S313, and the communication unit 504 is used to perform step S314. Figure 5B As shown, node 510 is used as the central node of the sample data annotation system 100. Node 510 includes a communication unit 511, a processing unit 512, and an annotation unit 513. The communication unit 511 is used to receive the first annotation result sent by the edge node 310, the processing unit 512 is used to execute step S322, and the annotation unit 513 is used to execute step S323. The constituent units of node 500 or 510 provided in this application embodiment are merely exemplary. Those skilled in the art can add or reduce constituent units as needed, or divide the function of one unit into multiple units.

[0061] This application also provides a computer-readable storage medium for storing program code implementing the above-described sample data annotation method. The program code includes instructions for executing the method flow described in any of the foregoing method embodiments. The aforementioned storage medium includes various non-transitory machine-readable media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, optical disks, random access memory (RAM), solid-state disks (SSDs), or non-volatile memory.

[0062] It should be noted that the embodiments provided in the present application are merely illustrative. It is obvious for those skilled in the art that, for the convenience and brevity of description, the description of each embodiment in the above embodiments is focused on, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. The features disclosed in the embodiments, claims and drawings of the present application can exist independently or in combination. The features described in the embodiments of the present application in the form of hardware can be executed by software, and vice versa. This is not limited here.

Claims

1. A sample data labeling system, characterized by, The system comprises an edge node and a center node; The edge node is configured to: obtain a key feature of sample data; determine whether the sample data is unknown sample data according to the key feature; when the sample data is unknown sample data, perform annotation processing on the unknown sample data to obtain a first annotation result; send the first annotation result to the center node; The center node is configured to: receive the first annotation result sent by the edge node; when the first annotation result indicates that the annotation on the unknown sample data is successful, perform consistency processing on the first annotation result to obtain a second annotation result; when the first annotation result indicates that the annotation on the unknown sample data fails, perform annotation processing on the unknown sample data to obtain a third annotation result.

2. The system of claim 1, wherein, When determining whether the sample data is unknown sample data according to the key feature, the edge node is configured to: determine whether the sample data is unknown sample data according to an unknown sample model and the key feature.

3. The system of claim 2, wherein, The unknown sample model is generated according to a plurality of key features obtained from a known annotation result set, and the known annotation result set comprises annotation results obtained by the center node before the unknown sample model is generated.

4. The system of claim 2, wherein, The unknown sample model comprises a multi-dimensional coordinate space, and a coordinate distance is obtained according to the key feature of the sample data or a mapping feature of the key feature. Whether the sample data is unknown sample data is determined according to the correlation represented by the coordinate distance.

5. The system of claim 2, wherein, The unknown sample model comprises a neural network model.

6. The system of any one of claims 1-5, wherein when the first annotation result indicates that the annotation on the unknown sample data is successful, the first annotation result comprises an identifier of the sample data, a sample feature of the sample data, and a label determined by the edge node for the sample data; when the first annotation result indicates that the annotation on the unknown sample data fails, the first annotation result comprises an identifier of the sample data and a sample feature of the sample data.

7. The system of claim 6, wherein, The sample feature of the first annotation result comprises the sample data and / or the key feature.

8. The system of any of claims 1-5, wherein, When performing consistency processing on the first annotation result, the center node is configured to cluster a plurality of first annotation results comprising the first annotation result by similarity division to obtain a group corresponding to the first annotation result.

9. The system of claim 8, wherein, The center node clusters the plurality of first annotation results comprising the first annotation result in a non-supervised manner, and the non-supervised manner comprises one or more of K-MEANS and KNN.

10. The system of claim 8, wherein, The center node is further configured to, when labels of the annotation results in the group are inconsistent, perform an integrated decision on labels of all the annotation results in the group to obtain a group label, and obtain the second annotation result according to the group label.

11. The system of claim 10, wherein, The integrated decision comprises a voting method or a weighted voting method.

12. The system of claim 3, wherein, The center node is further configured to generate a new unknown sample model or update the unknown sample data model according to the second annotation result and / or the third annotation result.

13. A sample data labeling method, characterized by, The method is used for an edge node of a sample data labeling system, and the method comprises: obtaining a key feature of sample data; judging whether the sample data is unknown sample data according to the key feature; when the sample data is unknown sample data, performing labeling processing on the unknown sample data to obtain a first labeling result; sending the first labeling result to a center node of the sample data labeling system.

14. The method of claim 13, wherein, The judging whether the sample data is unknown sample data according to the key feature comprises: judging whether the sample data is unknown sample data according to an unknown sample model and the key feature.

15. The method of claim 14, wherein, The unknown sample model is generated according to a plurality of key features obtained from a known labeling result set, and the known labeling result set comprises labeling results obtained by the center node before generating the unknown sample model.

16. The method of claim 14, wherein, The unknown sample model comprises a multi-dimensional coordinate space, and a coordinate distance is obtained according to the key feature of the sample data or a mapping feature of the key feature, and whether the sample data is unknown sample data is determined according to a correlation represented by the coordinate distance.

17. The method of claim 14, wherein, The unknown sample model comprises a neural network model.

18. The method according to any of claims 13-17, characterized by, When the first labeling result indicates that the unknown sample data is successfully labeled, the first labeling result comprises a sample identifier of the sample data, a sample feature of the sample data, and a label determined by the edge node for the sample data. When the first labeling result indicates that the unknown sample data is unsuccessfully labeled, the first labeling result comprises the sample identifier of the sample data and the sample feature of the sample data.

19. The method of claim 18, wherein, The sample feature of the first labeling result comprises the sample data and / or the key feature.

20. A sample data labeling method, characterized by, The method is used for a center node of a sample labeling system, and the method comprises: receiving a first labeling result sent by an edge node in the sample labeling system, the first labeling result being obtained by performing labeling processing on unknown sample data by the edge node; when the first labeling result indicates that the unknown sample data is successfully labeled, performing consistency processing on the first labeling result to obtain a second labeling result; when the labeling result indicates that the unknown sample data is unsuccessfully labeled, performing labeling processing on the unknown sample data to obtain a third labeling result.

21. The method of claim 20, wherein, When performing consistency processing on the first labeling result, a plurality of first registration results comprising the first labeling result are clustered by similarity division to obtain a group corresponding to the first labeling result.

22. The method of claim 21, wherein, The plurality of first registration results comprising the first labeling result are clustered in a non-supervised manner, and the non-supervised manner comprises one or more of K-MEANS and KNN.

23. The method of claim 20, wherein, When labels of the labeling results in the group are inconsistent, a group label is obtained by performing integrated decision on the labels of all the labeling results in the group, and the second labeling result is obtained according to the group label.

24. The method of claim 23, wherein, The integrated decision comprises voting or weighted voting.

25. The method of any of claims 20-24, wherein, The method further comprises generating a new unknown sample model or updating an existing unknown sample data model of the center node according to the second labeling result and / or the third labeling result.

26. An edge node, comprising: The edge node is deployed in a sample data labeling system, and the edge node comprises: An acquisition unit configured to acquire a key feature of sample data; An identification unit configured to identify unknown sample data, and determine whether the sample data is unknown sample data according to the key feature; A labeling unit configured to, when the sample data is unknown sample data, perform labeling processing on the unknown sample data to obtain a first labeling result; A communication unit configured to send the first labeling result to a center node of the sample data labeling system.

27. The edge node of claim 26, wherein, The identification unit is further configured to determine whether the sample data is unknown sample data according to an unknown sample model and the key feature.

28. The edge node of claim 27, wherein, The unknown sample model is generated according to a plurality of key features acquired from a known labeling result set, and the known labeling result set comprises labeling results acquired by the center node before the unknown sample model is generated.

29. The edge node of claim 28, wherein, The unknown sample model comprises a multi-dimensional coordinate space, a coordinate distance is obtained according to the key feature of the sample data or a mapping feature of the key feature, and whether the sample data is unknown sample data is determined according to a correlation represented by the coordinate distance.

30. The edge node of claim 29, wherein, The unknown sample model comprises a neural network model.

31. The edge node of any of claims 26-30, wherein, When the first labeling result indicates that the unknown sample data is successfully labeled, the first labeling result comprises a sample identifier of the sample data, a sample feature of the sample data, and a label determined by the edge node for the sample data. When the first labeling result indicates that the unknown sample data is unsuccessfully labeled, the first labeling result comprises the sample identifier of the sample data and the sample feature of the sample data.

32. The edge node of claim 31, wherein, The sample feature of the first labeling result comprises the sample data and / or the key feature.

33. A central node, characterized by The center node is deployed in a sample data labeling system, and the center node comprises: A communication unit configured to receive a first labeling result sent by an edge node in the sample data labeling system, the first labeling result being obtained by performing labeling processing on unknown sample data by the edge node; A processing unit configured to, when the first labeling result indicates that the unknown sample data is successfully labeled, perform consistency processing on the first labeling result to obtain a second labeling result; A labeling unit configured to, when the labeling result indicates that the unknown sample data is unsuccessfully labeled, perform labeling processing on the unknown sample data to obtain a third labeling result.

34. The central node of claim 33, characterized by When the consistency processing is performed on the first labeling result, the processing unit clusters a plurality of first labeling results comprising the first labeling result by similarity division to obtain a group corresponding to the first labeling result.

35. The central node of claim 34, characterized by The processing unit clusters a plurality of first labeling results comprising the first labeling result in a non-supervised manner, and the non-supervised manner comprises one or more of K-MEANS and KNN.

36. The central node of claim 34, characterized by When labels of the labeling results in the group are inconsistent, the processing unit performs integrated decision on the labels of all the labeling results in the group to obtain a group label, and obtains the second labeling result according to the group label.

37. The central node of claim 36, characterized by The integrated decision includes a voting method or a weighted voting method.

38. The central node of any of claims 33-37, characterized by The center node further includes a model unit configured to generate a new unknown sample model or update an existing unknown sample data model of the center node according to the second annotation result and / or the third annotation result.

Images