Man-machine collaborative labeling method and device, electronic equipment and storage medium

Abstract

The invention discloses a man-machine collaborative labeling method and device, electronic equipment and a storage medium, and relates to the technical field of man-machine collaborative labeling. The method comprises the steps that to-be-labeled data are obtained, multiple pieces of model labeling data obtained after the to-be-labeled data are labeled by a labeling model are obtained, and the labeling model is a model obtained by adopting training samples corresponding to different environment information to conduct pre-training; screening out to-be-corrected annotation data from the multiple pieces of model annotation data according to multiple screening rules; inputting the to-be-corrected annotation data into a candidate annotation model, inputting an output candidate annotation result and the plurality of model annotation data into a category recognition model, and outputting a candidate annotation type; and in response to an editing operation of the target account on the display interface of the client, updating the candidate labeling result and the candidate label to obtain a target labeling result. According to the method, the accuracy of labeling in different environments is ensured, and the manual labeling cost is reduced.

Description

Human-computer collaborative annotation methods, devices, electronic equipment and storage media Technical Field

[0001] This application relates to the field of human-computer collaborative annotation technology, and more specifically, to a human-computer collaborative annotation method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the continuous development of computer technology, the accuracy of product research (such as the tracking accuracy of tracking devices and the accuracy of product positioning in poster advertising) is constantly improving in various fields (such as camera tracking and poster advertising). In the process of improving the accuracy of product research, labeling data such as pictures, videos, and posters, and then conducting further research on the labeled products has become the mainstream method.

[0003] Existing technologies typically employ methods such as fully manual annotation and offline annotation. However, fully manual annotation requires a large amount of manpower and is costly in complex industrial image environments. Offline annotation, on the other hand, is prone to errors and omissions when conditions such as working conditions, lighting, materials, and batches change. The accuracy of both methods is affected by changes in the environment. In other words, existing annotation methods cannot adapt to annotation in dynamic environments. Summary of the Invention

[0004] The main purpose of this application is to provide a method, device, electronic device and storage medium for collaborative manual annotation, so as to solve the problems of high annotation cost and inability to adapt to dynamic environment in traditional annotation schemes, ensure the accuracy of annotation in different environments and reduce manual annotation cost.

[0005] To achieve the above objectives, according to one aspect of the embodiments of this application, a manual collaborative annotation method is proposed, comprising: acquiring data to be annotated, and acquiring multiple model annotation data obtained by annotating the data to be annotated using an annotation model deployed on a production line, wherein the annotation model is a model pre-trained using training samples corresponding to different environmental information; selecting annotation data to be corrected from the multiple model annotation data according to multiple filtering rules; inputting the annotation data to be corrected into a candidate annotation model, outputting candidate annotation results, and inputting the candidate annotation results and multiple model annotation data into a category recognition model, outputting candidate annotation types corresponding to the annotation data to be corrected; and updating the candidate annotation results and candidate annotation types in response to the editing operation of the candidate annotation results and the candidate annotation types on the display interface of the target account on the client, thereby obtaining the target annotation result.

[0006] According to another aspect of the embodiments of this application, a human-machine collaborative annotation device is also provided, comprising: an acquisition unit, configured to acquire data to be annotated, and acquire multiple model annotation data obtained by annotating the data to be annotated using an annotation model deployed on a production line, wherein the annotation model is a model pre-trained using training samples corresponding to different environmental information; a filtering unit, configured to filter out the annotation data to be corrected from the multiple model annotation data according to multiple filtering rules; an input unit, configured to input the annotation data to be corrected into a candidate annotation model, output candidate annotation results, and input the candidate annotation results and multiple model annotation data into a category recognition model, outputting the candidate annotation type corresponding to the annotation data to be corrected; and a response unit, configured to update the candidate annotation results and candidate annotation types in response to the editing operation of the candidate annotation results and the candidate annotation type corresponding to the target account on the client's display interface, thereby obtaining the target annotation result.

[0007] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, which stores computer instructions for causing a computer to perform the above-described manual collaborative annotation method.

[0008] According to another aspect of the embodiments of this application, an electronic device is also provided, the electronic device including: at least one processor, and a memory communicatively connected to the at least one processor; wherein, the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the above-described manual collaborative annotation method.

[0009] In this application, the above-mentioned manual collaborative annotation method, device, electronic device and storage medium solve the problems of high annotation cost and inability to adapt to dynamic environment in traditional annotation schemes, ensure the accuracy of annotation in different environments and reduce manual annotation cost. Attached Figure Description

[0010] The accompanying drawings, which form part of this application, are used to provide a further understanding of the application and to make other features, objects, and advantages of the application more apparent. The illustrative embodiments and descriptions of the accompanying drawings are used to explain the application and do not constitute an undue limitation of the application. In the drawings: Figure 1 is a schematic diagram of the hardware environment of an optional human collaborative annotation method provided by this application; Figure 2 is a flowchart of an optional human collaborative annotation method provided by this application; Figure 3 is a structural schematic diagram of an optional human collaborative annotation device provided by this application; Figure 4 is a structural schematic diagram of an optional electronic device provided by this application. Detailed Implementation

[0011] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0012] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in any order other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. Without conflict, the embodiments and features in the embodiments of this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0013] Industrial vision inspection systems commonly rely on deep learning models, but their performance is hampered by several issues: poor generalization (the model is prone to false positives or false negatives when operating conditions, lighting, materials, and batches change), high annotation costs (industrial images are complex, and manual annotation requires a large workforce), difficulty in updating data (the model cannot automatically determine which new data is worth annotating, resulting in training data that cannot cover more operating conditions), and a lack of automatic feedback loops (a complete automated system is lacking to handle "inference bias → automatic annotation → data update → model optimization"). Traditional methods typically employ offline annotation and manual sampling retraining, which are ill-suited for dynamic environments. Therefore, an intelligent system is needed that can automatically identify model performance biases, automatically generate annotations that can be quickly corrected manually, and automatically drive model iteration and upgrades to improve model generalization capabilities.

[0014] To address at least one of the problems of high annotation costs and inability to adapt to dynamic environments in traditional annotation schemes, this application provides a method for continuous optimization of visual models, including automatic detection of key samples for false positives / false negatives (identified through rules, statistics, or human feedback as described below), automatic generation of candidate annotations (boxes / masks / categories), lightweight human-machine collaborative correction, automatic updating of training datasets, automatic triggering of model training and evaluation, and automatic completion of model version upgrades. This method comprises a human collaborative annotation method, device, electronic device, and storage medium. As an optional embodiment, the aforementioned human collaborative annotation method can be applied, but is not limited to, to the human collaborative annotation system consisting of terminal device 102 and server 104 as shown in Figure 1. As shown in Figure 1, terminal device 102 is connected to server 104 via network 110. Network 110 can include, but is not limited to, wired networks and wireless networks. The wired network includes local area networks (LANs), metropolitan area networks (MANs), and wide area networks (WANs). The wireless network includes Bluetooth, Wi-Fi, and other networks that enable wireless communication. The aforementioned terminal device 102 may include, but is not limited to, at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, handheld computer, MID (Mobile Internet Devices), PAD, desktop computer, smart TV, in-vehicle device, etc.

[0015] The aforementioned terminal device 102 is also equipped with a display 106, a processor 108, and a memory 112. The display 106 can be used to display the manual collaborative annotation process and the target annotation structure, etc. The processor 108 can be used to process the collected data, and the memory 112 can be used to store relevant data. It is understood that when the terminal device 102 receives an annotation request from a user (target account), the terminal device 102 sends the annotation request to the server 104 via the network 110. The server 104 responds to the annotation request and implements the specific manual collaborative annotation process. The server 104 can be a single server, a server cluster composed of multiple servers, or a cloud server. The server 104 includes a database 114 and a processing engine 116. The database 114 can be used to store data, models, etc., and the processing engine 116 is used to process the aforementioned data. The aforementioned manual collaborative annotation system can also perform the following steps: First, the terminal device 102 executes S102 as shown in Figure 1; then, the server 104 executes S104 to S112 as shown in Figure 1, thereby realizing manual collaborative annotation.

[0016] This application relates to the fields of computer vision, machine learning, and industrial automation, and particularly to an automatic annotation method (i.e., human-machine collaborative annotation method) and system for visual models (annotation models) that can be implemented using multi-agent collaboration. The human-machine collaborative annotation system in this application is applicable to scenarios requiring high generalization capabilities, such as industrial defect detection (AOI), image segmentation, and object detection. In the embodiments of this invention, the above-mentioned manual collaborative annotation method solves the problems of high annotation costs and inability to adapt to dynamic environments inherent in traditional annotation schemes, ensuring the accuracy of annotation under different environments and reducing manual annotation costs.

[0017] As an optional implementation, please refer to Figure 2, which shows a flowchart of a manual collaborative annotation method provided in an embodiment of this application. The method includes at least one of the following steps (S202 to S208): S202, acquiring data to be annotated, and acquiring multiple model annotation data obtained by annotating the data to be annotated using an annotation model deployed on the production line, wherein the annotation model is a model pre-trained using training samples corresponding to different environmental information; S204, selecting annotation data to be corrected from multiple model annotation data according to multiple filtering rules; S206, inputting the annotation data to be corrected into a candidate annotation model, outputting candidate annotation results, and inputting the candidate annotation results and multiple model annotation data into a category recognition model, outputting the candidate annotation type corresponding to the annotation data to be corrected; S208, responding to the editing operation of the candidate annotation results and the candidate annotation type corresponding to the target account on the client's display interface, updating the candidate annotation results and the candidate annotation type to obtain the target annotation result.

[0018] The data to be labeled in S202 above can be understood as, but is not limited to, data generated in real time on the production line (also referred to as production line or offline data). This data can be product (e.g., phone case) images / continuous image streams (real-time images, historical images, video screenshots), or numerical data. When the data to be labeled is an image, image stream (video), etc., the method of acquiring the data can be a regular camera, stereo camera, LiDAR, scanner, etc., and this application does not impose any restrictions. The specific image dimension can be two-dimensional, three-dimensional, etc. (the image stream can also be processed to program images or videos). The application is not limited; the annotation model deployed on the production line will perform initial annotation on the data to be annotated, thereby generating annotation results. The annotation process of the annotation model in S202 above can be understood, but is not limited to, performing visual model inference on production line or offline data (the annotation model deployed on the production line is a lightweight model, which can be a single lightweight visual model or a combination of multiple visual models; this application does not limit this). The inference result (also called the prediction result or annotation result) is the aforementioned model annotation data. The output result of the annotation model (also called the visual model) includes not only the model annotation data but also the confidence level. The operation in S202 above can be understood, but is not limited to, relaxing the output standards of the annotation model deployed on the production line. For example, in actual application, the annotation model deployed on the production line will output annotation results with a confidence level greater than 0.5 on the data to be annotated. However, when obtaining multiple model annotation data, this application will modify the confidence level of the annotation model as needed, such as making the annotation model output annotation results with a confidence level greater than 0.3 (i.e., multiple model annotation data), in order to obtain more data than on the actual production line for subsequent analysis and research. The annotation model is a model deployed on the production line, responsible for annotating the data to be annotated (such as defect annotation, also known as defect identification), thereby obtaining the annotation results. The annotation in this application involves acquiring additional data on the production line, such as model annotation data with a confidence level of 0.3-0.5, which is not available on the production line. The annotation in this application can be for errors, defects, etc., in the data to be annotated, or annotation based on annotation requirements. Furthermore, the annotation data / annotation results in this application include both the data to be annotated and the corresponding annotation results.

[0019] The above-mentioned annotation model is a model pre-trained using training samples corresponding to different environmental information. This operation can be understood, but is not limited to, as follows: obtaining the model annotation data obtained by annotating the data to be annotated by the annotation model deployed in production under multiple annotation correction time periods, resulting in multiple sets of model annotation data. Each annotation correction time period corresponds to a set of model annotation data, and the multiple model annotation data obtained in S202 above is a set of model annotation data corresponding to one annotation correction time period. To obtain multiple sets of model annotation data, the annotation models deployed on the production line annotate the data to be annotated under multiple annotation correction time periods. Specifically, this involves: determining any annotation time period as the target correction time period and obtaining the target correction environment information corresponding to the target correction time period; segmenting the target correction time period according to the time period segmentation rules corresponding to the target correction environment information to obtain multiple reference time periods; obtaining the reference environment information corresponding to each reference time period, and obtaining multiple first model annotation data based on each reference time period and the reference environment information corresponding to each reference time period; deduplicating the multiple first model annotation data to obtain multiple model annotation data corresponding to the target correction time period (i.e., a set of model annotation data corresponding to the target correction time period), and then continuing to perform the above operations until a set of model annotation data corresponding to each annotation correction time period is obtained.

[0020] The target correction environment information indicates the weather environment (sunny, cloudy, rainy, etc.), light environment (specific information about light, which can be understood as, but is not limited to, specific light intensity information), and temperature and humidity environment (air temperature and humidity information) corresponding to the target correction time period. The reference environment information indicates the weather environment, light environment, temperature and humidity environment, etc., corresponding to the reference time period. When segmenting the target correction time period, the process specifically includes: constructing an environmental information change curve corresponding to the target correction time period based on the target correction environment information, where the horizontal axis of the environmental information change curve is the target correction time period, and the vertical axis is a comprehensive change curve that integrates the comprehensive change index of the weather environment, light environment, and temperature and humidity environment; obtaining the time period segmentation rules corresponding to the target correction time period, and determining at least one preset curvature change interval corresponding to the target correction time period based on the time period segmentation rules; and segmenting the target correction time period based on at least one preset curvature change interval to obtain multiple reference time periods. If there is only one predetermined curvature change interval, then the target correction time period is divided into time periods according to one curvature change interval. If there are multiple predetermined curvature change intervals, then the target correction time period is divided into time periods according to each of the multiple curvature change intervals to obtain the reference time period corresponding to each curvature change interval.

[0021] For example, when implementing the above-mentioned scheme for obtaining labeled model data on December 1, 2025 (which can be referred to as the overall correction period), multiple labeling correction periods (i.e., dividing the overall correction period into time periods, and the specific division rules can be set as needed, which is not limited in this application) can include time period A: 0:00-5:00 on December 1, 2025, time period B: 5:00-10:00 on December 1, 2025, time period C: 10:00-14:00 on December 1, 2025, time period D: 14:00-16:00 on December 1, 2025, time period E: 16:00-19:00 on December 1, 2025, and time period F: 19:00-24:00 on December 1, 2025. First, time period A can be used as the target correction period, and time period A can be obtained. The target correction environment information corresponding to the segment is obtained. Then, the time segment A is segmented according to the time segmentation rules corresponding to the target correction environment information. For example, the time segment A is divided into 5 reference time segments with an average interval of 1 hour: a, b, c, d, and e. Then, the reference environment information corresponding to each reference time segment is determined, and multiple first model annotation data are obtained according to each reference time segment and the corresponding reference environment information. Multiple first model annotation data can be understood as including multiple first model annotation data corresponding to multiple reference time segments. However, some data are not affected by environmental or time changes. Therefore, even if the time segment and reference environment information are changed, the annotation result will not change. So, there will be duplicate first model annotation data. Then, the duplicate first model annotation data will be deduplicated to obtain multiple model annotation data.

[0022] The above S204 includes: S1, obtaining the rule priorities corresponding to multiple filtering rules; S2, determining the target filtering rule set from multiple filtering rules according to the annotation requirements and multiple rule priorities, wherein the target filtering rule set includes at least one target filtering rule, and the multiple filtering rules include: confidence filtering rules based on the annotation model, statistical filtering rules based on the annotation type, and filtering rules based on human feedback; S3, filtering data from multiple model annotation data according to the target filtering rule set to obtain the annotation data to be corrected. S3 specifically includes: S3-1, when the target selection rule set includes one target selection rule, selecting data from multiple model annotation data according to the target selection rule to obtain the annotation data to be corrected; S3-2, when the target selection rule set includes multiple target selection rules, selecting data from multiple model annotation data using each target selection rule to obtain multiple sets of first selection data; S3-3, integrating the multiple sets of first selection data according to the multiple target selection rules and the rule priority corresponding to each target selection rule to obtain the first annotation data; S3-4, inputting the data to be labeled and the first annotation data into the correction and recognition model, and outputting the annotation data to be corrected.

[0023] The confidence-based filtering rules based on the annotation model can be simply referred to as rules, the statistical filtering rules based on annotation type can be simply referred to as statistics, and the filtering rules based on human feedback can be simply referred to as human feedback recognition. Each filtering rule corresponds to a rule priority. The operation in S2 above can be understood as determining at least one filtering rule from multiple filtering rules based on annotation requirements and multiple rule priorities. The operation in S3 above includes, when the target filtering rule set includes a target filtering rule, filtering the first labeled data from multiple model labeled data according to the target filtering rule, and inputting the data to be labeled and the first labeled data into the correction recognition. The model (pre-trained model) outputs labeled data to be corrected. The correction and recognition model is used to identify mislabeled data and mislabeled data in the first labeled data that are waiting to be corrected. When the target filtering rule set includes multiple filtering rules, it filters from multiple model labeled data according to each target filtering rule to obtain multiple sets of second labeled data, where each target filtering rule corresponds to a set of second labeled data. The multiple sets of second labeled data are integrated according to multiple target filtering rules to obtain the first labeled data. The data to be labeled and the first labeled data are input into the correction and recognition model, and the data to be corrected is output. For example, multiple target filtering rules include the three rules mentioned above; 80 (a group) of second-label data are filtered out using a confidence-based filtering rule (e.g., filtering for confidence levels above 0.6), while 20 second-label data are filtered out for each label type (or tag type, assuming there are 5 label types) using a statistical filtering rule based on label type (with a pre-defined number of filters for each label type). Therefore, the group of second-label data corresponding to the statistical filtering rule based on label type has a total of 100 data. 30 second-label data are filtered out (a group) using a filtering rule based on human feedback. If the priority order is: filtering rule based on human feedback > statistical filtering rule based on label type > confidence-based filtering rule based on label model; then, the rule corresponding to the highest priority is used as the basis. The 80, 100, and 30 second-label data sets are combined and deduplicated (i.e., the second-label data set with the highest priority is retained during the deduplication process). After deduplication, the second-label data set is filtered according to the filtering rules corresponding to the quantity limit (i.e., the statistical filtering rules based on the label type). (i.e., based on the data filtering, it is ensured that there are still 20 second-label data sets corresponding to each label type. If there are fewer than 20, the lowest priority filtering rule is modified, and the second-label data set corresponding to the lowest priority filtering rule is obtained again. For example, the confidence filtering rule based on the labeling model is modified from filtering for confidence levels above 0.6 to filtering for confidence levels above 0.5, or modified to filtering for confidence levels above 0.8, thus obtaining the final second-label data set.

[0024] Confidence-based filtering rules for annotation models include filtering data with a confidence level higher than 0.5, while data with a confidence level lower than 0.5 can be sent back to the annotation model for re-annotation; statistical filtering rules based on annotation type include limiting the amount of data collected for the same fixed defect (i.e., the above annotation type, such as defects in the same location and of the same type), and limiting the number of second annotation data for each annotation type to avoid overfitting of the subsequent model; and filtering rules based on human feedback include filtering second annotation data by human operation on the display interface showing multiple model annotation data. Assuming this application annotates images (i.e., data to be annotated), in addition to annotating the images themselves and retaining the annotation information, this application also annotates the annotation time and corresponding environmental information (i.e., the aforementioned reference environmental information) to avoid some annotation results changing with time or environment. For example, some products have no problems during the day but have problems at night, while some products have no problems in bright sunlight but have problems in dim light at night. That is, the annotation results of the same annotation type are inconsistent under different time or environmental conditions. In this application, the corresponding reference environmental information is obtained for different reference time periods. To a certain extent, the annotation time and annotation environment are changed to perform multiple tests, resulting in multiple sets of different annotation results (i.e., each reference time period and corresponding reference environmental information correspond to a set of annotation results). Finally, multiple sets are processed, such as removing duplicates, as described above, to obtain the final erroneous annotation samples (i.e., data to be corrected), such as false detections (not annotated what should not be annotated, such as annotated defective parts that are not actually defective) and missed detections (not annotated what should be annotated, such as actual defects that are not annotated). The data to be corrected can be manually verified before final determination to determine the accuracy of the data to be corrected. The samples to be corrected can form a bias sample pool. The corrected labeling data output by the recognition model needs to be manually recombined to check its accuracy. If inaccurate, adjustments are made, and the adjusted data is used as the corrected labeling data. The purpose of continuous iterative training and upgrading of the labeling model in this application can be understood, but is not limited to, reducing the confidence of the labeling model in false positives (i.e., minimizing or eliminating falsely labeled results) and increasing the confidence of the labeling model in false negatives (i.e., minimizing or eliminating false negatives). The continuous iterative training and upgrading scheme for the labeling model in this application establishes a self-operating automatic labeling and continuous learning closed loop, enabling the visual model (i.e., the labeling model) to continuously improve itself over time, enhancing its generalization ability.

[0025] The operation in S206 above specifically includes: S1, obtaining multiple reference candidate annotation models, including open vocabulary models and segmentation models, whereby the open vocabulary model is used to generate candidate bounding boxes and the segmentation model is used to generate candidate annotation masks; S2, determining candidate annotation models from the multiple reference candidate annotation models according to annotation requirements, and when the candidate annotation model is an open vocabulary model, determining the annotation results with candidate bounding boxes generated by the open vocabulary model as candidate annotation results; S3, when the candidate annotation model is a segmentation model, determining the annotation results with candidate masks generated by the segmentation model as candidate annotation results; S4, when the candidate annotation models are an open vocabulary model and a segmentation model, using the annotation results with candidate bounding boxes generated by the open vocabulary model as input to the segmentation model, and determining the annotation results with candidate masks generated by the segmentation model as candidate annotation results.

[0026] Open vocabulary models can employ various methods such as Grounding DINO and OWL-ViT. These models are used to generate candidate bounding boxes (which can be understood as: after identifying the target in the data to be labeled, candidate bounding boxes are used to enclose the target). Open vocabulary models can generate candidate bounding boxes or candidate masks. A model that generates candidate masks will always generate candidate bounding boxes, but a model that generates candidate bounding boxes may not necessarily generate candidate masks. In this application, when using an open vocabulary model as the candidate labeling model, the open vocabulary model that generates candidate masks is used. When using both an open vocabulary model and a segmentation model as candidate labeling models, the open vocabulary model that generates candidate bounding boxes is used. The advantage of using an open vocabulary model is that it allows direct text input, enabling users to input relevant content and achieve semi-automated labeling. Segmentation models can employ various methods such as SAM2 and HQ-SAM. Segmentation models can accurately separate the product from the background and generate candidate masks. The uses of open-vocabulary models and segmentation models can be understood through the following example: Suppose we have collected many images of phone cases, and the annotation is to mark the defects of the phone cases in the images. If the defect is a circle, the open-vocabulary model will outline the circumscribed rectangle of this circle. The segmentation model can directly outline the circle based on this circumscribed rectangle or directly on the original image. The output of each model is a candidate bounding box / candidate mask (i.e., candidate annotation result) plus the corresponding confidence score. The category recognition model is used to determine the corresponding annotation type (candidate annotation type, also called candidate label) for each annotation result. The annotation type can be classified according to the condition and severity of the defect. For example, the annotation types for phone cases include severe cracks, minor cracks, severe chips, and normal. For example, if the above annotation is an image (data to be annotated) containing many dogs, the annotation model, open vocabulary model, segmentation model, etc., will perform inference calculations on each pixel in the image, calculating the probability / likelihood of each pixel being a cat or a dog. If the probability of a certain area being a dog is very high, then that area will be bounded (i.e., an annotation result / candidate annotation result will be generated), and the corresponding annotation type / candidate annotation type will be either cat or dog.

[0027] The above-mentioned solutions S1-S4 can also be implemented by first using an open vocabulary model to output candidate boxes (outlining defects, i.e., the above candidate annotation results), and then considering the model requirements of the annotation model deployed on the production line. If segmentation is required, the segmentation model is used to generate candidate masks for the candidate boxes, and then the category recognition model generates candidate labels based on the candidate masks. If segmentation is not required, the category recognition model is directly used to generate candidate labels based on the candidate boxes (i.e., the above candidate annotation types).

[0028] The operation in S208 above can be understood, but is not limited to, the user using the target account to perform corresponding editing operations on the candidate annotation results and the first annotation update results on the display interface. Based on this editing operation, the server performs a second annotation update on the candidate annotation results and candidate labels to obtain the final target annotation results. This step is actually a lightweight annotation process involving human-computer collaboration. This application displays automatic candidate annotations through the front-end system, while the user performs minimal editing actions. The corresponding editing operations can include: deleting false positive candidates (which can be understood, but is not limited to, places / areas that should not have been annotated have been annotated), supplementing missing areas (which can be understood, but is not limited to, places / areas that should have been annotated have not been annotated), modifying the category (i.e., modifying the candidate annotation type), correcting the bounding box or mask (i.e., correcting the candidate bounding box or candidate mask), etc. The above false positives and missing positives are all erroneous annotations (i.e., the above annotations to be corrected).

[0029] The above-described embodiments of this application, employing the aforementioned collaborative manual annotation method, solve the problems of high annotation costs and inability to adapt to dynamic environments inherent in traditional annotation schemes. This ensures the accuracy of annotations under different conditions and reduces manual annotation costs. Furthermore, by using this human-computer collaborative annotation method, manual work is eliminated from the need for drawing and annotation from scratch, significantly reducing the workload.

[0030] In this application, after the target annotation result (including the annotation result and annotation type) is obtained after the annotation is completed, the target annotation result, the sample to be corrected corresponding to the target annotation result, and the non-sample to be corrected in the data to be labeled will be used to iteratively train the annotation model to obtain a new annotation model. Then, the user can choose whether to update the production line with this new annotation model to replace the old annotation model.

[0031] The overall steps for model updates include: S1, obtaining model annotation data from the annotation models deployed on the production line during multiple annotation correction time periods, resulting in multiple sets of model annotation data, with each annotation correction time period corresponding to one set of model annotation data; S2, determining the candidate annotation type corresponding to the annotation data to be corrected in each set of model annotation data; S3, revising the annotation data to be corrected in each set of model annotation data according to the candidate annotation type, obtaining multiple sets of reference model annotation data, and training the annotation model based on each set of reference model annotation data to obtain multiple reference annotation models; S4, based on the upgrade confirmation operation corresponding to the model update, replacing the reference annotation model corresponding to the upgrade confirmation operation with the annotation model and deploying it to the production line.

[0032] In S2 above, the annotations of the data to be corrected in each set of model annotation data are revised according to the candidate annotation type. The operation of obtaining multiple sets of reference model annotation data can be understood, but is not limited to, as updating the candidate annotation results and candidate annotation types in the above-mentioned editing operation of the candidate annotation results and candidate annotation types in response to the target account's display interface on the client, obtaining the target annotation results, and then replacing the data to be corrected in the multiple sets of model annotation data with the target annotation results to obtain multiple sets of reference model annotation data. In S3 above, the annotation model is trained according to each set of reference model annotation data to obtain multiple reference annotation models, including: S3-1, dividing the annotation model into model levels according to the model functions of the annotation model to obtain multiple model levels; S3-2, selecting multiple reference model levels from the multiple model levels according to the sub-functions of each model level; S3-3, iteratively training the multiple reference model levels using the reference model annotation data to obtain multiple target model levels; S3-4, updating the model architecture of the annotation model according to the multiple target model levels to obtain reference annotation models. In the above S3-3 to S3-4, when iteratively training the annotation model (multiple reference model levels), this application will also use a sample difficulty evaluation model to determine the sample difficulty value corresponding to each reference model annotation data; using multiple reference model annotation data, iteratively training the annotation model (multiple reference model levels) according to the sample difficulty value corresponding to each reference model annotation data to obtain the reference annotation model corresponding to each set of reference model annotation data.

[0033] The aforementioned sample difficulty assessment can be understood, but is not limited to, as the annotation results of some samples (i.e., the reference model's labeled data, including the data and the corresponding annotation results, with the sample being the data itself) changing over time or due to environmental changes. These samples are considered more difficult; for example, the annotation model might be able to annotate a defect, but not others. The sample difficulty assessment value in this application is a comprehensive evaluation based on multiple dimensions, including the confidence level of the annotation model's output, confidence stability, annotation error, uncertainty of the annotation category, degree of operational condition deviation (degree of annotation error), and frequency of manual correction. For highly difficult samples, the weight of that sample will be increased and / or the number of training rounds using that sample will be increased in subsequent model training. In other words, this application will sample highly difficult samples and train the annotation model for several more rounds. The aforementioned operational condition information records (including illumination, equipment number, batch number, etc.) are metadata tags added to the data to be labeled when the annotation model performs annotation.

[0034] This application updates the reference model annotation data periodically, that is, it updates the reference model annotation data regularly (e.g., daily, three days, or five days) according to multiple annotation correction time periods. When training the labeled model according to each set of reference model annotation data, this application adds the corrected annotations to the formal training set to obtain the reference model annotation data. Then, the model training operation is automatically driven by the Dataset Manager deployed in the system. Before model training, this application also performs data deduplication on the reference model annotation data (e.g., deleting duplicate or particularly similar images). The purpose of deduplication is to avoid the labeled model being "misled" by a large number of duplicate samples (i.e., reference model annotation data), which would lead to a decrease in model generalization ability, a decrease in training efficiency, or an increase in training bias.

[0035] Furthermore, this application also merges the original annotations (i.e., replaces the annotation data to be corrected (i.e., data with erroneous annotation results) in multiple model annotation data with new annotation information (i.e., the aforementioned target annotation results)) to form a new model annotation data group. During the merging of original annotations, this application re-merges them in different ways according to priority, such as whether the target annotation results are current or historical, whether they have been manually confirmed, etc. That is, this application merges based on the data attributes of the target annotation results and the corresponding priorities of those attributes. Data attributes include manual (target annotation results obtained only through manual annotation), automatic annotation model + manual confirmation (target annotation results obtained through collaboration between annotation model and manual confirmation), historical manual annotation (historical manual annotation results), and candidate annotations from the automatic annotation model (target annotation results obtained only through annotation model annotation). Based on the preset priorities of different data attributes corresponding to the annotation requirements, the highest priority merging method is selected from multiple merging methods for merging.

[0036] This application, when training the labeled model based on the labeled data of the reference model to obtain the reference labeled model, automatically divides the reference labeled model data into training set, validation set, and test set according to a ratio (in this application, the ratio is preferably 7 (training set): 2 (validation set): 1 (test set)). The training set is used for model learning, the validation set is used to examine the training results and does not participate in parameter tuning, and the test set is used for the final model evaluation and also does not participate in model parameter tuning. This application automatically divides the training set, validation set (the training set and validation set are used in the training phase, but the validation set is used to determine whether the current training meets the requirements and does not update the training parameters, while the test set is unrelated to the training phase) and test set, ensuring representative samples for different operating conditions, thereby guaranteeing the stability of the model optimization process and the reliable evaluation of the model's generalization ability. In this application, when the labeled model is deployed on the production line for actual use, the labeled model will also be automatically trained and optimized. That is, the production line system will automatically trigger the training task. The model training in this application adopts the following strategy: hard sample reinforcement: false positive samples are treated as negative samples and false negative samples as positive samples to strengthen learning (the weight of false positive samples, false negative samples, etc. in Nanyang is increased / or the number of training rounds in Nanyang is increased). In this application, the mislabeled samples (false positive samples, false negative samples, etc.) are regarded as high-weight samples. The model's learning of hard samples is strengthened through sample resampling, loss function weighting, and Focal Loss determination mechanism, so that the robustness of the model under certain working conditions is significantly improved.

[0037] Adaptive Enhancement: For samples susceptible to operating conditions (i.e., for the same data to be labeled, the first model labeled data is different for different reference environment information, and this data to be labeled is the sample susceptible to operating conditions), noise, brightness, and blur enhancement are used (i.e., actively simulating more severe imaging conditions for these samples susceptible to operating conditions, and using the processed samples as the final training samples (i.e., reference model labeled data) for model training, thereby forcibly improving the recognition accuracy and robustness of the model in complex environments. For example, if the model is sensitive to brightness, this application can use one image in ten images corresponding to different brightness levels for model training). Specifically, enhancement strategies are dynamically selected based on sample operating condition information (brightness, batch, equipment number, etc.). For samples susceptible to changes in operating conditions, at least one operation is selected from brightness perturbation, noise injection, blurring, color gamut shift, etc., to enhance the sample and improve the robustness of the model in complex environments.

[0038] The hierarchical training strategy (in this application, the entire model is not retrained, but a portion of the model is fine-tuned): Fine-tunable parts include the backbone (used to extract basic image features such as edges and textures) or the detection head (used for classification and localization based on features). For example, if an annotation is made at a specified location (the location of the annotation in the data to be annotated), but the annotation type is incorrect, this application will not train the backbone network, but only the detection head. That is, the scheme in this application will fine-tune the model layer corresponding to the incorrect annotation type based on the incorrect annotation type in the data to be corrected (including missing annotations, mis-annotations, incorrect annotation types, etc.). After training, this application will generate a new version of the annotation model. The model version includes: a readable version number (e.g., v1, v2, v3) and a hash value calculated based on the model file content (e.g., SHA-256). The hash value serves as a unique fingerprint identifier for the model content and can be used for integrity verification, rollback management, and traceability records, ensuring the uniqueness and immutability of the model version.

[0039] In this application, after training the annotation model based on the reference model's labeled data, an automatic evaluation and version decision (Gatekeeper Agent) is employed to automatically evaluate the reference annotation model. The specific evaluation process includes: global mAP (mean Average Precision, used to evaluate the predicted bounding box and mask; it can be understood as the accuracy of the bounding box in identifying defects; for example, if only a small part of the defect is bounded, the global mAP will be lower), recall rate for key unlabeled categories (such as the severity of key defect categories), mislabeling (false positive / false positive) reduction rate, missed labeling (missed positive / missed negative) reduction rate, and performance on the new validation set (the generalization ability of the reference annotation model on newly added scene data, such as originally detecting vehicle frames, but later detecting bearings). This application calculates global mAP, key defect category recall rate, missed positive reduction rate, and false positive reduction rate on the standard validation set and the new validation set, and comprehensively evaluates the performance of the new model based on a weighted scoring mechanism. If the comprehensive score exceeds the upgrade threshold, the model is marked as a candidate version. If the model performance meets the upgrade conditions, the model is marked as a candidate version. This application supports two modes: "one model covering multiple operating conditions" and "multiple models deployed in different scenarios." After the model training is completed and candidate versions (referencing the labeled model) are generated, the system automatically determines the applicability of the model based on the performance of the validation sets for each operating condition. When the candidate model significantly outperforms the old version in all operating conditions, the new model can replace the old model based on the user's "one-click confirmation" operation. If the candidate model only improves performance in specific operating conditions (such as different product scenarios like phone cases, vehicle frames, and bearings), a condition-specific model is automatically generated to achieve on-demand deployment. Finally, the version release is completed by manual "one-click confirmation," ensuring controllability and traceability. In this application, however, manual "one-click confirmation" is all that is needed to release the new model.

[0040] The above S4 specifically includes: S4-1, displaying the model editing interface, which includes multiple reference labeled models and the model comprehensive evaluation index corresponding to each reference labeled model. Each reference labeled model marked as a candidate labeled model corresponds to a triggerable model confirmation area, which allows the target account to initiate an upgrade confirmation operation; S4-2, based on the target account's upgrade confirmation operation for the candidate labeled model, replacing the labeled model with the candidate labeled model and deploying it to the production line. Specifically, S4-2 is: based on the target account's upgrade confirmation operation for any candidate labeled model, replacing the labeled model with the candidate labeled model corresponding to the upgrade confirmation operation and deploying it to the production line.

[0041] In this application, the model can be released and upgraded based on the user's "one-click confirmation" operation, realizing the release and closed-loop restart of the model. After the new labeled model is deployed, the labeling results (or inference results) continue to be detected by the system. If false positives / false negatives or other erroneous labeled samples are generated again, these samples will automatically enter the bias sample pool. Then, the process of labeling and training the samples to be corrected in the bias sample pool will continue to be executed, realizing a complete automatic closed loop.

[0042] After obtaining the reference annotation model corresponding to each set of reference model annotation data, this application further includes: S1, performing a comprehensive evaluation of the reference annotation model to obtain a comprehensive evaluation index; S2, marking the reference annotation model as a candidate annotation model if the comprehensive evaluation index is greater than or equal to a preset index; S3, marking the reference annotation model as an unqualified model if the comprehensive evaluation index is less than a preset index.

[0043] As an optional implementation, before displaying the model editing interface, the method further includes: S1, performing a comprehensive evaluation of multiple reference labeled models to obtain multiple comprehensive evaluation indices; S2, marking reference labeled models with comprehensive evaluation indices greater than or equal to preset indices as candidate labeled models; and S3, marking reference labeled models with comprehensive evaluation indices less than preset indices as unqualified models.

[0044] The system architecture of the human-machine collaborative annotation system shown in Figure 1 of this application is a multi-agent collaborative system architecture. The human-machine collaborative annotation system consists of several agents, including but not limited to: an inference (or annotation) agent (which executes model inference results and sends the results to the bias detection module), a bias detection agent (which automatically identifies false positives and false negatives), an automatic annotation agent (which generates candidate annotations using an open vocabulary model and a segmentation model), a manual correction agent (annotation UI) (which provides a human-machine collaborative interface for quick confirmation), and a data management agent (which manages the updating, cleaning, and stratification of the training dataset). The updating of the training dataset is automatically managed by the human-machine collaborative annotation system. When the number of new samples reaches a threshold, the cycle is reached, or a work condition deviation is detected, the system updates the data. The training set is automatically updated when there are anomalies or when the performance of the labeled model on the production line deteriorates; cleaning is an automatic data cleaning process performed by the human-machine collaborative annotation system (including: deleting duplicate or highly similar data / images; removing low-quality data / images such as blurry, overexposed, or damaged data / images; verifying the validity of the annotations; adjusting the distribution of categories and working conditions to avoid sample bias); the human-machine collaborative annotation system performs multi-dimensional stratification on the training data, including: stratification by sample difficulty (Easy / Medium / Hard), stratification by working condition (lighting, batch, equipment), stratification by product category (phone case / vehicle frame / bearing, etc.), stratification by data / image quality, stratification by annotation reliability (whether the annotation results are easily affected by factors such as time and working conditions), and stratification by time period (i.e., the reference time period corresponding to the training data), etc.

[0045] The human-machine collaborative annotation system in this application also includes a training density agent: which automatically executes the training process, manages GPU resources (resource allocation and control: multiple training tasks are triggered simultaneously, and the priority is automatically determined) and training parameters (parameter configuration: reads the parameters required for this training from a preset file); an evaluation and version decision agent: which automatically evaluates model performance and determines whether to upgrade; and a version deployment agent: which realizes model registration, release, and rollback. The multi-agent system in this application achieves end-to-end automation through messaging systems or API collaboration.

[0046] The overall process of this application includes: data (data to be labeled) input → inference agent (labeling the model) → bias detection (identifying erroneous labeled samples such as false positives / false negatives) → automatic labeling agent (using open vocabulary models, segmentation models, etc.) → manual correction → data management agent (updating the training set) → training density agent (starting retraining) → evaluation agent (determining whether generalization ability is improved, i.e., whether the model is labeled as a candidate model) → model deployment agent (deploying the model to the production line) → next inference (labeling) → repeating the closed loop.

[0047] The above-described embodiments of this application solve the problems of high annotation costs and inability to adapt to dynamic environments in traditional annotation schemes, ensuring the accuracy of annotation in different environments and reducing manual annotation costs.

[0048] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0049] According to another aspect of the present invention, a collaborative manual annotation apparatus for implementing the above-described collaborative manual annotation method is also provided. As shown in FIG3, the apparatus includes: an acquisition unit 302, used to acquire data to be annotated and to acquire multiple model annotation data obtained by annotating the data to be annotated using annotation models deployed on the production line, wherein the annotation models are models pre-trained using training samples corresponding to different environmental information; a filtering unit 304, used to filter out the annotation data to be corrected from the multiple model annotation data according to multiple filtering rules; an input unit 306, used to input the annotation data to be corrected into a candidate annotation model, output candidate annotation results, and input the candidate annotation results and multiple model annotation data into a category recognition model, outputting the candidate annotation type corresponding to the annotation data to be corrected; and a response unit 308, used to update the candidate annotation results and candidate annotation types in response to the editing operation of the target account on the client's display interface, thereby obtaining the target annotation result.

[0050] The specific methods of execution of each unit in the above device embodiments have been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0051] According to another aspect of the present invention, an electronic device for implementing a manual collaborative annotation method is also provided. This electronic device may be a terminal device or a server as shown in FIG. 4. This embodiment uses the electronic device as a terminal device as an example. As shown in FIG. 4, the electronic device includes: at least one processor 404; and a memory 402 communicatively connected to the at least one processor 404; wherein the memory 402 stores a computer program executable by the at least one processor 404, the computer program being executed by the at least one processor 404 to cause the at least one processor 404 to perform the steps in any of the above method embodiments. The electronic device may be located in at least one of a plurality of network devices in a computer network. The processor may be configured to execute the manual collaborative annotation method via the computer program.

[0052] Optionally, those skilled in the art will understand that the structure shown in Figure 4 is merely illustrative, and the electronic device may also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile internet device (MID), a PAD, or other terminal device. Figure 4 does not limit the structure of the aforementioned electronic device. For example, the electronic device may also include more or fewer components (such as a network interface) or have a different configuration than shown in Figure 4.

[0053] The memory 402 can be used to store software programs and modules, such as the program instructions / modules corresponding to the collaborative annotation method and apparatus in this embodiment of the invention. The processor 404 executes various functional applications and data processing by running the software programs and modules stored in the memory 402, thereby realizing the aforementioned collaborative annotation method. The memory 402 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 402 may further include memory remotely located relative to the processor 404, and these remote memories can be connected to the terminal via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. As an example, as shown in FIG4, the memory 402 may include, but is not limited to, the units in the aforementioned collaborative annotation apparatus. In addition, it may include, but is not limited to, other module units in the aforementioned collaborative annotation apparatus, which will not be described in detail in this example.

[0054] Optionally, the aforementioned transmission device 406 is used to receive or send data via a network. Specific examples of the network may include wired and wireless networks. In one example, the transmission device 406 includes a Network Interface Controller (NIC), which can be connected to other network devices and a router via a network cable to communicate with the Internet or a local area network. In another example, the transmission device 406 is a radio frequency (RF) module used for wireless communication with the Internet. Furthermore, the aforementioned electronic device also includes a display 408 and a connection bus 410 for connecting the various module components within the electronic device.

[0055] Obviously, those skilled in the art should understand that the various units or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0056] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for collaborative manual annotation, characterized in that, include: Obtain the data to be labeled, and obtain multiple model labeled data after the labeled data is labeled by the labeled model deployed on the production line. The labeled model is a model pre-trained using training samples corresponding to different environmental information. Select the labeled data to be corrected from the multiple model labeled data according to multiple screening rules. The annotation data to be corrected is input into the candidate annotation model, and the candidate annotation results are output. The candidate annotation results and multiple model annotation data are input into the category recognition model, and the candidate annotation type corresponding to the annotation data to be corrected is output. In response to the target account's editing operations on the client's display interface regarding the candidate annotation results and the corresponding candidate annotation types, the candidate annotation results and the candidate annotation types are updated to obtain the target annotation results.

2. The method according to claim 1, characterized in that, The process of selecting labeled data to be corrected from multiple model labeled data according to multiple filtering rules includes: obtaining the rule priorities corresponding to the multiple filtering rules; determining a target filtering rule set from the multiple filtering rules according to the annotation requirements and the multiple rule priorities, wherein the target filtering rule set includes at least one target filtering rule, and the multiple filtering rules include: a confidence filtering rule based on the labeled model, a statistical filtering rule based on the annotation type, and a filtering rule based on human feedback; and filtering data from the multiple model labeled data according to the target filtering rule set to obtain the labeled data to be corrected.

3. The method according to claim 2, characterized in that, The process of filtering data from multiple model annotation data according to the target filtering rule set to obtain the annotation data to be corrected includes: when the target filtering rule set includes one target filtering rule, filtering data from multiple model annotation data according to the target filtering rule to obtain the annotation data to be corrected; when the target filtering rule set includes multiple target filtering rules, filtering data from multiple model annotation data using each of the target filtering rules to obtain multiple sets of first filtered data; integrating the multiple sets of first filtered data according to the multiple target filtering rules and the rule priority corresponding to each target filtering rule to obtain first labeled data; and inputting the annotation data to be corrected and the first labeled data into the correction and recognition model to output the annotation data to be corrected.

4. The method according to claim 1, characterized in that, The process of inputting the annotation data to be corrected into a candidate annotation model and outputting candidate annotation results includes: obtaining multiple reference candidate annotation models, including an open vocabulary model and a segmentation model, wherein the open vocabulary model is used to generate candidate annotation boxes and the segmentation model is used to generate candidate annotation masks; determining a candidate annotation model from the multiple reference candidate annotation models according to the annotation requirements, and if the candidate annotation model is the open vocabulary model, determining the annotation result with candidate boxes generated by the open vocabulary model as the candidate annotation result; if the candidate annotation model is a segmentation model, determining the annotation result with candidate masks generated by the segmentation model as the candidate annotation result; if the candidate annotation model is both an open vocabulary model and a segmentation model, using the annotation result with candidate boxes generated by the open vocabulary model as input to the segmentation model, and determining the annotation result with candidate masks generated by the segmentation model as the candidate annotation result.

5. The method according to claim 1, characterized in that, The process of obtaining multiple model annotation data after the annotation model deployed on the production line annotates the data to be annotated includes: obtaining target correction time period and target correction environment information, and segmenting the target correction time period according to the time period segmentation rule corresponding to the target correction environment information to obtain multiple reference time periods; obtaining reference environment information corresponding to each reference time period, and obtaining multiple first model annotation data according to each reference time period and the reference environment information corresponding to each reference time period; and deduplicating the multiple first model annotation data to obtain multiple model annotation data.

6. The method according to claim 5, characterized in that, After obtaining the target annotation result, the method further includes: replacing the annotation data to be corrected in the multiple model annotation data with the target annotation result to obtain multiple reference model annotation data; using a sample difficulty assessment model to determine the sample difficulty value corresponding to each reference model annotation data; using the multiple reference model annotation data, iteratively training the annotation model according to the sample difficulty value corresponding to each reference model annotation data to obtain a reference annotation model.

7. The method according to claim 1, characterized in that, After obtaining the reference annotation model, the process further includes: performing a comprehensive evaluation of the reference annotation model to obtain a comprehensive evaluation index; marking the reference annotation model as a candidate annotation model if the comprehensive evaluation index is greater than or equal to a preset index; marking the reference annotation model as an unqualified model if the comprehensive evaluation index is less than the preset index; and deploying the candidate annotation model to the production line to replace the original annotation model based on the upgrade confirmation operation of the target account.

8. A collaborative labeling device, characterized in that, include: The acquisition unit is used to acquire the data to be labeled and to acquire multiple model labeled data obtained by the labeling model deployed on the production line after labeling the data to be labeled. The labeling model is a model pre-trained using training samples corresponding to different environmental information. The filtering unit is used to filter out the labeled data to be corrected from the multiple model labeled data according to multiple filtering rules. The input unit is used to input the annotation data to be corrected into the candidate annotation model, output the candidate annotation result, and input the candidate annotation result and multiple model annotation data into the category recognition model, and output the candidate annotation type corresponding to the annotation data to be corrected. The response unit is used to respond to the editing operations of the target account on the client's display interface on the candidate annotation results and the corresponding candidate annotation types, update the candidate annotation results and the candidate annotation types, and obtain the target annotation results.

9. An electronic device, characterized in that, The electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the at least one processor to perform the manual collaborative annotation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the human-computer collaborative annotation method according to any one of claims 1-7.

Images