Data annotation quality control method, device and equipment based on large model and medium

Abstract

The invention discloses a data annotation quality control method and device based on a large model, equipment and a medium, and relates to the field of big data, and the method comprises the steps: obtaining annotation behavior data and annotation result data in a process of carrying out data annotation on original data through a data annotation tool, analyzing the annotation behavior data, the annotation result data and the original data by using a preset multi-modal large model to obtain a first analysis result; determining an annotation error source by using the first analysis result, analyzing historical annotation data of a target annotation person when the annotation error source is the target annotation person, and triggering a corresponding control measure based on an obtained second analysis result; and when the first analysis result shows that the target image data exists in the labeling result data, assisting labeling personnel to perform data re-labeling on the target image data by using a preset image segmentation model. Therefore, the efficiency of quality control in the data labeling process can be improved.

Description

Technical Field

[0001] The present invention relates to the field of big data, and in particular to a method, device, equipment and medium for data annotation quality control based on a large model. Background Art

[0002] In the wave of digitalization, data labeling is the foundation of the development of artificial intelligence. With the widespread application of artificial intelligence in many fields such as autonomous driving and medical diagnosis, the demand for high-quality labeled data has increased sharply, which has led to the emergence of corresponding quality control solutions.

[0003] Currently, on the one hand, traditional rule-based annotation quality control relies on pre-set standards and is difficult to cope with complex and changing data scenarios. For example, in natural language processing annotation, semantic ambiguity and the diversity of contexts make it difficult for rules to cover all situations, leading to annotation errors. The manual sampling and review method is not only inefficient, but the limitations of the sampling samples make it difficult to fully reflect the overall annotation quality. At the same time, the existing simple verification tools have single functions and lack the ability to deeply analyze and intelligently judge the annotated data. When faced with the annotation of unstructured data such as images and videos, they are unable to accurately identify potential errors.

[0004] Therefore, how to improve the efficiency of quality control in the data labeling process is a technical problem that needs to be solved urgently. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a data annotation quality control method, device, equipment and medium based on a large model, which can improve the efficiency of quality control in the data annotation process. The specific scheme is as follows:

[0006] In a first aspect, this application provides a data annotation quality control method based on a large model, comprising:

[0007] In the process of annotating the original data using the data annotation tool, annotated behavior data and annotated result data are obtained, and the annotated behavior data, the annotated result data, and the original data are analyzed using a preset multimodal large model to obtain a first analysis result;

[0008] Determine the source of the labeling error using the first analysis result. If the source of the labeling error is the target labeler, analyze the historical labeling data of the target labeler and trigger corresponding control measures based on the obtained second analysis result.

[0009] When the first analysis result indicates that target image data exists in the annotation result data, a preset image segmentation model is used to assist the annotation personnel in re-annotating the target image data; the target image data is image data whose data features meet preset feature fuzzy conditions and has completed data annotation.

[0010] Optionally, in the process of annotating the original data using the data annotation tool, obtaining the annotation behavior data and the annotation result data includes:

[0011] Embedding a data collection plug-in into the data annotation tool;

[0012] In the process of data labeling the original data, the data acquisition plug-in is used to obtain a first timestamp for starting data labeling, a second timestamp for ending data labeling, a data labeling trajectory, and a data labeling operation to determine corresponding labeling behavior data;

[0013] The original data after data annotation, the coordinate position of the annotation frame and the annotation category are obtained to determine the corresponding annotation result data.

[0014] Optionally, the analyzing the labeling behavior data, the labeling result data, and the original data using a preset multimodal large model to obtain a first analysis result includes:

[0015] Analyzing the labeled behavior data using a preset multimodal large model to obtain behavioral abnormality indicators;

[0016] Aligning the labeled result data with the original data, extracting feature differences between the labeled result data and the original data using the preset multimodal large model, and determining the degree of deviation of the labeled result based on the feature differences and boundary data of the labeling box generated by the data labeling tool;

[0017] Cross-validating the abnormal behavior index and the deviation of the annotation result; when the abnormal behavior index and the deviation of the annotation result satisfy a preset positive correlation constraint condition, correcting the deviation of the annotation result using a preset dynamic compensation algorithm to obtain a corrected deviation;

[0018] Generating a feature fuzzy score based on the raw data and the preset multimodal macro model;

[0019] A first analysis result is generated based on the corrected deviation degree and the feature fuzziness score.

[0020] Optionally, determining a source of a labeling error by using the first analysis result includes:

[0021] If the feature fuzziness score in the first analysis result is greater than a preset first threshold, it indicates that there is data ambiguity in the original data, and the original data is determined to be a source of labeling error;

[0022] If the corrected deviation in the first analysis result is greater than a preset second threshold and the feature fuzziness score is less than the preset first threshold, it indicates that there is a tool defect in the data annotation tool, and the data annotation tool is determined to be the source of the annotation error;

[0023] If the corrected deviation in the first analysis result is greater than a preset third threshold and the data annotation tool does not have the tool defect, the target annotation person is determined as the source of the annotation error.

[0024] Optionally, analyzing the historical annotation data of the target annotator and triggering corresponding control measures based on the obtained second analysis result includes:

[0025] Determining historical annotation data of the target annotator, and determining the annotation trajectory characteristics of the target annotator, the misuse frequency of the data annotation tool, and the number of annotation adjustments based on the historical annotation data to determine corresponding annotation feature information;

[0026] Inputting the labeled feature information into the preset multimodal large model, and determining an output result of the preset multimodal large model as a second analysis result;

[0027] Trigger corresponding control measures based on the second analysis result.

[0028] Optionally, the using of a preset image segmentation model to assist annotators in re-labeling the target image data includes:

[0029] Determining a target area based on the target image data using a preset image segmentation model, and using the target area to determine a segmentation mask and a category prediction result; the category prediction result is a result of the preset image segmentation model predicting the category of the object in the target area;

[0030] Based on the segmentation mask and the category prediction result, an annotator is assisted to re-annotate the target image data to obtain a target annotated image.

[0031] In a second aspect, the present application provides a data annotation quality control device based on a large model, comprising:

[0032] a data analysis module, configured to obtain annotation behavior data and annotation result data during the process of annotating the original data using a data annotation tool, and analyze the annotation behavior data, the annotation result data, and the original data using a preset multimodal large model to obtain a first analysis result;

[0033] a measure triggering module, configured to determine the source of the labeling error using the first analysis result, and when the source of the labeling error is a target labeler, analyze the historical labeling data of the target labeler and trigger corresponding control measures based on the obtained second analysis result;

[0034] A data re-labeling module is used to assist the labeling personnel in re-labeling the target image data using a preset image segmentation model when the first analysis result indicates that the target image data exists in the labeling result data; the target image data is image data whose data features meet the preset feature fuzzy conditions and have completed data labeling.

[0035] In a third aspect, the present application provides an electronic device, comprising:

[0036] Memory, used to store computer programs;

[0037] A processor is used to execute the computer program to implement the aforementioned large model-based data annotation quality control method.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned large-model-based data annotation quality control method is implemented.

[0039] In this application, in the process of using a data annotation tool to annotate the original data, the annotation behavior data and the annotation result data are obtained, and the annotation behavior data, the annotation result data and the original data are analyzed using a preset multimodal large model to obtain a first analysis result; the source of the annotation error is determined using the first analysis result. When the source of the annotation error is the target annotation personnel, the historical annotation data of the target annotation personnel is analyzed, and corresponding control measures are triggered based on the obtained second analysis result; when the first analysis result shows that there is target image data in the annotation result data, the preset image segmentation model is used to assist the annotation personnel in re-annotating the target image data; the target image data is the image data whose data features meet the preset feature fuzzy conditions and has completed data annotation. As can be seen from the above, in the data annotation stage, the present application collects the annotation behavior data and the annotation result data, and analyzes the annotation behavior data, the annotation result data and the original data using a preset multimodal large model to generate a first analysis result. Then, based on the first analysis result, the source of the annotation error is located. When it is determined that the source of the annotation error is the target annotation personnel, the historical annotation data of the target annotation personnel is called for analysis, and the hierarchical control measures are triggered based on the generated second analysis result. If the analysis finds that the target image data in the annotation results has excessive feature ambiguity, the preset image segmentation model will be used to assist the annotator in completing the data re-annotation. In this way, this application can improve the efficiency of quality control during the data annotation process, thereby achieving accurate tracing and repair of annotation errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0041] Figure 1 This is a flow chart of a data annotation quality control method based on a large model disclosed in this application;

[0042] Figure 2 This is a schematic diagram of the structure of a data annotation quality control device based on a large model disclosed in this application;

[0043] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0045] Currently, for data annotation quality control, on the one hand, traditional rule-based annotation quality control relies on pre-set standards and is difficult to cope with complex and changeable data scenarios. For example, in natural language processing annotation, the ambiguity of semantics and the diversity of contexts make it difficult for rules to cover all situations, resulting in annotation errors. The method of manual sampling and review is not only inefficient, but also the limitations of the sampling samples make it difficult to fully reflect the overall annotation quality. At the same time, the existing simple verification tools have single functions and lack the ability to deeply analyze and intelligently judge the annotation data. When faced with the annotation of unstructured data such as images and videos, potential errors cannot be accurately identified. To this end, the present application provides a data annotation quality control method, device, equipment and medium based on a large model, which can improve the efficiency of quality control in the data annotation process.

[0046] See also Figure 1 As shown, an embodiment of the present invention discloses a data annotation quality control method based on a large model, comprising:

[0047] Step S11: In the process of annotating the original data using a data annotation tool, the annotation behavior data and the annotation result data are obtained, and the preset multimodal large model is used to analyze the annotation behavior data, the annotation result data and the original data to obtain a first analysis result.

[0048] In this embodiment, first, in the process of using the data annotation tool to annotate the original data, the annotation behavior data and the annotation result data are obtained. Before obtaining these data, it is also necessary to embed the data acquisition plug-in into the data annotation tool. Among them, the data annotation tool covers multiple types of annotation platforms such as image annotation tools, text annotation tools, audio annotation tools, etc., and supports annotation operations on multimodal original data such as images, text, audio, and video. In order to achieve comprehensive monitoring of the annotation process, a lightweight data acquisition plug-in is first embedded in the underlying architecture of the data annotation tool. The plug-in can use non-invasive data capture technology to ensure that the operation behavior of the annotation personnel is recorded without affecting the operating performance of the annotation tool.

[0049] During the annotation behavior data collection stage, when the annotator starts the data annotation tool and opens the original data, the data collection plug-in automatically triggers the timing module to record the first timestamp of the start of data annotation. During the annotation process, the plug-in captures the annotation trajectory data in real time by listening to mouse movement events or keyboard input events. Specifically, for image annotation scenarios, the coordinate sequence of the mouse movement trajectory in the image coordinate system can be recorded to form the basic data of the annotation browsing path heat map; for text annotation scenarios, the character sequence entered by the keyboard, the frequency of deletion operations, and the cursor positioning position can be recorded. When the annotator completes the annotation and submits the results, the plug-in records the second timestamp of the end of data annotation, and extracts key behavior parameters such as the annotation operation type, number of operations, and the time taken for a single operation to obtain the annotation behavior data.

[0050] In terms of labeling result data collection, the data collection plug-in synchronizes the labeling output submitted by the labeler in real time. Specifically, for image data, it collects information such as the coordinate position of the annotation box and the labeling category; for text data, it collects the annotated entity labels, entity boundary indexes, and relationship labeling results; for audio data, it collects the speech transcription text, emotion labels, and sound source location coordinates, and uses this collected information to determine the labeling result data.

[0051] Furthermore, the hierarchical feature extraction architecture of the preset multimodal large model is used to encode the time series features of the labeled behavior data. In other words, the labeled behavior data can be specifically analyzed through the long short-term memory network to obtain behavioral abnormality indicators such as "labeling time fluctuation rate", "operation frequency entropy value", and "trajectory deviation".

[0052] Next, align the annotation result data with the original data. For image data, the annotation box area can be spatially aligned with the visual features of the original image through computer vision algorithms; for text data, the annotation entity can be semantically aligned with the contextual semantics through natural language processing technology. The preset multimodal large model further extracts the feature differences between the annotation result data and the original data. For example, in medical image annotation, the pixel ratio of the lesion area covered by the annotation box, the cosine similarity of the lesion feature vector and the standard lesion feature library are calculated to form the annotation result deviation index. At the same time, the deviation is corrected in the spatial dimension by combining the annotation box boundary data output by the geometric calculation module of the data annotation tool.

[0053] To improve the reliability of analysis results, a cross-validation mechanism is introduced for behavioral anomaly indicators and the deviation of annotation results. A pre-set positive correlation constraint stipulates that when the "annotation duration fluctuation rate" in the behavioral anomaly indicator exceeds a threshold and the "trajectory deviation" increases simultaneously, the deviation of the annotation results should show a parallel growth trend. If the validation passes, a pre-set dynamic compensation algorithm is triggered. This algorithm, based on a deviation correction model trained on historical annotation data, corrects the deviation of the annotation results, resulting in a corrected deviation.

[0054] Furthermore, to address the inherent complexity of the raw data, a pre-set multimodal large model generates feature fuzzy scores through a self-attention mechanism. Ultimately, the first analysis result is generated by a weighted fusion of the corrected bias and the feature fuzzy scores, ensuring that the analysis balances the impact of annotation behavior with the inherent characteristics of the data.

[0055] Step S12: Determine the source of the labeling error using the first analysis result. If the source of the labeling error is the target labeler, analyze the historical labeling data of the target labeler and trigger corresponding control measures based on the obtained second analysis result.

[0056] In this embodiment, the source of the labeling error is located based on the first analysis result generated in step S11. First, the feature ambiguity score in the first analysis result is compared with a preset first threshold. If it exceeds the preset first threshold, it is determined that the original data has inherent ambiguity, such as blurred lesions in medical images or low-light areas in natural scene images. This type of data is prone to disagreements between different labelers due to its inherently unclear feature expression. In this case, the system automatically marks it as a data-type error source and triggers data preprocessing optimization processes, such as contrast enhancement and feature sharpening.

[0057] In a specific embodiment, if the feature fuzziness score is lower than a preset first threshold, but the deviation after correction exceeds a preset second threshold, it is determined that the data annotation tool has a tool defect and the data annotation tool is identified as the source of the annotation error. In this case, the tool upgrade process can be triggered and the patch package can be pushed to the annotation tool client.

[0058] In another specific embodiment, when the deviation degree after correction exceeds a preset third threshold and the data annotation tool does not have a tool defect, the target annotation person is identified as the source of the error. At this time, the system extracts all the annotation records of the person within a preset time period from the historical annotation database. The annotation trajectory characteristics can be analyzed by a time series feature extraction algorithm, the standard deviation of the moving speed of the annotation box center point can be calculated, and the stability of the annotation process can be evaluated; the trajectory curvature change rate can be extracted to identify whether there is a habitual misoperation pattern. At the same time, the frequency of misuse of the data annotation tool is counted, such as the percentage of times the rectangular annotation tool is used for polygon annotation, the frequency of unnecessary annotation box modification operations, etc.

[0059] In analyzing the number of annotation adjustments, a sliding window technique is used to calculate the peak number of adjustments per unit time. Feature engineering is performed on the aforementioned multi-dimensional indicators, including annotation trajectory characteristics, misuse frequency, and number of adjustments, to determine the corresponding annotation feature information. This annotation feature information is then input into a pre-set multimodal large model. After a series of processing, the pre-set multimodal large model will produce a model output result, which is used as the second analysis result. Based on this second analysis result, corresponding hierarchical control measures are then triggered.

[0060] Step S13: When the first analysis result indicates that target image data exists in the annotation result data, a preset image segmentation model is used to assist the annotation personnel in re-annotating the target image data; the target image data is image data whose data features meet the preset feature fuzzy conditions and has completed data annotation.

[0061] In this embodiment, when the first analysis results indicate the presence of target image data within the labeled data, a preset image segmentation model (i.e., the Segment Anything model) is used to assist the labeler in relabeling the target image data. The target image data is image data whose features meet the preset feature fuzziness conditions and has been labeled. Furthermore, due to the fuzzy nature of the target image data, the target image data may also be considered high-risk labeled data.

[0062] Specifically, we first need to determine the target area in the target image data through a preset image segmentation model, and then use the target area to determine the segmentation mask and category prediction results.

[0063] During segmentation mask generation, the pre-set image segmentation model can employ a deep supervision mechanism, introducing auxiliary loss functions at different encoder and decoder levels to optimize mask generation quality. For targets with blurred boundaries, the pre-set image segmentation model can extract target contour features and fuse them with semantic features to improve the accuracy of mask boundaries. During the category prediction phase, the pre-set image segmentation model classifies the feature vectors of the target region and outputs the corresponding category probability distribution.

[0064] Furthermore, in the assisted re-labeling stage, the annotator can adjust the segmentation mask boundaries by dragging and dropping. The system calculates the mask similarity before and after the adjustment in real time and provides semantic suggestions. For areas where the category prediction confidence is lower than the threshold, the system automatically marks them as suspicious areas and highlights them on the interface. When the re-labeling is completed, the target annotated image can be generated, which contains the accurate segmentation mask, category label and confidence score. In addition, the target annotated image can also be evaluated. If the evaluation index does not meet the preset standard, the secondary re-labeling process is automatically triggered until the quality requirements are met.

[0065] As can be seen from the above, in the data labeling stage, the present application collects labeling behavior data and labeling result data, and analyzes the labeling behavior data, labeling result data and original data through a preset multimodal large model to generate a first analysis result. Then, based on the first analysis result, the source of the labeling error is located. When it is determined that the source of the labeling error is the target labeler, the historical labeling data of the target labeler is called for analysis, and hierarchical control measures are triggered based on the generated second analysis result. If the analysis finds that there is target image data with excessive feature ambiguity in the labeling result, the preset image segmentation model is used to assist the labeler in completing the data re-labeling. In this way, the present application can improve the efficiency of quality control in the data labeling process, thereby achieving accurate tracing and repair of labeling errors.

[0066] The technical solutions of the embodiments of the present application are described in detail below in conjunction with specific application scenarios.

[0067] Specifically, in the field of intelligent security video surveillance, high-quality annotated data is crucial for training behavior recognition models. Taking the example of monitoring pedestrian flow in a shopping mall, a data collection plug-in is first embedded in the data annotation tool to record in real time the annotator's annotation of pedestrian behavior in video frames. When the annotator begins annotating a frame, the plug-in records the first timestamp and tracks the mouse's movement on the screen to form a browsing path heat map. During the annotation process, the system simultaneously collects the coordinate position, size changes, and final behavior category of the annotation box to determine the corresponding annotated behavior data and annotation result data.

[0068] Next, a pre-set multimodal large model conducted an in-depth analysis of the annotated behavior data, the annotated result data, and the original data. Through temporal feature encoding, the pre-set multimodal model discovered that when a certain annotator was annotating running behavior, the annotation duration fluctuation rate increased abnormally, while the operation frequency entropy value decreased, indicating that the annotator may have hesitated to annotate. Further analysis of the feature differences between the annotation result data and the original video frame revealed that the movement speed of the annotation box deviated from the actual pedestrian's motion trajectory. Combined with the feature fuzzy score, the system generated the first analysis result, indicating that the annotation had a high quality risk.

[0069] Furthermore, based on the first analysis results, the system activates the error source location mechanism. Since the feature fuzziness score does not exceed the preset first threshold, the data ambiguity factor is eliminated; further testing of the performance indicators of the annotation tool shows that the coordinate calculation error is within the normal range, eliminating tool defects. Ultimately, the system identifies the target annotation personnel as the source of the error and retrieves their historical annotation data for analysis. The system found that the target annotation personnel had obvious lags when annotating fast-moving targets, and the number of adjustments was significantly higher than the average level. The preset multimodal large model learns these historical features, outputs the corresponding second analysis results, and triggers targeted control measures.

[0070] Additionally, when the first analysis results contain previously annotated image data with ambiguous features, the system activates an image segmentation assistance mechanism. A pre-set image segmentation model processes the image data and generates a segmentation mask to reveal the pedestrian's outline. The pre-set image segmentation model also predicts the pedestrian's movement trajectory and outputs a position probability distribution for the next few frames, providing a reference for the annotator. The annotator can adjust the segmentation mask through an interactive interface, and the system calculates the adjusted quality indicators in real time to ensure that the re-annotation results meet standards.

[0071] Accordingly, see Figure 2 As shown, the embodiment of the present application provides a data annotation quality control device based on a large model, including:

[0072] The data analysis module 11 is configured to obtain annotation behavior data and annotation result data during the process of annotating the original data using a data annotation tool, and analyze the annotation behavior data, the annotation result data, and the original data using a preset multimodal large model to obtain a first analysis result;

[0073] A measure triggering module 12 is configured to determine the source of the labeling error using the first analysis result. If the source of the labeling error is a target labeler, the module analyzes the historical labeling data of the target labeler and triggers corresponding control measures based on the second analysis result.

[0074] The data re-labeling module 13 is used to use a preset image segmentation model to assist the labeling personnel in re-labeling the target image data when the first analysis result indicates that the target image data exists in the labeling result data; the target image data is image data whose data features meet the preset feature fuzzy conditions and the data labeling has been completed.

[0075] As can be seen from the above, in the data labeling stage, the present application collects labeling behavior data and labeling result data, and analyzes the labeling behavior data, labeling result data and original data through a preset multimodal large model to generate a first analysis result. Then, based on the first analysis result, the source of the labeling error is located. When it is determined that the source of the labeling error is the target labeler, the historical labeling data of the target labeler is called for analysis, and hierarchical control measures are triggered based on the generated second analysis result. If the analysis finds that there is target image data with excessive feature ambiguity in the labeling result, the preset image segmentation model is used to assist the labeler in completing the data re-labeling. In this way, the present application can improve the efficiency of quality control in the data labeling process, thereby achieving accurate tracing and repair of labeling errors.

[0076] In some specific implementations, the data analysis module 11 specifically includes:

[0077] A tool embedding unit, used to embed a data collection plug-in into the data annotation tool;

[0078] a behavior data determination unit, configured to use the data acquisition plug-in to obtain a first timestamp for starting data annotation, a second timestamp for ending data annotation, a data annotation trajectory, and a data annotation operation during data annotation of the original data, so as to determine corresponding annotated behavior data;

[0079] The result data determining unit is used to obtain the original data after data annotation, the coordinate position of the annotation frame and the annotation category to determine the corresponding annotation result data.

[0080] In some specific implementations, the data analysis module 11 specifically includes:

[0081] A data analysis unit, configured to analyze the labeled behavior data using a preset multimodal large model to obtain an abnormal behavior indicator;

[0082] a deviation determination unit, configured to align the annotation result data with the original data, extract feature differences between the annotation result data and the original data using the preset multimodal macro model, and determine the annotation result deviation based on the feature differences and boundary data of the annotation box generated by the data annotation tool;

[0083] a deviation correction unit, configured to cross-validate the abnormal behavior index and the deviation of the annotation result, and when the abnormal behavior index and the deviation of the annotation result satisfy a preset positive correlation constraint, correct the deviation of the annotation result using a preset dynamic compensation algorithm to obtain a corrected deviation;

[0084] a score determination unit, configured to generate a feature fuzzy score based on the original data and the preset multimodal macro model;

[0085] A result generating unit is used to generate a first analysis result based on the corrected deviation degree and the feature fuzzy score.

[0086] In some specific implementations, the measure triggering module 12 specifically includes:

[0087] a first error source determining unit, configured to indicate that data ambiguity exists in the original data and determine the original data as a source of labeling error if the feature fuzziness score in the first analysis result is greater than a preset first threshold;

[0088] a second error source determination unit, configured to indicate that the data annotation tool has a tool defect and determine the data annotation tool as the annotation error source if the corrected deviation degree in the first analysis result is greater than a preset second threshold and the feature fuzziness score is less than the preset first threshold;

[0089] The third error source determination unit is configured to determine the target annotator as the annotation error source if the corrected deviation in the first analysis result is greater than a preset third threshold and the data annotation tool does not have the tool defect.

[0090] In some specific implementations, the measure triggering module 12 specifically includes:

[0091] an information determination unit, configured to determine historical annotation data of the target annotator, and determine, based on the historical annotation data, annotation trajectory characteristics of the target annotator, a misuse frequency of the data annotation tool, and a number of annotation adjustments, to determine corresponding annotation feature information;

[0092] a result determination unit, configured to input the labeled feature information into the preset multimodal large model, and determine an output result of the preset multimodal large model as a second analysis result;

[0093] A measure triggering unit is used to trigger corresponding control measures based on the second analysis result.

[0094] In some specific implementations, the data re-labeling module 13 specifically includes:

[0095] a category prediction unit, configured to determine a target region based on the target image data using a preset image segmentation model, and determine a segmentation mask and a category prediction result using the target region; the category prediction result being a result of the preset image segmentation model predicting the category of the object in the target region;

[0096] A data re-labeling unit is used to assist a labeler in re-labeling the target image data based on the segmentation mask and the category prediction result to obtain a target labeled image.

[0097] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 shown according to an exemplary embodiment. The content in the figure cannot be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input and output interface 25 and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the large model-based data annotation quality control method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0098] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0099] In addition, the memory 22 as a carrier for resource storage can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0100] The operating system 221 is used to manage and control the hardware devices and computer program 222 on the electronic device 20, and can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to implement the large model-based data annotation quality control method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to perform other specific tasks.

[0101] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when executed by a processor, the computer program implements the aforementioned large-scale model-based data annotation quality control method. The specific steps of this method can be referred to the corresponding content disclosed in the aforementioned embodiments and will not be repeated here.

[0102] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0103] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0104] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0105] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0106] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data annotation quality control method based on a large model, characterized in that: include: In the process of annotating the original data using the data annotation tool, annotated behavior data and annotated result data are obtained, and the annotated behavior data, the annotated result data, and the original data are analyzed using a preset multimodal large model to obtain a first analysis result; Determine the source of the labeling error using the first analysis result. If the source of the labeling error is the target labeler, analyze the historical labeling data of the target labeler and trigger corresponding control measures based on the obtained second analysis result. When the first analysis result indicates that target image data exists in the annotation result data, a preset image segmentation model is used to assist the annotation personnel in re-annotating the target image data; the target image data is image data whose data features meet preset feature fuzzy conditions and has completed data annotation.

2. The data annotation quality control method based on a large model according to claim 1 is characterized in that: In the process of using the data annotation tool to annotate the original data, obtaining the annotation behavior data and the annotation result data includes: Embedding a data collection plug-in into the data annotation tool; In the process of data labeling the original data, the data acquisition plug-in is used to obtain a first timestamp for starting data labeling, a second timestamp for ending data labeling, a data labeling trajectory, and a data labeling operation to determine corresponding labeling behavior data; The original data after data annotation, the coordinate position of the annotation frame and the annotation category are obtained to determine the corresponding annotation result data.

3. The data annotation quality control method based on a large model according to claim 1 is characterized in that: The method of analyzing the labeling behavior data, the labeling result data, and the original data using a preset multimodal large model to obtain a first analysis result includes: Analyzing the labeled behavior data using a preset multimodal large model to obtain behavioral abnormality indicators; Aligning the labeled result data with the original data, extracting feature differences between the labeled result data and the original data using the preset multimodal large model, and determining the degree of deviation of the labeled result based on the feature differences and boundary data of the labeling box generated by the data labeling tool; Cross-validating the abnormal behavior index and the deviation of the annotation result; when the abnormal behavior index and the deviation of the annotation result satisfy a preset positive correlation constraint condition, correcting the deviation of the annotation result using a preset dynamic compensation algorithm to obtain a corrected deviation; Generating a feature fuzzy score based on the raw data and the preset multimodal macro model; A first analysis result is generated based on the corrected deviation degree and the feature fuzziness score.

4. The data annotation quality control method based on a large model according to claim 3 is characterized in that: The determining the source of the labeling error by using the first analysis result includes: If the feature fuzziness score in the first analysis result is greater than a preset first threshold, it indicates that there is data ambiguity in the original data, and the original data is determined to be a source of labeling error; If the corrected deviation in the first analysis result is greater than a preset second threshold and the feature fuzziness score is less than the preset first threshold, it indicates that there is a tool defect in the data annotation tool, and the data annotation tool is determined to be the source of the annotation error; If the corrected deviation in the first analysis result is greater than a preset third threshold and the data annotation tool does not have the tool defect, the target annotation person is determined as the source of the annotation error.

5. The data annotation quality control method based on a large model according to claim 1 is characterized in that: Analyzing the historical annotation data of the target annotator and triggering corresponding control measures based on the obtained second analysis result includes: Determining historical annotation data of the target annotator, and determining the annotation trajectory characteristics of the target annotator, the misuse frequency of the data annotation tool, and the number of annotation adjustments based on the historical annotation data to determine corresponding annotation feature information; Inputting the labeled feature information into the preset multimodal large model, and determining an output result of the preset multimodal large model as a second analysis result; Trigger corresponding control measures based on the second analysis result.

6. The method for data annotation quality control based on a large model according to any one of claims 1 to 5, characterized in that: The method of using a preset image segmentation model to assist annotators in re-labeling the target image data includes: Determining a target area based on the target image data using a preset image segmentation model, and using the target area to determine a segmentation mask and a category prediction result; the category prediction result is a result of the preset image segmentation model predicting the category of the object in the target area; Based on the segmentation mask and the category prediction result, an annotator is assisted to re-annotate the target image data to obtain a target annotated image.

7. A data annotation quality control device based on a large model, characterized in that: include: a data analysis module, configured to obtain annotation behavior data and annotation result data during the process of annotating the original data using a data annotation tool, and analyze the annotation behavior data, the annotation result data, and the original data using a preset multimodal large model to obtain a first analysis result; a measure triggering module, configured to determine the source of the labeling error using the first analysis result, and when the source of the labeling error is a target labeler, analyze the historical labeling data of the target labeler and trigger corresponding control measures based on the obtained second analysis result; A data re-labeling module is used to assist the labeling personnel in re-labeling the target image data using a preset image segmentation model when the first analysis result indicates that the target image data exists in the labeling result data; the target image data is image data whose data features meet the preset feature fuzzy conditions and have completed data labeling.

8. The data annotation quality control device based on a large model according to claim 7 is characterized in that: The data analysis module includes: A plug-in embedding unit, used to embed a data collection plug-in into the data annotation tool; a behavior data determination unit, configured to use the data acquisition plug-in to obtain a first timestamp for starting data annotation, a second timestamp for ending data annotation, a data annotation trajectory, and a data annotation operation during data annotation of the original data, so as to determine corresponding annotated behavior data; The result data determining unit is used to obtain the original data after data annotation, the coordinate position of the annotation frame and the annotation category to determine the corresponding annotation result data.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the large model-based data annotation quality control method as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, it implements the large model-based data annotation quality control method as described in any one of claims 1 to 6.

Images