Image data quality assessment and automatic labeling method and system

Abstract

The application discloses a kind of quality evaluation and automatic labeling method and system of image data, comprising the following steps: S1: the original image of input is carried out multidimensional quality evaluation, and the image that does not satisfy multidimensional quality evaluation requirement is filtered according to preset threshold value;S2: the image screened out is carried out automatic labeling, and the preliminary segmentation mask is post-processed optimization by cross-validation iteration mechanism, make the segmentation result of prompt type segmentation model generated reverse correction positioning frame generated by target detection model, then utilize the positioning frame corrected after guiding prompt type segmentation model to carry out higher quality segmentation, output optimized automatic labeling result;S3: the automatic labeling result after optimization is refined and smoothed, and the standardized automatic labeling result is output;S4: the standardized labeling result and quality evaluation information are pushed to man-machine verification interactive system, receive artificial review or correction, and output final labeling file.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically to a method and system for quality assessment and automatic annotation of image data. Background Technology

[0002] With the rapid development of computer vision technology, especially in tasks such as object detection and image segmentation, high-quality, high-precision labeled data has become a crucial foundation for model training and performance improvement. Traditional data labeling processes mainly rely on manual operation, which is not only time-consuming, labor-intensive, and costly, but also susceptible to the subjective influence of labelers, making it difficult to guarantee the consistency and quality stability of the labels. In recent years, although automatic labeling technologies based on deep learning models (such as using YOLO for object detection and SAM for instance segmentation) have gradually been applied, improving labeling efficiency to some extent, they still face the following prominent problems:

[0003] The quality of source data is inconsistent: raw image data collected in real-world applications often suffers from various quality issues, such as insufficient resolution, indistinct targets, blurred images, occluded targets, or irrelevant information to the task. Directly inputting such low-quality images into an automatic annotation model not only leads to unreliable annotation results but also increases the burden of subsequent manual verification, potentially misleading model training and impacting the final algorithm performance. Current technologies lack a mechanism for systematic, multi-dimensional quality assessment and cleaning of raw images at the front end of the annotation process, resulting in low-quality data being "mixed in" into the annotation stage, affecting the overall reliability and usability of the dataset.

[0004] The accuracy of automatic annotation results is limited: Existing automatic annotation methods mostly adopt a sequential pipeline of "detection-segmentation," that is, first using a target detection model (such as YOLO) to generate bounding boxes, and then calling a segmentation model (such as SAM) to generate masks based on these bounding boxes. This one-way process has obvious drawbacks: the initial localization boxes generated by the detection model may not be accurate enough (e.g., too large, too small, or positionally offset), and the segmentation model only works according to this inaccuracy, resulting in a deviation between the final segmentation mask and the true contour of the target. The lack of an effective feedback correction mechanism makes it difficult to further improve the geometric accuracy of the annotation results.

[0005] Insufficient efficiency of human-machine collaboration: Existing annotation systems often directly hand over automatic annotation results to human reviewers, lacking quantitative assessment and visualization of the credibility of the annotation results. Reviewers struggle to quickly distinguish between high-confidence results and questionable results, leading to low review efficiency and uneven distribution of manual correction workload. Furthermore, manual corrections are not effectively fed back to the automatic annotation model, and the system lacks the ability to continuously learn and optimize.

[0006] The quality assessment is limited to a single dimension: current methods for image quality screening tend to focus on a single metric (such as resolution or blur), lacking a mechanism to comprehensively evaluate whether an image is suitable for the current task from multiple dimensions such as semantic relevance, target completeness, and saliency. Especially in professional scenarios (such as rail transit vehicle recognition), relying solely on traditional image quality metrics cannot effectively filter images with irrelevant content or incomplete targets, affecting the relevance of subsequent model training.

[0007] Therefore, designing a comprehensive image data processing method that enables multi-dimensional intelligent image cleaning before annotation, closed-loop optimization of detection and segmentation models during annotation, and efficient human-machine collaboration and continuous learning after annotation has become a key technical challenge for improving annotation efficiency and quality and reducing labor costs. This invention is proposed against this backdrop. Summary of the Invention

[0008] The purpose of this invention is to provide a method and system for quality assessment and automatic annotation of image data, so as to at least solve the problem of relatively low efficiency and quality of current image annotation.

[0009] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for quality assessment and automatic annotation of image data, comprising the following steps:

[0010] S1: Perform multi-dimensional quality assessment on the input raw image and filter images that do not meet the multi-dimensional quality assessment requirements according to preset thresholds; the multi-dimensional quality assessment includes at least image base resolution assessment, target presence assessment, target salience assessment, target integrity pre-assessment, perceptual quality assessment and image-text consistency assessment.

[0011] S2: The selected images are automatically labeled, including using an object detection model to predict the bounding boxes of the images to generate initial localization boxes, using a cue-based segmentation model to generate a preliminary segmentation mask based on the initial localization boxes, and using a cross-validation iterative mechanism to post-process and optimize the preliminary segmentation mask so that the segmentation results generated by the cue-based segmentation model can inversely correct the localization boxes generated by the object detection model. Then, the corrected localization boxes are used to guide the cue-based segmentation model to perform higher quality segmentation and output optimized automatic labeling results.

[0012] S3: Refine and smooth the optimized automatic annotation results to output standardized automatic annotation results;

[0013] S4: Push the standardized annotation results and quality assessment information to the human-computer verification and interaction system, receive manual review or correction, and output the final annotation file.

[0014] Furthermore, the basic resolution assessment includes: determining whether the total number of pixels in the image is lower than a preset resolution threshold; if so, the image is deemed to have insufficient basic resolution and is filtered out.

[0015] Furthermore, the target existence assessment includes: using a pre-trained classification model to determine whether a target category exists in the image; if not, it is determined that there is no clear target in the image, and the image is filtered out.

[0016] Furthermore, the target saliency assessment includes: using a target detection model to calculate the area ratio of the target in the image; if the area ratio is lower than a preset saliency threshold, the target is deemed to be insufficiently salient in the image, and the image is filtered out.

[0017] Furthermore, the target integrity pre-assessment includes: generating an initial mask using a segmentation model, calculating the area ratio of the mask to the corresponding bounding box, and if the area ratio is lower than a preset integrity threshold, the image is determined to be incomplete and the image is filtered out.

[0018] Furthermore, the perceptual quality assessment includes: calculating an image quality score using a no-reference image quality assessment algorithm; if the image quality score is higher than a preset image quality threshold, the image is determined to have insufficient perceptual quality and is filtered out.

[0019] Furthermore, the image-text consistency assessment includes: using a visual language model to calculate the semantic similarity between the image and the target category text; if the semantic similarity is lower than a preset similarity threshold, the image is determined to be irrelevant to the task and the image is filtered out.

[0020] Furthermore, step S2 specifically includes the following steps:

[0021] S21: Use the object detection model to generate initial bounding boxes B_initial, and use the spatial cue input of each initial bounding box B_initial to generate a preliminary segmentation mask M_current from the cue-type segmentation model;

[0022] S22: Perform connectivity analysis on the initial segmentation mask, remove connected regions with areas smaller than a set ratio, and obtain the optimized mask M_optimized;

[0023] S23: Check whether the optimized mask M_optimized still contains valid foreground pixels. If there are no valid foreground pixels in the mask M_optimized, terminate the iteration for the target and retain the initial localization box B_initial of the target detection model and its corresponding segmentation mask M_current as the automatic annotation result of the target; otherwise, proceed to step S24.

[0024] S24: Calculate the minimum bounding rectangle of the optimized mask M_optimized and use it as the corrected new bounding box B_refined;

[0025] S25: Calculate the intersection-union ratio (IU) between the new bounding box B_refined and the bounding box used in the previous cue-based segmentation model. If the IU reaches the preset convergence threshold or the maximum number of iterations has been reached, the bounding box is considered stable and the iteration terminates. At this point, the mask M_optimized obtained from the last optimization and the finally corrected bounding box B_refined are the automatic annotation results for the target. If the IU does not reach the preset convergence threshold and does not exceed the maximum number of iterations, the corrected bounding box B_refined is used as the new spatial cue input cue-based segmentation model, and the process jumps to step S22 to start a new round of iteration.

[0026] Furthermore, step S3 specifically includes the following steps:

[0027] S31: For each segmentation mask in the optimized automatic annotation result, extract its outermost contour to obtain an initial polygon composed of a sequence of pixels; then, use a key point filtering method based on the Douglas-Puk algorithm to simplify the initial polygon according to a preset distance tolerance threshold, remove redundant intermediate pixels, and filter out the key boundary point sequence that represents the shape features of the mask body.

[0028] S32: Based on the key boundary point sequence, and combined with the edge gradient information of the image itself as geometric constraints, the key boundary point sequence is smoothly fitted using a spline interpolation algorithm to generate a smooth and continuous closed contour curve, thereby optimizing the mask contour. At the same time, the area of ​​all connected regions within the mask is calculated, and isolated small regions with an area smaller than the noise threshold are removed according to the area threshold rule, and internal non-semantic holes with an area smaller than the hole threshold are filled.

[0029] S33: Convert the smooth closed contour curve into the final refined segmentation mask, and encapsulate the refined segmentation mask and its corresponding target category and confidence information together to output standardized automatic annotation results.

[0030] Secondly, the present invention provides an image data quality assessment and automatic annotation system, comprising:

[0031] The multi-dimensional quality assessment and filtering module is used to perform multi-dimensional quality assessment on the input raw image and filter images that do not meet the multi-dimensional quality assessment requirements according to preset thresholds. The multi-dimensional quality assessment includes at least image base resolution assessment, target existence assessment, target saliency assessment, target integrity pre-assessment, perceptual quality assessment, and image-text consistency assessment.

[0032] The automatic annotation module is used to automatically annotate the selected images. This includes generating initial bounding boxes using an object detection model, generating a preliminary segmentation mask using a cue-based segmentation model, and post-processing and optimizing the preliminary segmentation mask through a cross-validation iterative mechanism. This allows the segmentation results generated by the cue-based segmentation model to inversely correct the bounding boxes generated by the object detection model. The corrected bounding boxes are then used to guide the cue-based segmentation model to perform higher-quality segmentation.

[0033] The manual review module is used to receive and visualize the automatic annotation results and quality assessment information, receive manual review or correction, and output the final annotation file.

[0034] The beneficial effects of this invention are as follows:

[0035] 1. This method fundamentally solves the problem of the disconnect between cleaning and annotation stages in existing pipelines by designing multi-dimensional, pre-processed image quality assessment and cleaning steps. By simultaneously considering basic image quality, target saliency and completeness, and image-text consistency, and employing a strict threshold determination mechanism, it achieves a leap from "simple filtering" to "intelligent cleaning." Its direct benefit is that it proactively and accurately eliminates various low-value images (such as those with insufficient resolution, no target, overly small target, incomplete segmentation, blurred images, and semantically irrelevant images) before the high-cost annotation stage begins, greatly improving the "purity" of the data entering the annotation process. This not only lays a solid foundation for producing high-quality training sets from the source but also significantly saves the computational and human resources required for subsequent annotation and model training, avoiding ineffective investment in low-quality data.

[0036] 2. This method breaks the traditional unidirectional operation mode of detection and segmentation modules in pipelines by introducing a cross-validation iterative mechanism. It uses the segmentation results of the cue-based segmentation model to back-correct the bounding boxes generated by the target detection model, and then uses the corrected, more accurate bounding boxes to guide the cue-based segmentation model for higher-quality segmentation. This closed-loop process effectively corrects the deviations of the initial bounding boxes in the target detection model (such as boxes that are too large, too small, or inaccurate), ensuring that the bounding boxes in the final output annotation closely match the true contour of the segmented target (i.e., the final segmentation mask and bounding boxes are spatially highly consistent), significantly improving the geometric accuracy of the annotation. This precise spatial alignment is crucial for vision tasks requiring high-precision positioning (such as measurement and precision assembly vision guidance), and also reduces the workload of subsequent manual correction of the bounding boxes and masks.

[0037] 3. This method employs a dedicated post-processing optimization workflow for the mask. This workflow automatically removes isolated connected regions that are too small and may be caused by noise, and smooths and simplifies the remaining main body contours. It can automatically generate segmentation masks with clearer boundaries, more accurate contours, and better conformity to human annotation habits without relying on manual intervention. This not only directly improves the visual quality and geometric accuracy of the annotation results and reduces the workload of subsequent manual correction, but more importantly, it provides a higher-quality "standard answer," enabling segmentation models trained on such data to learn more accurate edge features, thereby improving their performance in practical applications.

[0038] 4. This method employs a human-computer verification interaction system to automatically load the aforementioned results and present them to human reviewers in a visual manner. The review interface clearly displays the images, automatically generated annotations, and scores for each quality indicator. For annotations with high quality assessment scores and good confidence levels, reviewers can quickly perform batch verification. For annotations that raise questions (such as a quality score being at a critical point), have conflicting confidence levels in the automated process, or have failed cross-validation iterations, reviewers can use the system's convenient graphical tools (such as boundary point drag-and-drop correction, label modification, and result acceptance / rejection buttons) for fine-tuning or decision-making. All manual correction actions are recorded by the system and can be used as feedback data to optimize the parameters of the automatic annotation model (such as the object detection model and the prompting segmentation model), enabling continuous iterative improvement of the system. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, use the same reference numerals to denote the same or similar parts. The illustrative embodiments of this application and their descriptions are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0041] like Figure 1 The method for quality assessment and automatic annotation of the image data shown includes the following steps:

[0042] S1: Perform multi-dimensional quality assessment on the input raw image and filter images that do not meet the multi-dimensional quality assessment requirements according to preset thresholds; the multi-dimensional quality assessment includes at least image base resolution assessment, target presence assessment, target salience assessment, target integrity pre-assessment, perceptual quality assessment and image-text consistency assessment.

[0043] S2: The selected images are automatically labeled, including using an object detection model to predict the bounding boxes of the images to generate initial localization boxes, using a cue-based segmentation model to generate a preliminary segmentation mask based on the initial localization boxes, and using a cross-validation iterative mechanism to post-process and optimize the preliminary segmentation mask so that the segmentation results generated by the cue-based segmentation model can inversely correct the localization boxes generated by the object detection model. Then, the corrected localization boxes are used to guide the cue-based segmentation model to perform higher quality segmentation and output optimized automatic labeling results.

[0044] S3: Refine and smooth the optimized automatic annotation results to output standardized automatic annotation results;

[0045] S4: Push the standardized annotation results and quality assessment information to the human-computer verification and interaction system, receive manual review or correction, and output the final annotation file.

[0046] This method fundamentally solves the problem of the disconnect between cleaning and annotation in existing pipelines by designing multi-dimensional, pre-processed image quality assessment and cleaning steps. By simultaneously considering basic image quality, target saliency and completeness, and image-text consistency, and employing a strict threshold determination mechanism, it achieves a leap from "simple filtering" to "intelligent cleaning." Its direct benefit is that it proactively and accurately eliminates various low-value images (such as those with insufficient resolution, no target, overly small target, incomplete segmentation, blurred images, and semantically irrelevant images) before the high-cost annotation process begins, greatly improving the "purity" of the data entering the annotation process. This not only lays a solid foundation for producing high-quality training sets from the source but also significantly saves the computational and human resources required for subsequent annotation and model training, avoiding ineffective investment in low-quality data.

[0047] This method breaks away from the traditional unidirectional operation of detection and segmentation modules in pipelines by introducing a cross-validation iterative mechanism. It uses the segmentation results from the cue-based segmentation model to back-correct the bounding boxes generated by the target detection model, and then uses the corrected, more accurate bounding boxes to guide the cue-based segmentation model for higher-quality segmentation. This closed-loop process effectively corrects deviations in the initial bounding boxes of the target detection model (such as boxes that are too large, too small, or inaccurate), ensuring that the bounding boxes in the final annotation results closely match the true contours of the segmented target (i.e., the final segmentation mask and bounding boxes are spatially highly consistent), significantly improving the geometric accuracy of the annotation. This precise spatial alignment is crucial for vision tasks requiring high-precision positioning (such as measurement and precision assembly vision guidance), and also reduces the workload of subsequent manual correction of the bounding boxes and masks.

[0048] This method employs a human-machine verification interaction system to automatically load the aforementioned results and present them to human reviewers in a visual manner. The review interface clearly displays the images, automatically generated annotations, and scores for each quality indicator. For annotations with high quality assessment scores and good confidence levels, reviewers can quickly perform batch verification. For annotations that raise questions (such as a quality score being at a critical point), have conflicting confidence levels in the automated process, or have failed cross-validation iterations, reviewers can use the system's convenient graphical tools (such as boundary point drag-and-drop correction, label modification, and result acceptance / rejection buttons) for fine-tuning or decision-making. All manual correction actions are recorded by the system and can be used as feedback data to optimize the parameters of the automatic annotation model (such as object detection models and suggestive segmentation models), enabling continuous iterative improvement of the system.

[0049] Finally, the annotation results, after manual verification or correction, are output by the system as standard annotation files (such as JSON format). This pipeline processes all images sequentially, thereby effectively balancing processing efficiency and annotation accuracy through a human-machine collaboration mechanism while ensuring high-quality output, achieving automated, closed-loop production from raw images to finely annotated datasets.

[0050] According to one embodiment of this application, the basic resolution evaluation includes: determining whether the total number of pixels in an image is lower than a preset resolution threshold; if so, the image is determined to have insufficient basic resolution and is filtered out. Specifically, the evaluation method involves: the system reading the pixel dimensions (width and height) of the image and calculating its total number of pixels. This total number of pixels is compared with a preset minimum resolution threshold set according to task requirements (e.g., 640x480 pixels for general object detection; 1024x768 pixels for fine segmentation). If the total number of pixels in the image is lower than the threshold, it is determined to have insufficient resolution and is filtered out.

[0051] According to one embodiment of this application, target existence assessment includes: using a pre-trained classification model to determine whether a target category exists in an image; if not, the image is determined to lack a definite target and is filtered out. Specifically, the assessment method involves the system using a lightweight, pre-trained image classification convolutional neural network (e.g., a simplified version of MobileNet or ResNet) to perform forward inference on the input image. This network has been trained on a dataset containing the target category and the "background" category required for this task. Its output is the probability distribution of the image belonging to each category. The system takes the highest probability of the target category; if this probability is lower than a preset existence confidence threshold (e.g., 0.7), the image is determined to lack a definite target and is filtered out.

[0052] According to one embodiment of this application, target saliency evaluation includes: calculating the area proportion of the target in an image using a target detection model; if the area proportion is lower than a preset saliency threshold, the target is deemed insufficiently salient in the image and the image is filtered out. Specifically, the evaluation method is as follows: the system uses a lightweight version of the YOLO target detection model to perform fast inference on the image and obtain all detected target bounding boxes. The area (width x height) of each bounding box is calculated and divided by the total area of ​​the image to obtain the area proportion of a single target. The maximum value of the area proportions of all detected boxes is taken; if this maximum value is lower than a preset saliency threshold (e.g., 0.1, meaning the target occupies at least 10% of the image), the target is deemed too small and insufficiently salient in the image and is filtered out.

[0053] According to one embodiment of this application, the target integrity pre-assessment includes: generating a preliminary mask using a segmentation model, calculating the area ratio of the mask to the corresponding bounding box, and if the area ratio is lower than a preset integrity threshold, the image is determined to have insufficient integrity and is filtered out. Specifically, the assessment method is as follows: the system uses the target bounding box with the largest area detected by YOLO as a cue, inputs a lightweight SAM segmentation model (or its fast variant), and generates a preliminary segmentation mask. The total area of ​​the foreground pixels in the mask is calculated, and its ratio to the area of ​​the corresponding bounding box is calculated. If this ratio is lower than a preset integrity threshold (e.g., 0.6), it indicates that the target may be severely occluded or have a special shape, resulting in incomplete segmentation. Such images are judged as potentially low-quality samples and filtered out. This step is a pre-judgment, designed to identify targets that may be difficult to segment completely in advance.

[0054] According to one embodiment of this application, perceptual quality assessment includes: calculating an image quality score using a no-reference image quality assessment algorithm; if the image quality score is higher than a preset image quality threshold, the image is determined to have insufficient perceptual quality and is filtered out. Specifically, the assessment method involves the system using the no-reference image spatial quality evaluator (BRISQUE) algorithm to evaluate image quality. This algorithm first extracts the natural scene statistical features of the brightness coefficients of the image after local normalization, and then uses a support vector machine regression model pre-trained on an image quality database to map these features into a perceptual quality score. The lower the score, the better the image quality (less distortion). The system sets a perceptual quality threshold; if the BRISQUE score of an image is higher than this threshold (i.e., the quality is lower than the standard), the image is determined to have perceptual quality problems such as blurring or noise and is filtered out. Besides using the BRISQUE algorithm, different or no-reference image quality assessment algorithms can be used to replace the BRISQUE algorithm, such as using the Natural Image Quality Evaluator (NIQE) or the Perceptual Image Quality Evaluator (PIQE) to evaluate the perceptual quality of an image.

[0055] According to one embodiment of this application, image-text consistency evaluation includes: calculating the semantic similarity between an image and target category text using a visual language model; if the semantic similarity is lower than a preset similarity threshold, the image is determined to be irrelevant to the task and is filtered out. Specifically, the evaluation method uses a contrastive language-image pre-trained model based on Visual Transformer (CLIP-ViT) for image-text consistency evaluation. The image to be evaluated is input into the CLIP image encoder to obtain image feature embeddings. Simultaneously, the target category name expected by the task (e.g., "locomotive," "track," "pedestrian," "urban rail transit vehicle") is constructed into a text prompt (e.g., "a photo about [urban rail transit vehicles]") and input into the CLIP text encoder to obtain text feature embeddings for each category. Subsequently, the cosine similarity between the image feature embedding and the text feature embedding for each category is calculated, and the maximum value is taken as the image-text consistency score of the image. The higher this score, the more the image content matches the expected target category. The system sets an image-text consistency threshold; if the image score is lower than this threshold, the image content is determined to be irrelevant to the task and is filtered out. The prompts used in this implementation are shown in Table 1 below.

[0056] Table 1. CLIP-ViT Model Image-Text Consistency Assessment Cue Word Design

[0057]

[0058] For image-text consistency evaluation, other large-scale visual language models or multimodal models can be used instead of CLIP-ViT. For example, the BLIP series models can be used to generate image descriptions and evaluate their consistency with the expected target category. In target presence and saliency evaluation, other lightweight detectors (such as SSD Lite, EfficientNet-Det) or classifiers can be used instead of YOLO and MobileNet.

[0059] The system sets strict thresholds for all six evaluation indicators. Input images must pass all six criteria to be deemed "annotable images" and enter the subsequent automatic annotation process; any image failing to meet the standard is immediately filtered out. This "one-vote veto" mechanism ensures that only high-quality, highly relevant images proceed to the more costly annotation stage. Furthermore, the fusion and judgment strategy for the evaluation indicators can be adjusted. For example, instead of using a strict "one-vote veto" threshold method, a weighted scoring method can be adopted, assigning different weights to each indicator, calculating a comprehensive quality score, and then filtering based on the comprehensive score. While this method changes the strictness of the standards, it still achieves the fundamental purpose of intelligent image cleaning.

[0060] According to one embodiment of this application, step S2 specifically includes the following steps:

[0061] S21: Use the object detection model (YOLO) to generate initial bounding boxes B_initial, and use the spatial cue input cueing segmentation model (SAM) for each initial bounding box B_initial to generate an initial segmentation mask M_current;

[0062] S22: Perform connected component analysis on the initial segmentation mask M_curren, identify all independent connected components, calculate the pixel area of ​​each connected component, and remove connected components with an area less than a set proportion (e.g., 10%) to obtain the optimized mask M_optimized.

[0063] S23: Check whether the optimized mask M_optimized still contains valid foreground pixels. If there are no valid foreground pixels in the mask M_optimized, terminate the iteration for the target and retain the initial localization box B_initial of the target detection model and its corresponding segmentation mask M_current as the automatic annotation result of the target, or mark it as needing manual verification; otherwise, proceed to step S24.

[0064] S24: Calculate the minimum bounding rectangle of the optimized mask M_optimized and use it as the corrected new positioning box B_refined to accurately define the target body range segmented by the SAM segmentation model;

[0065] S25: Calculate the intersection-over-union (IoU) ratio between the new bounding box B_refined and the bounding box used in the previous cue-based segmentation model. If the IoU ratio reaches the preset convergence threshold (e.g., 0.95) or the maximum number of iterations (e.g., 3), the bounding box is considered stable and the iteration terminates. At this point, the mask M_optimized obtained from the last optimization and the finally corrected bounding box B_refined are the automatic annotation results for the target. If the IoU ratio does not reach the preset convergence threshold and does not exceed the maximum number of iterations, the corrected bounding box B_refined is used as the new spatial cue input cue-based segmentation model, and the process jumps to step S22 to start a new round of iteration.

[0066] YOLO is a one-stage object detection method. Its fundamental innovation lies in reconstructing the object detection task into a single regression problem, directly predicting the location (bounding box) and class probability of all objects in an image from the input image pixels in a single operation using a neural network model. This contrasts sharply with the traditional two-stage method (first extracting candidate regions, then classifying them), thus achieving extremely fast processing speed. The implementation of the OLO model can be broken down into the following key steps:

[0067] (1) Input and preprocessing

[0068] The input image size is fixedly scaled to a standard size (e.g., 640x640 pixels). The pixel values ​​are then normalized.

[0069] (2) Feature extraction

[0070] The image is fed into a backbone network, such as Darknet or CSPNet. This network is a deep convolutional neural network (CNN) responsible for extracting multi-level, highly semantic feature maps from the image.

[0071] (3) Detection head prediction

[0072] The extracted feature maps are fed into the detection head. The detection head consists of a series of convolutional layers, and its final output tensor has a size of S × S × (B × 5 + C). S × S represents dividing the input image into S × S grid cells. Each grid cell is responsible for predicting the target falling into its central region. B represents the number of bounding boxes predicted by each grid cell. 5 represents the five basic parameters of each bounding box: the x-coordinate offset of the bounding box center point, the y-coordinate offset, the width w, the height h, and the objectness score (the probability that the box contains an object). C represents the number of all classes to be detected in the dataset. Each grid cell also predicts a set of conditional class probabilities, i.e., the probability that the object belongs to each class if the box contains an object.

[0073] (4) Nonmaximum suppression

[0074] The model generates a large number of candidate bounding boxes, but many boxes repeatedly predict the same target. Therefore, non-maximum suppression is used to filter the bounding boxes.

[0075] First, the class-specific confidence score (SPS) for each bounding box is calculated based on the confidence score and class probability, i.e., confidence score × class probability. Then, a non-maximum suppression algorithm is applied: (1) boxes with class confidence scores below a set threshold (e.g., 0.5) are filtered out; (2) for each class, the box with the highest confidence score is selected; (3) the intersection-union ratio (IoU) of the box with other boxes of the same class is calculated; (4) all boxes with IoU above a set threshold (e.g., 0.45) are deleted. This process is iterated, and finally, the most accurate and least redundant predicted bounding box is retained for each target.

[0076] (5) Output results

[0077] The final output of the YOLO method is a list, each item in the list representing a detected target, containing the following information: (1) bounding box: a rectangular box (x_min, y_min, width, height) represented in absolute or relative coordinates; (2) class label: the class name or ID of the target; (3) confidence score: the model’s overall confidence score for the existence of the target and its class.

[0078] The YOLO solution achieves a good balance between speed and accuracy through its unique unified regression framework, making it one of the most widely used target detection technologies in industry.

[0079] The core objective of the SAM segmentation model is to build a general model capable of generating high-quality pixel-level segmentation masks for any object in an image based on simple user-provided interactive prompts. The SAM segmentation model is based on an encoder-decoder architecture, and its workflow begins with feature extraction from the input image by the image encoder. This image encoder abandons the standard Vision Transformer and instead adopts a novel design based on a lightweight convolutional backbone and label distillation techniques. Specifically, the input image is first segmented into blocks and then fed into this optimized encoder. This encoder is trained through distillation, learning to generate image embeddings that are semantically similar to the output of a large teacher model but with lower dimensionality and less computational cost. The final output is a dense image feature map, where each location encodes rich semantic information of the corresponding image region. This improvement significantly reduces image encoding time and is key to achieving real-time performance.

[0080] After acquiring the image embeddings, the system's prompt encoder begins its work. This component is responsible for transforming various user interaction prompts into machine-understandable embedding vectors. When the user inputs a point prompt, the encoder encodes its location information and combines it with learnable "foreground" or "background" label embeddings; when the input is a rectangular box, it is represented using the location encoding of its pair of diagonal points; for more complex coarse mask prompts, a small convolutional network is used for downsampling and feature encoding. This design allows SAM to flexibly handle various forms of interactive input.

[0081] Finally, the mask decoder undertakes the crucial transformation from cue to segmentation result. This lightweight Transformer decoder takes the image embedding output from the image encoder and the cue embedding generated by the cue encoder as input, and uses a cross-attention mechanism to allow the cue information to query the most relevant image region features. The decoder dynamically outputs multiple possible segmentation masks and predicts the IoU confidence score for each mask. This design allows it to effectively handle the inherent ambiguity of the cue. The final output of the entire process is one or more pixel-level binary masks, where white pixels represent the segmented target object and black pixels represent the background.

[0082] Besides using a combination of YOLO and SAM for automatic annotation, other types of object detection models can be combined with segmentation models. For example, two-stage detectors such as Faster R-CNN or anchor-free detectors such as FCOS can be used to provide initial bounding boxes, which may provide more accurate initial boxes and potentially reduce the number of iterations. For segmentation models, other general or domain-specific segmentation models (such as Mask R-CNN, variants of U-Net) can be used instead of SAM, as long as the model can accept bounding boxes as cues and generate segmentation masks. Furthermore, the convergence criteria can be adjusted: in addition to the intersection-over-union ratio (IoU), the offset distance of the bounding box center points or changes in the mask itself (such as mask IoU) can be considered as convergence criteria. Even more complex strategies can be employed, such as dynamically adjusting the convergence threshold, using a more lenient threshold in the early stages of iteration to accelerate convergence and a stricter threshold in later stages to pursue accuracy.

[0083] According to one embodiment of this application, step S3 specifically includes the following steps:

[0084] S31: For each segmentation mask in the optimized automatic annotation result, extract its outermost contour to obtain an initial polygon composed of a sequence of pixels; then, use a key point filtering method based on the Douglas-Puk algorithm to simplify the initial polygon according to a preset distance tolerance threshold, remove redundant intermediate pixels, and filter out the key boundary point sequence that represents the shape features of the mask body.

[0085] S32: Based on the key boundary point sequence, and combined with the edge gradient information of the image itself as geometric constraints, the key boundary point sequence is smoothly fitted using a spline interpolation algorithm to generate a smooth and continuous closed contour curve, thereby optimizing the mask contour. At the same time, the area of ​​all connected regions within the mask is calculated. According to the area threshold rule, isolated small regions with an area smaller than the noise threshold are removed, and internal non-semantic holes with an area smaller than the hole threshold are filled to improve the physical consistency and integrity of the mask.

[0086] S33: Convert the smooth closed contour curve into the final refined segmentation mask, and encapsulate the refined segmentation mask and its corresponding target category and confidence information together to output standardized automatic annotation results.

[0087] This embodiment addresses the uncontrollable quality of the original mask boundaries output by general segmentation models (such as SAM 2.0) by designing a specialized mask post-processing optimization workflow. This workflow automatically removes isolated connected regions that are too small and may be caused by noise, and smooths and simplifies the retained main body contours. The core benefit of this technique is that it can automatically generate segmentation masks with clearer boundaries, more accurate contours, and better conformity to human annotation habits without relying on manual intervention. This not only directly improves the visual quality and geometric accuracy of the annotation results and reduces the workload of subsequent manual correction, but more importantly, it provides a higher-quality "standard answer," enabling segmentation models trained on this type of data to learn more accurate edge features, thereby improving their performance in practical applications.

[0088] Regarding the implementation of human-computer interaction verification, this invention automatically pushes the verification tasks to the reviewers. An alternative is to use an asynchronous batch processing mode, where the system generates a batch of annotation results, and then reviewers log in to the system for centralized review and correction, rather than strictly real-time task pushes. Another alternative is to introduce a multi-person collaborative review and arbitration mechanism. When a single person is uncertain about the judgment of a sample, it can be submitted to multiple reviewers for joint judgment. The system determines the final result based on the majority principle or arbitration principle. This approach may have advantages in dealing with extremely difficult cases. Furthermore, the feedback from manual corrections can be used not only to optimize model parameters but also directly to create a small-scale, high-quality "gold standard" dataset for rapid calibration of automatically labeled results or as priority samples for model retraining.

[0089] Secondly, the present invention provides an image data quality assessment and automatic annotation system, comprising:

[0090] The multi-dimensional quality assessment and filtering module is used to perform multi-dimensional quality assessment on the input raw image and filter images that do not meet the multi-dimensional quality assessment requirements according to preset thresholds. The multi-dimensional quality assessment includes at least image base resolution assessment, target existence assessment, target saliency assessment, target integrity pre-assessment, perceptual quality assessment, and image-text consistency assessment.

[0091] The automatic annotation module is used to automatically annotate the selected images. This includes generating initial bounding boxes using the object detection model (YOLO), generating preliminary segmentation masks using the cue-based segmentation model (SAM), and post-processing and optimizing the preliminary segmentation masks through a cross-validation iterative mechanism. This allows the segmentation results generated by the cue-based segmentation model to inversely correct the bounding boxes generated by the object detection model. The corrected bounding boxes are then used to guide the cue-based segmentation model to perform higher-quality segmentation.

[0092] The manual review module is used to receive and visualize the automatic annotation results and quality assessment information, receive manual review or correction, and output the final annotation file.

[0093] This invention constructs an automated annotation system with intelligent quality cleaning, cross-validation iterative annotation, result optimization, and human-machine collaboration capabilities. Its beneficial effects are reflected in four aspects: proactive quality control, spatial consistency and geometric accuracy of annotation results, optimized mask boundary quality, and system terminal reliability. These effects work together to enable this invention to produce annotated data of far superior quality to existing automated methods, while significantly reducing the time and economic costs required for relying entirely on manual annotation or post-processing corrections. This provides a practical and feasible technical solution for achieving large-scale, industrial-level high-quality data production.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for quality assessment and automatic annotation of image data, characterized in that, Includes the following steps: S1: Perform multi-dimensional quality assessment on the input raw image and filter images that do not meet the multi-dimensional quality assessment requirements; the multi-dimensional quality assessment includes at least image base resolution assessment, target existence assessment, target salience assessment, target integrity pre-assessment, perceptual quality assessment and image-text consistency assessment. S2: The selected images are automatically labeled, including using an object detection model to predict the bounding boxes of the images to generate initial localization boxes, using a cue-based segmentation model to generate a preliminary segmentation mask based on the initial localization boxes, and using a cross-validation iterative mechanism to post-process and optimize the preliminary segmentation mask so that the segmentation results generated by the cue-based segmentation model can inversely correct the localization boxes generated by the object detection model. Then, the corrected localization boxes are used to guide the cue-based segmentation model to perform higher quality segmentation and output optimized automatic labeling results. S3: Refine and smooth the optimized automatic annotation results to output standardized automatic annotation results; S4: Push the standardized annotation results and quality assessment information to the human-computer verification and interaction system, receive manual review or correction, and output the final annotation file.

2. The image data quality assessment and automatic annotation method according to claim 1, characterized in that, The basic resolution assessment includes: determining whether the total number of pixels in the image is lower than a preset resolution threshold; if so, the image is determined to have insufficient basic resolution and is filtered out.

3. The image data quality assessment and automatic annotation method according to claim 1, characterized in that, The target existence assessment includes: using a pre-trained classification model to determine whether a target category exists in the image; if not, it is determined that there is no clear target in the image, and the image is filtered out.

4. The image data quality assessment and automatic annotation method according to claim 1, characterized in that, The target saliency assessment includes: using a target detection model to calculate the area ratio of the target in the image; if the area ratio is lower than a preset saliency threshold, it is determined that the target is not saliency enough in the image, and the image is filtered out.

5. The image data quality assessment and automatic annotation method according to claim 1, characterized in that, The target integrity pre-assessment includes: generating an initial mask using a segmentation model, calculating the area ratio of the mask to the corresponding bounding box, and if the area ratio is lower than a preset integrity threshold, the image is determined to be incomplete and the image is filtered out.

6. The image data quality assessment and automatic annotation method according to claim 1, characterized in that, The perceptual quality assessment includes: calculating an image quality score using a no-reference image quality assessment algorithm; if the image quality score is higher than a preset image quality threshold, the image is determined to have insufficient perceptual quality and is filtered out.

7. The image data quality assessment and automatic annotation method according to claim 1, characterized in that, The image-text consistency assessment includes: using a visual language model to calculate the semantic similarity between the image and the target category text; if the semantic similarity is lower than a preset similarity threshold, the image is determined to be irrelevant to the task and the image is filtered out.

8. The image data quality assessment and automatic annotation method according to any one of claims 1-7, characterized in that, Step S2 specifically includes the following steps: S21: Use the object detection model to generate an initial bounding box B_initial, and use the spatial cue input of each initial bounding box B_initial to generate a preliminary segmentation mask M_current from the cue-type segmentation model; S22: Perform connectivity analysis on the initial segmentation mask, remove connected regions with areas smaller than a set ratio, and obtain the optimized mask M_optimized; S23: Check whether the optimized mask M_optimized still contains valid foreground pixels. If there are no valid foreground pixels in the mask M_optimized, terminate the iteration for the target and retain the initial localization box B_initial of the target detection model and its corresponding segmentation mask M_current as the automatic annotation result of the target; otherwise, proceed to step S24. S24: Calculate the minimum bounding rectangle of the optimized mask M_optimized and use it as the corrected new positioning box B_refined; S25: Calculate the intersection-union ratio (IUR) between the new bounding box B_refined and the bounding box used in the previous cue-based segmentation model. If the IUR reaches a preset convergence threshold or the maximum number of iterations has been reached, the bounding box is considered stable, and the iteration terminates. At this point, the mask M_optimized obtained from the last optimization and the finally corrected bounding box B_refined are the automatic annotation results for the target. If the IUR does not reach the preset convergence threshold and does not exceed the maximum number of iterations, the corrected bounding box B_refined is used as the new spatial cue input cue-based segmentation model, and the process jumps to step S22 to start a new round of iteration.

9. The image data quality assessment and automatic annotation method according to claim 8, characterized in that, Step S3 specifically includes the following steps: S31: For each segmentation mask in the optimized automatic annotation result, extract its outermost contour to obtain an initial polygon composed of a sequence of pixels; then, use a key point filtering method based on the Douglas-Puk algorithm to simplify the initial polygon according to a preset distance tolerance threshold, remove redundant intermediate pixels, and filter out the key boundary point sequence that represents the shape features of the mask body. S32: Based on the key boundary point sequence, and combined with the edge gradient information of the image itself as geometric constraints, the key boundary point sequence is smoothly fitted using a spline interpolation algorithm to generate a smooth and continuous closed contour curve, thereby optimizing the mask contour. At the same time, the area of ​​all connected regions within the mask is calculated, and isolated small regions with an area smaller than the noise threshold are removed according to the area threshold rule, and internal non-semantic holes with an area smaller than the hole threshold are filled. S33: Convert the smooth closed contour curve into the final refined segmentation mask, and encapsulate the refined segmentation mask and its corresponding target category and confidence information together to output standardized automatic annotation results.

10. A system for quality assessment and automatic annotation of image data, characterized in that, include: The multi-dimensional quality assessment and filtering module is used to perform multi-dimensional quality assessment on the input raw image and filter images that do not meet the multi-dimensional quality assessment requirements according to a preset threshold. The multi-dimensional quality assessment includes at least image base resolution assessment, target existence assessment, target saliency assessment, target integrity pre-assessment, perceptual quality assessment, and image-text consistency assessment. The automatic annotation module is used to automatically annotate the selected images, including generating initial bounding boxes using an object detection model, generating a preliminary segmentation mask using a cue-based segmentation model, and post-processing and optimizing the preliminary segmentation mask through a cross-validation iterative mechanism. This allows the segmentation results generated by the cue-based segmentation model to inversely correct the bounding boxes generated by the object detection model, and then uses the corrected bounding boxes to guide the cue-based segmentation model to perform higher-quality segmentation. The manual review module is used to receive and visualize the automatic annotation results and quality assessment information, receive manual review or correction, and output the final annotation file.

Images