A visual target labeling method, device, equipment and readable storage medium

Abstract

A visual target labeling method, device and equipment and readable storage medium, the method comprises: obtaining a to-be-labeled image and a candidate label, inputting the to-be-labeled image into a pre-trained detection model, and extracting at least one candidate region; the candidate region is processed based on a plurality of semantic generation strategies to obtain a plurality of groups of semantic description information, each group of semantic description information is matched with the candidate label to obtain a pseudo-label pairing of the candidate region and the candidate label; the pseudo-label pairing is evaluated based on a pre-trained cross-modal matching model to obtain an association confidence, the pseudo-label pairing obtained by the plurality of semantic generation strategies is evaluated for consistency to obtain a consistency score, and the comprehensive confidence is determined according to the association confidence and the consistency score; according to the comprehensive confidence, the pseudo-label pairing is divided into a plurality of confidence levels, and according to the divided confidence levels, a corresponding labeling correction strategy is executed to obtain a labeling result. The present application effectively improves the reliability of visual target labeling.

Description

Technical Field

[0001] This application relates to the field of image annotation technology, specifically to a visual target annotation method, apparatus, device, and readable storage medium. Background Technology

[0002] Visual object annotation is a fundamental step in model training in the field of computer vision, and its data quality directly determines the performance of downstream tasks such as object detection, image segmentation, and visual localization. With the rapid development of artificial intelligence technology and increasingly complex application scenarios, the demand for large-scale, high-precision, and semantically informative labeled data has increased dramatically. However, traditional fully manual annotation methods are time-consuming, labor-intensive, costly, and struggle to guarantee annotation consistency, failing to meet the demands of rapid iteration with massive amounts of data. Therefore, how to leverage automation technology to assist manual work and achieve efficient, high-quality semi-automatic visual object annotation has become a critical issue urgently needing to be addressed in the current technological field.

[0003] In existing technologies, semi-automatic visual object annotation typically employs pre-trained object detection or segmentation models to extract candidate regions from images. These candidate regions are then linked to natural language queries using cross-modal matching techniques to generate pseudo-labels for manual verification. During this process, the matching score or probability value output by the model is generally used as a confidence index to filter the generated pseudo-labels, eliminating low-scoring samples and retaining high-scoring samples for subsequent manual verification or direct use. This filtering mechanism, based on the model's direct output score, reduces the amount of data processed manually to some extent.

[0004] However, existing semi-automatic annotation methods have some drawbacks: Firstly, for data of different modalities, there is a cross-modal heterogeneity gap, and the semantic descriptions of candidate regions generated by existing annotation methods often lack comprehensive semantic expression and are prone to generating erroneous pseudo-associations, resulting in low accuracy in pseudo-label pairing between candidate regions and candidate labels. Secondly, existing confidence assessments usually rely on only a single association matching result, leading to insufficient robustness of the confidence assessment results and difficulty in effectively distinguishing between high-quality pseudo-labels and randomly matched noise, thus resulting in a lack of reliable confidence basis for subsequent annotation processing. Therefore, existing technologies suffer from low accuracy in pseudo-label pairing between candidate regions and candidate labels, and insufficient robustness of confidence assessment results, leading to the inability to obtain accurate and reliable target annotation results, which require improvement. Summary of the Invention

[0005] This application provides a visual target annotation method, apparatus, device, and readable storage medium, which can solve the technical problems in the prior art, such as the low accuracy of pseudo-label matching between candidate regions and candidate labels, and the insufficient robustness of confidence evaluation results, which leads to the inability to obtain accurate and reliable target annotation results.

[0006] In a first aspect, embodiments of this application provide a visual target annotation method, including: Obtain the image to be labeled and candidate labels. Input the image to be labeled into a pre-trained detection model to extract at least one candidate region. Candidate regions are processed based on multiple semantic generation strategies to obtain multiple sets of semantic description information. Each set of semantic description information is matched with candidate labels to obtain pseudo-label pairings between candidate regions and candidate labels. The confidence of pseudo-label pairing is evaluated based on the pre-trained cross-modal matching model to obtain the association confidence. The consistency of pseudo-label pairing obtained by various semantic generation strategies is evaluated to obtain the consistency score. The comprehensive confidence is determined based on the association confidence and the consistency score. Based on the overall confidence level, the pseudo-label pairings are divided into multiple confidence levels, and the corresponding labeling correction strategy is executed according to the divided confidence levels to obtain the labeling results.

[0007] In conjunction with the first aspect, in one implementation, candidate regions are processed based on multiple semantic generation strategies to obtain multiple sets of semantic description information, including: Based on the first semantic generation strategy, the predicted category and spatial location information of the candidate region are extracted, and basic description information is generated based on the predicted category and spatial location information. Based on the second semantic generation strategy, the candidate region is input into the pre-trained cross-modal generation model, and feature description information is generated based on the preset prompt word template. Based on the third semantic generation strategy, the image to be labeled, the predicted category and spatial location information are combined as conditional prompts and input into the pre-trained cross-modal generation model to obtain contextual description information. Basic description information, feature description information, and contextual description information are used as multiple sets of semantic description information.

[0008] In conjunction with the first aspect, in one implementation, each set of semantic description information is matched with candidate tags to obtain pseudo-tag pairings between candidate regions and candidate tags, including: Transform semantic description information and candidate labels into text embedding vectors of a uniform dimension; For each semantic description information group, calculate the cosine similarity and word overlap between the semantic description information and the candidate labels; A matching score is obtained by weighted fusion of cosine similarity and word overlap. The matching relationship between semantic description information and candidate labels is determined based on the matching score, and the pseudo-label pairing between candidate regions and candidate labels is determined based on the matching relationship.

[0009] In conjunction with the first aspect, in one implementation, determining the matching relationship between semantic description information and candidate tags based on matching scores, and determining pseudo-tag pairings between candidate regions and candidate tags based on the matching relationship, includes: For each candidate label, retain the preset number of semantic descriptions with the highest matching score, and generate pseudo-labels by pairing the corresponding candidate regions with the candidate labels. If multiple candidate labels are matched to the same candidate region, retain the pseudo-label pair with the highest matching score and mark it as a conflict pair.

[0010] In conjunction with the first aspect, in one implementation, a confidence assessment of pseudo-label pairings based on a pre-trained cross-modal matching model is performed to obtain an association confidence score; a consistency assessment of pseudo-label pairings obtained from multiple semantic generation strategies is performed to obtain a consistency score; and a comprehensive confidence score is determined based on the association confidence score and the consistency score, including: The pseudo-label pair is input into the pre-trained cross-modal matching model to perform image-text matching and obtain the association confidence. The consistency score is determined based on the number of times a candidate region successfully matches the same candidate label under different semantic generation strategies. The overall confidence score is obtained by weighting and fusing the association confidence score and the consistency score.

[0011] In conjunction with the first aspect, in one implementation, the image to be labeled is input into a pre-trained detection model to extract at least one candidate region, including: The image to be labeled is input into the pre-trained detection model, and the target bounding box is identified and redundant filtering is performed on the image to be labeled to obtain the identified bounding box. Image segmentation is performed based on the identified bounding box as input prompts to obtain a segmentation mask. The image to be labeled is then cropped based on the segmentation mask to obtain the target image. Candidate regions are constructed based on the identified borders, segmentation masks, and target images.

[0012] In conjunction with the first aspect, in one implementation, based on the overall confidence level, the pseudo-label pairings are divided into multiple confidence levels, and a corresponding annotation correction strategy is executed according to the divided confidence levels to obtain the annotation results, including: Based on the overall confidence level, pseudo-label pairings are divided into three confidence levels: the first confidence level, the second confidence level, and the third confidence level. For the first confidence level, the pseudo-label pairing is directly determined as the labeling result; For the second confidence level, a correction prompt signal is generated, and a correction instruction based on the correction prompt signal is received. The pseudo-label pairing is updated according to the correction instruction to obtain the labeling result. For the third confidence level, a re-annotation prompt signal is generated, and a re-annotation instruction based on the re-annotation prompt signal is received. The annotation result is then generated according to the re-annotation instruction.

[0013] Secondly, embodiments of this application provide a visual target annotation device, comprising: The candidate region identification module is used to acquire the image to be labeled and candidate labels, input the image to be labeled into the pre-trained detection model, and extract at least one candidate region; The tag matching module is used to process candidate regions based on multiple semantic generation strategies to obtain multiple sets of semantic description information. Each set of semantic description information is matched with candidate tags to obtain pseudo-tag matching between candidate regions and candidate tags. The label confidence evaluation module is used to evaluate the confidence of pseudo-label pairings based on the pre-trained cross-modal matching model to obtain the association confidence, evaluate the consistency of pseudo-label pairings obtained by various semantic generation strategies to obtain the consistency score, and determine the comprehensive confidence based on the association confidence and the consistency score. The annotation correction module is used to divide pseudo-label pairs into multiple confidence levels based on the overall confidence level, and execute the corresponding annotation correction strategy according to the divided confidence level to obtain the annotation results.

[0014] Thirdly, embodiments of this application provide a visual target annotation device, which includes a processor, a memory, and a visual target annotation program stored in the memory and executable by the processor, wherein when the visual target annotation program is executed by the processor, it implements the steps of the visual target annotation method as described above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a visual target annotation program, wherein when the visual target annotation program is executed by a processor, it implements the steps of the visual target annotation method as described above.

[0016] The beneficial effects of the technical solutions provided in this application include: This application obtains multiple sets of semantic description information through various semantic generation strategies, which enhances the comprehensiveness of semantic expression and significantly improves the accuracy of pseudo-label pairing between candidate regions and candidate labels. Furthermore, by combining the association confidence and the consistency score based on multiple sets of semantic description information to determine the comprehensive confidence, the robustness of confidence evaluation is improved, which can effectively distinguish between high-quality pseudo-labels and noise that is matched correctly by chance, and effectively improve the reliability of visual target annotation. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating an embodiment of the visual target annotation method of this application; Figure 2 This is a schematic diagram of the process for generating candidate regions according to an embodiment of this application; Figure 3 This is a schematic diagram illustrating the process of generating semantic description information according to an embodiment of this application; Figure 4 This is a schematic diagram illustrating the process of generating pseudo-tag pairings according to an embodiment of this application; Figure 5 For this application Figure 4 A detailed flowchart of step S404; Figure 6 This is a schematic diagram illustrating the process of calculating the overall confidence level in an embodiment of this application; Figure 7 This is a schematic diagram illustrating the confidence level grading and label correction process in an embodiment of this application; Figure 8 This is a schematic diagram of the functional modules of an embodiment of the visual target annotation device of this application; Figure 9 This is a schematic diagram of the hardware structure of the visual target annotation device involved in the embodiments of this application. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0020] Firstly, embodiments of this application provide a visual target annotation method.

[0021] In one embodiment, reference is made to Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the visual target annotation method of this application. Figure 1 As shown, visual target annotation methods include: S101. Obtain the image to be labeled and candidate labels. Input the image to be labeled into the pre-trained detection model and extract at least one candidate region. S102. Based on multiple semantic generation strategies, the candidate regions are processed to obtain multiple sets of semantic description information. Each set of semantic description information is matched with the candidate labels to obtain pseudo-label pairings between the candidate regions and the candidate labels. S103. Based on the pre-trained cross-modal matching model, the confidence of pseudo-label pairing is evaluated to obtain the association confidence. The consistency of pseudo-label pairing obtained by various semantic generation strategies is evaluated to obtain the consistency score. The comprehensive confidence is determined based on the association confidence and the consistency score. S104. Based on the overall confidence level, the pseudo-label pairings are divided into multiple confidence levels, and the corresponding labeling correction strategy is executed according to the divided confidence levels to obtain the labeling results.

[0022] Specifically, in the visual image annotation method, for the acquired image to be annotated and candidate labels, the embodiment first needs to identify each target object in the image to be annotated and determine its corresponding candidate region. In the embodiment, a pre-trained detection model is used to generate candidate regions. The pre-trained detection model first identifies each target in the image to be annotated and generates corresponding bounding boxes. Simultaneously, redundancy filtering is performed on the identified bounding boxes; for example, based on the IOU value, obviously overlapping bounding boxes are redundantly filtered. Next, the embodiment uses the identified bounding boxes as input prompts to perform image segmentation, obtaining a pixel-level segmentation mask. The original image is then cropped based on the segmentation mask to obtain the target image. Thus, the constructed data structure includes the identified bounding boxes, the segmentation mask, and the candidate regions of the target image.

[0023] Then, for each candidate region, multiple semantic generation strategies are needed to generate semantic description information to enhance the comprehensiveness of semantic expression. In the embodiment, semantic description can be performed from three branches: attribute rules, local features, and scene relationships.

[0024] In the attribute rule generation branch, the embodiment extracts the predicted category and spatial location information of candidate regions, and generates structured text descriptions based on these information as basic descriptive information. In the local feature generation branch, the embodiment inputs candidate regions into a pre-trained cross-modal generation model. The pre-trained cross-modal generation model generates feature description information focusing on the object's own features based on the overlay image composed of the segmentation mask and the target image, combined with a preset prompt word template. In the scene relationship generation branch, the embodiment combines the image to be labeled, the predicted category, and the spatial location information as conditional prompts, inputting them into the pre-trained cross-modal generation model to obtain contextual description information containing the spatial relationships of multiple objects and the scene context. Through these multiple branches, the embodiment can obtain semantic description information containing comprehensive semantic information.

[0025] The next step is to match a label to each candidate region based on the obtained semantic description information.

[0026] In this embodiment, the semantic description information and candidate tags are first converted into text embedding vectors under a unified dimension. Then, for each semantic description information, the cosine similarity and word overlap between it and each candidate tag are calculated. The cosine similarity and word overlap are then weighted and fused to obtain the matching score between the semantic description information and the candidate tags.

[0027] Then, considering that each candidate tag can match multiple candidate regions and each candidate region can only match one candidate tag when establishing matching relationships, in order to identify conflicting matches during the matching process, the embodiment retains a preset number of semantic description information with the highest matching score for each candidate tag based on the matching score, and generates pseudo-tag pairings between the candidate region corresponding to the semantic description information and the candidate tag. Then, it detects whether multiple candidate tags are paired with the same candidate region in the pseudo-tag pairings. If such a situation is detected, the embodiment retains the pseudo-tag pairing with the highest matching score and marks it as a conflicting pairing, thereby identifying conflicting pairings while matching.

[0028] After generating pseudo-label pairings, it is also necessary to verify whether the paired labels are reliable. In this embodiment, confidence and consistency are used to conduct a comprehensive confidence assessment.

[0029] In the comprehensive confidence assessment, the example first inputs the pseudo-label pairings into a pre-trained cross-modal matching model, and uses the image-text matching branch to calculate the association confidence of each pseudo-label pairing. Then, the example calculates the consistency score of the pseudo-label pairings based on the number of times each candidate region successfully matches the same candidate label under different semantic generation strategies.

[0030] In other embodiments, when calculating the consistency score, the stability of candidate regions being assigned to the same candidate label in different iteration rounds can also be used as the consistency score of the pseudo-label.

[0031] The final embodiment weighted and fused the association confidence score and the consistency score to obtain a comprehensive confidence score for pseudo-label pairings, and combined with this comprehensive confidence score, determined whether the pseudo-label pairings needed to be corrected.

[0032] In this embodiment, the confidence score can be divided into three confidence levels. When the pseudo-label pairing is classified into the first confidence level, which has the highest confidence, it is considered to have high credibility after verification, and the obtained pseudo-label pairing can be directly determined as the annotation result without additional annotation. When the pseudo-label pairing is classified into the second confidence level, it has a certain confidence, and a correction prompt signal will be generated. After the annotator corrects the pseudo-label pairing, the annotation result is obtained. When the pseudo-label pairing is classified into the third confidence level, it is considered to have low credibility. At this time, only the segmented candidate regions are displayed, and a re-annotation prompt signal is generated, requiring the annotator to manually re-annotate.

[0033] Furthermore, in this embodiment, samples can be randomly drawn from the manually corrected annotation results classified under the second confidence level to construct a fine-tuning dataset, resulting in data containing both positive and negative samples. For the bounding box recognition and localization part, the embodiment uses a confidence-weighted selective localization loss for model fine-tuning. For the segmentation mask generation part, the embodiment uses a combination of Dice loss and cross-entropy loss for fine-tuning. Through regular model fine-tuning, an iterative optimization loop can be formed in the visual target annotation process, continuously improving the accuracy of annotation.

[0034] In this embodiment, multiple semantic description information is obtained through various semantic generation strategies, which enhances the comprehensiveness of semantic expression and significantly improves the accuracy of pseudo-label pairing between candidate regions and candidate labels. Furthermore, by combining the association confidence and the consistency score based on multiple sets of semantic description information to determine the comprehensive confidence, the robustness of confidence evaluation is improved, which can effectively distinguish between high-quality pseudo-labels and noise that is matched correctly by chance, and effectively improve the reliability of visual target annotation.

[0035] Furthermore, in one embodiment, Figure 2 This is a schematic diagram of the process for generating candidate regions according to an embodiment of this application, such as... Figure 2 As shown, the image to be labeled is input into a pre-trained detection model, and at least one candidate region is extracted, including: S201. Input the image to be labeled into the pre-trained detection model, and perform target bounding box recognition and redundancy filtering on the image to be labeled to obtain the recognized bounding box. S202. Perform image segmentation based on the identified border as input prompts to obtain a segmentation mask, and crop the image to be labeled according to the segmentation mask to obtain the target image; S203. Construct candidate regions based on the identified borders, segmentation masks, and target images.

[0036] Specifically, in this embodiment, for the acquired images to be labeled, the image format and resolution must first be standardized, such as common formats like JPG, PNG, and BMP, and the longer side of the image must be scaled to no more than 1024 pixels. For candidate labels, it is necessary to ensure that they are short descriptions in Chinese or English.

[0037] In the process of generating candidate regions, the pre-trained detection model includes two parts: recognition and cropping. In this embodiment, it is first determined whether the target label category belongs to a general category. For general categories, a pre-trained general object detection model can be used to extract the top-k most confident bounding boxes. For custom non-general categories, a pre-trained open-set object detection model is used to extract the top-k most confident bounding boxes. Simultaneously, the pre-trained general object detection model or the pre-trained open-set object detection model can also perform preliminary identification of the predicted category of the target corresponding to each bounding box.

[0038] Then, the embodiment uses an IoU-Graph to perform redundancy filtering on the extracted bounding boxes. In the embodiment, using the bounding boxes as nodes, edges are established between nodes with an IoU greater than 0.9. The graph is traversed to cluster overlapping regions, and the bounding boxes with the highest confidence are retained in each cluster.

[0039] For the retained recognition bounding boxes, the embodiment uses them as prompt input for the SAM3 segmentation model to generate a pixel-level segmentation mask for each recognition bounding box. Based on the segmentation mask, the target image of the corresponding region is cropped from the original image to be labeled and uniformly scaled to a preset size. Then, the recognition bounding box, segmentation mask and target image are bound together as the obtained candidate region.

[0040] Furthermore, in one embodiment, Figure 3 This is a schematic diagram illustrating the process of generating semantic description information according to an embodiment of this application, such as... Figure 3 As shown, candidate regions are processed based on various semantic generation strategies to obtain multiple sets of semantic description information, including: S301. Based on the first semantic generation strategy, extract the predicted category and spatial location information of the candidate region, and generate basic description information based on the predicted category and spatial location information; S302. Based on the second semantic generation strategy, the candidate region is input into the pre-trained cross-modal generation model, and feature description information is generated based on the preset prompt word template. S303. Based on the third semantic generation strategy, the image to be labeled, the predicted category, and the spatial location information are combined as conditional prompts and input into the pre-trained cross-modal generation model to obtain contextual description information. S304. Treat basic description information, feature description information, and context description information as multiple sets of semantic description information.

[0041] Specifically, in order to improve the comprehensiveness of semantic expression, the embodiment adopts three semantic generation strategies in the semantic description of candidate regions, respectively from the three levels of attribute rules, local features and scene relationships, to obtain multiple sets of semantic description information.

[0042] Under the attribute rule branch, the implementation example filters out abnormal regions and generates structured descriptions based on the predicted category, relative coordinates, area ratio, and contour features of the candidate regions.

[0043] In this embodiment, the image is divided into nine orientations: top left, top center, top right, left center, center, right center, bottom left, bottom center, and bottom right, and the relative coordinates of each segmentation instance are determined. Then, by calculating the Jaccard similarity between the predicted category and the candidate label of the candidate region, abnormal regions with a similarity of less than 0.2 are filtered out. Next, subordinate objects whose area is less than 10% of the main object and whose IoU with the main object is greater than 0.3 are used as attribute supplements to the main object. A structured description containing basic attributes and spatial orientation is generated according to a preset format as basic descriptive information, such as "the red car in the top left corner" or "the black cup on the table."

[0044] Under the local feature branch, the embodiment generates a mask overlay map of the image to be labeled based on the obtained segmentation mask, wherein the mask region transparency is set to 0.5 and the color is red. Then, it is input into the pre-trained cross-modal generation model, using the first type of cue template to guide the model to generate an object appearance feature description: "describe in detail the color, shape, material and size of {object category} in the image"; using the second type of cue template to guide the model to generate an object action state description: "describe what action {object category} in the image is performing and what state it is in".

[0045] In the scene relationship branch, the implementation generates a spatial orientation description based on the relative coordinates of the candidate region in the image to be labeled. Then, the image to be labeled, the predicted category, and the spatial location information are combined as conditional prompts. Through a pre-trained cross-modal generation model, a comprehensive description containing the spatial relationships of multiple objects and scene context is generated as contextual description information, such as "the person in blue standing next to the car" or "the book placed on the third shelf of the bookshelf".

[0046] In the embodiments, multiple sets of semantic description information are obtained through various semantic generation strategies, which effectively enhances the comprehensiveness of semantic expression.

[0047] Furthermore, in one embodiment, Figure 4 This is a schematic diagram of the process for generating pseudo-tag pairings according to an embodiment of this application, as shown below. Figure 4 As shown, each set of semantic description information is matched with candidate labels to obtain pseudo-label pairings between candidate regions and candidate labels, including: S401. Convert semantic description information and candidate labels into text embedding vectors of uniform dimension; S402. For each semantic description information group, calculate the cosine similarity and word overlap between the semantic description information and the candidate labels; S403: Weighted fusion of cosine similarity and word overlap is used to obtain a matching score; S404. Determine the matching relationship between semantic description information and candidate labels based on the matching score, and determine the pseudo-label pairing between candidate regions and candidate labels based on the matching relationship.

[0048] Furthermore, in one embodiment, Figure 5 For this application Figure 4 The detailed process diagram for step S404 involves determining the matching relationship between semantic description information and candidate tags based on the matching score, and determining the pseudo-label pairing between candidate regions and candidate tags based on the matching relationship, including: S501. For each candidate label, retain the preset number of semantic description information with the highest matching score, and generate pseudo-label pairings between the corresponding candidate region and the candidate label. S502. If multiple candidate labels are matched to the same candidate region, retain the pseudo-label pair with the highest matching score and mark it as a conflict pair.

[0049] Specifically, in the process of pairing candidate labels with candidate regions, the embodiment obtains pseudo-label pairings after semantic description under different dimensions by matching the semantic description information of the candidate regions with the candidate labels under different branches.

[0050] During the pairing process, the embodiment first converts the semantic description information and candidate labels of each candidate region into text embedding vectors of a uniform dimension. This step can be achieved by pre-training a BERT-based Chinese text encoder, which can unify the dimension of the text embedding vectors to 768 dimensions.

[0051] Then, using the transformed text embedding vectors, the cosine similarity between the candidate labels and the semantic description information is calculated. Then, the word overlap between candidate labels and semantic description information in the text is calculated. The word overlap is calculated using the Jaccard coefficient. The final matching score is obtained through weighted averaging. .For example:

[0052] Then, considering that each candidate label can match multiple candidate regions when establishing matching relationships, and each candidate region can only match one candidate label, in order to identify conflicting matches during the matching process, the embodiment does not directly retain the candidate label with the highest matching score with each semantic description information. Instead, it sorts the semantic description information corresponding to each candidate label according to the matching score, retains the top k semantic description information, and generates pseudo-label pairings between the candidate region corresponding to the semantic description information and the candidate label. At the same time, the obtained pseudo-label pairings are detected to check whether multiple candidate labels are paired with the same candidate region. If so, the embodiment retains the pseudo-label pairing with the highest matching score and marks it as a conflicting pairing. This allows for cross-validation, identifying conflicting pairings while matching.

[0053] Furthermore, in one embodiment, Figure 6 This is a flowchart illustrating the calculation of the overall confidence level in an embodiment of this application, as shown below. Figure 6 As shown, a pre-trained cross-modal matching model is used to evaluate the confidence of pseudo-label pairings to obtain the association confidence score. Consistency evaluation is performed on pseudo-label pairings obtained from various semantic generation strategies to obtain the consistency score. The comprehensive confidence score is determined based on the association confidence score and the consistency score, including: S601. Input the pseudo-label pairing into the pre-trained cross-modal matching model and perform image-text matching to obtain the association confidence. S602. Determine the consistency score based on the number of times the candidate region successfully matches the same candidate label under different semantic generation strategies; S603. The association confidence score and consistency score are weighted and fused to obtain the overall confidence score.

[0054] Specifically, after generating pseudo-label pairs, it is necessary to determine the confidence level of the pseudo-label pairs in order to identify which pseudo-label pairs are highly reliable and can be used as annotation results, and which pseudo-label pairs are less reliable and need to be corrected or even re-annotated.

[0055] In this embodiment, the image-text matching branch in the pre-trained cross-modal matching model is first used to perform image-text matching on the pseudo-label pairings, and the matching score, which is between 0 and 1, is output as the association confidence of the pairing.

[0056] Next, by counting the number of times each candidate region successfully matches the same candidate label in at least two different semantic branches, or by counting the stability of the region being assigned the same candidate label in multiple consecutive iterations, the number of times or stability is normalized to the [0,1] interval as the consistency score of the pseudo label.

[0057] Then, the implementation example weights and fuses the association confidence score and the consistency score to obtain the overall confidence score:

[0058] in, Indicates the overall confidence level. Indicates the confidence level of association. Represents the consistency score. As the weighting coefficient, this embodiment takes... .

[0059] Furthermore, based on the obtained overall confidence level, the implementation example can also dynamically calculate the noise filtering threshold. The overall confidence level is lower than The pairs are marked as noise and discarded.

[0060] In the example, the dynamic threshold The update method is as follows: When starting a labeling task for the first time, use the baseline noise threshold. This serves as the initial dynamic threshold. After every three complete iterations of annotation fine-tuning, the following sub-steps are performed: collect the effective comprehensive confidence scores of all pairs in the current batch to form a confidence score set. ; set Sort in ascending order and calculate the 20th percentile of the confidence distribution. ; Calculate the percentage of high-confidence samples with a comprehensive confidence level greater than or equal to 0.85 in the current batch. Based on a target percentage range of 70%-80%, the following is applied: Perform calibration, in order to This represents the updated noise threshold. ,but ,like ,but ,otherwise And on Apply boundary constraints The final updated dynamic threshold is obtained. .

[0061] Furthermore, if the current batch sample size is less than 50, the confidence standard deviation is less than 0.05, or the proportion of high confidence is less than 30%, the threshold update is paused, and the threshold from the previous iteration is used. .

[0062] In this embodiment, after eliminating invalid noise pairs using a dynamic noise threshold, the overall confidence scores of the remaining valid pairs are batch-normalized and used as weights for subsequent labeling and model training. The normalization step can be expressed by the formula:

[0063] in, This represents the minimum overall confidence level among valid pairings. This represents the maximum overall confidence level among valid pairings. This represents the original overall confidence level. This represents the confidence level after normalization.

[0064] Then the implementation example will use the normalized version. Mapped to The interval is used as the weight for subsequent labeling and classification, and model training.

[0065] In this embodiment, the comprehensive confidence level is determined by combining the association confidence level and the consistency score based on multiple sets of semantic description information, which improves the robustness of the confidence level assessment, effectively distinguishes high-quality pseudo-labels from noise that is matched correctly by chance, and improves the reliability of visual target annotation.

[0066] Furthermore, in one embodiment, Figure 7 This is a schematic diagram illustrating the confidence level grading and label correction process in an embodiment of this application, as shown below. Figure 7 As shown, based on the overall confidence level, the pseudo-label pairings are divided into multiple confidence levels, and the corresponding annotation correction strategy is executed according to the divided confidence levels to obtain the annotation results, including: S701. Based on the overall confidence level, the pseudo-label pairings are divided into the first confidence level, the second confidence level, and the third confidence level. S702. For the first confidence level, the pseudo-label pairing is directly determined as the labeling result; S703. For the second confidence level, generate a correction prompt signal and receive a correction instruction based on the correction prompt signal. Update the pseudo-label pairing according to the correction instruction to obtain the labeling result. S704. For the third confidence level, generate a re-annotation prompt signal, receive a re-annotation instruction based on the re-annotation prompt signal, and generate annotation results according to the re-annotation instruction.

[0067] Specifically, in this embodiment, for pseudo-label pairings that have been verified based on overall confidence, the embodiment classifies the pairings according to the overall confidence level to determine whether label correction is needed. The embodiment uses three confidence levels.

[0068] In the first confidence level, for high-confidence samples with a confidence level greater than or equal to 0.85, the implementation example can directly determine the pseudo-label pairing as the labeling result, automatically generate the corresponding bounding box and segmentation mask, as well as the labeling label, without manual intervention.

[0069] For samples with a confidence level greater than or equal to 0.6 but less than 0.85 in the second confidence level, the example generates a correction prompt signal, sends the pseudo-label pairing to the manual processing sequence, and the labeler determines whether to save it directly as the final result, to perform label correction, or to identify it as a low-quality sample and transfer it to the low confidence processing flow.

[0070] For low-confidence samples (less than 0.6) in the third confidence level, a re-annotation prompt signal is generated. Only the obtained candidate regions are displayed, and manual annotation is required. Additionally, if there are mismatched segmented regions, annotators can add new regions or manually correct them.

[0071] In another embodiment, for the annotation results that have been manually corrected at the second confidence level, samples can be randomly extracted to construct a fine-tuning dataset, thereby continuously iteratively optimizing the object detection model and the SAM3 segmentation model.

[0072] In this embodiment, 5%-10% of the samples are randomly selected from the manually confirmed and selected annotation results as positive samples, and the unselected high-confidence candidate regions are used as negative samples to construct a joint fine-tuning dataset containing positive and negative samples. Segmentation error samples are selected first as training data for the SAM3 segmentation model.

[0073] Based on the fine-tuning dataset, the example uses confidence-weighted selective localization loss to fine-tune the target detection model. At the same time, it uses a manually corrected segmentation mask as a supervision signal and a combination of Dice loss and cross-entropy loss to fine-tune the mask decoder and cue encoder of the SAM3 model.

[0074] The confidence-weighted selective localization loss formula is expressed as:

[0075] in, This represents a set of manually labeled samples. For the sample Normalized composite confidence weights, The bounding box is determined manually. Predict bounding boxes for the model.

[0076] The example fine-tunes the target detection model's recognition and localization functions based on the above loss function, while also fine-tuning the mask segmentation function of the SAM3 model.

[0077] Furthermore, for samples with low overall confidence or low pseudo-label consistency scores, generalized cross-entropy loss can be used instead of standard cross-entropy loss during fine-tuning. For example, for samples with an overall confidence score below 0.6 or a pseudo-label consistency score below 0.5, generalized cross-entropy loss can be enabled. The generalized cross-entropy loss function is as follows:

[0078] in, In the example, take .

[0079] Then, the generalized cross-entropy loss and selective localization loss are weighted and combined for optimization of low-quality samples during the fine-tuning process.

[0080] Based on the fine-tuned model, the target annotation steps can be re-executed to obtain higher quality pre-annotation results.

[0081] In this embodiment, pseudo-label pairings are divided into three levels by comprehensive confidence, and automatic approval, manual confirmation and correction, and manual re-annotation strategies are adopted respectively. This effectively reduces the workload of manual review in visual target annotation. At the same time, by constructing a fine-tuning dataset for model iterative fine-tuning, the manual correction information is effectively utilized, and continuous optimization can be carried out to obtain higher pre-annotation results.

[0082] Secondly, embodiments of this application also provide a visual target annotation device.

[0083] In one embodiment, reference is made to Figure 8 , Figure 8 This is a schematic diagram of the functional modules of an embodiment of the visual target annotation device of this application. Figure 8 As shown, the visual target annotation device includes: The candidate region recognition module 801 is used to acquire the image to be labeled and candidate labels, input the image to be labeled into the pre-trained detection model, and extract at least one candidate region. The tag matching module 802 is used to process candidate regions based on multiple semantic generation strategies to obtain multiple sets of semantic description information, and match each set of semantic description information with candidate tags to obtain pseudo-tag matching between candidate regions and candidate tags. The label confidence evaluation module 803 is used to evaluate the confidence of pseudo-label pairing based on the pre-trained cross-modal matching model to obtain the association confidence, evaluate the consistency of pseudo-label pairing obtained by multiple semantic generation strategies to obtain the consistency score, and determine the comprehensive confidence based on the association confidence and the consistency score. The annotation correction module 804 is used to divide the pseudo-label pairings into multiple confidence levels based on the overall confidence level, and execute the corresponding annotation correction strategy according to the divided confidence level to obtain the annotation results.

[0084] The functions of each module in the above-mentioned visual target annotation device correspond to the steps in the above-mentioned visual target annotation method embodiment, and their functions and implementation processes will not be described in detail here.

[0085] Thirdly, embodiments of this application provide a visual target annotation device, which can be a personal computer (PC), laptop computer, server, or other device with data processing capabilities.

[0086] Reference Figure 9 , Figure 9 This is a schematic diagram of the hardware structure of the visual target annotation device involved in the embodiments of this application. In the embodiments of this application, the visual target annotation device may include a processor, a memory, a communication interface, and a communication bus.

[0087] The communication bus can be of any type and is used to interconnect the processor, memory, and communication interface.

[0088] Communication interfaces include input / output (I / O) interfaces, physical interfaces, and logical interfaces used for interconnecting internal components of the visual target annotation device, as well as interfaces used for interconnecting the visual target annotation device with other devices (such as other computing devices or user equipment). Physical interfaces can be Ethernet interfaces, fiber optic interfaces, ATM interfaces, etc.; user equipment can be displays, keyboards, etc.

[0089] Memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), etc.

[0090] The processor can be a general-purpose processor, which can call a visual target annotation program stored in memory and execute the visual target annotation method provided in the embodiments of this application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the visual target annotation program is called can be referred to in the various embodiments of the visual target annotation method of this application, and will not be repeated here.

[0091] Those skilled in the art will understand that Figure 9 The hardware structure shown does not constitute a limitation of this application and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0092] Fourthly, embodiments of this application also provide a computer-readable storage medium.

[0093] The present application provides a computer-readable storage medium storing a visual target annotation program, wherein when the visual target annotation program is executed by a processor, it implements the steps of the visual target annotation method described above.

[0094] The method implemented when the visual target annotation program is executed can be referred to in various embodiments of the visual target annotation method of this application, and will not be repeated here.

[0095] It should be noted that the sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0096] The terms "comprising" and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to such process, method, product, or apparatus. The terms "first," "second," and "third," etc., are used to distinguish different objects, etc., and do not indicate a sequence, nor do they limit "first," "second," and "third" to different types.

[0097] In the description of the embodiments of this application, terms such as "exemplary," "for example," or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for instance" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary," "for example," or "for instance" is intended to present the relevant concepts in a concrete manner.

[0098] In the description of the embodiments of this application, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0099] In some processes described in the embodiments of this application, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order they appear in the embodiments of this application, or they may be executed in parallel. The sequence number of the operation is only used to distinguish different operations, and the sequence number itself does not represent any execution order. In addition, these processes may include more or fewer operations, and these operations or steps may be executed sequentially or in parallel, and these operations or steps may be combined.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device to execute the methods of the various embodiments of this application.

[0101] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A visual target annotation method, characterized in that, include: Obtain the image to be labeled and candidate labels, input the image to be labeled into a pre-trained detection model, and extract at least one candidate region; The candidate region is processed based on multiple semantic generation strategies to obtain multiple sets of semantic description information. Each set of semantic description information is matched with the candidate label to obtain pseudo-label pairing of the candidate region and the candidate label. The association confidence is obtained by evaluating the confidence of the pseudo-label pairing based on the pre-trained cross-modal matching model, and the consistency score is obtained by evaluating the consistency of the pseudo-label pairing obtained by multiple semantic generation strategies. The comprehensive confidence is determined based on the association confidence and the consistency score. Based on the overall confidence level, the pseudo-label pairings are divided into multiple confidence levels, and the corresponding annotation correction strategy is executed according to the divided confidence levels to obtain the annotation results.

2. The visual target annotation method according to claim 1, characterized in that, The candidate regions are processed based on multiple semantic generation strategies to obtain multiple sets of semantic description information, including: Based on the first semantic generation strategy, the predicted category and spatial location information of the candidate region are extracted, and basic description information is generated according to the predicted category and spatial location information. Based on the second semantic generation strategy, the candidate region is input into a pre-trained cross-modal generation model, and feature description information is generated based on a preset prompt word template. Based on the third semantic generation strategy, the image to be labeled, the predicted category, and the spatial location information are combined as conditional prompts and input into a pre-trained cross-modal generation model to obtain contextual description information. The basic description information, the feature description information, and the context description information are used as multiple sets of semantic description information.

3. The visual target annotation method according to claim 1, characterized in that, The step of matching each group of semantic description information with the candidate labels to obtain pseudo-label pairings between the candidate regions and the candidate labels includes: The semantic description information and the candidate labels are converted into text embedding vectors of a uniform dimension; In each semantic description information group, the cosine similarity and word overlap between the semantic description information and the candidate label are calculated; The cosine similarity and word overlap are weighted and fused to obtain a matching score; The matching relationship between the semantic description information and the candidate label is determined based on the matching score, and the pseudo-label pairing between the candidate region and the candidate label is determined based on the matching relationship.

4. The visual target annotation method according to claim 3, characterized in that, The step of determining the matching relationship between the semantic description information and the candidate label based on the matching score, and determining the pseudo-label pairing between the candidate region and the candidate label based on the matching relationship, includes: For each candidate label, retain the preset number of semantic descriptions with the highest matching score, and generate pseudo-labels by pairing the corresponding candidate regions with the candidate labels. If multiple candidate labels are paired to the same candidate region, the pseudo-label pair with the highest matching score is retained and marked as a conflict pair.

5. The visual target annotation method according to claim 1, characterized in that, The process involves evaluating the confidence of the pseudo-label pairings based on a pre-trained cross-modal matching model to obtain an association confidence score, evaluating the consistency of the pseudo-label pairings obtained from multiple semantic generation strategies to obtain a consistency score, and determining a comprehensive confidence score based on the association confidence score and the consistency score, including: The pseudo-label pairings are input into a pre-trained cross-modal matching model to perform image-text matching and obtain the association confidence score. The consistency score is determined based on the number of times the candidate region successfully matches the same candidate label under different semantic generation strategies. The overall confidence level is obtained by weighting and fusing the association confidence level and the consistency score.

6. The visual target annotation method according to claim 1, characterized in that, The step of inputting the image to be labeled into a pre-trained detection model to extract at least one candidate region includes: The image to be labeled is input into a pre-trained detection model, and the image to be labeled is subjected to target bounding box recognition and redundancy filtering to obtain the recognition bounding box; Image segmentation is performed based on the identified border as a prompt input to obtain a segmentation mask, and the image to be labeled is cropped based on the segmentation mask to obtain the target image; Candidate regions are constructed based on the identified bounding box, the segmentation mask, and the target image.

7. The visual target annotation method according to claim 1, characterized in that, The process involves dividing the pseudo-label pairings into multiple confidence levels based on the overall confidence level, and executing corresponding annotation correction strategies according to the divided confidence levels to obtain annotation results, including: Based on the overall confidence level, the pseudo-label pairings are divided into a first confidence level, a second confidence level, and a third confidence level. For the first confidence level, the pseudo-label pairing is directly determined as the annotation result; For the second confidence level, a correction prompt signal is generated, and a correction instruction based on the correction prompt signal is received. The pseudo-label pairing is updated according to the correction instruction to obtain the labeling result. For the third confidence level, a re-annotation prompt signal is generated, and a re-annotation instruction based on the re-annotation prompt signal is received. Annotation results are generated according to the re-annotation instruction.

8. A visual target annotation device, characterized in that, include: The candidate region identification module is used to acquire the image to be labeled and candidate labels, input the image to be labeled into a pre-trained detection model, and extract at least one candidate region; The tag matching module is used to process the candidate region based on multiple semantic generation strategies to obtain multiple sets of semantic description information, and match each set of semantic description information with the candidate tag to obtain a pseudo tag pairing between the candidate region and the candidate tag. The label confidence evaluation module is used to evaluate the confidence of the pseudo-label pairing based on the pre-trained cross-modal matching model to obtain the association confidence, evaluate the consistency of the pseudo-label pairing obtained by multiple semantic generation strategies to obtain the consistency score, and determine the comprehensive confidence based on the association confidence and the consistency score. The annotation correction module is used to divide the pseudo-label pairings into multiple confidence levels based on the comprehensive confidence level, and execute the corresponding annotation correction strategy according to the divided confidence level to obtain the annotation result.

9. A visual target annotation device, characterized in that, The visual target annotation device includes a processor, a memory, and a visual target annotation program stored in the memory and executable by the processor, wherein when the visual target annotation program is executed by the processor, it implements the steps of the visual target annotation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a visual target annotation program, wherein when the visual target annotation program is executed by a processor, it implements the steps of the visual target annotation method as described in any one of claims 1 to 7.

Images