Multi-target sample automatic labeling method based on picture text large model

Abstract

The invention relates to the technical field of multi-target sample automatic labeling, in particular to a multi-target sample automatic labeling method based on a picture text large model. The method comprises the following steps: S1, detecting a picture text by using an open set target detection algorithm Ground DIN0, and outputting a corresponding detection frame in a picture by using a model weight; s2, de-weighting the detection frame by using a non-maximum suppression method, and reserving the category with the maximum confidence in the Grouping DINO output target categories and the detection frame; and S3, storing the detection frame information and the label information in a format required by a labeling platform. According to the multi-target sample automatic labeling method based on the picture text large model, the multi-target samples in the picture are effectively and accurately labeled automatically through the open set target detection algorithm GrondingDIN0, and only a detection frame which is not too close needs to be slightly adjusted, so that the workload of labeling personnel is greatly reduced, and the labeling efficiency is improved. And the marking accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of automated annotation technology for multi-target samples, and more specifically, to an automated annotation method for multi-target samples based on a large image-text model. Background Technology

[0002] With the development of deep learning, artificial intelligence is needed in more and more situations. The most basic and essential part of using artificial intelligence is the collection and annotation of sample data. Depending on the specific task, the dataset may require tens of thousands to millions of images. For such a large amount of data, annotators need to annotate the images one by one, which is not only very labor-intensive but also time-consuming. Therefore, this paper provides an automated annotation method for multi-target samples based on a large image-text model. Summary of the Invention

[0003] The purpose of this invention is to provide an automated annotation method for multi-target samples based on a large image-text model, so as to solve the problems of large workload and long time consumption when annotating massive data as mentioned in the background art.

[0004] To achieve the above objectives, the present invention aims to provide an automated annotation method for multi-target samples based on a large image-text model, comprising the following steps:

[0005] S1. Use the open set object detection algorithm GroundingDINO to detect text in the image, and output the corresponding detection box in the image using the model weights;

[0006] S2. Use the non-maximum suppression method to remove duplicate detection boxes, and retain the category with the highest confidence among the target categories output by GroundingDINO and the detection box.

[0007] S3. Store the detection box information and label information into the format required by the annotation platform.

[0008] As a further improvement to this technical solution, in S1, the open set target detection algorithm GroundingDINO is formed by a detector, ground truth pre-training, cross-modal fusion mechanism, language-guided query selection, feature enhancer, and dynamic box generation and optimization components.

[0009] As a further improvement to this technical solution, in step S1, the open set target detection algorithm GroundingDINO is used to detect text in the image, involving the following steps:

[0010] S3.1, Receive a text message for an image;

[0011] S3.2. Use a convolutional neural network to extract visual feature maps from images;

[0012] S3.3. The input text prompts are converted into dense vector representations using the Transformer encoder;

[0013] S3.4. The extracted image feature maps and text vectors are fused using an attention mechanism;

[0014] S3.5 Initialize a set of decoder queries that are aligned with the text prompts and are dynamically updated during the Transformer decoder process. Through interaction with image features, the queries gradually focus on image regions that match the text descriptions.

[0015] As a further improvement to this technical solution, in S3.3, the extracted image feature map and text vector are fused through the encoder part of the Transformer, involving the following steps:

[0016] S3.31. Each input text is converted into its corresponding word embedding vector;

[0017] S3.32. Add positional encoding, and add the positional encoding vector to the word embedding vector;

[0018] S3.33 After adding the embedding vector with location information, the next step is the multi-head self-attention layer;

[0019] S3.34. Each encoder layer also includes a feedforward neural network, which processes the vector at each position independently;

[0020] S3.35. Add a residual connection to sum the input and output, and then normalize the result using layer normalization.

[0021] As a further improvement to this technical solution, in step S3.4, the extracted image feature map and text vector are fused through an attention mechanism, involving the following steps:

[0022] S3.41. Extract semantic features of text using Transformer;

[0023] S3.42. Fuse the extracted image features and text features;

[0024] S3.43. During the fusion process, the input is weighted according to the importance weight of the input data. In the attention-based feature fusion, each input feature is converted into a vector and multiplied by a learnable attention weight vector. Then, these weighted vectors are summed and processed through an activation function to obtain the final fused feature vector.

[0025] As a further improvement to this technical solution, step S1, which uses model weights to output the corresponding detection boxes in the image, involves the following steps:

[0026] S6.1 Load model weights;

[0027] S6.2, Read the photo you want to detect;

[0028] S6.3 Define the image text query to be detected;

[0029] S6.4. Use the model to infer the image and obtain the detection box;

[0030] S6.5 Draw the detected bounding boxes onto the original image and display the results.

[0031] As a further improvement to this technical solution, in step S6.4, the inference process using the model to analyze the image involves the following steps:

[0032] S6.41. The image needs to be converted to a specific format and adjusted to the size expected by the model.

[0033] S6.42. Determine the text query that the model is looking for;

[0034] S6.43 The model performs inference on image and text queries to obtain prediction results for relevant regions.

[0035] As a further improvement to this technical solution, in step S2, the detection boxes are deduplicated using a non-maximum suppression method, retaining the category and detection box with the highest confidence among the target categories output by GroundingDINO. This involves the following steps:

[0036] S8.1 Sort all detection boxes from highest to lowest confidence level according to their respective categories;

[0037] S8.2 Select the first box from the sorted list (i.e. the box with the highest current confidence) and add it to the final set of retained detection boxes;

[0038] S8.3 For each remaining box in the sorted list, calculate the mask area IoU between that box and each box in the selected box set;

[0039] S8.4 Set a threshold of 0.5. If the IoU between a certain box and any box in the selected box set is greater than the preset threshold, it is considered that the box overlaps too much with the selected boxes and should be removed.

[0040] S8.5. Iterate through all remaining boxes, and each time select the box with the highest confidence that has not been removed and add it to the retention set, until all boxes have been checked.

[0041] As a further improvement to this technical solution, the specific calculation formula for the IoU of the mask area in S8.3 is as follows:

[0042]

[0043] Where X represents the predicted bounding box, Y represents the ground truth bounding box, |X\capY| represents the intersection area of ​​the two boxes, and |X\cupY| represents the union area of ​​the two boxes.

[0044] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0045] 1. This method for automated annotation of multi-target samples based on a large image-text model utilizes the open-set target detection algorithm GroundingDINO to effectively and accurately annotate multi-target samples in images. Only slight adjustments are needed for detection boxes that do not fit well, greatly reducing the workload of annotators and improving the accuracy of annotation. This method can be freely adjusted according to a suitable annotation format and is compatible with all annotation platforms.

[0046] 2. In this automated annotation method for multi-target samples based on a large image-text model, the non-maximum suppression algorithm is used to remove duplicate detection boxes from the model output, and the category with the highest confidence among the target categories output by GroundingDINO is retained, which reduces the amount of modification by annotators and lowers the duplication rate of detection boxes. Attached Figure Description

[0047] Figure 1 This is a flowchart of the overall method of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Example:

[0050] Please see Figure 1 As shown, this embodiment provides an automated annotation method for multi-target samples based on a large image-text model, including the following steps:

[0051] S1. Use the open set object detection algorithm GroundingDINO to detect text in the image, and output the corresponding detection box in the image using the model weights;

[0052] In this embodiment, the open set object detection algorithm GroundingDINO is composed of a detector, ground truth pre-training, cross-modal fusion mechanism, language-guided query selection, feature enhancer, and dynamic bounding box generation and optimization components. It aims to solve the object detection problem in the open world, that is, to locate and identify any object in an image without limiting the candidate categories.

[0053] The GroundingDINO open-set object detection algorithm is used to detect text in images, which involves the following steps:

[0054] S3.1, Receive a text message for an image;

[0055] S3.2 Use convolutional neural networks to extract visual feature maps from images. These feature maps contain the location and semantic information of all pixels in the image.

[0056] Among them, convolutional neural networks extract visual feature maps through convolutional layers, pooling layers, multi-layer stacking, and fully connected layers;

[0057] S3.3. The input text prompts are converted into dense vector representations using the Transformer encoder;

[0058] The process of fusing the extracted image feature maps and text vectors through the encoder part of the Transformer involves the following steps:

[0059] S3.31. Each input text is converted into its corresponding word embedding vector. These word embeddings are usually learned during training. The word embeddings are designed to capture the semantic information of the words.

[0060] S3.32. Since the Transformer model is based on the self-attention mechanism, it does not contain recurrent or convolutional structures to explicitly encode positional information. Therefore, positional encoding needs to be added. The positional encoding vector is added to the word embedding vector to ensure that the model can recognize the relative position of words in the sequence. The positional encoding can be calculated by a fixed function, such as a combination of sine and cosine functions.

[0061] S3.33. After adding the embedding vector with location information, the next step is a multi-head self-attention layer. This layer allows the model to pay attention to multiple positions in the input sequence at the same time. Specifically, it includes the following steps: linear projection, dot product attention, output of the combined head, residual connection and layer normalization.

[0062] S3.34. Each encoder layer also includes a feedforward neural network, which processes the vector at each position independently;

[0063] S3.35. Add a residual connection to sum the input and output, and then normalize the result using layer normalization;

[0064] S3.36. The extracted image feature maps and text vectors are fused through an attention mechanism. This step aims to establish the correspondence between image regions and text descriptions.

[0065] The process of fusing the extracted image feature maps and text vectors using an attention mechanism involves the following steps:

[0066] S3.41. Extract semantic features of text using Transformer;

[0067] S3.42. The extracted image features and text features are fused together, which can be achieved through simple methods such as concatenation and weighted averaging.

[0068] S3.43. During the fusion process, the input is weighted according to the importance weight of the input data. In the attention-based feature fusion, each input feature is converted into a vector and multiplied by a learnable attention weight vector. Then, these weighted vectors are summed and processed through an activation function to obtain the final fused feature vector.

[0069] S3.5 Initialize a set of decoder queries that are aligned with the text prompts and are dynamically updated during the Transformer decoder process. Through interaction with image features, the queries gradually focus on image regions that match the text descriptions.

[0070] The process of outputting the corresponding detection boxes in the image using model weights involves the following steps:

[0071] S6.1 Load model weights;

[0072] S6.2, Read the photo you want to detect;

[0073] S6.3 Define the image text query to be detected;

[0074] S6.4. Use the model to infer the image and obtain the detection box;

[0075] The process of using a model to reason about images involves the following steps:

[0076] S6.41. Images need to be converted to a format, usually PIL Image or Tensor, and adjusted to the size expected by the model.

[0077] S6.42. Determine the text query that the model is looking for;

[0078] S6.43. The model performs inference on image and text queries to obtain prediction results for relevant regions. The output usually includes information such as the coordinates of the prediction box and the score.

[0079] S6.5 Draw the detected bounding boxes onto the original image and display the results.

[0080] S2. Use the non-maximum suppression method to remove duplicate detection boxes, and retain the category with the highest confidence among the target categories output by GroundingDINO and the detection box.

[0081] The purpose of the non-maximum suppression method is to reduce redundancy and ensure that each target corresponds to only one detection box, which is usually the one with the highest confidence.

[0082] In this embodiment, the non-maximum suppression method is used to remove duplicate detection boxes, retaining the category and detection box with the highest confidence among the target categories output by GroundingDINO. This involves the following steps:

[0083] S8.1 Sort all detection boxes by confidence level of their respective categories from highest to lowest. This ensures that the most likely correct detection results are considered first.

[0084] S8.2 Select the first box from the sorted list (i.e. the box with the highest current confidence) and add it to the final set of retained detection boxes;

[0085] S8.3 For each remaining box in the sorted list, calculate the mask area IoU between that box and each box in the selected box set;

[0086] S8.4 Set threshold B. If the IoU between a certain box and any box in the selected box set is greater than the preset threshold, then the box is considered to have too much overlap with the selected boxes and should be removed.

[0087] S8.5. Iterate through all remaining boxes, and each time select the box with the highest confidence that has not been removed and add it to the retention set, until all boxes have been checked.

[0088] In S8.3, the specific formula for calculating the IoU of the mask area is as follows:

[0089]

[0090] Where X represents the predicted bounding box, Y represents the ground truth bounding box, |X\capY| represents the intersection area of ​​the two boxes, and |X\cupY| represents the union area of ​​the two boxes.

[0091] When two boxes do not intersect, the IoU is 0; when two boxes completely overlap, the IoU is 1. The value of IoU is between 0 and 1, reflecting the degree of overlap between the two boxes. The higher the value, the greater the degree of overlap.

[0092] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely preferred examples and are not intended to limit the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. An automated annotation method for multi-target samples based on a large image-text model, characterized in that: Includes the following steps: S1. Use the open set object detection algorithm GroundingDINO to detect text in the image, and output the corresponding detection box in the image using the model weights; S2. Use the non-maximum suppression method to remove duplicate detection boxes, and retain the category with the highest confidence among the target categories output by GroundingDINO and the detection box. S3. Store the detection box information and label information into the format required by the annotation platform.

2. The automated annotation method for multi-target samples based on a large image-text model according to claim 1, characterized in that: In S1, the open set object detection algorithm GroundingDINO is composed of a detector, ground truth pre-training, a cross-modal fusion mechanism, language-guided query selection, a feature enhancer, and a dynamic box generation and optimization component.

3. The automated annotation method for multi-target samples based on a large image-text model according to claim 1, characterized in that: In step S1, the GroundingDINO open-set object detection algorithm is used to detect text in the image, involving the following steps: S3.1, Receive a text message for an image; S3.

2. Use a convolutional neural network to extract visual feature maps from images; S3.

3. The input text prompts are converted into dense vector representations using the Transformer encoder; S3.

4. The extracted image feature maps and text vectors are fused using an attention mechanism; S3.5 Initialize a set of decoder queries that are aligned with the text prompts and are dynamically updated during the Transformer decoder process.

4. The automated annotation method for multi-target samples based on a large image-text model according to claim 3, characterized in that: In step S3.3, the extracted image feature map and text vector are fused through the encoder part of the Transformer, involving the following steps: S3.

31. Each input text is converted into its corresponding word embedding vector; S3.

32. Add positional encoding, and add the positional encoding vector to the word embedding vector; S3.33 After adding the embedding vector with location information, the next step is the multi-head self-attention layer; S3.

34. Each encoder layer also includes a feedforward neural network, which processes the vector at each position independently; S3.

35. Add a residual connection to sum the input and output, and then normalize the result using layer normalization.

5. The automated annotation method for multi-target samples based on a large image-text model according to claim 3, characterized in that: In step S3.4, the extracted image feature map and text vector are fused through an attention mechanism, involving the following steps: S3.

41. Extract semantic features of text using Transformer; S3.

42. Fuse the extracted image features and text features; S3.

43. During the fusion process, the input is weighted according to the importance weight of the input data. In the attention-based feature fusion, each input feature is converted into a vector and multiplied by a learnable attention weight vector. Then, these weighted vectors are summed and processed through an activation function to obtain the final fused feature vector.

6. The automated annotation method for multi-target samples based on a large image-text model according to claim 1, characterized in that: In step S1, the corresponding detection box in the image is output using the model weights, which involves the following steps: S6.1 Load model weights; S6.2, Read the photo you want to detect; S6.3 Define the image text query to be detected; S6.

4. Use the model to infer the image and obtain the detection box; S6.5 Draw the detected bounding boxes onto the original image and display the results.

7. The automated annotation method for multi-target samples based on a large image-text model according to claim 6, characterized in that: In step S6.4, the inference process using the model to analyze the image involves the following steps: S6.

41. The image needs to be converted to a specific format and adjusted to the size expected by the model. S6.

42. Determine the text query that the model is looking for; S6.43 The model performs inference on image and text queries to obtain prediction results for relevant regions.

8. The automated annotation method for multi-target samples based on a large image-text model according to claim 1, characterized in that: In step S2, the detection boxes are deduplicated using a non-maximum suppression method, retaining the category with the highest confidence among the target categories output by GroundingDIN0. This involves the following steps: S8.1 Sort all detection boxes from highest to lowest confidence level according to their respective categories; S8.2 Select the first box from the sorted list and add it to the final set of retained detection boxes; S8.3 For each remaining box in the sorted list, calculate the mask area IoU between that box and each box in the selected box set; S8.4 Set a threshold of 0.

5. If the IoU between a certain box and any box in the selected box set is greater than the preset threshold, it is considered that the box overlaps too much with the selected boxes and should be removed. S8.

5. Iterate through all remaining boxes, and each time select the box with the highest confidence that has not been removed and add it to the retention set, until all boxes have been checked.

9. The automated annotation method for multi-target samples based on a large image-text model according to claim 8, characterized in that: In S8.3, the specific formula for calculating the IoU of the mask area is as follows: Where X represents the predicted bounding box, Y represents the ground truth bounding box, |X\capY| represents the intersection area of ​​the two boxes, and |X\cupY| represents the union area of ​​the two boxes.

Images