Target component detection and segmentation method based on self-evolution prompt learning

Abstract

The invention belongs to the technical field of computer vision, and particularly discloses a target component detection and segmentation method based on self-evolution prompt learning, which comprises the following steps: acquiring a component data set; training a neural network by using the component data set to obtain a target component detection segmentation model; calculating a loss function according to a detection segmentation result input by the target component detection segmentation model and a real sample result, and performing single-stage end-to-end training fine tuning to obtain a fine-tuned target component detection segmentation model; and inputting the target component image data into the fine-tuned target component detection segmentation model, and outputting a target segmentation result corresponding to the target component image data through the fine-tuned target component detection segmentation model. The method can autonomously evolve and learn, is suitable for prompt information of different granularities, and improves the generalization ability and detection segmentation accuracy of the model. The problem that in the prior art, precision is not high in the aspect of fine-grained detection and segmentation is solved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a target component detection and segmentation method based on self-evolutionary cue learning. Background Art

[0002] Fine-grained object component detection and segmentation are basic tasks in the field of computer vision and are of great significance in this field. Fine-grained object component detection refers to the ability to accurately identify and locate different object categories in an image, or even subcategories within the same category. Fine-grained object component segmentation is to further segment the image at the pixel level to obtain the precise contours and regions of each object. These technologies have broad application prospects in multiple application scenarios such as autonomous driving, medical image analysis, and security monitoring. In recent years, deep learning-based methods have made significant progress in object detection and segmentation tasks, but they still face challenges such as coarse detection and segmentation granularity, strong data dependence, high model complexity, and insufficient robustness and generalization capabilities. In order to address the above challenges, the application of large models has become an important breakthrough in recent years.

[0003] Large models have demonstrated strong learning and generalization capabilities in tasks such as image classification, object detection, and segmentation. Although the image-text registration capabilities of large visual language models such as the Contrastive Language-Image Pre-training (CLIP) model have shown good performance in object-level detection and segmentation tasks, it is difficult to perform more fine-grained detection and segmentation of objects because the structure of object components is more complex than the objects themselves. The large model DINOV2 (Data INTensive Object detection with self-Supervision version 2) that relies on self-supervised learning does not learn as clear and precise semantic information as supervised learning models. In fine-grained component-level detection and segmentation tasks, DINOV2 may not provide sufficiently accurate semantic representations. Summary of the invention

[0004] In view of the above-mentioned deficiencies in the prior art, the present invention provides a method for detecting and segmenting target components based on self-evolutionary prompt learning, which solves the problem of low accuracy in fine-grained detection and segmentation in the prior art.

[0005] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: a target component detection and segmentation method based on self-evolution hint learning, comprising the following steps: S1, obtain component data set; S2. Use the component data set to train the neural network to obtain a target component detection and segmentation model; The target component detection and segmentation model includes an image feature extraction module, a text prompt feature extraction module, a multimodal feature fusion module, and an image detection and segmentation module; The image feature extraction module is used to extract features from the component data set to obtain image features; The text prompt feature extraction module is used to receive object-level text prompt information and component-level text prompt information, convert the object-level text prompt information and component-level text prompt information into an object-level feature vector and a component-level feature vector respectively, obtain a fusion vector according to the object-level feature vector and the component-level feature vector, and use a text encoder to encode the fusion vector to obtain a text prompt feature; The multimodal feature fusion module is used to obtain fused image features based on image features and text prompt features; The image detection and segmentation module is used to obtain the detection and segmentation results based on the fused image features; S3, calculating the loss function through the detection and segmentation results input by the target component detection and segmentation model and the real results of the samples, and performing single-stage end-to-end training fine-tuning to obtain a fine-tuned target component detection and segmentation model; The loss function is:

[0006] in, and represents the weight parameter, represents the cross entropy loss, express loss, and , , represents the number of pixels in the image, Indicates The predicted value of pixels, Indicates The true value of pixels, represents the predicted value of a pixel in the image, Represents the true value of the pixel in the image; S4. Input the target component image data into the fine-tuned target component detection and segmentation model, and output the target segmentation result corresponding to the target component image data through the fine-tuned target component detection and segmentation model.

[0007] The beneficial effects of the above scheme are: The target component detection and segmentation method based on self-evolutionary prompt learning of the present invention can effectively perform fine-grained component-level detection and segmentation of the target in different scenarios, such as urban or rural environments, and expands the application scope of the fine-grained detection and segmentation system. The self-evolutionary prompt learning module can generate adaptive prompts and enhance the distinction between the same component categories in different categories. Through the multimodal feature fusion module, the present invention can autonomously evolve and learn to adapt to prompt information of different granularities, thereby improving the generalization ability of the model and the detection and segmentation accuracy.

[0008] Furthermore, the method further comprises: According to the detection segmentation results and the true labeling results, the detection segmentation score is obtained.

[0009] The beneficial effect of the above further scheme is that by comparing the degree of overlap between the bounding box or pixel mask predicted by the segmentation model and the true annotation, the performance of the model on the object detection or image segmentation task can be quantified. The segmentation score can help researchers or engineers identify the shortcomings of the model and then adjust the network architecture, training strategy or loss function to improve the model accuracy.

[0010] Furthermore, the multimodal feature fusion module is also used to denoise the image features to obtain denoised image features; The fused image features are obtained based on the denoised image features and the text prompt features.

[0011] The beneficial effect of the above further scheme is that denoising can reduce random noise and interference in the image, making the image features clearer, which is beneficial to subsequent tasks such as image recognition, classification or object detection, and improving the recognition accuracy of the system.

[0012] Furthermore, in step S1, the part dataset includes a PartImageNet dataset and a PascalPart dataset, the PartImageNet dataset includes 11 object categories and 40 part categories, and the PascalPart dataset includes 17 object categories and 116 part categories.

[0013] The beneficial effects of the above further scheme are: the two datasets cover a wide range of object categories and component categories, providing rich training samples for the model, helping to improve the model's ability to recognize various complex components, and enhancing the model's versatility and robustness.

[0014] Furthermore, the image feature extraction module is a pre-trained DINOV2 image encoder; The text prompt feature extraction module includes the CLIP text encoder; The multimodal feature fusion module is the Transform network; The image detection and segmentation module is a CNN network.

[0015] The beneficial effect of the above further scheme is that the pre-trained model has learned a general visual representation on massive image data, which enables the model to quickly adapt and achieve good performance when fine-tuning on specific tasks, reducing dependence on large amounts of labeled data and reducing training costs.

[0016] Furthermore, in step S2, the training process of the target component detection and segmentation model is a single-stage end-to-end fine-tuning training; The single-stage end-to-end fine-tuning training used the Adam optimizer and the maximum number of training rounds was 20. The initial learning rate of the single-stage end-to-end fine-tuning training is 1e-3, and it is reduced to half of the original rate every 5 rounds; The batch size for single-stage end-to-end fine-tuning training is 32; During the single-stage end-to-end fine-tuning training, the pre-trained DINOV2 image encoder and CLIP text encoder were used, and all modules in the target part detection and segmentation model except the pre-trained DINOV2 image encoder and CLIP text encoder were trained successively in the PartImageNet dataset and the PascalPart dataset.

[0017] The beneficial effect of the above further scheme is that the feature extraction module integrating the CLIP text encoder not only improves the model's ability to understand the correspondence between image content and text description, but also greatly expands its potential and flexibility in cross-modal applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 Shown is a flowchart of a target component detection and segmentation method based on self-evolving cue learning. DETAILED DESCRIPTION

[0019] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0020] like Figure 1 As shown, a target component detection and segmentation method based on self-evolving hint learning includes the following steps: S1. Obtain component data set.

[0021] In this embodiment, the component datasets include the PartImageNet dataset and the PascalPart dataset. The PartImageNet dataset is a large-scale image dataset specifically used for object component level analysis, designed for the segmentation, detection, and classification tasks of object components, including 11 object categories and 40 component categories. The PascalPart dataset is an image dataset specifically used for computer vision research, mainly focusing on object component segmentation and fine-grained recognition tasks, including 17 object categories and 116 component categories.

[0022] For example, after obtaining the component data set, in order to avoid different sizes of image data in the component data set, before using the component data set to train the neural network, the component data set can be preprocessed, that is, the size of the image data in the component data set can be uniformly adjusted to 512×512 using bilinear interpolation.

[0023] S2. Use the component data set to train the neural network to obtain a target component detection and segmentation model; The target component detection and segmentation model includes an image feature extraction module, a text prompt feature extraction module, a multimodal feature fusion module, and an image detection and segmentation module; The image feature extraction module is used to extract features from the component data set to obtain image features; The text prompt feature extraction module is used to receive object-level text prompt information and component-level text prompt information, convert the object-level text prompt information and component-level text prompt information into an object-level feature vector and a component-level feature vector respectively, obtain a fusion vector according to the object-level feature vector and the component-level feature vector, and use a text encoder to encode the fusion vector to obtain a text prompt feature; The multimodal feature fusion module is used to obtain fused image features based on image features and text prompt features; The image detection and segmentation module is used to obtain detection and segmentation results based on fused image features.

[0024] In this embodiment, the multimodal feature fusion module is also used to denoise the image features to obtain denoised image features; The fused image features are obtained based on the denoised image features and the text prompt features.

[0025] In this embodiment, the image feature extraction module is a pre-trained DINOV2 image encoder; the text prompt feature extraction module includes a CLIP text encoder; the multimodal feature fusion module is a Transform network; and the image detection and segmentation module is a CNN network.

[0026] Exemplarily, the pre-trained DINOV2 (Data INTensive Object detection with self-Supervision version 2, DINOV2) image encoder can be a large self-supervised model pre-trained on 120 million images, whose architecture is based on the Vision Transformer (ViT).

[0027] For example, the component dataset may include an image of a cat, denoted as , this image After entering the parameter-frozen DINOV2 image encoder, we can extract dimensional image feature tensor (i.e., image feature), and is recorded as ,in, R represents the set of real numbers, , are the height and width of the input image, , is the height and width of the output image feature tensor. This process can be expressed as the following expression:

[0028] in, Represents the DINOV2 image encoder.

[0029] Taking the above image data with a uniform size of 512×512 as an example, the image feature tensor that can be output by the DINOV2 image encoder is expressed as .

[0030] Exemplarily, the CLIP text encoder can introduce a multimodal big data pre-training model to perform text encoding on self-evolving prompt parameters. The CLIP text encoder can continuously evolve and learn autonomously to generate adaptive prompts for different scenarios and target granularity inputs.

[0031] For example, the text prompt feature extraction module can receive n sets of object-level text prompt information and component-level text prompt information, and embed the n sets of object-level text prompt information and component-level text prompt information respectively, and convert them into object-level feature vectors and component-level feature vectors. The object-level feature vector and component-level feature vector can be recorded as , The object-level feature vector and the component-level feature vector are fused using learnable vector parameters to obtain a fusion vector, which can be recorded as , The fused vector can be input into the CLIP text encoder and output through the CLIP text encoder The text prompt feature of the dimension can be recorded as . This step can be expressed as the following expression:

[0032]

[0033] in, Represents a hint learner, Represents the CLIP text encoder.

[0034] Taking the above image data with uniform size adjustment to 512×512 as an example, the text prompt feature output by the CLIP text encoder can be expressed as .

[0035] For example, the multimodal feature fusion module can combine image features and text prompt features Fusion is performed to obtain fused image features. This process can be expressed as the following expression:

[0036] in, Represents the Transformer encoder layer in the multimodal feature fusion module.

[0037] Optionally, before the multimodal feature fusion module fuses the image features and the text prompt features, the image features may be denoised to obtain denoised image features. , this process can be expressed as the following expression:

[0038] in, represents the fully connected layer. Represents the Rectified Linear Unit function.

[0039] After obtaining the denoised image features Afterwards, the denoised image features and text prompt features can be Fusion is performed to obtain fused image features. This process can be expressed as the following expression:

[0040] in, Represents the Transformer encoder layer in the multimodal feature fusion module.

[0041] Exemplarily, the image detection and segmentation module can be a general vision device. , through the universal visual The fused image features are processed to obtain the detection and segmentation results.

[0042] S3. Calculate the loss function through the detection and segmentation results of the target component detection and segmentation model input and the real results of the samples, and perform single-stage end-to-end training fine-tuning to obtain the fine-tuned target component detection and segmentation model.

[0043] In this embodiment, the loss function is:

[0044] in, and represents the weight parameter, represents the cross entropy loss, express loss, and , , represents the number of pixels in the image, Indicates The predicted value of pixels, Indicates The true value of pixels, represents the predicted value of a pixel in the image, Represents the true value of a pixel in an image.

[0045] For example, in fine-grained target detection and segmentation, the loss function of the target component detection and segmentation model is not easy to optimize during training because some categories of components may have fewer components than other categories, or the number of pixels occupied by different component categories is too different. Therefore, it is necessary to combine Dice loss and cross entropy loss to fully utilize the advantages of both. Dice loss focuses on the matching of boundary areas, which helps to improve the accuracy of segmentation, especially in the detection and segmentation tasks of small and complex boundaries. Cross entropy loss provides stable gradient information, which helps the model converge faster in the early stages of training. Combining cross entropy loss with Dice loss can make up for the shortcomings of a single loss function, so that the model can be optimized more stably during training.

[0046] S4. Input the target component image data into the fine-tuned target component detection and segmentation model, and output the target segmentation result corresponding to the target component image data through the fine-tuned target component detection and segmentation model.

[0047] In this embodiment, the method further includes: According to the detection segmentation results and the true labeling results, the detection segmentation score is obtained.

[0048] Exemplarily, the real annotation results can be obtained by professional annotators through manual annotation of the component data set, and are used to compare with the output results of the target component detection and segmentation model, so as to determine whether the detection results of the target component detection and segmentation model achieve the expected training effect.

[0049] For example, in this embodiment, the detection and segmentation results obtained by the image detection and segmentation module can be recorded as , the true annotation result can be recorded as , detection segmentation score The calculation expression can be:

[0050] in, Indicates the intersection of the detection segmentation result and the true annotation result. It means to obtain the union of the detection segmentation result and the true annotation result.

[0051] Optionally, in an embodiment of the present invention, when training the target component detection segmentation model, an Adam optimizer can be used and the maximum number of training rounds is 20. The initial learning rate is set to 1e-3, and is reduced to half of the original rate every 5 rounds. In addition, the batch size can be set to 32. The data sets select the PartImageNet data set and the PascalPart data set. In addition, in the training stage, the pre-trained DINOV2 image encoder and CLIP text encoder can be used, and the remaining module parameters of the entire target component detection segmentation model except the pre-trained DINOV2 image encoder and CLIP text encoder are successively trained on the two data sets to obtain the target component detection segmentation model.

[0052] For example, the initial learning rate is set to 1e-3, that is, the value of the initial learning rate is 0.001. In machine learning and deep learning, the learning rate is a hyperparameter used to control the speed of model weight update. Specifically, it determines the step size of the weight update based on the gradient of the loss function in each iteration. 1e-3 is scientific notation, which is equal to 1 times 10 to the power of -3, that is, 0.001.

[0053] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for detecting and segmenting target components based on self-evolving cue learning, characterized in that: The method comprises: S1, obtain component data set; S2. Using the component data set to train a neural network to obtain a target component detection and segmentation model; The target component detection and segmentation model includes an image feature extraction module, a text prompt feature extraction module, a multimodal feature fusion module and an image detection and segmentation module; The image feature extraction module is used to extract features from the component data set to obtain image features; The text prompt feature extraction module is used to receive object-level text prompt information and component-level text prompt information, convert the object-level text prompt information and the component-level text prompt information into an object-level feature vector and a component-level feature vector respectively, obtain a fusion vector according to the object-level feature vector and the component-level feature vector, and encode the fusion vector using a text encoder to obtain a text prompt feature; The multimodal feature fusion module is used to obtain fused image features according to the image features and the text prompt features; The image detection and segmentation module is used to obtain detection and segmentation results according to the fused image features; S3, calculating the loss function through the detection and segmentation results input by the target component detection and segmentation model and the real results of the samples, and performing single-stage end-to-end training fine-tuning to obtain a fine-tuned target component detection and segmentation model; The loss function is: in, and represents the weight parameter, represents the cross entropy loss, express loss, and , , represents the number of pixels in the image, Indicates The predicted value of pixels, Indicates The true value of pixels, represents the predicted value of a pixel in the image, Represents the true value of the pixel in the image; S4. Input the target component image data into the fine-tuned target component detection and segmentation model, and output the target segmentation result corresponding to the target component image data through the fine-tuned target component detection and segmentation model.

2. The method according to claim 1, characterized in that The method further comprises: A detection segmentation score is obtained according to the detection segmentation result and the true annotation result.

3. The method according to claim 1, characterized in that The multimodal feature fusion module is also used to denoise the image features to obtain denoised image features; A fused image feature is obtained according to the denoised image feature and the text prompt feature.

4. The method according to claim 1, characterized in that: In the step S1, the part dataset includes a PartImageNet dataset and a PascalPart dataset, the PartImageNet dataset includes 11 object categories and 40 part categories, and the PascalPart dataset includes 17 object categories and 116 part categories.

5. The method according to claim 1, characterized in that The image feature extraction module is a pre-trained DINOV2 image encoder; The text prompt feature extraction module includes a CLIP text encoder; The multimodal feature fusion module is a Transform network; The image detection and segmentation module is a CNN network.

6. The method according to claim 4, characterized in that In step S2, the training process of the target component detection and segmentation model is a single-stage end-to-end fine-tuning training; The single-stage end-to-end fine-tuning training uses the Adam optimizer during training and the maximum number of training rounds is 20; The initial learning rate of the single-stage end-to-end fine-tuning training is 1e-3, and is reduced to half of the original rate every 5 rounds; The batch size of the single-stage end-to-end fine-tuning training is 32; The single-stage end-to-end fine-tuning training used the pre-trained DINOV2 image encoder and CLIP text encoder during training, and all modules in the target part detection and segmentation model except the pre-trained DINOV2 image encoder and CLIP text encoder were trained successively in the PartImageNet dataset and the PascalPart dataset.

Images