A method, device and medium for single-sample generalized target segmentation based on input data guidance

Abstract

The present invention provides a single-sample generalized target segmentation method, device and medium based on input data guidance, belonging to the field of single-sample generalization technology. The method includes: collecting training set data and performing preprocessing; constructing and training a single-sample generalized target segmentation model, the single-sample generalized target segmentation model includes a target segmentation framework and a decoder; adjusting a prompt image and its corresponding mask data, and generating a random vector of the same size as the training image as the guide data; combining the guide data with the prompt image and inputting it into the single-sample generalized target segmentation model to obtain the trained guide data; collecting the image to be detected, and outputting the target segmentation result based on the guide data through the model. The present invention utilizes the strong context association and reasoning ability of VIT to enhance the model feature extraction ability; by directly using the input data as a guide, it reduces the dependence on external semantic information and improves the generalization ability and robustness of the model.

Description

Technical Field

[0001] The present invention relates to a single-sample generalization target segmentation method, device and medium based on input data guidance, belonging to the technical field of single-sample generalization. Background Art

[0002] With the rapid development of machine learning technology, single-sample generalization has become a research field that has attracted much attention. Single-sample generalization refers to the ability of a machine learning model to maintain a certain level of prediction accuracy when faced with previously unseen data categories. This ability is particularly important when processing new categories of data, especially when there is a lack of training data. Among the existing single-sample generalization technology solutions, there are mainly the following strategies:

[0003] Leverage semantic information: This includes using attributes to describe each category and using these attributes to infer unseen categories. For example, in the animal classification task, attributes such as "feathered" and "flying" are used to describe birds. The characteristics of the category are expressed through text descriptions, and the model learns the mapping from text to visual features to identify new categories.

[0004] Shared feature space: Mapping visual features and semantic features (such as attribute vectors or text embeddings) into the same space enables the model to make predictions for unseen categories based on semantic information. Combining information from different modalities such as images and text enables the model to understand data from multiple perspectives and enhances the model's generalization ability.

[0005] Cross-modal learning: Enhance the generalization ability of the model by sharing information between different modalities. Consider data from multiple modalities simultaneously during training to improve the generalization performance of the model.

[0006] However, although these techniques have solved the problem of single-sample generalization to a certain extent, there are still some significant limitations and challenges:

[0007] Regarding reliance on semantic information, attribute descriptions may not be accurate enough or may not fully capture all key features of the category, causing the model to perform poorly on unseen categories. If the text description is of low quality, the model may not be able to correctly learn the semantic information, which in turn affects the prediction accuracy.

[0008] Regarding the alignment problem in feature space, visual features and semantic features may be distributed in different spaces, making alignment difficult. In high-dimensional feature space, the alignment process may become complex and computationally expensive.

[0009] Regarding the challenges of cross-modal learning, the correlation between different modalities may not be strong enough, affecting the learning effect of the model. There may be inconsistencies between images and text descriptions, causing the model to learn incorrect associations. Summary of the invention

[0010] The purpose of the present invention is to provide a single-sample generalized target segmentation method, device and medium based on input data guidance, so as to improve the segmentation performance of the model in a zero-sample scenario by means of data guidance.

[0011] In order to achieve the above object, the present invention is implemented through the following technical solutions:

[0012] A single-sample generalized target segmentation method based on input data guidance includes the following steps:

[0013] Collect training set data and perform preprocessing;

[0014] Constructing and training a single-sample generalized target segmentation model, the single-sample generalized target segmentation model includes a target segmentation framework and a decoder; the target segmentation framework selects a Vision Transformer model with the MLP Head deleted, and the output features of the Vision Transformer model are input into the decoder through a fully connected layer;

[0015] Adjust the prompt image and its corresponding mask data, and generate a random vector of the same size as the training image as the guide data;

[0016] The guidance data is combined with the prompt image and then input into the single-sample generalized target segmentation model to obtain the trained guidance data;

[0017] Collect the image to be detected, splice the trained guidance data with the image to be detected after preprocessing, and then input it into the single-sample generalized target segmentation model. The model outputs the target segmentation result based on the guidance data.

[0018] Preferably, the Vision Transformer model is pre-trained using the ImageNet dataset, and the trained model parameters are used as parameters of the target segmentation framework.

[0019] Preferably, the decoder is a 3-layer upsampling decoder, which restores the spatial resolution through transposed convolution, and its output resolution corresponds to the mask size of the training set.

[0020] Preferably, the loss function of the single-sample generalized target segmentation model is as follows:

[0021] ,

[0022] ,

[0023] ,

[0024]

[0025] in, is the set of positions of all pixels in the image, Represents the category index, Pixel The first of the true label vectors at elements, is the probability vector predicted by the model. elements, is the coefficient weighted according to the importance of the edge, is the probability predicted by the model, The scale set is in the scale The loss function on is in scale The loss function on is the importance weight of the corresponding scale, and is a hyperparameter.

[0026] Preferably, the trained guidance data is obtained through a single-sample generalized target segmentation model, specifically in the following manner:

[0027] The guidance data and the prompt image are combined and input into the single-sample generalized target segmentation model, and the loss function of the model output and the prompt image mask is calculated; the value of the guidance data is adjusted using the gradient descent method until the loss is less than the set value, and the trained guidance data is obtained.

[0028] Preferably, the set value is 0.0001.

[0029] Preferably, the preprocessing method is as follows: preprocessing small-sized data through upsampling and bilinear interpolation, preprocessing large-sized data through downsampling and bilinear interpolation, and unifying the dimensions of all data sets and masks.

[0030] A single-sample generalized target segmentation device based on input data guidance comprises a processor and a memory storing program instructions, wherein the processor is configured to execute the single-sample generalized target segmentation method based on input data guidance when running the program instructions.

[0031] A computer-readable storage medium stores a computer program, which is used by a processor to implement a single-sample generalized target segmentation method guided by input data.

[0032] The advantages of the present invention are:

[0033] The present invention reduces the reliance on external semantic information and improves the generalization and robustness of the model by directly using the input data (i.e., the prompt image and its corresponding mask data) as a guide. By utilizing the powerful feature extraction capability of the VisionTransformer (ViT) model and combining it with a training strategy based on input data guidance, the alignment problem of the feature space is effectively improved, and the prediction accuracy of the model for unseen categories is improved. Using the target to be detected directly as prompt data simplifies the complexity of cross-modal learning, avoids the problem of inconsistency between modalities, and makes it easier for the model to learn effective feature representations from the input data.

[0034] The present invention introduces edge-sensitive loss and multi-scale loss in the loss function design, so that the model pays more attention to the accuracy of edge pixels and the retention of multi-scale information in the segmentation task. This design effectively improves the accuracy and edge sensitivity of the segmentation results, making the segmentation results more accurate and detailed.

[0035] Compared with the single-sample generalization technology based on text prompts, the present invention does not rely on complex prompt writing skills, digitizes and fixes the prompt steps, and greatly reduces the dependence on prompt words. At the same time, the model parameter scale is small, which is more conducive to deployment and optimization in practical applications.

[0036] In summary, the single-sample generalized target segmentation method based on input data guidance proposed in the present invention shows significant beneficial effects in reducing semantic information dependence, enhancing feature space alignment capability, simplifying cross-modal learning complexity, improving segmentation accuracy and edge sensitivity, and reducing model parameters. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0038] Figure 1 It is a schematic diagram of the process of the present invention.

[0039] Figure 2 This is a flow chart of the guiding data training phase of the present invention.

[0040] Figure 3 It is a schematic diagram of the process flow of the application stage of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] Example 1

[0043] like Figure 1-Figure 3 As shown, a single-sample generalization target segmentation method based on input data guidance includes the following steps:

[0044] Model training phase

[0045] S1: Collect training set data and perform preprocessing.

[0046] S2: Construct and train a single-sample generalized target segmentation model, which includes a target segmentation framework and a decoder; the target segmentation framework selects a Vision Transformer model with the MLP Head deleted, and the output features of the Vision Transformer model are input into the decoder through a fully connected layer.

[0047] Bootstrap data training phase

[0048] S3: Adjust the prompt image and its corresponding mask data, and generate a random vector of the same size as the training image as the guide data.

[0049] S4: Combine the guidance data with the prompt image and input them into the single-sample generalized target segmentation model to obtain the trained guidance data.

[0050] Application Phase

[0051] S5: Collect the image to be detected, splice the trained guidance data with the image to be detected after preprocessing, and input them into the single-sample generalized target segmentation model. The model outputs the target segmentation result based on the guidance data.

[0052] As a refinement of the above embodiment, the Vision Transformer model in the model training stage is pre-trained with the ImageNet dataset, and the trained model parameters are used as the parameters of the target segmentation framework. The target segmentation framework selects the Vision Transformer model with the MLP Head deleted, that is, selects the feature parameters of its penultimate layer as the target segmentation framework of the segmentation model, and its size is [batchsize, 197, 768], where batchsize is a training parameter.

[0053] The size of the fully connected layer is 1024*1. The decoder is a 3-layer upsampling decoder. The spatial resolution is restored by transposed convolution. The output resolution corresponds to the mask size of the training set. The output of the last layer is [batchsize, 640, 640].

[0054] The training data can use open source object segmentation datasets to train the above networks, PASCAL VOC 2012, Cityscapes, COCO, ADE20K, SUN-RGBD, ADE20K-SceneParsing.

[0055] As a refinement of the above embodiment, since the sizes of the images and the corresponding masks of the above data sets are inconsistent, in order to make the data sets adapt to the same segmentation model, the data sets are preprocessed, and the preprocessing method is as follows: the small-sized data is preprocessed by upsampling and bilinear interpolation, and the large-sized data is preprocessed by downsampling and bilinear interpolation, and all data sets and masks are unified in dimension. For example: for pictures with a size less than 640*640, upsampling is used, and the interpolation method is selected as bilinear interpolation; for pictures with a size greater than 640*640, downsampling is used, and the interpolation method is selected as bilinear interpolation. All pictures and masks are unified in dimension to 640*640.

[0056] As a refinement of the above embodiment, the loss function of the single-sample generalized target segmentation model is as follows:

[0057] ,

[0058] in, Cross entropy loss is a commonly used loss function, which is widely used in classification tasks, including but not limited to image classification, speech recognition, natural language processing and other fields. In the field of computer vision, especially in image segmentation tasks, cross entropy loss is also frequently used. In image segmentation tasks, each pixel needs to be classified into a specific category. The details are as follows:

[0059] .

[0060] To make the loss function more sensitive to edge pixels, an edge-aware loss term is added to Dice Loss to penalize those cases where prediction errors are made in edge areas. The details are as follows:

[0061] .

[0062] For multi-scale loss, although the input image has been unified to a size of 640x640, information at different scales is still important. This embodiment designs a multi-scale loss function, calculates the loss at multiple scales, and then combines them to help the model learn feature representations at different scales. The details are as follows:

[0063] .

[0064] in, is the set of positions of all pixels in the image, Represents the category index, Pixel The first of the true label vectors at elements, is the probability vector predicted by the model. elements, is the coefficient weighted according to the importance of the edge, is the probability predicted by the model, The scale set is in the scale The loss function on is in scale The loss function on is the importance weight of the corresponding scale, and is a hyperparameter.

[0065] It should be noted that the loss function is designed to comprehensively consider pixel-level classification, multi-scale information retention, and edge sensitivity, so as to improve the overall performance of the segmentation task.

[0066] As a refinement of the above embodiment, the single-sample generalization target segmentation model optimizer selects the Adam optimizer, which is a very popular adaptive learning rate optimization algorithm that combines the advantages of AdaGrad and RMSProp and is widely used in the training of deep learning models. A key feature of the Adam optimizer is that it can automatically adjust the learning rate of each parameter, which enables it to perform well in many cases, especially when dealing with high-dimensional and sparse data.

[0067] The model training process is as follows: Use the pre-trained ViT model as the initial weight of the target segmentation framework (encoder). Load data in batches, forward propagate the input image, calculate the loss, and perform backpropagation to update the weights. At the end of each epoch, evaluate the performance of the model on the validation set to monitor the progress of training. Adjust hyperparameters such as learning rate and batch size based on the performance of the validation set; use evaluation indicators such as Dice Coefficient to evaluate the performance of the model on the test set, and finally obtain a single-sample generalized target segmentation model.

[0068] As a refinement of the above embodiment, in the guidance data training stage, the prompt image and its corresponding mask data are adjusted, upsampled or downsampled to 640*640, and a random vector of the same size as the training image is generated as the guidance data.

[0069] Optionally, the guidance data is combined with the prompt image and input into the single-sample generalized object segmentation model to obtain the trained guidance data. The specific method is as follows:

[0070] The guidance data is combined with the prompt image into one image by concat method, and input into the VIT-based target segmentation model to obtain the output of the model and the corresponding mask data of the prompt image to perform Dice Loss.

[0071] The model parameters for this round of training are the guide data. The gradient descent method is used to adjust the value of the guide data until the output of the model, the loss of the mask value corresponding to the prompt image, is less than the set value, which is set to 0.0001 here.

[0072] Finally, the guidance data corresponding to the prompt picture is obtained.

[0073] Taking pest and disease data images as an example: the single-sample generalized target segmentation method guided by input data has very high value in practical applications. For application scenarios that require rapid response to new categories, especially small sample scenarios, this method can identify the target object in a timely and accurate manner. Timely and accurate identification and prevention of pests and diseases are of great significance to ensuring food security and reducing economic losses. However, due to the wide variety of pests and diseases, complex occurrence patterns, and differences in different regions and seasons, it is very difficult to obtain comprehensive and detailed pest and disease image data. Using the single-sample generalized target segmentation method guided by input data, even with limited data, an effective model can be quickly trained to respond to sudden pest and disease situations in a timely manner. Through early identification and early warning, it helps farmers take timely measures to reduce economic losses caused by pests and diseases. The automated recognition system can greatly improve the efficiency of pest and disease monitoring and reduce the workload of manual inspections.

[0074] Specifically,

[0075] (1) Model training phase

[0076] Collect existing pest and disease image data as training set data and perform preprocessing. Build and train a single-sample generalized target segmentation model.

[0077] (2) Guided data training phase

[0078] Adjust the prompt image and its corresponding mask data, and generate a random vector of the same size as the training image as the guide data. Combine the guide data with the prompt image and input it into the single-sample generalized target segmentation model to obtain the trained guide data.

[0079] (3) Application stage

[0080] The image data of pests and diseases to be detected are collected, and after preprocessing, the trained guidance data is spliced ​​with the image to be detected and input into the single-sample generalized target segmentation model. The model outputs the target segmentation result based on the guidance data.

[0081] Example 2

[0082] The disclosed embodiment also provides a single-sample generalized target segmentation device based on input data guidance, including a processor and a memory. Optionally, the device may also include a communication interface and a bus. The processor, the communication interface, and the memory may communicate with each other through the bus. The communication interface may be used for information transmission. The processor may call the logic instructions in the memory to execute the single-sample generalized target segmentation method based on input data guidance of the above embodiment.

[0083] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product.

[0084] The memory, as a computer-readable storage medium, can be used to store software programs and computer executable programs, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes the function application and data processing by running the program instructions / modules stored in the memory, that is, the single-sample generalized target segmentation method guided by input data in the above embodiment is implemented.

[0085] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the terminal device, etc. In addition, the memory may include a high-speed random access memory and may also include a non-volatile memory.

[0086] An embodiment of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are configured to execute the above-mentioned single-sample generalized target segmentation method guided by input data.

[0087] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0088] The technical solution of the embodiment of the present disclosure can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for enabling a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in the embodiment of the present disclosure. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0089] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A single-sample generalized target segmentation method based on input data guidance, characterized in that: The following steps are involved: Collect training set data and perform preprocessing; Constructing and training a single-sample generalized target segmentation model, the single-sample generalized target segmentation model includes a target segmentation framework and a decoder; the target segmentation framework selects a Vision Transformer model with the MLP Head deleted, and the output features of the Vision Transformer model are input into the decoder through a fully connected layer; Adjust the prompt image and its corresponding mask data, and generate a random vector of the same size as the training image as the guide data; The guidance data is combined with the prompt image and then input into the single-sample generalized target segmentation model to obtain the trained guidance data; Collect the image to be detected, and after preprocessing, splice the trained guide data with the image to be detected and input them into the single-sample generalized target segmentation model. The model outputs the target segmentation result based on the guide data; The loss function of the single-sample generalized target segmentation model is as follows: ， , ， in, is the set of positions of all pixels in the image, Represents the category index, Pixel The first of the true label vectors at elements, is the probability vector predicted by the model. elements, is the coefficient weighted according to the importance of the edge, is the probability predicted by the model, The scale set is in the scale The loss function on is in scale The loss function on is the importance weight of the corresponding scale, and is a hyperparameter; Obtain the trained guidance data through the single-sample generalized target segmentation model as follows: The guidance data and the prompt image are combined and input into the single-sample generalized target segmentation model, and the loss function of the model output and the prompt image mask is calculated; the value of the guidance data is adjusted using the gradient descent method until the loss is less than the set value, and the trained guidance data is obtained.

2. The single-sample generalized target segmentation method based on input data guidance according to claim 1, characterized in that: The Vision Transformer model is pre-trained using the ImageNet dataset, and the trained model parameters are used as parameters of the target segmentation framework.

3. The single-sample generalized target segmentation method based on input data guidance according to claim 1, characterized in that: The decoder is a 3-layer upsampling decoder that restores the spatial resolution through transposed convolution, and its output resolution corresponds to the mask size of the training set.

4. The single-sample generalized target segmentation method based on input data guidance according to claim 1, characterized in that: The set value is 0.0001.

5. The single-sample generalized target segmentation method based on input data guidance according to claim 1, characterized in that: The training set data is preprocessed as follows: small-sized data is preprocessed by upsampling and bilinear interpolation, large-sized data is preprocessed by downsampling and bilinear interpolation, and all data sets and masks are unified in dimension.

6. A single-sample generalized target segmentation device guided by input data, comprising a processor and a memory storing program instructions, characterized in that: The processor is configured to execute the single-sample generalized target segmentation method based on input data guidance according to any one of claims 1 to 5 when running the program instructions.

7. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method described in any one of claims 1 to 5 is implemented.

Images