Edge-based open vocabulary segmentation model target identification method

Abstract

The invention discloses an edge-based open vocabulary segmentation model target identification method. The method comprises the steps of obtaining a target image; initializing Gaussian white noise as a noise contour mask pattern of a target; inputting the target image and the noise contour mask image into an open vocabulary segmentation model to execute the following operations: extracting multi-scale features of the target from the target image and the noise contour mask image; encoding the noise contour mask pattern to obtain encoding features; the multi-scale features are used as condition features to be fused with the coding features, and fusion features are obtained; predicting a de-noised contour mask graph of the target according to the fusion features; and according to the de-noised contour mask graph and the multi-scale features, a single full connection layer is used to predict classification and positioning of the target. According to the method, target identification based on the open vocabulary segmentation model can be effectively and quickly completed.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to an edge-based open vocabulary segmentation model target recognition method. Background Art

[0002] The object segmentation model is a technology in the field of computer vision, which is used to segment and mark different objects or regions in an image at the pixel level, and has broad application prospects. In the field of traditional image processing, object edge detection, segmentation and classification are crucial tasks, but traditional methods have some limitations. In general, these methods usually rely on manually defined feature extraction and rules, and cannot fully capture complex target structures and changes. Traditional edge detection methods are generally based on features such as edge strength, gradient or texture, and use techniques such as filters and thresholds to extract edge information in images. However, these methods often do not work well for image areas with complex textures, blurred edges or low contrast. In addition, traditional segmentation methods are often based on manually defined rules such as thresholds, region growing or graph cuts, and the segmentation effect is limited for objects with complex shapes and changes. Traditional classification methods require feature vectors and classifiers to be defined in advance, and cannot automatically learn and adapt to the feature representation of different objects. In summary, traditional image processing methods have many challenges in processing complex images, and more advanced technologies are needed to overcome these limitations.

[0003] With the rapid development of deep learning technology, deep learning models can automatically learn the feature representation of images from large-scale labeled data by using multi-layer neural network structures, showing strong pattern recognition capabilities. In particular, in target edge detection, segmentation and classification tasks, deep learning-based methods have achieved remarkable results, greatly improving the accuracy and robustness of these tasks.

[0004] The existing open vocabulary segmentation task based on deep learning models usually uses the existing large open source datasets for model training, such as the COCO dataset, the PASCAL VOC dataset, etc. Its implementation process is generally to use the deep learning model structure to capture multi-level image features, and then restore the features to spatial resolution, so as to achieve pixel-level segmentation of targets and backgrounds, and adjust and optimize the model parameters through the loss function. There are also multiple implementation schemes for the existing open vocabulary segmentation task based on deep learning models: the COD model implements the camouflaged object detection task based on the conditional diffusion model, and processes camouflaged objects that are highly similar to the surrounding environment through a progressive denoising process, optimizing the difference between the object boundary and the background; the COD model uses the Adaptive Transformer Conditional Network (ATCN) network structure to extract multi-scale features of the image, and combines the diffusion model to effectively improve the ability to handle subtle boundary differences, segment the location of the camouflaged object, and avoid the problem of overconfident point estimation. The SAM model combines image data and interactive prompts (such as clicks and box selections) and other multi-modal input forms. It extracts multi-position features in the image based on the Transformer network and generates accurate segmentation masks through the decoder combined with the input. In addition, the model adopts an interactive learning mechanism, quickly optimizes through user feedback, enhances the adaptability to new scenes, and achieves fast processing and efficient segmentation. When the above two models perform target segmentation tasks, the results obtained do not contain target semantic information. The result returned by the COD model is a binary mask map containing all target positions in the image. The result returned by the SAM model is a target mask map at the prompt location or a mask map of all pixels in the image divided by targets when there is no prompt. The target in the image cannot be effectively identified based on the result mask map that does not contain target semantic information. Existing methods all combine the target segmentation model with some semantic feature pre-training models with large parameters to provide target semantic information for the segmentation result map for target recognition. The Open-Vocabulary SAM model combines the SAM model and the CLIP model, and achieves efficient knowledge fusion through innovative bidirectional knowledge transfer modules - SAM2CLIP and CLIP2SAM - so that the model can effectively realize interactive image segmentation and recognition without pre-set categories.

[0005] However, current open vocabulary segmentation models based on deep learning have the following drawbacks:

[0006] (1) Most existing open vocabulary segmentation models focus on pixel-level image segmentation, which can subdivide images into different regions, but often lack the ability to classify and identify these segmented regions. This single segmentation task is not enough to handle complex scenes in the real world, and also limits the adaptability and scalability of the model. In a changing application environment, it is difficult to meet a wider range of scene analysis and decision support needs by simply segmenting without recognition.

[0007] (2) When dealing with blurred boundaries, overlapping objects, or occlusions, the existing open vocabulary segmentation model will have reduced recognition accuracy when faced with objects with severe occlusions, resulting in incorrect segmentation decisions or blurred segmentation boundaries. This not only affects the recognition accuracy of the model, but also reduces its reliability in practical applications.

[0008] (3) Although integrating the recognition task into the open vocabulary segmentation model with the help of a pre-trained model with large parameters and high complexity can assist the segmentation model in object recognition, thereby significantly improving the functionality of the model, this is usually accompanied by an increase in model size and complexity. Such a complex model requires more computing resources for training and reasoning, which may cause deployment difficulties in resource-constrained environments or application scenarios that require high real-time performance.

[0009] Therefore, there is an urgent need for an effective and fast object recognition method based on an open vocabulary segmentation model. Summary of the invention

[0010] In order to solve the above problems existing in the prior art, the present invention provides an edge-based open vocabulary segmentation model target recognition method.

[0011] The technical problem to be solved by the present invention is achieved through the following technical solutions:

[0012] An edge-based open vocabulary segmentation model object recognition method, comprising:

[0013] Acquire a target image; the target image contains a target to be identified;

[0014] Initialize Gaussian white noise as the noise contour mask map of the target;

[0015] The target image and the noise contour mask are input into a pre-trained open vocabulary segmentation model, so that the open vocabulary segmentation model performs the following operations:

[0016] Extracting multi-scale features of the target from the target image and the noise contour mask map; encoding the noise contour mask map to obtain encoded features; fusing the multi-scale features with the encoded features as conditional features to obtain fused features; predicting a denoised contour mask map of the target based on the fused features; and predicting the classification and positioning of the target using a single fully connected layer based on the denoised contour mask map and the multi-scale features.

[0017] Optionally, the open vocabulary segmentation model includes: a PVT network, a DN network and a classification and positioning module;

[0018] The PVT network is used to extract multi-scale features of the target from the target image and the noise contour mask map; the DN network is a denoising diffusion model based on Unet, including an encoder, a feature fusion module and a decoder; the encoder is used to encode the noise contour mask map to obtain encoded features; the feature fusion module is used to fuse the multi-scale features with the encoded features as conditional features to obtain fused features; the decoder is used to predict the denoised contour mask map of the target based on the fused features; the classification and positioning module is used to predict the classification and positioning of the target based on the denoised contour mask map and the multi-scale features using a single fully connected layer.

[0019] Optionally, the classification and positioning module predicts the classification and positioning of the target using a single fully connected layer, including:

[0020] Generating a proposed region corresponding to each potential target by performing edge detection on the denoised contour mask image;

[0021] For each of the proposed regions, extracting features of the region corresponding to the potential target from the multi-scale features according to the proposed region, and predicting the classification and location of the potential target and the corresponding confidence using a single fully connected layer according to the features;

[0022] The classification and positioning of potential targets with a confidence level higher than a confidence threshold are regarded as the classification and positioning of the target in the target image.

[0023] Optionally, the method of training the open vocabulary segmentation model includes:

[0024] Acquire multiple sample images and corresponding annotation information; the sample images contain sample targets, and the annotation information includes position information, classification information, and mask contour information of the sample targets in the sample images;

[0025] Obtain a noise contour mask image of the corresponding sample target from each sample image;

[0026] Constructing a data set using the plurality of sample images and corresponding annotation information and a noise contour mask map;

[0027] The open vocabulary segmentation model is trained using the dataset.

[0028] Optionally, constructing a data set using the plurality of sample images and corresponding annotation information and a noise contour mask map includes:

[0029] Perform boundary preprocessing on the noise contour mask image corresponding to part or all of the sample image to randomly destroy the sample target contour;

[0030] A data set is constructed using the multiple sample images, the corresponding annotation information, and the noise contour mask image after boundary preprocessing.

[0031] Optionally, using the data set to train an open vocabulary segmentation model comprises:

[0032] Inputting the sample image and the corresponding noise contour mask image into the PVT network being trained so that the PVT network outputs multi-scale features of the sample target;

[0033] Inputting the noise contour mask image corresponding to the sample image into the DN network being trained, so that the DN network predicts the denoised contour mask image of the sample target;

[0034] Based on the denoised contour mask and multi-scale features of the sample target, a single fully connected layer is used to predict the classification and location of the sample target.

[0035] The model loss of the open vocabulary segmentation model in training is calculated based on the annotation information corresponding to the sample image, the denoising contour mask map, the noise processed by the denoising diffusion model of the DN network in training during the noisy and denoising processes, and the classification and positioning of the sample target predicted by a single fully connected layer, and the network parameters of the open vocabulary segmentation model are adjusted according to the model loss until the training is completed.

[0036] Optionally, the model loss of the open vocabulary segmentation model in training is calculated based on the annotation information corresponding to the sample image, the denoised contour mask map, the noise processed by the denoising diffusion model of the DN network in training during the denoising and denoising process, and the classification and location of the sample target predicted by a single fully connected layer, including:

[0037] Calculate the segmentation loss of the open vocabulary segmentation model in training based on the mask contour information and denoised contour mask map corresponding to the sample image, and the noise processed by the diffusion model of the DN network in training during the denoising and denoising processes;

[0038] Calculate the classification and positioning loss of the open vocabulary segmentation model in training based on the position information and classification information corresponding to the sample image and the classification and positioning of the sample target predicted by a single fully connected layer in training;

[0039] The segmentation loss and the classification localization loss are combined to obtain the loss of the open vocabulary segmentation model in training.

[0040] Optionally, the segmentation loss is calculated by the following formula:

[0041]

[0042] in, represents the segmentation loss, f represents the mask contour information corresponding to the sample image; f θ represents the denoising contour mask corresponding to the sample image; n represents the noise added by the diffusion model during the denoising process, n θ represents the noise removed by the diffusion model during the denoising process; δ t =(1-t) 2 Expressed as the weight loss rate varying with time t; is the sample target margin loss.

[0043] Optionally, The calculation formula is:

[0044]

[0045] in, E + Indicates the number of pixels belonging to the edge of the sample target in the denoised contour mask corresponding to the sample image, E - represents the number of pixels in the denoised contour mask corresponding to the sample image that do not belong to the edge of the sample target, and λ is the balance E + With E - The weight factor of is the i-th pixel of the j-th edge in the mask contour information corresponding to the sample image The actual pixel value, is the denoised contour mask corresponding to the sample image, and The probability value that the pixel corresponding to the position belongs to the edge of the sample target.

[0046] Optionally, the classification positioning loss is calculated by the following formula:

[0047]

[0048] in, represents the classification and positioning loss, y i is the true classification of the sample target; p iis the classification of the sample target predicted by a single fully connected layer; C is the total number of categories of all sample targets; n represents the number of sample targets predicted by a single fully connected layer; p j Represents the sample target location predicted by a single fully connected layer; y j Indicates the true location of the sample target.

[0049] Compared with the prior art, the edge-based open vocabulary segmentation model target recognition method provided by the present invention has the following beneficial effects:

[0050] (1) The present invention inputs the noise contour mask of the target together with the original target image into the open vocabulary segmentation model. The noise contour mask contains the target edge information, which enables the model to learn and identify the edge features of the target in the image, thereby segmenting the image into different areas with clear boundaries, and using the connectivity of the areas contained in the boundaries to achieve accurate classification and positioning of the target. As a result, the present invention improves the accuracy of the open vocabulary segmentation model in processing blurred boundaries, overlapping or occluded objects in image segmentation tasks, greatly improves the ability to segment the target edge, enables the open vocabulary segmentation model to effectively segment and identify the category and location of the target, and improves the accuracy of the target recognition method of the open vocabulary segmentation model.

[0051] (2) The present invention combines the multi-scale features extracted from the target image and the noise contour mask map as conditional features with the coded features extracted from the noise contour mask map. The comprehensive application of this multi-scale feature allows the model to capture various details and contextual information from basic to advanced levels, greatly enriching the model's visual information processing capabilities at different resolution levels and effectively improving the accuracy of the output.

[0052] (3) The present invention uses a lightweight single fully connected layer to predict the classification and location of the target, and does not require a complex model to complete the classification, thereby significantly improving the computational efficiency of the model, reducing deployment costs, and reducing the demand for hardware resources, allowing the model to run smoothly on low-power devices, thereby improving the usability of the open vocabulary segmentation model.

[0053] In summary, the present invention realizes an effective and fast object recognition method based on an open vocabulary segmentation model.

[0054] The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 It is an internal workflow diagram of an edge-based open vocabulary segmentation model provided by an embodiment of the present invention;

[0056] Figure 2is a structural diagram of an edge-based open vocabulary segmentation model provided by an embodiment of the present invention;

[0057] Figure 3 It is a schematic diagram of the training process of an edge-based open vocabulary segmentation model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0058] The present invention is further described in detail below with reference to specific embodiments, but the embodiments of the present invention are not limited thereto.

[0059] In order to effectively and quickly identify an object based on an open vocabulary segmentation model, an embodiment of the present invention provides an edge-based open vocabulary segmentation model object recognition method, comprising the following steps:

[0060] Step 1: Get the target image.

[0061] Specifically, the target image can be obtained by importing an image, or by collecting an image of the target in a specific scene, thereby obtaining the target image. The target image contains the target to be identified, which can be one target or multiple targets.

[0062] In addition, after obtaining the target image, the target image may be preprocessed to a certain extent. For example, the target image may be converted to a specific size to match the input of a subsequent open vocabulary segmentation model.

[0063] Step 2: Initialize Gaussian white noise as the target noise contour mask.

[0064] Step 3: Input the target image and the noise contour mask into the pre-trained open vocabulary segmentation model so that the open vocabulary segmentation model performs as follows: Figure 1 The steps shown are:

[0065] S10, extracting multi-scale features of the target from the target image and the noise contour mask image;

[0066] S20, encoding the noise contour mask image to obtain a coding feature;

[0067] S30, fusing the multi-scale features as conditional features with the coding features to obtain fused features;

[0068] S40, predicting a denoised contour mask image of the target according to the fused features;

[0069] S50. Based on the denoised contour mask image and multi-scale features, a single fully connected layer is used to predict the classification and location of the target.

[0070] Specifically, the target image is an RGB three-channel image, and the three-channel image and the single-channel noise contour mask image are input into a pre-trained open vocabulary segmentation model, so that the open vocabulary segmentation model predicts the classification of the target according to the above steps S10 to S40.

[0071] Here, in the pre-trained open vocabulary segmentation model, the network parameters have been optimized through pre-training, which can ensure that the open vocabulary segmentation model can accurately predict the classification and location of the target based on the target image and the noise contour mask map.

[0072] After obtaining the classification and localization of the target using the open vocabulary segmentation model, the classification and localization results are combined with the denoised contour mask Figure 1 And output. On this basis, the target image can be further processed to be applied to tasks such as image segmentation and classification prediction in different scenarios.

[0073] Regarding the specific structure and training method of the open vocabulary segmentation model, in order to make the layout of the manual clearer, examples are given later.

[0074] The embodiment of the present invention inputs the noise contour mask of the target together with the original target image into the open vocabulary segmentation model. The noise contour mask contains the target edge information, which enables the model to learn and identify the edge features of the target in the image, thereby segmenting the image into different areas with clear boundaries, and using the connectivity of the areas contained in the boundaries to achieve accurate classification and positioning of the target. As a result, the embodiment of the present invention improves the accuracy of the open vocabulary segmentation model in processing blurred boundaries, overlapping or occluded objects in image segmentation tasks, greatly improves the ability to segment the edge of the target, enables the open vocabulary segmentation model to effectively segment and identify the category and location of the target, and improves the accuracy of the target recognition method of the open vocabulary segmentation model.

[0075] The embodiment of the present invention combines the multi-scale features extracted from the target image and the noise contour mask map as conditional features with the coding features extracted from the noise contour mask map. The comprehensive application of such multi-scale features allows the model to capture various details and contextual information from basic to advanced levels, greatly enriching the model's visual information processing capabilities at different resolution levels and effectively improving the accuracy of the output.

[0076] The embodiment of the present invention uses a lightweight single fully connected layer to predict the classification and positioning of the target, and does not require a complex model to complete the classification, thereby significantly improving the computational efficiency of the model, reducing deployment costs, and reducing the demand for hardware resources, so that the model can run smoothly on low-power devices, thereby improving the availability of the open vocabulary segmentation model.

[0077] Therefore, the embodiment of the present invention realizes an effective and fast object recognition method based on an open vocabulary segmentation model, and solves the problems existing in the prior art.

[0078] The following is an example of the model structure and training method of the open vocabulary segmentation model.

[0079] For example, see Figure 2 As shown, the open vocabulary segmentation model used in the embodiment of the present invention may include: a Pyramid Vision Transformer (PVT) network, a DeNoising (DN) network, and a classification and positioning module.

[0080] Among them, the PVT network is used to extract the multi-scale features of the target from the target image and the noise contour mask map. Specifically, the PVT model is a variant model of the Transformer, see Figure 2 As shown in the figure, the PVT network includes layer1 to layer4; layer1 includes the Input Embedding module and the Transformer Encoder; Input Embedding is used to implement the input embedding of the model, and the Transformer Encoder uses 3*3 convolution to capture the local structure and texture information in the image, and keeps the image data size unchanged through parameters such as padding and step size. Layers 2 to 4 constitute a hierarchical feature extractor, which is used to obtain feature representations of different resolutions and levels, and combined with the Transformer's self-attention mechanism, it can capture the global context information of the image at different scales.

[0081] Specifically, the model structure of the PVT network adopts a pyramid-shaped Transformer design to process multi-scale image features. Each level in the model contains a series of Transformer encoders that use a self-attention mechanism to fuse and encode input data. Specifically, the calculation process is as follows: PVT converts the input into query, key, and value vectors through three weight matrices. For each query vector, it calculates the dot product with all key vectors to obtain an attention score vector, which indicates the importance of each element to the current element; the attention score vector is softmaxed to make its sum equal to 1, and an attention weight vector is obtained, which indicates the contribution ratio of each element to the current element; for each value vector, multiply it by the corresponding attention weight and sum it to obtain an output vector. The final output vector is generated by weighted summation, thereby providing rich contextual information for each processing element.

[0082] The DN network is a denoising diffusion model based on Unet, including an encoder, a feature fusion module and a decoder; the encoder is used to encode the noise contour mask map to obtain the encoded features; the feature fusion module is used to fuse the multi-scale features as conditional features with the encoded features to obtain the fused features; the decoder is used to predict the denoising contour mask map of the target according to the fused features.

[0083] Specifically, the encoder of the DN network adopts a single-channel convolutional neural network including convolutional layers, activation layers, and pooling layers. It uses 3*3 convolutions to fuse the global information of the image, and keeps the structure of the image data unchanged through parameters such as padding and step size. The decoder of the DN network extracts and processes the input image through a combination of depth-separable convolutional layers, activation functions, and pooling layers. By analyzing the image after diffusion processing, the encoding features of the image are effectively extracted.

[0084] The feature fusion module concatenates multi-scale features of the same size with the encoding features, and uses the concatenated features as fusion features.

[0085] In the embodiment of the present invention, the denoising diffusion model DN based on Unet adopts a classic architecture to process the denoising and generation tasks of the image. The model structure includes two main stages of downsampling and upsampling, and each stage uses a series of convolution encoders and decoders to process image features. The specific implementation steps are as follows: the model first downsamples the input image through a series of convolutional layers and pooling layers, which increase the depth and complexity of the feature map by continuously reducing the size of the feature map, capturing the bottom to high-level features in the image. After the downsampling process, the model gradually restores the size and details of the image through the corresponding upsampling path. In this stage, each decoder layer uses the skip connection feature from the corresponding downsampling layer, which helps to retain the detailed information of the image, and gradually expands the size of the feature map through the deconvolution layer, thereby using the deep learning ability of Unet to capture and reconstruct the image details, and the diffusion model enables the gradual generation of clear images from the noisy state, and accurately obtains the denoising contour mask map containing the edge of the target.

[0086] The classification and positioning module is used to predict the classification and positioning of the target using a single fully connected layer based on the denoised contour mask image and multi-scale features.

[0087] Specifically, the classification and positioning module predicts the classification and positioning of the target using a single fully connected layer according to the denoised contour mask image and the multi-scale features, including:

[0088] (1) Generate a proposed region corresponding to each potential target by performing edge detection on the denoised contour mask image.

[0089] It can be understood that the proposed area corresponding to the potential target is also the area where the potential target is located in the denoised contour mask image.

[0090] (2) For each proposed region, the features of the corresponding potential target area are extracted from the multi-scale features according to the proposed region, and based on the features, a single fully connected layer is used to predict the classification and location of the potential target and the corresponding confidence.

[0091] Among them, after extracting the features of the area where the corresponding potential target is located from the multi-scale features according to the proposed area, the features are input into a single fully connected layer, so that the single fully connected layer can predict the classification and positioning of the potential target and the corresponding confidence.

[0092] Here, the segmentation mask image is processed in detail by a single fully connected layer to identify and classify the specific category of each instance. The single fully connected layer adopts a single-layer convolutional neural network structure containing a linear layer and an activation pooling layer, and its network structure is simple. In a single fully connected layer, the features of the denoised contour mask image are first standardized to optimize the network learning efficiency, and then after passing through the fully connected layer, the image features are converted into the final category output. Among them, a single fully connected layer performs a series of linear transformations (weight matrix multiplication plus bias), followed by a nonlinear activation function (such as ReLU or Sigmoid). Such a structure helps the network capture complex patterns and relationships. Through the processing of a single fully connected layer, useful information can be gradually extracted from the input features, and finally a classification decision vector is generated. Each dimension of the vector represents the possibility of a category, and the classification result is determined by the category with the highest probability, thereby obtaining the category information of the target object. The probability mentioned here is also the confidence mentioned above.

[0093] (3) The classification and positioning of potential targets with confidence levels higher than a confidence threshold are regarded as the classification and positioning of targets in the target image.

[0094] It can be understood that potential targets whose confidence is not higher than the confidence threshold are not sufficient to be determined as targets in the target image because of the low confidence of their classification and positioning results, so they are screened out.

[0095] Based on the above network structure, it can be seen that the open vocabulary segmentation model used in the embodiment of the present invention can be regarded as a COD (camouflaged object detection) model, and the model gives the classification and positioning results of the target by learning image features and target edge contour features, thereby improving the segmentation accuracy and target recognition ability of the open vocabulary segmentation model.

[0096] The following is a detailed description of the implementation mechanism of how the open vocabulary segmentation model is trained in the embodiment of the present invention so that it has the above functions and effects.

[0097] Specifically, in an embodiment of the present invention, the method of training the open vocabulary segmentation model includes:

[0098] (1) Acquire multiple sample images and corresponding annotation information; the sample images contain sample objects, and the annotation information includes position information, classification information, and mask contour information of the sample objects in the sample images.

[0099] Specifically, sample images and their corresponding annotation information may be selected according to the actual application scenario. For example, if the actual application scenario is to identify animal species, animal photos may be selected as sample images, and the animal's location information, classification information, and mask contour information may be annotated. If the actual application scenario is to classify and identify vehicles in traffic scenes, photos in traffic scenes may be selected as sample images, and the location information, classification information, and mask contour information of the vehicles may be annotated. The source of the sample images in the embodiments of the present invention is not limited.

[0100] In addition, for sample images of different sizes, they are changed to a uniform size through preprocessing to adapt to the input of the open vocabulary segmentation model.

[0101] Optionally, a ready-made data set such as a PIDXray data set may be obtained as the above-mentioned multiple sample images. The PIDXray data set contains a known annotation file, and no additional annotation information is required. Moreover, the sizes of the images are the same, and no image size preprocessing operation is required.

[0102] (2) Obtain the noise contour mask map of the corresponding sample target from each sample image.

[0103] Specifically, for a sample image generated by oneself, the edge of the sample target can be detected from the sample image by edge detection, and then the contour mask of the sample target is generated. For an existing data set such as a PIDXray data set, since it usually contains a known annotation file, the required contour mask can be directly generated according to the annotation file. Then, by adding noise to the contour mask of the sample target, a noise contour mask of the sample target is obtained.

[0104] (3) Construct a dataset using multiple sample images, corresponding annotation information and noise contour mask images.

[0105] Specifically, each sample image and its corresponding annotation information and noise contour mask image are taken as a training sample, thereby constructing a data set containing multiple training samples.

[0106] In addition, in an optional implementation, the noise contour mask image corresponding to part or all of the sample images may be subjected to boundary preprocessing to randomly destroy the sample target contour.

[0107] The boundary preprocessing of the noise contour mask image is to erode the noise contour mask image through pixel-level structure. The specific processing means may include data enhancement and expansion erosion, etc., which will not be elaborated in the embodiment of the present invention.

[0108] Therefore, a dataset is constructed using multiple sample images and corresponding annotation information and noise contour mask images after boundary preprocessing. Using such a dataset to train an open vocabulary segmentation model can effectively correct the problem of model deviation in prediction.

[0109] (4) Use the dataset to train an open vocabulary segmentation model.

[0110] Specifically, first initialize the network parameters and iteration times of the open vocabulary segmentation model. The initial network parameters of the PVT network, DN network, and a single fully connected layer can all use the parameters trained on their own public datasets. Then, refer to Figure 3 , training samples are obtained from the dataset in batches, so that the open vocabulary segmentation model performs corresponding processing on each training sample. The specific training process is as follows:

[0111] (4-1) Inputting the sample image and the corresponding noise contour mask image into the PVT network being trained so that the PVT network outputs multi-scale features of the sample target;

[0112] (4-2) inputting the noise contour mask image corresponding to the sample image into the DN network being trained, so that the DN network predicts the denoised contour mask image of the sample target;

[0113] (4-3) Based on the denoised contour mask image and multi-scale features of the sample target, a single fully connected layer is used to predict the classification and location of the sample target;

[0114] (4-4) Calculate the model loss of the open vocabulary segmentation model in training based on the annotation information corresponding to the sample image, the denoised contour mask image, the noise processed by the diffusion model of the DN network in training during the denoising and denoising process, and the classification of the sample target predicted by a single fully connected layer, and adjust the network parameters of the open vocabulary segmentation model based on the model loss until the training is completed.

[0115] Specifically, the segmentation loss of the open vocabulary segmentation model in training is calculated based on the mask contour information and denoised contour mask map corresponding to the sample image, as well as the noise processed by the diffusion model of the DN network in training during the denoising and denoising process; the classification and positioning loss of the open vocabulary segmentation model in training is calculated based on the position information and classification information corresponding to the sample image and the classification and positioning of the sample target predicted by the single fully connected layer in training; the model loss of the open vocabulary segmentation model in training is obtained by combining the segmentation loss and the classification and positioning loss. The network parameters of the open vocabulary segmentation model are adjusted according to this model loss, so that the network parameters of the open vocabulary segmentation model are adjusted and optimized in the direction of reducing the model loss. This process is repeated until a preset number of iterations is reached, or the model loss converges and no longer decreases, and the training is completed. The current network parameters are saved to obtain the trained open vocabulary segmentation model.

[0116] The segmentation loss is calculated as follows:

[0117]

[0118] in, represents the segmentation loss, f represents the mask contour information corresponding to the sample image; f θ represents the denoised contour mask corresponding to the sample image, ||f θ -f|| is used to quantize f and f θ By introducing this difference into the segmentation loss, the difference between the mask contour information corresponding to the sample image and the denoised contour mask map can be minimized, thereby minimizing the model's error in data reconstruction or prediction; n represents the noise added by the diffusion model during the denoising process, n θ represents the noise removed by the diffusion model during the denoising process, ||n θ -n|| is used to quantize n and n θ The difference between them can be introduced into the segmentation loss to improve the accuracy of the model in noise modeling or denoising tasks; t =(1-t) 2 Expressed as the weight loss rate varying with time t; is the sample target margin loss.

[0119] here, The calculation formula is:

[0120]

[0121] in, E + Indicates the number of pixels belonging to the edge of the sample target in the denoised contour mask corresponding to the sample image, E -represents the number of pixels in the denoised contour mask corresponding to the sample image that do not belong to the edge of the sample target, and λ is the balance E + With E - The weight factor of is the i-th pixel of the j-th edge in the mask contour information corresponding to the sample image The actual pixel value, is the denoised contour mask corresponding to the sample image, and The probability value that the pixel corresponding to the position belongs to the edge of the sample target.

[0122] The classification loss is calculated as follows:

[0123]

[0124] Among them, y i is the true classification of the sample target; p i is the classification of the sample target predicted by a single fully connected layer; C is the total number of categories of all sample targets, n represents the number of sample targets predicted by a single fully connected layer; p j Represents the sample target location predicted by a single fully connected layer; y j represents the true location of the sample target, Represents the classification and localization loss. By calculating the classification and localization loss, the difference between the classification and localization of the sample target predicted by the open vocabulary segmentation model and the true classification and localization of the sample target is minimized.

[0125] After obtaining the classification and positioning loss and the segmentation loss, the two are combined. There are many specific ways to combine them, for example, you can calculate As model loss, or computing and As the model loss, this is all OK.

[0126] Therefore, in each round of training, the network parameters of the open vocabulary segmentation model are adjusted according to the model loss, so that the network parameters of the open vocabulary segmentation model are adjusted and optimized in the direction of reducing the model loss. This process is repeated until the preset number of iterations is reached, or the model loss converges and no longer decreases, and the training is completed. The current network parameters are saved to obtain the trained open vocabulary segmentation model.

[0127] In addition, after training is completed, if you find that the prediction accuracy of the open vocabulary segmentation model is not high in actual testing, you can continue training by optimizing the data set or optimizing the training process.

[0128] It should be noted that it is possible to achieve Figure 1 The open vocabulary segmentation model of the operation flow shown is not limited to Figure 2The model structure shown, any solution that transforms the existing model so that it can realize the function of the edge-based open vocabulary segmentation model in the embodiment of the present invention, belongs to the protection scope of the embodiment of the present invention.

[0129] The method provided in the embodiment of the present invention can be applied to electronic devices. Specifically, the electronic device can be: a desktop computer, a portable computer, an intelligent mobile terminal, a server, etc. This is not limited here, and any electronic device that can implement the present invention belongs to the protection scope of the present invention.

[0130] It should be noted that the terms "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification.

[0132] Although the present application is described herein in conjunction with various embodiments, in the process of implementing the claimed application, those skilled in the art can understand and implement other changes to the disclosed embodiments by viewing the drawings and the disclosed content. In the description of the present invention, the term "comprising" does not exclude other components or steps, "one" or "an" does not exclude multiple situations, and the meaning of "multiple" is two or more, unless otherwise clearly and specifically limited. In addition, certain measures are recorded in different embodiments, but this does not mean that these measures cannot be combined to produce good results.

[0133] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment), or computer program products. Therefore, the present application may adopt the form of complete hardware embodiments, complete software embodiments, or embodiments in combination with software and hardware, which are collectively referred to as "modules" or "systems" herein. Moreover, the present application may adopt the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program codes. The computer program is stored / distributed in a suitable medium, provided together with other hardware or as a part of hardware, or may adopt other distribution forms, such as by Internet or other wired or wireless telecommunication systems.

[0134] The present application is described with reference to the flowcharts and / or block diagrams of the methods, apparatuses (devices) and computer program products of the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0135] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0136] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0137] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A target recognition method based on an edge-based open vocabulary segmentation model, characterized in that: include: Get the target image; The target image contains a target to be identified; Initialize Gaussian white noise as the noise contour mask map of the target; The target image and the noise contour mask are input into a pre-trained open vocabulary segmentation model, so that the open vocabulary segmentation model performs the following operations: Extracting multi-scale features of the target from the target image and the noise contour mask image; The noise contour mask image is encoded to obtain encoding features; the multi-scale features are fused with the encoding features as conditional features to obtain fused features; the denoised contour mask image of the target is predicted based on the fused features; and the classification and positioning of the target are predicted using a single fully connected layer based on the denoised contour mask image and the multi-scale features.

2. The edge-based open vocabulary segmentation model target recognition method according to claim 1, characterized in that: The open vocabulary segmentation model includes: a PVT network, a DN network and a classification and positioning module; The PVT network is used to extract multi-scale features of the target from the target image and the noise contour mask map; the DN network is a denoising diffusion model based on Unet, including an encoder, a feature fusion module and a decoder; the encoder is used to encode the noise contour mask map to obtain encoded features; the feature fusion module is used to fuse the multi-scale features with the encoded features as conditional features to obtain fused features; the decoder is used to predict the denoised contour mask map of the target based on the fused features; the classification and positioning module is used to predict the classification and positioning of the target based on the denoised contour mask map and the multi-scale features using a single fully connected layer.

3. The edge-based open vocabulary segmentation model target recognition method according to claim 2, characterized in that: The classification and positioning module predicts the classification and positioning of the target using a single fully connected layer, including: Generating a proposed region corresponding to each potential target by performing edge detection on the denoised contour mask image; For each of the proposed regions, extracting features of the region corresponding to the potential target from the multi-scale features according to the proposed region, and predicting the classification and location of the potential target and the corresponding confidence using a single fully connected layer according to the features; The classification and positioning of potential targets with a confidence level higher than a confidence threshold are regarded as the classification and positioning of the target in the target image.

4. The edge-based open vocabulary segmentation model target recognition method according to claim 2, characterized in that: The method of training the open vocabulary segmentation model includes: Acquire multiple sample images and corresponding annotation information; the sample images contain sample targets, and the annotation information includes position information, classification information, and mask contour information of the sample targets in the sample images; Obtain a noise contour mask image of the corresponding sample target from each sample image; Constructing a data set using the plurality of sample images and corresponding annotation information and a noise contour mask map; The open vocabulary segmentation model is trained using the dataset.

5. The edge-based open vocabulary segmentation model target recognition method according to claim 4, characterized in that: Constructing a data set using the plurality of sample images and corresponding annotation information and a noise contour mask map, including: Perform boundary preprocessing on the noise contour mask image corresponding to part or all of the sample image to randomly destroy the sample target contour; A data set is constructed using the multiple sample images, the corresponding annotation information, and the noise contour mask image after boundary preprocessing.

6. The edge-based open vocabulary segmentation model target recognition method according to claim 4, characterized in that: Using the dataset to train an open vocabulary segmentation model, including: Inputting the sample image and the corresponding noise contour mask image into the PVT network being trained so that the PVT network outputs multi-scale features of the sample target; Inputting the noise contour mask image corresponding to the sample image into the DN network being trained, so that the DN network predicts the denoised contour mask image of the sample target; Based on the denoised contour mask and multi-scale features of the sample target, a single fully connected layer is used to predict the classification and location of the sample target. The model loss of the open vocabulary segmentation model in training is calculated based on the annotation information corresponding to the sample image, the denoising contour mask map, the noise processed by the denoising diffusion model of the DN network in training during the noisy and denoising processes, and the classification and positioning of the sample target predicted by a single fully connected layer, and the network parameters of the open vocabulary segmentation model are adjusted according to the model loss until the training is completed.

7. The edge-based open vocabulary segmentation model object recognition method according to claim 6, characterized in that: Based on the annotation information corresponding to the sample image, the denoised contour mask map, the noise processed by the denoising diffusion model of the DN network in training during the denoising and denoising process, and the classification and positioning of the sample target predicted by a single fully connected layer, the model loss of the open vocabulary segmentation model in training is calculated, including: Calculate the segmentation loss of the open vocabulary segmentation model in training based on the mask contour information and denoised contour mask map corresponding to the sample image, and the noise processed by the diffusion model of the DN network in training during the denoising and denoising processes; Calculate the classification and positioning loss of the open vocabulary segmentation model in training based on the position information and classification information corresponding to the sample image and the classification and positioning of the sample target predicted by a single fully connected layer in training; The segmentation loss and the classification localization loss are combined to obtain the loss of the open vocabulary segmentation model in training.

8. The edge-based open vocabulary segmentation model object recognition method according to claim 7, characterized in that: The segmentation loss is calculated by the following formula: in, represents the segmentation loss, f represents the mask contour information corresponding to the sample image; f θ represents the denoising contour mask corresponding to the sample image; n represents the noise added by the diffusion model during the denoising process, n θ represents the noise removed by the diffusion model during the denoising process; δ t =(1-t) 2 Expressed as the weight loss rate varying with time t; is the sample target margin loss.

9. The edge-based open vocabulary segmentation model object recognition method according to claim 8, characterized in that: The calculation formula is: in, E + Indicates the number of pixels belonging to the edge of the sample target in the denoised contour mask corresponding to the sample image, E - represents the number of pixels in the denoised contour mask corresponding to the sample image that do not belong to the edge of the sample target, and λ is the balance E + With E - The weight factor of is the i-th pixel of the j-th edge in the mask contour information corresponding to the sample image The actual pixel value, is the denoised contour mask corresponding to the sample image, and The probability value that the pixel corresponding to the position belongs to the edge of the sample target.

10. The edge-based open vocabulary segmentation model object recognition method according to claim 7, characterized in that: The classification and positioning loss is calculated by the following formula: in, represents the classification and positioning loss, y i is the true classification of the sample target; p i is the classification of the sample target predicted by a single fully connected layer; C is the total number of categories of all sample targets; n represents the number of sample targets predicted by a single fully connected layer; p j Represents the sample target location predicted by a single fully connected layer; y j Indicates the true location of the sample target.

Images