An edge-based open-vocabulary segmentation model object recognition method

Abstract

The application discloses an edge-based open-vocabulary segmentation model target recognition method, comprising the following steps: acquiring a target image; initializing a Gaussian white noise as a noise contour mask graph of the target; inputting the target image and the noise contour mask graph into an open-vocabulary segmentation model to perform the following operations: extracting multi-scale features of the target from the target image and the noise contour mask graph; encoding the noise contour mask graph to obtain encoded features; fusing the multi-scale features as conditional features with the encoded features to obtain fused features; predicting a denoising contour mask graph of the target according to the fused features; and predicting the classification and positioning of the target by using a single full connection layer according to the denoising contour mask graph and the multi-scale features. The application can effectively and quickly complete the target recognition based on the open-vocabulary segmentation model.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to an edge-based open vocabulary segmentation model for target recognition. Background Technology

[0002] Object segmentation is a technique in computer vision used to segment and label different objects or regions in an image at the pixel level, with broad application prospects. In traditional image processing, object edge detection, segmentation, and classification are crucial tasks, but traditional methods have limitations. Generally, these methods rely on manually defined feature extraction and rules, failing to fully capture complex object structures and variations. Traditional edge detection methods typically rely on features such as edge strength, gradient, or texture, using techniques like filters and thresholding to extract edge information from images. However, these methods often perform poorly on image regions with complex textures, blurred edges, or low contrast. Furthermore, traditional segmentation methods often rely on manually defined rules such as thresholding, region growing, or graph cutting, resulting in limited segmentation effectiveness for objects with complex shapes and variations. Traditional classification methods require pre-defined feature vectors and classifiers, failing to automatically learn and adapt to the feature representations of different objects. In summary, traditional image processing methods face numerous challenges when processing complex images, necessitating more advanced techniques to overcome these limitations.

[0003] With the rapid development of deep learning technology, deep learning models, through the use of multi-layer neural network structures, can automatically learn feature representations of images from large-scale labeled data, demonstrating powerful pattern recognition capabilities. Especially in tasks such as object edge detection, segmentation, and classification, deep learning-based methods have achieved remarkable results, greatly improving the accuracy and robustness of these tasks.

[0004] Existing deep learning-based open vocabulary segmentation tasks typically employ large open-source datasets for model training, such as the COCO and PASCAL VOC datasets. The general process involves using deep learning models to capture multi-level image features, then restoring the spatial resolution of these features to achieve pixel-level object and background segmentation, and finally adjusting and optimizing model parameters using a loss function. Several implementation schemes exist for existing deep learning-based open vocabulary segmentation tasks: the COD model implements a camouflaged object detection task based on a conditional diffusion model. It processes camouflaged objects highly similar to their surroundings through a progressive denoising process, optimizing the distinction between object boundaries and background. The COD model uses an Adaptive Transformer Conditional Network (ATCN) network structure to extract multi-scale features from images and combines it with a diffusion model, effectively improving the ability to handle subtle boundary differences, segmenting the location of camouflaged objects, and avoiding the problem of overconfident point estimation. The SAM model combines image data with multimodal input formats such as interactive prompts (clicks, selection boxes, etc.). Based on a Transformer network, it extracts multi-location features from the image and generates an accurate segmentation mask by combining these features with the input through a decoder. Furthermore, the model employs an interactive learning mechanism, rapidly optimizing based on user feedback to enhance its adaptability to new scenes, achieving fast processing and efficient segmentation. In object segmentation tasks, the results obtained by the two models mentioned above do not contain semantic information about the target. The COD model returns a binary mask containing all target locations in the image, while the SAM model returns a target mask at the prompt location or, in the absence of a prompt, a mask that divides all pixels in the image according to the target. Based on the resulting mask without semantic information, it is impossible to effectively identify targets in the image. Existing methods combine object segmentation models with pre-trained models using semantic features with large parameters to provide semantic information about the target in the segmentation result image for target recognition. The Open-Vocabulary SAM model combines the SAM and CLIP models, and achieves efficient knowledge fusion through innovative bidirectional knowledge transfer modules—SAM2CLIP and CLIP2SAM. This enables the model to effectively perform interactive image segmentation and recognition even without pre-defined categories.

[0005] However, current deep learning-based open vocabulary segmentation models have the following drawbacks:

[0006] (1) Most existing open vocabulary segmentation models focus on pixel-level image segmentation, which can subdivide images into different regions, but often lack subsequent classification and recognition of these segmented regions. This single segmentation task is inadequate when dealing with complex scenes in the real world, and also limits the adaptability and scalability of the model. In the ever-changing application environment, it is difficult to meet the broader needs of scene analysis and decision support by simply segmenting without recognition.

[0007] (2) When dealing with blurred boundaries, overlapping objects, or occlusion, existing open vocabulary segmentation models experience a decrease in recognition accuracy when faced with objects that are severely occluded, leading to incorrect segmentation decisions or blurry segmentation boundaries. This not only affects the model's recognition accuracy but also reduces its reliability in practical applications.

[0008] (3) Although integrating the recognition task into the open vocabulary segmentation model with a pre-trained model that has a large number of parameters and high complexity can assist the segmentation model in object recognition, thereby significantly improving the model's functionality, this is usually accompanied by an increase in model size and complexity. Such complex models require more computing resources for training and inference, which may cause deployment difficulties in resource-constrained environments or application scenarios that require high real-time performance.

[0009] Therefore, there is an urgent need for an effective and fast target recognition method based on an open vocabulary segmentation model. Summary of the Invention

[0010] To address the aforementioned problems in the existing technology, this invention provides an edge-based open vocabulary segmentation model for target recognition.

[0011] The technical problem to be solved by this invention is achieved through the following technical solution:

[0012] An edge-based open vocabulary segmentation model for object recognition includes:

[0013] Acquire a target image; the target image contains the target to be identified;

[0014] Initialize Gaussian white noise as a noise contour mask for the target;

[0015] The target image and the noise contour mask are input into a pre-trained open vocabulary segmentation model, so that the open vocabulary segmentation model performs the following operations:

[0016] Multi-scale features of the target are extracted from the target image and the noise contour mask; the noise contour mask is encoded to obtain encoded features; the multi-scale features are fused with the encoded features as conditional features to obtain fused features; a denoised contour mask of the target is predicted based on the fused features; and the classification and localization of the target are predicted using a single fully connected layer based on the denoised contour mask and the multi-scale features.

[0017] Optionally, the open vocabulary segmentation model includes: a PVT network, a DN network, and a classification and localization module;

[0018] The PVT network is used to extract multi-scale features of the target from the target image and the noise contour mask; the DN network is a denoising diffusion model based on Unet, including an encoder, a feature fusion module, and a decoder; the encoder is used to encode the noise contour mask to obtain encoded features; the feature fusion module is used to fuse the multi-scale features as conditional features with the encoded features to obtain fused features; the decoder is used to predict the denoised contour mask of the target based on the fused features; the classification and localization module is used to predict the classification and localization of the target using a single fully connected layer based on the denoised contour mask and the multi-scale features.

[0019] Optionally, the classification and localization module uses a single fully connected layer to predict the classification and localization of the target, including:

[0020] By performing edge detection on the denoised contour mask image, a proposed region corresponding to each potential target is generated;

[0021] For each proposed region, features of the region where the corresponding potential target is located are extracted from the multi-scale features based on the proposed region, and the classification and location of the potential target and the corresponding confidence level are predicted using a single fully connected layer based on the features.

[0022] The classification and localization of potential targets with confidence levels higher than the confidence threshold are used as the classification and localization of targets in the target image.

[0023] Optionally, the open vocabulary segmentation model can be trained in the following ways:

[0024] Multiple sample images and corresponding annotation information are acquired; the sample images contain sample targets, and the annotation information includes the location information, classification information, and mask contour information of the sample targets in the sample images.

[0025] Obtain the noise contour mask of the corresponding sample target from each sample image;

[0026] A dataset is constructed using the multiple sample images, their corresponding annotation information, and noise contour mask images.

[0027] The dataset was used to train an open vocabulary segmentation model.

[0028] Optionally, a dataset is constructed using the multiple sample images, corresponding annotation information, and noise contour mask images, including:

[0029] Boundary preprocessing is performed on the noise contour mask map corresponding to some or all of the sample images to randomly destroy the target contour of the sample;

[0030] A dataset is constructed using the multiple sample images, their corresponding annotation information, and the noise contour mask image after boundary preprocessing.

[0031] Optionally, an open-vocabulary segmentation model is trained using the dataset, including:

[0032] The sample image and the corresponding noise contour mask are input into the PVT network being trained, so that the PVT network outputs multi-scale features of the sample target.

[0033] The noise contour mask corresponding to the sample image is input into the DN network under training so that the DN network can predict the denoised contour mask of the sample target.

[0034] Based on the denoised contour mask and multi-scale features of the sample target, a single fully connected layer is used to predict the classification and localization of the sample target.

[0035] Based on the annotation information corresponding to the sample image, the denoised contour mask, the noise processed by the denoising diffusion model of the DN network during the denoising and denoising process, and the classification and localization of the sample target predicted by a single fully connected layer, the model loss of the open vocabulary segmentation model during training is calculated. The network parameters of the open vocabulary segmentation model are adjusted according to the model loss until training is completed.

[0036] Optionally, based on the annotation information corresponding to the sample image, the denoised contour mask, the noise processed by the denoising diffusion model of the trained DN network during the denoising and denoising processes, and the classification and localization of the sample target predicted by a single fully connected layer, the model loss of the trained open vocabulary segmentation model is calculated, including:

[0037] Based on the mask contour information and denoised contour mask image corresponding to the sample image, as well as the noise processed by the diffusion model of the DN network during the noise addition and denoising process, calculate the segmentation loss of the open vocabulary segmentation model in training.

[0038] Based on the location information and classification information of the sample image and the classification and localization of the sample target predicted by a single fully connected layer in training, calculate the classification and localization loss of the open vocabulary segmentation model in training;

[0039] Combining the segmentation loss and the classification and localization loss, we obtain the loss of the open vocabulary segmentation model during training.

[0040] Optionally, the segmentation loss is calculated using the following formula:

[0041]

[0042] in, The segmentation loss is represented by f, where f represents the mask contour information corresponding to the sample image; f θ This represents the denoised contour mask corresponding to the sample image; n represents the noise added by the diffusion model during the noise addition process. θ δ represents the noise removed during the denoising process by the diffusion model. t =(1-t) 2 Expressed as the rate of weightlessness as a function of time t; This is the edge loss for the sample target.

[0043] Optionally, The calculation formula is:

[0044]

[0045] in, E + E represents the number of pixels belonging to the edge of the sample target in the denoised contour mask image corresponding to the sample image. - λ represents the number of pixels in the denoised contour mask corresponding to the sample image that do not belong to the edge of the sample target, and λ is the balance E + With E - Weighting factors It is the i-th pixel of the j-th edge in the mask contour information corresponding to the sample image. The actual pixel value, In the denoised contour mask image corresponding to the sample image, and The probability value that the pixel corresponding to the location belongs to the edge of the sample target.

[0046] Optionally, the classification and localization loss is calculated using the following formula:

[0047]

[0048] in, y represents the classification and localization loss. i It is the true classification of the sample target; p iis the classification of the sample target predicted by a single fully connected layer; C is the total number of categories for all sample targets; n represents the number of sample targets predicted by a single fully connected layer; p j This represents the target localization predicted by a single fully connected layer; y j This indicates the true location of the target in the sample.

[0049] Compared with existing technologies, the edge-based open vocabulary segmentation model target recognition method provided by this invention has the following beneficial effects:

[0050] (1) This invention inputs the noise contour mask of the target along with the original target image into the open vocabulary segmentation model. The noise contour mask contains target edge information, enabling the model to learn and recognize the edge features of the target in the image, thereby segmenting the image into different regions with clear boundaries. By utilizing the connectivity of the regions contained within the boundaries, accurate classification and localization of the target can be achieved. Thus, this invention improves the accuracy of the open vocabulary segmentation model in handling situations such as blurred boundaries, overlapping objects, or occlusions in image segmentation tasks, greatly enhancing the segmentation ability for target edges. This allows the open vocabulary segmentation model to effectively segment and identify the category and localization of the target, improving the accuracy of the target recognition method of the open vocabulary segmentation model.

[0051] (2) The present invention uses the multi-scale features extracted from the target image and the noise contour mask as conditional features and fuses them with the encoded features extracted from the noise contour mask. This comprehensive application of multi-scale features allows the model to capture various details and contextual information from basic to advanced levels, greatly enriching the model's visual information processing capabilities at different resolution levels and effectively improving the accuracy of the output.

[0052] (3) This invention uses a lightweight single fully connected layer to predict the classification and localization of targets, without the need for a complex model to complete the classification. This significantly improves the computational efficiency of the model, reduces deployment costs, reduces the demand for hardware resources, and enables the model to run smoothly on low-power devices, thereby improving the availability of open vocabulary segmentation models.

[0053] In summary, this invention provides an effective and fast target recognition method based on an open vocabulary segmentation model.

[0054] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0055] Figure 1 This is an internal workflow diagram of an edge-based open vocabulary segmentation model provided in an embodiment of the present invention;

[0056] Figure 2This is a schematic diagram of the structure of an edge-based open vocabulary segmentation model provided in an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram illustrating the training process of an edge-based open vocabulary segmentation model provided in an embodiment of the present invention. Detailed Implementation

[0058] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0059] To achieve effective and rapid target recognition based on open vocabulary segmentation models, this invention provides an edge-based target recognition method using open vocabulary segmentation models, comprising the following steps:

[0060] Step 1: Obtain the target image.

[0061] Specifically, target images can be obtained by importing images or by capturing images of the target in a specific scene. The target image contains the target to be identified, which can be one or more targets.

[0062] In addition, after obtaining the target image, it can be preprocessed, such as converting it to a specific size to match the input of the subsequent open vocabulary segmentation model.

[0063] Step 2: Initialize Gaussian white noise as the noise contour mask for the target.

[0064] Step 3: Input the target image and noise contour mask into the pre-trained open vocabulary segmentation model, so that the open vocabulary segmentation model can perform the following... Figure 1 The operation steps shown are as follows:

[0065] S10. Extract multi-scale features of the target from the target image and the noise contour mask;

[0066] S20. Encode the noise contour mask to obtain the encoded features;

[0067] S30. Fuse the multi-scale features as conditional features with the encoded features to obtain the fused features;

[0068] S40. Predict the denoised contour mask of the target based on the fusion features;

[0069] S50. Based on the denoised contour mask and multi-scale features, a single fully connected layer is used to predict the classification and localization of the target.

[0070] Specifically, the target image is an RGB three-channel image. The three-channel image and the single-channel noise contour mask are input into a pre-trained open vocabulary segmentation model so that the open vocabulary segmentation model can predict the classification of the target according to the above steps S10 to S40.

[0071] Here, the network parameters of the pre-trained open vocabulary segmentation model have been optimized through pre-training, which can ensure that the open vocabulary segmentation model can accurately predict the classification and localization of the target based on the target image and the noise contour mask.

[0072] After obtaining the classification and localization of the target using an open vocabulary segmentation model, the classification and localization results are combined with a denoised contour mask. Figure 1 The output is then processed. Based on this, the target image can be further processed for applications such as image segmentation and classification prediction in different scenarios.

[0073] To clarify the layout of the instruction manual, examples will be provided later regarding the specific structure and training methods of the open vocabulary segmentation model.

[0074] In this embodiment of the invention, a noise contour mask of the target, along with the original target image, is input into an open vocabulary segmentation model. This noise contour mask contains target edge information, enabling the model to learn and recognize the edge features of the target in the image, thereby segmenting the image into different regions with clear boundaries. Utilizing the connectivity of the regions contained within these boundaries, accurate classification and localization of the target are achieved. Therefore, this embodiment of the invention improves the accuracy of the open vocabulary segmentation model in handling situations such as blurred boundaries, overlapping objects, or occlusion in image segmentation tasks, significantly enhancing its ability to segment target edges. This allows the open vocabulary segmentation model to effectively segment and identify the category and localization of targets, improving the accuracy of the target recognition method using the open vocabulary segmentation model.

[0075] In this embodiment of the invention, multi-scale features extracted from the target image and the noise contour mask are fused with coded features extracted from the noise contour mask as conditional features. This comprehensive application of multi-scale features allows the model to capture various details and contextual information from basic to advanced levels, greatly enriching the model's visual information processing capabilities at different resolution levels and effectively improving the accuracy of the output.

[0076] The embodiments of the present invention utilize a lightweight single fully connected layer to predict the classification and localization of targets, eliminating the need for complex models to perform classification. This significantly improves the computational efficiency of the model, reduces deployment costs, and decreases the demand for hardware resources, enabling the model to run smoothly on low-power devices and improving the usability of open vocabulary segmentation models.

[0077] Therefore, the embodiments of the present invention realize an effective and fast target recognition method based on an open vocabulary segmentation model, which solves the problems existing in the prior art.

[0078] The following example illustrates the model structure and training method of the open vocabulary segmentation model.

[0079] For example, see Figure 2 As shown, the open vocabulary segmentation model used in this embodiment of the invention may include: a PVT (Pyramid Vision Transformer) network, a DN (DeNoising) network, and a classification and localization module.

[0080] The PVT network is used to extract multi-scale features of the target from the target image and the noise contour mask. Specifically, the PVT model is a variant of the Transformer model; see [link to relevant documentation]. Figure 2 As shown, the PVT network consists of layers 1 through 4. Layer 1 includes an Input Embedding module and a Transformer Encoder. Input Embedding is used to embed the model's input, while the Transformer Encoder uses 3x3 convolutions to capture local structure and texture information in the image, and maintains the image data size by adjusting parameters such as padding and stride. Layers 2 through 4 constitute a hierarchical feature extractor, used to obtain feature representations at different resolutions and levels. Combined with the Transformer's self-attention mechanism, it can capture the global contextual information of the image at different scales.

[0081] Specifically, the PVT network model structure adopts a pyramid-shaped Transformer design to handle multi-scale image features. Each layer in the model contains a series of Transformer encoders that use a self-attention mechanism to fuse and encode the input data. The computation process is as follows: PVT transforms the input into query, key, and value vectors through three weight matrices. For each query vector, it calculates the dot product with all key vectors to obtain an attention score vector, which represents the importance of each element to the current element. A softmax operation is performed on the attention score vectors to sum them to 1, resulting in an attention weight vector, which represents the proportion of each element's contribution to the current element. For each value vector, it is multiplied by the corresponding attention weight and summed to obtain an output vector. Weighted summation generates the final output vector, thus providing rich contextual information for each processed element.

[0082] The DN network is a denoising diffusion model based on Unet, which includes an encoder, a feature fusion module, and a decoder. The encoder is used to encode the noisy contour mask to obtain encoded features. The feature fusion module is used to fuse multi-scale features as conditional features with the encoded features to obtain fused features. The decoder is used to predict the denoised contour mask of the target based on the fused features.

[0083] Specifically, the encoder of the DN network uses a single-channel convolutional neural network containing convolutional layers, activation layers, and pooling layers. It uses 3*3 convolutions to fuse global information of the image and maintains the structure of the image data by using parameters such as padding and stride. The decoder of the DN network uses a combination of depthwise separable convolutional layers, activation functions, and pooling layers to extract and process features from the input image. By analyzing the diffused image, it effectively extracts the encoded features of the image.

[0084] The feature fusion module concatenates multi-scale features of the same size with encoded features, and uses the concatenated features as the fused features.

[0085] In this embodiment of the invention, the Unet-based denoising diffusion model (DN) employs a classic architecture to handle image denoising and generation tasks. The model structure comprises two main stages: downsampling and upsampling. Each stage utilizes a series of convolutional encoders and decoders to process image features. Specifically, the model first downsamples the input image through a series of convolutional and pooling layers. These layers increase the depth and complexity of the feature map by continuously reducing its size, capturing low-to-high-level features in the image. After downsampling, the model gradually restores the image size and details through corresponding upsampling paths. In this stage, each decoder layer uses skip connections from the corresponding downsampling layer, which helps preserve image details. Simultaneously, the size of the feature map is gradually expanded through deconvolutional layers, leveraging Unet's deep learning capabilities to capture and reconstruct image details. The diffusion model then gradually generates a clear image from a noisy state, accurately obtaining a denoised contour mask map containing the target edges.

[0086] The classification and localization module is used to predict the classification and localization of the target using a single fully connected layer based on the denoised contour mask image and multi-scale features.

[0087] Specifically, the classification and localization module predicts the classification and localization of the target using a single fully connected layer based on the denoised contour mask image and multi-scale features, including:

[0088] (1) By performing edge detection on the denoised contour mask, a proposed region corresponding to each potential target is generated.

[0089] It is understandable that the proposed region corresponding to the potential target is the region where the potential target is located in the denoised contour mask image.

[0090] (2) For each proposed region, extract the features of the region where the potential target is located from the multi-scale features based on the proposed region, and use a single fully connected layer to predict the classification and location of the potential target and the corresponding confidence level based on the features.

[0091] Specifically, after extracting the features of the region where the potential target is located from the multi-scale features based on the proposed region, the features are input into a single fully connected layer, which can then predict the classification and location of the potential target and the corresponding confidence level.

[0092] Here, a single fully connected layer meticulously processes the segmentation mask image to identify and classify the specific category of each instance. This single fully connected layer employs a simple single-layer convolutional neural network structure containing linear layers and activation pooling layers. In this single fully connected layer, the features of the denoised contour mask image are first standardized to optimize network learning efficiency. Then, after passing through the fully connected layer, the image features are transformed into the final category output. Specifically, the single fully connected layer performs a series of linear transformations (weight matrix multiplication plus bias), followed by a non-linear activation function (such as ReLU or Sigmoid). This structure helps the network capture complex patterns and relationships. Through the processing of this single fully connected layer, useful information can be gradually extracted from the input features, ultimately generating a classification decision vector. Each dimension of this vector represents the probability of a category, and the classification result is determined by the category with the highest probability, thus obtaining the category information of the target object. The probability mentioned here is the confidence level mentioned above.

[0093] (3) The classification and localization of potential targets with confidence scores higher than the confidence threshold are used as the classification and localization of targets in the target image.

[0094] Understandably, potential targets with a confidence level not higher than the confidence threshold are not considered targets in the target image because their classification and localization results have low confidence levels, and therefore are filtered out.

[0095] Based on the above network structure, the open vocabulary segmentation model used in this embodiment of the invention can be regarded as a COD (Coolie Detection) model. This model learns image features and target edge contour features to give the classification and localization results of the target, thereby improving the segmentation accuracy and target recognition capability of the open vocabulary segmentation model.

[0096] The following is a detailed explanation of the implementation mechanism of how the embodiments of the present invention train the open vocabulary segmentation model, thereby enabling it to have the above-mentioned functions and effects.

[0097] Specifically, in this embodiment of the invention, the method for training the open vocabulary segmentation model includes:

[0098] (1) Obtain multiple sample images and corresponding annotation information; the sample images contain sample targets, and the annotation information includes the location information, classification information and mask contour information of the sample targets in the sample images.

[0099] Specifically, sample images and their corresponding annotation information can be selected based on the actual application scenario. For example, if the actual application scenario is to identify animal species, animal photos can be selected as sample images, and the animal's location information, classification information, and mask contour information can be annotated. If the actual application scenario is to classify and identify vehicles in a traffic scene, photos of traffic scenes can be selected as sample images, and the vehicle's location information, classification information, and mask contour information can be annotated. The source of the sample images in this embodiment of the invention is not limited.

[0100] In addition, for sample images of different sizes, they are preprocessed to be transformed to a uniform size in order to adapt to the input of the open vocabulary segmentation model.

[0101] Alternatively, an existing dataset, such as the PIDXray dataset, can be used as the multiple sample images mentioned above. The PIDXray dataset contains known annotation files, eliminating the need for additional annotation information, and all images are the same size, eliminating the need for image size preprocessing.

[0102] (2) Obtain the noise contour mask of the corresponding sample target from each sample image.

[0103] Specifically, for self-generated sample images, edge detection can be used to detect the edges of the target objects, thereby generating a contour mask of the target objects. For existing datasets such as the PIDXray dataset, since they usually contain known annotation files, the required contour mask can be directly generated based on these annotation files. Then, noise is added to the contour mask of the target objects to obtain a noisy contour mask of the target objects.

[0104] (3) Construct a dataset using multiple sample images, corresponding annotation information, and noise contour mask images.

[0105] Specifically, each sample image, its corresponding annotation information, and the noise contour mask are used as a training sample to construct a dataset containing multiple training samples.

[0106] In an alternative implementation, boundary preprocessing can be performed on the noise contour mask maps corresponding to some or all of the sample images to randomly destroy the sample target contours.

[0107] The boundary preprocessing of the noise contour mask involves performing pixel-level structural erosion on the noise contour mask. Specific processing methods may include data augmentation and dilation erosion, which will not be elaborated in the embodiments of this invention.

[0108] Therefore, a dataset is constructed using multiple sample images, corresponding annotation information, and noise contour masks after boundary preprocessing. Training an open-vocabulary segmentation model using such a dataset can effectively correct the bias in the model's predictions.

[0109] (4) Use the dataset to train an open vocabulary segmentation model.

[0110] Specifically, first, initialize the network parameters and iteration count of the open vocabulary segmentation model. The initial network parameters for the PVT network, DN network, and a single fully connected layer can all be pre-trained parameters on their respective public datasets. Then, see... Figure 3 Training samples are obtained from the dataset in batches, allowing the open vocabulary segmentation model to process each training sample accordingly. The specific training process is as follows:

[0111] (4-1) Input the sample image and the corresponding noise contour mask into the PVT network during training so that the PVT network outputs the multi-scale features of the sample target.

[0112] (4-2) Input the noise contour mask corresponding to the sample image into the DN network in training so that the DN network can predict the denoised contour mask of the sample target;

[0113] (4-3) Based on the denoised contour mask and multi-scale features of the sample target, a single fully connected layer is used to predict the classification and localization of the sample target;

[0114] (4-4) Based on the annotation information corresponding to the sample image, the denoised contour mask, the noise processed by the diffusion model of the DN network during the denoising and denoising process, and the classification of the sample target predicted by a single fully connected layer, calculate the model loss of the open vocabulary segmentation model during training, and adjust the network parameters of the open vocabulary segmentation model according to the model loss until training is completed.

[0115] Specifically, based on the mask contour information and denoised contour mask image corresponding to the sample image, and the noise processed by the diffusion model of the DN network during denoising and denoising, the segmentation loss of the open vocabulary segmentation model in training is calculated. Based on the location information and classification information corresponding to the sample image, and the classification and localization of the sample target predicted by a single fully connected layer in training, the classification and localization loss of the open vocabulary segmentation model in training is calculated. Combining the segmentation loss and the classification and localization loss, the model loss of the open vocabulary segmentation model in training is obtained. The network parameters of the open vocabulary segmentation model are adjusted according to this model loss to optimize the network parameters in the direction of reducing the model loss. This process is repeated until a pre-set number of iterations is reached, or the model loss converges and no longer decreases, thus completing the training. The current network parameters are saved, resulting in the trained open vocabulary segmentation model.

[0116] The segmentation loss is calculated using the following formula:

[0117]

[0118] in, The segmentation loss is represented by f, where f represents the mask contour information corresponding to the sample image; θ This represents the denoised contour mask image corresponding to the sample image, ||f θ -f|| is used to quantize f and f θ The difference between the sample image and the denoised contour mask is minimized by introducing this difference into the segmentation loss, thereby aiming to minimize the model's error in data reconstruction or prediction; n represents the noise added by the diffusion model during the noise addition process. θ ||n represents the noise removed during the denoising process by the diffusion model. θ -n|| is used to quantize n and n θ The difference between them, which can be introduced into the segmentation loss, can affect the accuracy of the model in noise modeling or denoising tasks; δ t =(1-t) 2 Expressed as the rate of weightlessness as a function of time t; This is the edge loss for the sample target.

[0119] here, The calculation formula is:

[0120]

[0121] in, E + E represents the number of pixels belonging to the edge of the sample target in the denoised contour mask image corresponding to the sample image. -λ represents the number of pixels in the denoised contour mask corresponding to the sample image that do not belong to the edge of the sample target, and λ is the balance E + With E - Weighting factors It is the i-th pixel of the j-th edge in the mask contour information corresponding to the sample image. The actual pixel value, In the denoised contour mask image corresponding to the sample image, and The probability value that the pixel corresponding to the location belongs to the edge of the sample target.

[0122] The classification loss is calculated using the following formula:

[0123]

[0124] Among them, y i It is the true classification of the sample target; p i is the classification of the sample target predicted by a single fully connected layer; C is the total number of categories of all sample targets, and n represents the number of sample targets predicted by a single fully connected layer; p j This represents the target localization predicted by a single fully connected layer; y j This indicates the true location of the target in the sample. This represents the classification and localization loss. By calculating this loss, the difference between the classification and localization of the sample target predicted by the open vocabulary segmentation model and the true classification and localization of the sample target is minimized.

[0125] After obtaining the classification localization loss and segmentation loss, the two are combined. There are various ways to combine them, such as calculating... As model loss, or calculation and The weighted sum can be used as the model loss, which is acceptable.

[0126] Therefore, in each training round, the network parameters of the open vocabulary segmentation model are adjusted based on the model loss, optimizing the network parameters in a direction that reduces the model loss. This process is repeated until a pre-set number of iterations is reached, or the model loss converges and no longer decreases, thus completing the training. Saving the current network parameters yields the trained open vocabulary segmentation model.

[0127] In addition, if the prediction accuracy of the open vocabulary segmentation model is found to be low in actual testing after training, it can be further trained by optimizing the dataset or the training process.

[0128] It should be noted that it is possible to achieve Figure 1 The open vocabulary segmentation model shown in the operation flow is not limited to Figure 2The model structure shown, any scheme that modifies an existing model to achieve the functionality of the edge-based open vocabulary segmentation model in the embodiments of the present invention, falls within the protection scope of the embodiments of the present invention.

[0129] The method provided in this invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc. No limitation is made herein; any electronic device that can implement this invention falls within the protection scope of this invention.

[0130] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0131] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0132] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings and the disclosure, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the description of this invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects, all of which are collectively referred to herein as "modules" or "systems." Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The computer program may be stored / distributed in a suitable medium, provided with or as part of other hardware, or may take other distribution forms, such as via the Internet or other wired or wireless telecommunications systems.

[0134] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0135] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0136] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0137] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A target recognition method based on an edge-based open vocabulary segmentation model, characterized in that, include: Acquire the target image; The target image contains a target to be identified; Initialize Gaussian white noise as a noise contour mask for the target; The target image and the noise contour mask are input into a pre-trained open vocabulary segmentation model, so that the open vocabulary segmentation model performs the following operations: Extract multi-scale features of the target from the target image and the noise contour mask; The noise contour mask is encoded to obtain encoded features; the multi-scale features are fused with the encoded features as conditional features to obtain fused features; the denoised contour mask of the target is predicted based on the fused features; the classification and localization of the target are predicted using a single fully connected layer based on the denoised contour mask and the multi-scale features. During the training of the open vocabulary segmentation model, the model loss of the open vocabulary segmentation model during training is calculated, and the network parameters of the open vocabulary segmentation model are adjusted according to the model loss until training is completed; the model loss combines the segmentation loss and the classification and localization loss; the segmentation loss is calculated by the following formula: ； in, Indicates the partition loss. This represents the mask contour information corresponding to the sample image; This represents the denoised contour mask image corresponding to the sample image. This represents the noise added during the noise addition process in the diffusion model. This represents the noise removed during the denoising process by the diffusion model; Represented as over time The changing rate of weightlessness; The sample target edge loss is calculated using the following formula: ， ； in, , , This represents the number of pixels belonging to the edge of the target object in the denoised contour mask image corresponding to the sample image. This represents the number of pixels in the denoised contour mask image corresponding to the sample image that do not belong to the edge of the sample target. It is balance and Weighting factors It is the mask contour information corresponding to the sample image, the first The first edge 1 pixel The actual pixel value, In the denoised contour mask image corresponding to the sample image, and The probability value that the pixel corresponding to the location belongs to the edge of the sample target.

2. The edge-based open vocabulary segmentation model target recognition method according to claim 1, characterized in that, The open vocabulary segmentation model includes: a PVT network, a DN network, and a classification and localization module; The PVT network is used to extract multi-scale features of the target from the target image and the noise contour mask; the DN network is a denoising diffusion model based on Unet, including an encoder, a feature fusion module, and a decoder; the encoder is used to encode the noise contour mask to obtain encoded features; the feature fusion module is used to fuse the multi-scale features as conditional features with the encoded features to obtain fused features; the decoder is used to predict the denoised contour mask of the target based on the fused features; the classification and localization module is used to predict the classification and localization of the target using a single fully connected layer based on the denoised contour mask and the multi-scale features.

3. The edge-based open vocabulary segmentation model target recognition method according to claim 2, characterized in that, The classification and localization module uses a single fully connected layer to predict the classification and localization of the target, including: By performing edge detection on the denoised contour mask image, a proposed region corresponding to each potential target is generated; For each proposed region, features of the region where the corresponding potential target is located are extracted from the multi-scale features based on the proposed region, and the classification and location of the potential target and the corresponding confidence level are predicted using a single fully connected layer based on the features. The classification and localization of potential targets with confidence levels higher than the confidence threshold are used as the classification and localization of targets in the target image.

4. The edge-based open vocabulary segmentation model target recognition method according to claim 2, characterized in that, The methods for training the open vocabulary segmentation model include: Multiple sample images and corresponding annotation information are acquired; the sample images contain sample targets, and the annotation information includes the location information, classification information, and mask contour information of the sample targets in the sample images. Obtain the noise contour mask of the corresponding sample target from each sample image; A dataset is constructed using the multiple sample images, their corresponding annotation information, and noise contour mask images. An open vocabulary segmentation model was trained using the dataset.

5. The edge-based open vocabulary segmentation model target recognition method according to claim 4, characterized in that, A dataset is constructed using the multiple sample images, their corresponding annotation information, and noise contour mask images, including: Boundary preprocessing is performed on the noise contour mask map corresponding to some or all of the sample images to randomly destroy the target contour of the sample; A dataset is constructed using the multiple sample images, their corresponding annotation information, and the noise contour mask image after boundary preprocessing.

6. The edge-based open vocabulary segmentation model target recognition method according to claim 4, characterized in that, Training an open vocabulary segmentation model using the dataset includes: The sample image and the corresponding noise contour mask are input into the PVT network being trained, so that the PVT network outputs multi-scale features of the sample target. The noise contour mask corresponding to the sample image is input into the DN network under training so that the DN network can predict the denoised contour mask of the sample target. Based on the denoised contour mask and multi-scale features of the sample target, a single fully connected layer is used to predict the classification and localization of the sample target. Based on the annotation information corresponding to the sample image, the denoised contour mask, the noise processed by the denoising diffusion model of the DN network during the denoising and denoising process, and the classification and localization of the sample target predicted by a single fully connected layer, the model loss of the open vocabulary segmentation model during training is calculated. The network parameters of the open vocabulary segmentation model are adjusted according to the model loss until training is completed.

7. The edge-based open vocabulary segmentation model target recognition method according to claim 6, characterized in that, Based on the annotation information corresponding to the sample images, the denoised contour mask, the noise processed by the denoising diffusion model of the trained DN network during the denoising and denoising processes, and the classification and localization of the sample targets predicted by a single fully connected layer, the model loss of the trained open vocabulary segmentation model is calculated, including: Based on the mask contour information and denoised contour mask image corresponding to the sample image, as well as the noise processed by the diffusion model of the DN network during the noise addition and denoising process, calculate the segmentation loss of the open vocabulary segmentation model in training. Based on the location information and classification information of the sample image and the classification and localization of the sample target predicted by a single fully connected layer in training, calculate the classification and localization loss of the open vocabulary segmentation model in training; Combining the segmentation loss and the classification and localization loss, we obtain the loss of the open vocabulary segmentation model during training.

8. The edge-based open vocabulary segmentation model target recognition method according to claim 7, characterized in that, The classification and localization loss is calculated using the following formula: ； in, This represents the classification and localization loss. It is the true classification of the sample target; It is the classification of the sample target predicted by a single fully connected layer; It is the total number of categories for all sample targets; This represents the number of target samples predicted by a single fully connected layer. This indicates the target localization of a sample predicted by a single fully connected layer. This indicates the true location of the target in the sample.

Images