An image instance segmentation method, device, storage medium and electronic equipment

Abstract

The specification discloses an image instance segmentation method and device, a storage medium and an electronic device. The method comprises the following steps: obtaining an image to be segmented and a plurality of positioning boxes to be adjusted; inputting the image to be segmented into an encoding end to extract multi-scale features of the image to be segmented; inputting the multi-scale features into a decoding end to detect each target in the image to be segmented; for each target, inputting each positioning box to be adjusted as a positioning box containing noise into the decoding end; removing noise from each positioning box to be adjusted to obtain a standard positioning box for positioning the target; the noise is noise that causes each positioning box to be adjusted to differ from the standard positioning box of the target; and segmenting the target from the image to be segmented according to the standard positioning box of the target and the multi-scale features. The method realizes instance segmentation of an image by removing image noise.

Description

Technical Field

[0001] This specification relates to the field of image processing, and in particular to an image instance segmentation method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the development of internet technology, more and more scenarios require the application of instance segmentation, which involves processing images to obtain images containing only the target object, which may contain personal privacy data. For example, obtaining an image of only a specific target object within an image. Therefore, how to perform instance segmentation on images is a problem that urgently needs to be solved.

[0003] Based on this, this specification provides a method for instance segmentation of images. Summary of the Invention

[0004] This specification provides an image instance segmentation method, apparatus, storage medium, and electronic device to at least partially solve the aforementioned problems existing in the prior art.

[0005] The following technical solution is adopted in this specification:

[0006] This specification provides an instance segmentation method for images. The pre-trained segmentation model includes an encoder and a decoder, wherein the decoder is implemented by a diffusion model. The method includes:

[0007] Obtain the image to be segmented and several bounding boxes to be adjusted;

[0008] The image to be segmented is input into the encoding end so that multi-scale features of the image to be segmented can be extracted through the encoding end;

[0009] Multi-scale features are input to the decoding end so that the decoding end can detect each target in the image to be segmented based on the multi-scale features;

[0010] For each target, each positioning box to be adjusted is input as a positioning box containing noise into the decoding end, so that the decoding end can denoise each positioning box to be adjusted based on the target to obtain a standard positioning box for locating the target; the noise is the noise that causes each positioning box to be adjusted to differ from the standard positioning box of the target.

[0011] The target is segmented from the image to be segmented based on its standard bounding box and the multi-scale features.

[0012] Optionally, the target is segmented from the image to be segmented based on its standard bounding box and the multi-scale features, specifically including:

[0013] The multi-scale features are fused to obtain a fused feature map, which is used as the fused image.

[0014] The target is segmented from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image.

[0015] Optionally, the target is segmented from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image, specifically including:

[0016] Based on the portion of the standard localization box located in the target in the image to be segmented and the multi-scale features, the feature vector of the target is determined;

[0017] The target is segmented from the image to be segmented based on the feature vector and the fused image.

[0018] Optionally, segmenting the target from the image to be segmented based on the feature vector and the fused image specifically includes:

[0019] The convolution kernel corresponding to the target is determined based on the feature vector;

[0020] The fused image is convolved using the convolution kernel to determine the binarization mask of the target;

[0021] The target is segmented from the image to be segmented based on the target's binarized mask.

[0022] Optionally, training the segmentation model specifically includes:

[0023] Acquire a sample image, a label for the sample image, and a bounding box for each sample to be adjusted. The label includes a mask for each target in the sample image.

[0024] The sample image is input into the encoding end of the segmentation model to extract multi-scale features of the sample image through the encoding end;

[0025] The multi-scale features of the sample image are input into the decoding end of the segmentation model so that the decoding end can detect each target in the sample image based on the multi-scale features of the sample image.

[0026] For each target in the sample image, the localization box of each sample to be adjusted is input into the decoding end, so that the decoding end can denoise each localization box of the sample to be adjusted based on the target to obtain a first localization box for locating the target.

[0027] Based on the first bounding box of the target and the multi-scale features of the sample image, the mask of the target output by the segmentation model is determined;

[0028] The segmentation model is trained based on the mask and label of each target in the sample image.

[0029] Optionally, the bounding boxes of each sample to be adjusted are obtained, specifically including:

[0030] Obtain the standard bounding boxes corresponding to each labeled target in the sample image;

[0031] Noise is added to the standard positioning frame to obtain the positioning frames of each sample to be adjusted.

[0032] Optionally, the segmentation model is also used to classify each target in the sample image;

[0033] The label also includes the standard bounding box corresponding to each target in the sample image and the category of each target;

[0034] The segmentation model is trained based on the mask and label of each target in the sample image, specifically including:

[0035] Based on the classification results of the segmentation model for each target detected in the sample image and the labels, the classification loss is determined;

[0036] The first loss is determined based on the mask and label of each target in the sample image;

[0037] The second loss is determined based on the label and the first bounding box of each target in the sample image;

[0038] Based on the classified loss, the first loss, and the second loss, determine the comprehensive loss;

[0039] The segmentation model is trained with the goal of minimizing the overall loss.

[0040] This specification provides an image instance segmentation apparatus. The pre-trained segmentation model includes an encoder and a decoder, wherein the decoder is implemented by a diffusion model. The apparatus includes:

[0041] The image acquisition module is used to acquire the image to be segmented and several positioning boxes to be adjusted;

[0042] The feature extraction module is used to input the image to be segmented into the encoding end, so as to extract multi-scale features of the image to be segmented through the encoding end;

[0043] The target determination module is used to input multi-scale features into the decoding end so that the decoding end can detect each target in the image to be segmented based on the multi-scale features.

[0044] A denoising module is used to input each positioning box to be adjusted as a positioning box containing noise into the decoding end for each target, so that the decoding end can denoise each positioning box to be adjusted based on the target to obtain a standard positioning box for locating the target; the noise is noise that causes each positioning box to be adjusted to differ from the standard positioning box of the target.

[0045] The target segmentation module is used to segment the target from the image to be segmented based on the target's standard bounding box and the multi-scale features.

[0046] Optionally, the target segmentation module is specifically used to: fuse the multi-scale features to obtain a fused feature map as a fused image; and segment the target from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image.

[0047] Optionally, the target segmentation module is specifically used to: determine the feature vector of the target based on the portion of the standard localization box located in the target in the image to be segmented and the multi-scale features; and segment the target from the image to be segmented based on the feature vector and the fused image.

[0048] Optionally, the target segmentation module is specifically used to: determine the convolution kernel corresponding to the target based on the feature vector; perform convolution on the fused image using the convolution kernel to determine the binarization mask of the target; and segment the target from the image to be segmented based on the binarization mask of the target.

[0049] Optionally, the device further includes:

[0050] The model training module is used to acquire sample images, labels for the sample images, and bounding boxes for each target to be adjusted, wherein the labels include masks for each target in the sample images; input the sample images into the encoding end of the segmentation model to extract multi-scale features of the sample images; input the multi-scale features of the sample images into the decoding end of the segmentation model to detect each target in the sample images based on the multi-scale features; for each target in the sample images, input the bounding boxes for each target to be adjusted into the decoding end to denoise the bounding boxes based on the target, obtaining a first bounding box for locating the target; determine the mask for the target output by the segmentation model based on the first bounding box of the target and the multi-scale features of the sample images; and train the segmentation model based on the mask of each target in the sample images and the labels.

[0051] Optionally, the model training module is specifically used to obtain the standard localization boxes corresponding to each target already labeled in the sample image; and to add noise to the standard localization boxes to obtain the sample localization boxes to be adjusted.

[0052] Optionally, the label further includes the standard bounding box corresponding to each target in the sample image and the category of each target; the model training module is specifically used to: determine a classification loss based on the classification result of the segmentation model classifying each target detected in the sample image and the label; determine a first loss based on the mask of each target in the sample image and the label; determine a second loss based on the label and the first bounding box of each target in the sample image; determine a comprehensive loss based on the classification loss, the first loss and the second loss; and train the segmentation model with minimizing the comprehensive loss as the training objective.

[0053] This specification provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image instance segmentation method.

[0054] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described image instance segmentation method.

[0055] The above-mentioned technical solutions adopted in this specification can achieve the following beneficial effects:

[0056] In the image instance segmentation method provided in this specification, the pre-trained segmentation model includes an encoder and a decoder. The decoder is implemented by a diffusion model to acquire the image to be segmented and several localization boxes to be adjusted. The image to be segmented is input into the encoder to extract multi-scale features of the image to be segmented. The multi-scale features are input into the decoder to detect each target in the image to be segmented based on the multi-scale features. For each target, each localization box to be adjusted is input into the decoder as a localization box containing noise, so that the decoder denoises each localization box to be adjusted based on the target to obtain a standard localization box for locating the target. The noise is noise that causes the localization boxes to be adjusted to differ from the standard localization box of the target. The target is segmented from the image to be segmented based on the standard localization box of the target and the multi-scale features.

[0057] As can be seen from the above method, this method removes noise from the image containing the localization box to be adjusted by using a diffusion model, and obtains the standard localization box of each target in the image to be segmented. Through the standard localization box and multi-scale features, instance segmentation of the image to be segmented is achieved. Attached Figure Description

[0058] The accompanying drawings, which are included to provide a further understanding of this specification and form part of this specification, illustrate exemplary embodiments and their descriptions, serving to explain this specification and do not constitute an undue limitation thereof.

[0059] In the picture:

[0060] Figure 1 This is a flowchart illustrating an image instance segmentation method provided in this specification.

[0061] Figure 2 This specification provides a structural diagram of the segmentation model;

[0062] Figure 3 A schematic diagram of the training segmentation model provided in this manual;

[0063] Figure 4 This is a schematic diagram of an image segmentation device provided in this specification;

[0064] Figure 5 The corresponding information provided in this specification Figure 1 A schematic diagram of an electronic device. Detailed Implementation

[0065] To make the objectives, technical solutions, and advantages of this specification clearer, the technical solutions of this specification will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this specification, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without creative effort are within the scope of protection of this application.

[0066] The technical solutions provided in the various embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0067] Figure 1 This document provides a flowchart illustrating an image instance segmentation method, which includes the following steps:

[0068] S100: Obtain the image to be segmented and several positioning boxes to be adjusted.

[0069] In one or more embodiments of this specification, when a user performs various services, it may be necessary to segment an image containing only the target objects from an original image containing multiple target objects. This necessitates instance segmentation of the original image. Among existing methods for instance segmentation of images, the accuracy of segmentation results obtained using single-stage instance segmentation algorithms is relatively low. Therefore, this specification proposes a method for instance segmentation of images using a diffusion model. Image segmentation can typically be performed by a server or other computing-enabled devices. For ease of explanation, the following description uses a server as the entity performing instance segmentation.

[0070] Before performing instance segmentation on the image, the server must first obtain the image to be segmented and a number of positioning boxes to be adjusted. The server can randomly generate (i.e., the size is random) a certain number of positioning boxes. These randomly generated positioning boxes are the positioning boxes to be adjusted. The number of positioning boxes to be adjusted can be randomly generated by the server or a preset number. This manual does not impose any restrictions on this, but the number of randomly generated positioning boxes to be adjusted must be at least equal to the number of targets in the image to be segmented.

[0071] S102: Input the image to be segmented into the encoding end to extract multi-scale features of the image to be segmented through the encoding end.

[0072] Figure 2 This specification provides a structural diagram of the segmentation model, such as... Figure 2 As shown, the segmentation model has an encoding end and a decoding end. The encoding end is used to extract multi-scale features of the image to be segmented, and the decoding end is implemented by a diffusion model to determine each target in the image to be segmented and to denoise each localization box to be adjusted.

[0073] When performing instance segmentation on an image, the server also needs to obtain the multi-scale features of the image to be segmented in order to subsequently determine the targets in the image. Therefore, the server can input the image to be segmented into the encoding end of the segmentation model, which extracts the multi-scale features of the image to be segmented. The encoding end can be composed of a backbone network and a feature pyramid network (FPN), and this specification does not impose any restrictions on this.

[0074] S104: Input the multi-scale features into the decoding end so that the decoding end can detect each target in the image to be segmented based on the multi-scale features.

[0075] The server inputs multi-scale features into the diffusion model, which then detects each target in the image to be segmented based on these multi-scale features. The diffusion model can also classify the detected targets to obtain the classification results for each target.

[0076] S106: For each target, each positioning box to be adjusted is input as a positioning box containing noise into the decoding end, so that the decoding end can denoise each positioning box to be adjusted based on the target to obtain a standard positioning box for locating the target.

[0077] like Figure 2 As shown, to more accurately segment each target, the standard bounding box corresponding to each target can be determined first. The server then segments the image based on the standard bounding box and multi-scale features. In one or more embodiments of this specification, the decoding end is implemented by a diffusion model. This is because the diffusion model can be used to remove noise from the image to obtain a realistic image. However, in this specification, it is necessary to first determine the standard bounding box of the target based on each bounding box to be adjusted. Each bounding box to be adjusted can be understood as a standard bounding box affected by noise. It is precisely because of the influence of noise that each bounding box to be adjusted differs from the standard bounding box of the target. Therefore, the diffusion model is used to denoise the first model to obtain the standard bounding box corresponding to each target. That is, for each target, the server inputs each bounding box to be adjusted into the decoding end. The decoding end denoises each bounding box to be adjusted based on the target to obtain the standard bounding box used to locate the target. The server inputs each bounding box to be adjusted into the decoding end based on its position. The position of each bounding box to be adjusted can be its coordinates. For example, the server can input each bounding box to be adjusted into the decoding end based on the coordinates of its vertices.

[0078] Specifically, for each target, the diffusion model determines whether a standard positioning frame for the target exists in each positioning frame to be adjusted. If it exists, the positioning frame to be adjusted is determined as the standard positioning frame for the target. If it does not exist, the target positioning frame closest to the standard positioning frame for the target is determined, and the position, size, etc. of the target positioning frame are adjusted to obtain the standard positioning frame used to locate the target. Then, other positioning frames that are removed from the standard positioning frame are removed.

[0079] S108: Segment the target from the image to be segmented based on the target's standard bounding box and the multi-scale features.

[0080] Specifically, firstly, the server fuses multi-scale features to obtain a fused feature map, which serves as the fused image. Because this fused image incorporates features from multiple scales of the image to be segmented, it better represents the image to be segmented. For example... Figure 2 As shown, the server can upsample / downsample feature maps at multiple scales through a fusion network to obtain feature maps of the same size as the image to be segmented. Then, the sampled feature maps are concatenated by channels, and convolution is performed in the channel dimension using convolution kernels to obtain a single-channel fusion feature map as the fusion image. This specification does not limit the method of obtaining the fusion image.

[0081] Next, based on the portion of the standard localization box located in the target in the image to be segmented and the multi-scale features, the feature vector of the target is determined. The feature vector of the target can be 169-dimensional or other dimensions, and this specification does not limit it.

[0082] Then, the convolution kernel corresponding to the target is determined based on the feature vector. This specification does not limit the number of convolution kernels corresponding to the target.

[0083] Finally, the fused image is convolved using a convolution kernel to determine the binarization mask of the target. Based on the binarization mask of the target, the target is segmented from the image to be segmented. For example, the binarization mask of the target is multiplied by a matrix to obtain an image containing only the target.

[0084] based on Figure 1 The image instance segmentation method shown uses a diffusion model to remove noise from the image containing the localization box to be adjusted, and obtains the standard localization box of each target in the image to be segmented. The standard localization box and multi-scale features are used to achieve instance segmentation of the image to be segmented.

[0085] Figure 3 This is a schematic diagram of the training segmentation model provided in this manual, as shown below. Figure 3 As shown.

[0086] This specification also provides a method for training the segmentation model. When training the segmentation model, the server first obtains sample images, labels of the sample images, and bounding boxes of each sample to be adjusted. The labels include the masks of each target in the sample image.

[0087] Specifically, when the server obtains the localization boxes of each sample to be adjusted, it needs to acquire the standard localization boxes in the sample image that are already labeled with the corresponding targets, and add noise to the standard localization boxes corresponding to each target to obtain the localization boxes of each sample to be adjusted. The noise can be added to the standard localization boxes in the standard image by adjusting their position, size, or by adding a preset number of localization boxes to be adjusted; this specification does not restrict this. Of course, the server can also directly generate a certain number of localization boxes randomly as the localization boxes of the samples to be adjusted; this specification does not restrict this either.

[0088] Next, the sample image is input into the encoder of the segmentation model to extract multi-scale features from the sample image. These multi-scale features are then input into the decoder of the segmentation model to detect targets in the sample image based on these features.

[0089] Then, for each target in the sample image, the localization box of each sample to be adjusted is input into the decoding end, so that the decoding end can denoise each localization box of the sample to be adjusted based on the target to obtain the first localization box used to locate the target.

[0090] Finally, a fusion network is used to fuse the multi-scale features of the sample image to obtain a fused sample image. Based on the portion of the first bounding box located at the target in the sample image and the multi-scale features of the sample image, the target feature vector is determined. Based on the target feature vector, the sample convolution kernel for the target is determined, and this kernel is used to convolve the fused sample image to determine the sample binarized mask of the target output by the segmentation model. Since this segmentation model can also be used to classify each target in the sample image, and the labels can include the standard bounding boxes corresponding to each target in the sample image and the category of each target, the server can train the segmentation model based on the mask and label of each target in the sample image.

[0091] Specifically, the server first determines the difference between the classification results and the categories corresponding to the detected targets in the sample image based on the segmentation model and the corresponding categories in the labels. The classification loss is positively correlated with this difference. Next, the server determines the difference between the masks of each target in the sample image and the corresponding mask in the labels. A first loss is determined based on this difference, and it is also positively correlated with the difference. Then, the server determines the difference between the standard bounding boxes corresponding to the targets in the labels and the first bounding boxes of each target in the sample image. A second loss is determined based on this difference, and it is also positively correlated with the difference. Finally, a comprehensive loss is determined based on the classification loss, the first loss, and the second loss. This specification does not restrict the order in which the classification loss, the first loss, and the second loss are determined. Finally, the segmentation model is trained with the goal of minimizing this comprehensive loss. The trained segmentation model is then used for instance segmentation of the image. The system allows setting weights for the classification loss, the first loss, and the second loss. The weighted losses are then summed, and the sum is used as the comprehensive loss. Alternatively, the system can directly sum the losses and use the sum as the comprehensive loss. This specification does not impose any restrictions on this approach.

[0092] The above describes one or more embodiments of image instance segmentation methods provided in this specification. Based on the same idea, this specification also provides corresponding image instance segmentation devices, such as... Figure 4 As shown.

[0093] Figure 4This specification provides a schematic diagram of an image instance segmentation device. The pre-trained segmentation model includes an encoder and a decoder, wherein the decoder is implemented by a diffusion model. The device includes:

[0094] Image acquisition module 400 is used to acquire the image to be segmented and several positioning boxes to be adjusted;

[0095] Feature extraction module 402 is used to input the image to be segmented into the encoding end, so as to extract multi-scale features of the image to be segmented through the encoding end;

[0096] The target determination module 404 is used to input multi-scale features into the decoding end so that the decoding end can detect each target in the image to be segmented based on the multi-scale features.

[0097] The denoising module 406 is used to input each positioning frame to be adjusted as a positioning frame containing noise into the decoding end for each target, so that the decoding end can denoise each positioning frame to be adjusted based on the target to obtain a standard positioning frame for locating the target; the noise is noise that causes each positioning frame to be adjusted to differ from the standard positioning frame of the target.

[0098] The target segmentation module 408 is used to segment the target from the image to be segmented based on the target's standard bounding box and the multi-scale features.

[0099] Optionally, the target segmentation module 408 is specifically used to fuse the multi-scale features to obtain a fused feature map as a fused image; and to segment the target from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image.

[0100] Optionally, the target segmentation module 408 is specifically used to: determine the feature vector of the target based on the portion of the standard localization box of the target in the image to be segmented and the multi-scale features; and segment the target from the image to be segmented based on the feature vector and the fused image.

[0101] Optionally, the target segmentation module 408 is specifically used to: determine the convolution kernel corresponding to the target based on the feature vector; perform convolution on the fused image using the convolution kernel to determine the binarization mask of the target; and segment the target from the image to be segmented based on the binarization mask of the target.

[0102] Optionally, the device further includes:

[0103] The model training module 410 is used to acquire sample images, labels of the sample images, and localization boxes of each target to be adjusted, wherein the labels include masks of each target in the sample images; input the sample images into the encoding end of the segmentation model to extract multi-scale features of the sample images through the encoding end; input the multi-scale features of the sample images into the decoding end of the segmentation model to detect each target in the sample images based on the multi-scale features of the sample images through the decoding end; for each target in the sample images, input each localization box to be adjusted into the decoding end to denoise each localization box based on the target, thereby obtaining a first localization box for locating the target; determine the mask of the target output by the segmentation model based on the first localization box of the target and the multi-scale features of the sample images; and train the segmentation model based on the mask of each target in the sample images and the labels.

[0104] Optionally, the model training module 410 is specifically used to obtain the standard localization boxes corresponding to each target that has been labeled in the sample image; add noise to the standard localization boxes to obtain each sample localization box to be adjusted.

[0105] Optionally, the label further includes the standard bounding box corresponding to each target in the sample image and the category of each target; the model training module 410 is specifically used to: determine a classification loss based on the classification result of the segmentation model classifying each target detected in the sample image and the label; determine a first loss based on the mask of each target in the sample image and the label; determine a second loss based on the label and the first bounding box of each target in the sample image; determine a comprehensive loss based on the classification loss, the first loss and the second loss; and train the segmentation model with minimizing the comprehensive loss as the training objective.

[0106] This specification also provides a computer-readable storage medium storing a computer program that can be used to execute the above-described... Figure 1 The provided image instance segmentation method.

[0107] This instruction manual also provides Figure 5 The diagram shows the structure of the electronic device. Figure 5 As shown, at the hardware level, this autonomous driving device includes a processor, internal bus, network interface, memory, and non-volatile memory, and may also include other hardware required for other operations. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to achieve the above-mentioned functions. Figure 1The image instance segmentation method described above. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0108] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0109] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0110] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0111] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0112] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0113] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0117] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0118] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0119] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0120] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0121] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0122] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0123] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this application.

Claims

1. An instance segmentation method for an image, wherein a pre-trained segmentation model includes an encoder and a decoder, the decoder being implemented by a diffusion model, the method comprising: Obtain the image to be segmented and several bounding boxes to be adjusted; The image to be segmented is input into the encoding end so that multi-scale features of the image to be segmented can be extracted through the encoding end; Multi-scale features are input to the decoding end so that the decoding end can detect each target in the image to be segmented based on the multi-scale features; For each target, each positioning box to be adjusted is input as a positioning box containing noise into the decoding end, so that the decoding end can denoise each positioning box to be adjusted based on the target to obtain a standard positioning box for locating the target; the noise is the noise that causes each positioning box to be adjusted to differ from the standard positioning box of the target. The target is segmented from the image to be segmented based on its standard bounding box and the multi-scale features.

2. The method as described in claim 1, wherein the target is segmented from the image to be segmented based on the target's standard bounding box and the multi-scale features, specifically includes: The multi-scale features are fused to obtain a fused feature map, which is used as the fused image. The target is segmented from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image.

3. The method as described in claim 2, wherein the target is segmented from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image, specifically includes: Based on the portion of the standard localization box located in the target in the image to be segmented and the multi-scale features, the feature vector of the target is determined; The target is segmented from the image to be segmented based on the feature vector and the fused image.

4. The method as described in claim 3, wherein segmenting the target from the image to be segmented based on the feature vector and the fused image, specifically includes: The convolution kernel corresponding to the target is determined based on the feature vector; The fused image is convolved using the convolution kernel to determine the binarization mask of the target; The target is segmented from the image to be segmented based on the target's binarized mask.

5. The method as described in claim 1, wherein training the segmentation model specifically includes: Acquire a sample image, a label for the sample image, and a bounding box for each sample to be adjusted. The label includes a mask for each target in the sample image. The sample image is input into the encoding end of the segmentation model to extract multi-scale features of the sample image through the encoding end; The multi-scale features of the sample image are input into the decoding end of the segmentation model so that the decoding end can detect each target in the sample image based on the multi-scale features of the sample image. For each target in the sample image, the localization box of each sample to be adjusted is input into the decoding end, so that the decoding end can denoise each localization box of the sample to be adjusted based on the target to obtain a first localization box for locating the target. Based on the first bounding box of the target and the multi-scale features of the sample image, the mask of the target output by the segmentation model is determined; The segmentation model is trained based on the mask and label of each target in the sample image.

6. The method as described in claim 5, specifically including: obtaining the positioning boxes of each sample to be adjusted, comprising: Obtain the standard bounding boxes corresponding to each labeled target in the sample image; Noise is added to the standard positioning frame to obtain the positioning frames of each sample to be adjusted.

7. The method of claim 5, wherein the segmentation model is further used to classify each target in the sample image; The label also includes the standard bounding box corresponding to each target in the sample image and the category of each target; The segmentation model is trained based on the mask and label of each target in the sample image, specifically including: Based on the classification results of the segmentation model for each target detected in the sample image and the labels, the classification loss is determined; The first loss is determined based on the mask and label of each target in the sample image; The second loss is determined based on the label and the first bounding box of each target in the sample image; Based on the classified loss, the first loss, and the second loss, determine the comprehensive loss; The segmentation model is trained with the goal of minimizing the overall loss.

8. An apparatus for instance segmentation of an image, wherein a pre-trained segmentation model includes an encoder and a decoder, the decoder being implemented by a diffusion model, the apparatus comprising: The image acquisition module is used to acquire the image to be segmented and several positioning boxes to be adjusted; The feature extraction module is used to input the image to be segmented into the encoding end, so as to extract multi-scale features of the image to be segmented through the encoding end; The target determination module is used to input multi-scale features into the decoding end so that the decoding end can detect each target in the image to be segmented based on the multi-scale features. A denoising module is used to input each positioning box to be adjusted as a positioning box containing noise into the decoding end for each target, so that the decoding end can denoise each positioning box to be adjusted based on the target to obtain a standard positioning box for locating the target; the noise is noise that causes each positioning box to be adjusted to differ from the standard positioning box of the target. The target segmentation module is used to segment the target from the image to be segmented based on the target's standard bounding box and the multi-scale features.

9. The apparatus of claim 8, wherein the target segmentation module is specifically configured to fuse the multi-scale features to obtain a fused feature map as a fused image; and to segment the target from the image to be segmented based on the target's standard bounding box, the multi-scale features, and the fused image.

10. The apparatus of claim 9, wherein the target segmentation module is specifically configured to determine the feature vector of the target based on the portion of the standard bounding box of the target in the image to be segmented and the multi-scale features; and to segment the target from the image to be segmented based on the feature vector and the fused image.

11. The apparatus of claim 9, wherein the target segmentation module is specifically configured to: determine the convolution kernel corresponding to the target based on the feature vector; perform convolution on the fused image using the convolution kernel to determine the binarization mask of the target; and segment the target from the image to be segmented based on the binarization mask of the target.

12. The apparatus of claim 8, further comprising: The model training module is used to acquire sample images, labels of the sample images, and localization boxes of each sample to be adjusted. The labels include masks of each target in the sample images. The sample image is input into the encoding end of the segmentation model to extract multi-scale features of the sample image through the encoding end; The multi-scale features of the sample image are input into the decoding end of the segmentation model to detect each target in the sample image based on the multi-scale features of the sample image. For each target in the sample image, each sample localization box to be adjusted is input into the decoding end to denoise each sample localization box to be adjusted based on the target, so as to obtain a first localization box for locating the target. Based on the first localization box of the target and the multi-scale features of the sample image, the mask of the target output by the segmentation model is determined. The segmentation model is trained based on the mask and label of each target in the sample image.

13. The apparatus of claim 12, wherein the model training module is specifically configured to obtain standard bounding boxes corresponding to each target already labeled in the sample image; and to add noise to the standard bounding boxes to obtain each sample bounding box to be adjusted.

14. The apparatus of claim 12, wherein the label further includes a standard bounding box corresponding to each target in the sample image and a category for each target; The model training module is specifically used to determine the classification loss based on the classification results of the segmentation model for each target detected in the sample image and the label. A first loss is determined based on the mask and label of each target in the sample image; a second loss is determined based on the label and the first bounding box of each target in the sample image; and a comprehensive loss is determined based on the classification loss, the first loss, and the second loss. The segmentation model is trained with the goal of minimizing the overall loss.

15. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in any one of claims 1 to 7.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method described in any one of claims 1 to 7.

Images