Instance segmentation model determination method and device, equipment and storage medium

Abstract

The invention discloses an instance segmentation model determination method and device, equipment and a storage medium, and the method comprises the steps: inputting a sample image and object prompt information into a target detection model in an initial instance segmentation model, and obtaining a first target query vector and prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image; acquiring real information of the object; matching the real information with the prediction information by using a matching algorithm to obtain semantic information corresponding to the real mask and a second target query vector; processing the first target query vector and a part of feature maps in the plurality of first scale feature maps through an initial mask segmentation model in the initial instance segmentation model to obtain a prediction mask of the object; determining mask loss of the predicted mask and the real mask; and training the initial instance segmentation model by using the mask loss to obtain a target instance segmentation model. The target instance segmentation model with a relatively good segmentation effect is obtained on the basis of obtaining the real mask containing semantic information at low cost.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence technology, and in particular, relates to a method, device, equipment and storage medium for determining an instance segmentation model. Background Art

[0002] After the weakly supervised instance segmentation model is obtained based on the incomplete supervision information training model, the effect of instance segmentation using the weakly supervised instance segmentation model is worse than that of the fully supervised instance segmentation model. However, the fully supervised instance segmentation model is trained based on sample information accurately annotated to the pixel level. However, it takes a lot of manpower and time to obtain sample information accurately annotated to the pixel level, so the cost of obtaining sample information accurately annotated to the pixel level is high. Therefore, there is an urgent need for a way to train an instance segmentation model with better results at a low cost. Summary of the invention

[0003] The embodiment of the present application provides an implementation solution different from the prior art to solve the technical problem of urgently needing a method to train an instance segmentation model with better effect at low cost.

[0004] In a first aspect, the present application provides an instance segmentation model determination method, comprising: inputting a sample image and object prompt information into a target detection model in an initial instance segmentation model, obtaining a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and multiple first-scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object border prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object; obtaining real information of the object in the sample image, wherein the real information includes real border information and real mask; and matching the real information with the predicted information using a preset matching algorithm. The method comprises the steps of: performing row matching to obtain semantic information corresponding to the true mask and a second target query vector corresponding to the true mask; processing the first target query vector and a plurality of second-scale feature maps in the plurality of first-scale feature maps through an initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of an object in the sample image; determining a mask loss between the predicted mask and the true mask; and training the initial instance segmentation model using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image according to input prompt information of the object to be segmented and a target image to be instance segmented.

[0005] In a second aspect, the present application provides an instance segmentation model determination device, comprising: an input unit, used to input a sample image and object prompt information into a target detection model in an initial instance segmentation model, to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and multiple first-scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object border prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object; an acquisition unit, used to acquire real information of the object in the sample image, wherein the real information includes real border information and real mask; a matching unit, used to match the real information with the predicted information using a preset matching algorithm. row matching to obtain semantic information corresponding to the true mask and a second target query vector corresponding to the true mask; a prediction unit, used to process the first target query vector and multiple second scale feature maps in the multiple first scale feature maps through an initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image; a determination unit, used to determine a mask loss between the predicted mask and the true mask; a training unit, used to train the initial instance segmentation model using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image according to input prompt information of the object to be segmented and a target image to be instance segmented.

[0006] In a third aspect, the present application provides an electronic device comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any method in the first aspect or any possible implementation manner of the first aspect by executing the executable instructions.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the computer program implements any method in the first aspect or any possible implementation manner of the first aspect.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method described in the first aspect or any possible implementation manner of the first aspect.

[0009] The solution provided by the present application inputs a sample image and object prompt information into a target detection model in an initial instance segmentation model, obtains a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and multiple first scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object border prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object; obtains the real information of the object in the sample image, wherein the real information includes real border information and real mask; uses a preset matching algorithm to match the real information with the predicted information to obtain the real The method comprises the following steps: the first target query vector and the plurality of second scale feature maps in the plurality of first scale feature maps are processed by the initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image; the mask loss between the predicted mask and the real mask is determined; the initial instance segmentation model is trained by the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of the instance segmentation of the target image according to the input prompt information of the object to be segmented and the target image to be segmented. The semantic information is matched to the real mask (i.e., the sample information with pixel-level annotations that is not related to the category) by a matching algorithm, the predicted mask of the object in the image is predicted by the initial mask segmentation model in the initial instance segmentation model, and the mask loss between the predicted mask and the corresponding real mask is calculated to train the initial mask segmentation model, thereby obtaining a target instance segmentation model with better segmentation effect. The efficiency of obtaining the sample information with pixel-level annotations containing semantic information is high and the cost is low. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0011] Figure 1 A schematic diagram of a flow chart of a method for determining an instance segmentation model provided in one embodiment of the present application;

[0012] Figure 2 A schematic diagram of the structure of a target detection model is provided for an embodiment of the present application;

[0013] Figure 3A schematic diagram of the structure of an initial mask segmentation model provided in one embodiment of the present application;

[0014] Figure 4 A schematic diagram of the structure of a target instance segmentation model is provided for an embodiment of the present application;

[0015] Figure 5 A schematic diagram of the structure of an instance segmentation model determination device provided in one embodiment of the present application;

[0016] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0018] The terms "first" and "second" etc. in the specification, claims and drawings of the embodiments of the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or equipment comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0019] First, some terms in the embodiments of the present application are explained below to facilitate understanding by those skilled in the art.

[0020] Object category: used to represent the types of different objects, such as people, cars, roads, etc.

[0021] Semantic information: used to describe the object category corresponding to the true mask.

[0022] Mask: used to display a specific area in an image, such as an object in an image, but the object category of the object is unknown.

[0023] Pixel-level annotation: A fine-grained annotation method in the field of image processing and machine learning that is used to annotate each pixel in an image to determine the object category to which each pixel belongs, where the object category is used to mark which object the pixel belongs to, such as a person, car, road, etc.

[0024] Inception Instance Segmentation Model: A weakly supervised instance segmentation model.

[0025] Object detection models: used to predict the categories and locations of various objects in images.

[0026] After the weakly supervised instance segmentation model is obtained based on the incomplete supervision information training model, the effect of instance segmentation using the weakly supervised instance segmentation model is worse than that of the fully supervised instance segmentation model. However, the fully supervised instance segmentation model is trained based on sample information accurately annotated to the pixel level. However, it takes a lot of manpower and time to obtain sample information accurately annotated to the pixel level, so the cost of obtaining sample information accurately annotated to the pixel level is high. Therefore, there is an urgent need for a way to train an instance segmentation model with better results at a low cost.

[0027] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0028] Figure 1 A flowchart of a method for determining an instance segmentation model provided by an exemplary embodiment of the present application is provided. The execution subject of the method may be a server. The method at least includes the following steps S101-S106:

[0029] S101, inputting a sample image and object prompt information into a target detection model in an initial instance segmentation model, obtaining a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object bounding box prediction information, and the object prompt information is used to indicate the position of an object in the sample image, or the position of an object in the sample image and the category of the object;

[0030] Optionally, the target detection model may be DETR (Detection Transformer, a target detection model based on Transformer).

[0031] Optionally, the initial detection model can be trained by using existing large-scale image classification data or target detection data sets to obtain a target detection model with a specific and powerful object detection capability. The large-scale image classification data can be ImageNet, and the target detection data set can be Object365.

[0032] Optionally, the first target query vector refers to position information of an object, typically position information of four points around the object, or position information of four vertices of a rectangular box that selects the object.

[0033] Optionally, the scales in the plurality of first scale feature maps are different, wherein the scales include a 1 / 4 scale, a 1 / 8 scale, a 1 / 16 scale, a 1 / 32 scale and a 1 / 64 scale.

[0034] Optionally, the multiple first-scale feature maps refer to feature representations generated for objects or structures of different scales in the sample image.

[0035] In optional embodiment 1, the sample image may include multiple sub-objects, and the object prompt information may include multiple sub-object prompt information. In the aforementioned S101, the sample image and the object prompt information are input into the target detection model in the initial instance segmentation model to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and multiple first scale feature maps corresponding to the sample image, including:

[0036] A sample image and multiple sub-object hint information are input into a target detection model in an initial instance segmentation model to obtain multiple first sub-target query vectors corresponding to the multiple sub-object hint information, multiple sub-prediction information corresponding to the multiple sub-object hint information, and multiple first scale feature maps corresponding to the sample image, wherein the sub-object hint information corresponds to the first sub-target query vector in one-to-one correspondence, the sub-object hint information corresponds to the sub-prediction information in one-to-one correspondence, the sub-prediction information includes sub-object category prediction information and sub-object bounding box prediction information, and the sub-object category prediction information corresponds to the sub-object bounding box prediction information in one-to-one correspondence.

[0037] Optionally, the structure of the object detection model can be found in Figure 2 , Figure 2 A schematic diagram of the structure of a target detection model provided for an exemplary embodiment of the present application, wherein in the aforementioned S101, the target detection model includes an image backbone network, an encoder, a decoder, and a prediction head, and in the aforementioned S101, the sample image and the object prompt information are input into the target detection model in the initial instance segmentation model to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image, including the following steps S1011-S1014:

[0038] S1011, inputting a sample image into the image backbone network to obtain a plurality of first-scale feature maps corresponding to the sample image;

[0039] Optionally, the image backbone network is used to extract high-level features from an input image.

[0040] S1012, processing the plurality of first scale feature maps by the encoder to obtain a first encoding result;

[0041] Optionally, in the aforementioned S1012, the encoder processes the plurality of first scale feature maps to obtain a first encoding result, including:

[0042] The encoder processes feature maps of 1 / 8 scale, 1 / 16 scale, 1 / 32 scale and 1 / 64 scale in the multiple first-scale feature maps to obtain a first encoding result.

[0043] In the aforementioned optional embodiment 1, in the aforementioned S1012, the processing of the multiple first-scale feature maps by the encoder to obtain a first encoding result includes: processing the multiple first-scale feature maps by an attention mechanism in the encoder to obtain a first encoding result.

[0044] Optionally, the first encoding result is represented in the form of a cross-attention mechanism of Key and Value.

[0045] Optionally, the explanation and use of the encoder can be found in the prior art and will not be elaborated here.

[0046] S1013, inputting the object prompt information and the first encoding result into the decoder to obtain a first target query vector corresponding to the object prompt information;

[0047] Optionally, the explanation and use of the decoder can be found in the prior art and will not be elaborated here.

[0048] S1014: Process the first target query vector through the prediction head to obtain prediction information corresponding to the object prompt information.

[0049] Optionally, the explanation and use of the prediction head can be found in the prior art and will not be elaborated here.

[0050] S102, obtaining real information of the object in the sample image, where the real information includes real frame information and real mask;

[0051] Optionally, the real mask is mask data of an area corresponding to an object, but it is unknown what the object is, and the real frame information is a circumscribed rectangle corresponding to the object.

[0052] Optionally, the real information is information in a semantically irrelevant SA1B data set.

[0053] In the aforementioned optional embodiment 1, the real information may include a plurality of sub-real information, and the sub-real information includes sub-real bounding box information and sub-real mask, wherein the sub-real information corresponds to the real information of a sub-object.

[0054] S103, using a preset matching algorithm, matching the real information with the predicted information to obtain semantic information corresponding to the real mask and a second target query vector corresponding to the real mask;

[0055] Optionally, the second target query vector has the same meaning as the first target query vector, and reference may be made to the description of the first target query vector.

[0056] In an optional embodiment, the matching algorithm in the aforementioned S103 includes a first matching algorithm and a second matching algorithm. If the object prompt information is used to prompt the position of the object in the sample image, the object category prediction information is the category prediction information of the object. The aforementioned S103 uses a preset matching algorithm to match the real information with the predicted information to obtain the semantic information corresponding to the real mask and the second target query vector corresponding to the real mask, including the following steps S11-S12:

[0057] S11, using a preset first matching algorithm to calculate the overlap between the object frame prediction information in the prediction information and the real frame information in the real information;

[0058] Optionally, the overlap is IoU.

[0059] In the aforementioned optional embodiment 1, the matching algorithm includes a first matching algorithm and a second matching algorithm, and if the sub-object prompt information is used to prompt the position of the sub-object in the sample image, then the sub-object category prediction information is the sub-category prediction information of the sub-object, and in the aforementioned S11, the overlap between the object border prediction information in the prediction information and the real border information in the real information is calculated using the preset first matching algorithm, including: for each sub-prediction information in a plurality of sub-prediction information, the sub-overlap between the sub-object border prediction information in the sub-prediction information and the sub-real border information in each sub-real information in a plurality of sub-real information is calculated using the preset first matching algorithm;

[0060] S12: If the overlap degree is higher than a preset overlap degree threshold, the object category prediction information is used as the semantic information corresponding to the true mask, and the first target query vector is used as the second target query vector corresponding to the true mask.

[0061] In the aforementioned optional embodiment 1, the aforementioned S12, if the overlap is higher than a preset overlap threshold, using the object category prediction information as the semantic information corresponding to the true mask, and using the first target query vector as the second target query vector corresponding to the true mask, includes:

[0062] If the sub-overlap degree is higher than a preset overlap degree threshold, the sub-object category prediction information corresponding to the sub-object bounding box prediction information corresponding to the sub-overlap degree is used as the semantic information corresponding to the sub-true mask corresponding to the sub-overlap degree, and the first sub-target query vector corresponding to the sub-object bounding box prediction information is used as the second sub-target query vector corresponding to the sub-true mask.

[0063] Specifically, the real information is the object's category information, that is, it does not contain the semantic information of the real bounding box. The real bounding box information is simply perturbed as a query vector and predicted by the initial instance segmentation model to obtain a prediction result. If the predicted bounding box in the prediction result is close enough to the real bounding box, it is considered a match. The semantic information in the prediction result and the real mask corresponding to the real bounding box are used as training targets to train the initial instance segmentation model.

[0064] In another optional embodiment, the matching algorithm in the aforementioned S103 includes a first matching algorithm and a second matching algorithm. If the object prompt information is used to prompt the position of the object and the category of the object in the sample image, the object category prediction information is the credibility value of the category prediction information of the object. The aforementioned S103 uses a preset matching algorithm to match the real information with the predicted information to obtain the semantic information corresponding to the real mask and the second target query vector corresponding to the real mask, including the following steps S01-S02:

[0065] S01, using a preset second matching algorithm to calculate the overlap between the object frame prediction information in the prediction information and the real frame information in the real information;

[0066] In the aforementioned optional embodiment 1, the matching algorithm includes a first matching algorithm and a second matching algorithm, and if the sub-object prompt information is used to prompt the position of the sub-object and the category of the object in the sample image, then the sub-object category prediction information is the credibility value of the sub-category prediction information of the sub-object, and in the aforementioned S01, the overlap between the object border prediction information in the prediction information and the real border information in the real information is calculated using the preset second matching algorithm, including: for each sub-prediction information in a plurality of sub-prediction information, the sub-overlap between the sub-object border prediction information in the sub-prediction information and the sub-real border information in each sub-real information in a plurality of sub-real information is calculated using the preset second matching algorithm;

[0067] S02. If the overlap is higher than a preset overlap threshold and the credibility value is higher than a preset category prediction threshold, the category prediction information of the object is used as the semantic information corresponding to the true mask, and the first target query vector is used as the second target query vector corresponding to the true mask.

[0068] Specifically, the real information does not know the category information of the object, that is, it does not contain the semantic information of the real border. Some specific categories (such as cats and dogs) are used as prediction objects to obtain prediction results. If the prediction result is close enough to the real border, it is considered a successful match. The semantic information corresponding to the prediction result and the real mask corresponding to the real border are used as training targets to train the initial instance segmentation model.

[0069] Therefore, it can be seen that the first matching algorithm and the second matching algorithm of this scheme can also be trained to obtain an instance segmentation model without knowing the category information of the true mask, and can achieve similar training effects when matching using the category information of the known true mask, thereby significantly reducing the requirements for mask data labeling.

[0070] In the aforementioned optional embodiment 1, the aforementioned S02, if the overlap is higher than a preset overlap threshold, and the credibility value is higher than a preset category prediction threshold, then using the category prediction information of the object as the semantic information corresponding to the true mask, and using the first target query vector as the second target query vector corresponding to the true mask, includes:

[0071] If the sub-overlap degree is higher than a preset overlap degree threshold, and the credibility value is higher than a preset category prediction threshold, the sub-category prediction information of the sub-object corresponding to the sub-object bounding box prediction information corresponding to the sub-overlap degree is used as the semantic information corresponding to the sub-true mask corresponding to the sub-overlap degree, and the first sub-target query vector corresponding to the sub-object bounding box prediction information is used as the second sub-target query vector corresponding to the sub-true mask.

[0072] S104, processing the first target query vector and a plurality of second scale feature maps in the plurality of first scale feature maps by using an initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image;

[0073] Optionally, the structure of the initial mask segmentation model can be found in Figure 3 , Figure 3A schematic diagram of the structure of an initial mask segmentation model provided for an exemplary embodiment of the present application, wherein in S104, the initial mask segmentation model includes a mask encoder, a 2x sampler, a mask decoder, a multiplier, and an adder, and the initial mask segmentation model in the initial instance segmentation model is used in S104 to process the first target query vector and the multiple second scale feature maps in the multiple first scale feature maps to obtain a predicted mask of the object in the sample image, including the following steps S1041-S1046:

[0074] S1041, processing the second scale feature map of the first preset size ratio in the multiple first scale feature maps by the mask encoder to obtain a second encoding result;

[0075] Optionally, the first preset size ratio is 1 / 8 size, and the second scale feature map is a 1 / 8 scale feature map.

[0076] Optionally, the second encoding result has the same meaning as the first encoding result, and reference may be made to the aforementioned description of the first encoding result.

[0077] S1042, sampling the second encoding result by the 2x sampler to obtain a sampling result;

[0078] Optionally, the size ratio of the sampling result is 1 / 4 scale.

[0079] Optionally, the 2x sampler converts features at a 1 / 8 scale into features at a 1 / 4 scale to obtain a sampling result, wherein the sampling result is a feature map.

[0080] S1043, summing the sampling result and another second scale feature map with a second preset size ratio in the plurality of first scale feature maps by the adder to obtain a summation result;

[0081] Optionally, the summation result is represented in the form of a cross-attention mechanism of Key and Value.

[0082] Optionally, the second preset size ratio is 1 / 4 scale.

[0083] S1045. Decoding the first target query vector and the summation result by using the mask decoder to obtain a decoding result;

[0084] S1046: Utilize the multiplier to multiply the sum result and the decoding result to obtain a prediction mask of the object in the sample image.

[0085] In the aforementioned optional embodiment 1, the aforementioned S104 processes the first target query vector and a plurality of second scale feature maps in the plurality of first scale feature maps by using an initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image, including:

[0086] For each first sub-target query vector in the multiple first sub-target query vectors, the first sub-target query vector and multiple second-scale feature maps in the multiple first-scale feature maps are processed by an initial mask segmentation model in the initial instance segmentation model to obtain a sub-prediction mask of a sub-object in the sample image.

[0087] S105, determining a mask loss between the predicted mask and the true mask;

[0088] In the aforementioned optional embodiment 1, in the aforementioned S105, the determining of the mask loss between the predicted mask and the real mask includes the following steps S1051-S1053:

[0089] S1051, obtaining sub-object prompt information corresponding to the sub-object corresponding to the sub-prediction mask;

[0090] S1052, obtaining sub-object prompt information corresponding to the second sub-target query vector corresponding to the sub-real mask;

[0091] S1053: Calculate the mask loss of the sub-prediction mask and the sub-true mask with the same sub-object hint information.

[0092] Optionally, in the aforementioned S105, the determining of the mask loss between the predicted mask and the real mask comprises the following steps S1051-S1052:

[0093] S1051, obtaining a binary cross entropy loss and a similarity loss between the predicted mask and the true mask;

[0094] Optionally, the binary cross entropy loss is denoted by The similarity loss is recorded as

[0095] S1052: Use a preset mask loss function to process the binary cross entropy loss, the similarity loss, the predicted mask, and the true mask to obtain a mask loss of the predicted mask and the true mask.

[0096] Optionally, in the aforementioned S1052, the binary cross entropy loss, the similarity loss, the predicted mask, and the true mask are processed using a preset mask loss function to obtain a mask loss of the predicted mask and the true mask.

[0097]

[0098] Among them, the is the mask loss, the M pred is the prediction mask, the M gt is a real mask, and the object prompt information corresponding to the predicted mask and the real mask is the same.

[0099] S106. Use the mask loss to train the initial instance segmentation model to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image based on input prompt information of the object to be segmented and the target image to be instance segmented.

[0100] Optionally, the structure of the target instance segmentation model can be found in Figure 4 , Figure 4 A structural schematic diagram of a target instance segmentation model provided for an exemplary embodiment of the present application, wherein in the aforementioned S106, the initial instance segmentation model is trained using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, comprising the following steps: determining whether the mask loss is less than a preset threshold, if not, adjusting parameters of the initial mask segmentation model, returning to execute the step of inputting the sample image and object prompt information into the target detection model in the initial instance segmentation model, until the mask loss is less than the preset threshold, and determining the target instance segmentation model corresponding to the initial instance segmentation model based on the target detection model and the initial mask segmentation model after the most recent parameter adjustment.

[0101] In summary, the solution provided by the present application inputs the sample image and the object prompt information into the target detection model in the initial instance segmentation model, obtains a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and multiple first scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object border prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object; obtains the real information of the object in the sample image, wherein the real information includes real border information and real mask; uses a preset matching algorithm to match the real information with the predicted information to obtain the The method comprises the following steps: the first target query vector and the plurality of second scale feature maps in the plurality of first scale feature maps are processed by the initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image; the mask loss between the predicted mask and the true mask is determined; the initial instance segmentation model is trained by the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image according to the input prompt information of the object to be segmented and the target image to be segmented. The semantic information is matched to the true mask (i.e., the sample information with pixel-level annotations that is not related to the category) by a matching algorithm, the predicted mask of the object in the image is predicted by the initial mask segmentation model in the initial instance segmentation model, the mask loss between the predicted mask and the corresponding true mask is calculated to train the initial mask segmentation model, and thus a target instance segmentation model with better segmentation effect is obtained. The efficiency of obtaining the sample information with pixel-level annotations containing semantic information is high and the cost is low.

[0102] Specifically, by using the existing large-scale category-independent mask dataset, the semantic information of the category-independent mask data is determined through the target detection model and matching algorithm, so that the category-independent mask dataset becomes a mask dataset containing semantic information. The mask dataset containing semantic information is used to assist in training the weakly supervised instance segmentation model, which not only greatly improves the performance of the weakly supervised instance segmentation model (initial instance segmentation model), but also reduces the workload and difficulty of data annotation and reduces the cost. By creating a matching pair between the predicted mask and the real mask, it is used to train the segmentation branch. The performance of the weakly supervised instance segmentation model is effectively improved, providing new ideas for further research and practical applications in the field of weakly supervised instance segmentation.

[0103] Figure 5 A schematic diagram of a structure of an instance segmentation model determination device provided for an exemplary embodiment of the present application; wherein the device comprises:

[0104] An input unit 21 is used to input the sample image and the object prompt information into the target detection model in the initial instance segmentation model to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object bounding box prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object;

[0105] An acquisition unit 22, configured to acquire real information of the object in the sample image, wherein the real information includes real frame information and real mask;

[0106] A matching unit 23, configured to match the real information with the predicted information by using a preset matching algorithm to obtain semantic information corresponding to the real mask and a second target query vector corresponding to the real mask;

[0107] A prediction unit 24 is configured to process the first target query vector and a plurality of second scale feature maps in the plurality of first scale feature maps by using an initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image;

[0108] A determination unit 25, configured to determine a mask loss between the predicted mask and the true mask;

[0109] The training unit 26 is used to train the initial instance segmentation model using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image based on the input prompt information of the object to be segmented and the target image to be instance segmented.

[0110] Optionally, the device is used to train the initial instance segmentation model using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, specifically for: determining whether the mask loss is less than a preset threshold, if not, adjusting parameters of the initial mask segmentation model, returning to execute the step of inputting the sample image and object prompt information into the target detection model in the initial instance segmentation model, until the mask loss is less than the preset threshold, and then determining the target instance segmentation model corresponding to the initial instance segmentation model based on the target detection model and the initial mask segmentation model after the most recent parameter adjustment.

[0111] Optionally, the target detection model in the device includes an image backbone network, an encoder, a decoder and a prediction head. The device is used to input the sample image and object prompt information into the target detection model in the initial instance segmentation model to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and multiple first-scale feature maps corresponding to the sample image. It is specifically used to: input the sample image into the image backbone network to obtain multiple first-scale feature maps corresponding to the sample image; process the multiple first-scale feature maps through the encoder to obtain a first encoding result; input the object prompt information and the first encoding result into the decoder to obtain a first target query vector corresponding to the object prompt information; process the first target query vector through the prediction head to obtain prediction information corresponding to the object prompt information.

[0112] Optionally, the initial mask segmentation model in the device includes a mask encoder, a 2x sampler, a mask decoder, a multiplier, and an adder. The device is used to process the first target query vector and multiple second scale feature maps in the multiple first scale feature maps through the initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image. Specifically, it is used to: process the second scale feature map of a first preset size ratio in the multiple first scale feature maps through the mask encoder to obtain a second encoding result; sample the second encoding result through the 2x sampler to obtain a sampling result; sum the sampling result and another second scale feature map of a second preset size ratio in the multiple first scale feature maps through the adder to obtain a summation result; decode the first target query vector and the summation result through the mask decoder to obtain a decoding result; and multiply the summation result and the decoding result by the multiplier to obtain a predicted mask of the object in the sample image.

[0113] Optionally, the matching algorithm in the device includes a first matching algorithm and a second matching algorithm. If the object prompt information is used to prompt the position of the object in the sample image, the object category prediction information is the category prediction information of the object. The device is used to match the real information with the predicted information using the preset matching algorithm to obtain the semantic information corresponding to the real mask and the second target query vector corresponding to the real mask. It is specifically used to: calculate the overlap between the object border prediction information in the predicted information and the real border information in the real information using the preset first matching algorithm; if the overlap is higher than a preset overlap threshold, use the object category prediction information as the semantic information corresponding to the real mask, and use the first target query vector as the second target query vector corresponding to the real mask.

[0114] Optionally, the matching algorithm in the device includes a first matching algorithm and a second matching algorithm. If the object prompt information is used to prompt the position and category of the object in the sample image, then the object category prediction information is the credibility value of the category prediction information of the object. The device is used to match the real information with the predicted information using the preset matching algorithm to obtain the semantic information corresponding to the real mask and the second target query vector corresponding to the real mask. Specifically, it is used to: calculate the overlap between the object border prediction information in the predicted information and the real border information in the real information using the preset second matching algorithm; if the overlap is higher than a preset overlap threshold and the credibility value is higher than a preset category prediction threshold, then the category prediction information of the object is used as the semantic information corresponding to the real mask, and the first target query vector is used as the second target query vector corresponding to the real mask.

[0115] Optionally, the device is used to determine the mask loss between the predicted mask and the real mask, specifically for: obtaining the binary cross entropy loss and similarity loss between the predicted mask and the real mask; using a preset mask loss function to process the binary cross entropy loss, the similarity loss, the predicted mask, and the real mask to obtain the mask loss between the predicted mask and the real mask.

[0116] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, no further description is given here. Specifically, the device may perform the above method embodiment, and the above and other operations and / or functions of each module in the device are the corresponding processes in each method in the above method embodiment, respectively, and no further description is given here for the sake of brevity.

[0117] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0118] Figure 6 is a schematic block diagram of an electronic device provided in an embodiment of the present application, and the electronic device may include:

[0119] The memory 301 and the processor 302, the memory 301 is used to store the computer program and transmit the program code to the processor 302. In other words, the processor 302 can call and run the computer program from the memory 301 to implement the method in the embodiment of the present application.

[0120] For example, the processor 302 may be configured to execute the above method embodiments according to instructions in the computer program.

[0121] In some embodiments of the present application, the processor 302 may include but is not limited to:

[0122] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0123] In some embodiments of the present application, the memory 301 includes but is not limited to:

[0124] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0125] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 301 and executed by the processor 302 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0126] like Figure 6 As shown, the electronic device may also include:

[0127] The transceiver 303 may be connected to the processor 302 or the memory 301 .

[0128] The processor 302 may control the transceiver 303 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 303 may include a transmitter and a receiver. The transceiver 303 may further include an antenna, and the number of antennas may be one or more.

[0129] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0130] The present application also provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0131] When software is used for implementation, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integration. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0132] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0134] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0135] The above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for determining an instance segmentation model, characterized in that: include: Inputting the sample image and the object prompt information into the target detection model in the initial instance segmentation model, obtaining a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object bounding box prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object; Acquire real information of the object in the sample image, wherein the real information includes real frame information and real mask; Using a preset matching algorithm, the real information is matched with the predicted information to obtain semantic information corresponding to the real mask and a second target query vector corresponding to the real mask; Processing the first target query vector and a plurality of second scale feature maps in the plurality of first scale feature maps by an initial mask segmentation model in the initial instance segmentation model to obtain a predicted mask of the object in the sample image; Determining a mask loss between the predicted mask and the true mask; The initial instance segmentation model is trained using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image based on input prompt information of the object to be segmented and the target image to be instance segmented.

2. The method according to claim 1, characterized in that The step of training the initial instance segmentation model using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model includes: Determine whether the mask loss is less than a preset threshold. If not, adjust the parameters of the initial mask segmentation model, return to the step of inputting the sample image and object prompt information into the target detection model in the initial instance segmentation model, until the mask loss is less than the preset threshold, and determine the target instance segmentation model corresponding to the initial instance segmentation model based on the target detection model and the initial mask segmentation model after the most recent parameter adjustment.

3. The method according to claim 1, characterized in that The target detection model includes an image backbone network, an encoder, a decoder, and a prediction head. The target detection model inputs the sample image and the object prompt information into the initial instance segmentation model to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image, including: Inputting a sample image into the image backbone network to obtain a plurality of first-scale feature maps corresponding to the sample image; Processing the plurality of first scale feature maps by the encoder to obtain a first encoding result; Inputting the object prompt information and the first encoding result into the decoder to obtain a first target query vector corresponding to the object prompt information; The first target query vector is processed by the prediction head to obtain prediction information corresponding to the object prompt information.

4. The method according to claim 1, characterized in that: The initial mask segmentation model includes a mask encoder, a 2x sampler, a mask decoder, a multiplier, and an adder. The initial mask segmentation model in the initial instance segmentation model is used to process the first target query vector and a plurality of second scale feature maps in the plurality of first scale feature maps to obtain a predicted mask of the object in the sample image, including: Processing, by the mask encoder, a second scale feature map of a first preset size ratio in the plurality of first scale feature maps to obtain a second encoding result; Sampling the second encoding result by the 2x sampler to obtain a sampling result; The sampling result and another second scale feature map of a second preset size ratio in the plurality of first scale feature maps are summed by the adder to obtain a summation result; Decoding the first target query vector and the summation result by the mask decoder to obtain a decoding result; The multiplier is used to multiply the sum result and the decoding result to obtain a prediction mask of the object in the sample image.

5. The method according to claim 1, characterized in that The matching algorithm includes a first matching algorithm and a second matching algorithm. If the object prompt information is used to prompt the position of the object in the sample image, the object category prediction information is the category prediction information of the object. The preset matching algorithm is used to match the real information with the prediction information to obtain the semantic information corresponding to the real mask and the second target query vector corresponding to the real mask, including: Calculating the degree of overlap between the object frame prediction information in the prediction information and the real frame information in the real information by using a preset first matching algorithm; If the overlap is higher than a preset overlap threshold, the object category prediction information is used as the semantic information corresponding to the true mask, and the first target query vector is used as the second target query vector corresponding to the true mask.

6. The method according to claim 1, characterized in that The matching algorithm includes a first matching algorithm and a second matching algorithm. If the object prompt information is used to prompt the position of the object and the category of the object in the sample image, the object category prediction information is the credibility value of the category prediction information of the object. The preset matching algorithm is used to match the real information with the predicted information to obtain the semantic information corresponding to the real mask and the second target query vector corresponding to the real mask, including: Calculating the degree of overlap between the object frame prediction information in the prediction information and the real frame information in the real information by using a preset second matching algorithm; If the overlap is higher than a preset overlap threshold and the credibility value is higher than a preset category prediction threshold, the category prediction information of the object is used as the semantic information corresponding to the true mask, and the first target query vector is used as the second target query vector corresponding to the true mask.

7. The method according to claim 1, characterized in that The determining of the mask loss between the predicted mask and the true mask comprises: Obtaining a binary cross entropy loss and a similarity loss between the predicted mask and the true mask; The binary cross entropy loss, the similarity loss, the predicted mask, and the true mask are processed using a preset mask loss function to obtain a mask loss of the predicted mask and the true mask.

8. An instance segmentation model determination device, characterized in that: include: An input unit, used to input the sample image and the object prompt information into the target detection model in the initial instance segmentation model, to obtain a first target query vector corresponding to the object prompt information, prediction information corresponding to the object prompt information, and a plurality of first scale feature maps corresponding to the sample image, wherein the prediction information includes object category prediction information and object bounding box prediction information, and the object prompt information is used to indicate the position of the object in the sample image, or the position of the object in the sample image and the category of the object; An acquisition unit, used for acquiring real information of the object in the sample image, wherein the real information includes real frame information and real mask; A matching unit, configured to match the real information with the predicted information by using a preset matching algorithm to obtain semantic information corresponding to the real mask and a second target query vector corresponding to the real mask; A prediction unit, configured to process the first target query vector and a plurality of second scale feature maps in the plurality of first scale feature maps by using an initial mask segmentation model in the initial instance segmentation model, so as to obtain a predicted mask of the object in the sample image; A determining unit, configured to determine a mask loss between the predicted mask and the true mask; A training unit is used to train the initial instance segmentation model using the mask loss to obtain a target instance segmentation model corresponding to the initial instance segmentation model, wherein the target instance segmentation model is used to output a segmentation result of instance segmentation of the target image based on input prompt information of the object to be segmented and the target image to be instance segmented.

9. An electronic device, characterized in that: include: processor; and a memory for storing executable instructions for the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images