An example segmentation model training method, an example segmentation method, and an example segmentation device

Abstract

The application discloses an instance segmentation model training method. Training samples are obtained, the training samples include multiple training pictures, the multiple training pictures correspond to the same original picture, and the original picture includes multiple instances. The annotation information corresponding to different training pictures is different, the annotation information includes pixel-level mask annotation, and the mask annotation reflects the instances included in the training picture. The mask annotations corresponding to the multiple training pictures are superimposed to obtain foreground annotation. For any training picture, the training picture is input into an initial model to obtain a prediction result, the prediction result includes mask prediction results corresponding to N instances and a first foreground prediction result. A loss function corresponding to the initial model is determined based on the prediction result, the annotation information and the foreground annotation. The parameters of the initial model are adjusted to minimize the loss function until the initial model converges, and an instance segmentation model is obtained.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to an instance segmentation model training method, instance segmentation method, and apparatus. Background Technology

[0002] Instance segmentation is one of the fundamental tasks in computer vision. Current instance segmentation models can only segment object types that exist in the training set. For object types that do not exist in the training set or are unlabeled, the model will consider them as background. Summary of the Invention

[0003] In view of this, embodiments of this application provide an instance segmentation model training method, an instance segmentation method, and an apparatus, so as to enable the trained instance segmentation model to segment various objects included in an image.

[0004] To achieve the above objectives, the technical solution provided in this application is as follows:

[0005] In a first aspect of this application, a method for training an instance segmentation model is provided, the method comprising:

[0006] Obtain training samples, which include multiple training images corresponding to the same original image. The original image includes multiple instances. Each training image has different annotation information, including pixel-level mask annotations, which reflect the instances included in the training image.

[0007] The mask annotations corresponding to the multiple training images are superimposed to obtain the foreground annotation;

[0008] For any one of the multiple training images, the training image is input into an initial model to obtain the prediction result output by the initial model. The prediction result includes the mask prediction result corresponding to each of the N predicted instances and the first foreground prediction result, where N is greater than the number of instances labeled in the training image.

[0009] Based on the prediction results, the annotation information, and the foreground annotation, the loss function corresponding to the initial model is determined. With the goal of minimizing the loss function, the parameters of the initial model are adjusted until the initial model converges, thereby obtaining an instance segmentation model.

[0010] In a second aspect of this application, an instance segmentation method is provided, the method comprising:

[0011] Obtain the image to be processed, which includes the instance to be segmented;

[0012] The image to be processed is input into the instance segmentation model to obtain the output result, which includes the mask prediction results corresponding to N instances. The instance segmentation model is trained and generated based on the method described in the first aspect.

[0013] The instances included in the image to be processed are determined based on the output results and the mask threshold.

[0014] In a third aspect of this application, an instance segmentation model training apparatus is provided, the apparatus comprising:

[0015] The first acquisition unit is used to acquire training samples, which include multiple training images, each training image corresponding to the same original image, each original image including multiple instances, and each training image having different annotation information, including pixel-level mask annotations, which reflect the instances included in the training images.

[0016] The second acquisition unit is used to superimpose the mask annotations corresponding to the multiple training images to obtain foreground annotations;

[0017] The third acquisition unit is used to input any training image from the plurality of training images into an initial model and obtain the prediction result output by the initial model. The prediction result includes the mask prediction result corresponding to each of the N predicted instances and the first foreground prediction result, where N is greater than the number of instances labeled in the training image.

[0018] The fourth acquisition unit is used to determine the loss function corresponding to the initial model based on the prediction result, the annotation information and the foreground annotation, and to adjust the parameters of the initial model with the goal of minimizing the loss function, until the initial model converges to obtain an instance segmentation model.

[0019] In a fourth aspect of this application, an instance segmentation apparatus is provided, the apparatus comprising:

[0020] The first acquisition unit is used to acquire an image to be processed, the image to be processed including instances to be segmented;

[0021] The second acquisition unit is used to input the image to be processed into the instance segmentation model and obtain the output result. The output result includes the mask prediction results corresponding to N instances. The instance segmentation model is trained and generated based on the method described in the first aspect.

[0022] A determining unit is used to determine the instances included in the image to be processed based on the output result and the mask threshold.

[0023] In a fifth aspect of this application, an electronic device is provided, the device comprising: a processor and a memory;

[0024] The memory is used to store instructions or computer programs;

[0025] The processor is configured to execute the instructions or computer program in the memory to enable the electronic device to perform the method described in the first or second aspect.

[0026] In a sixth aspect of this application, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a device, cause the device to perform the method described in the first or second aspect.

[0027] In a seventh aspect of this application, a computer program product is provided, the computer program product comprising a computer program / instructions that, when executed by a processor, implement the method described in the first aspect or the second aspect.

[0028] Therefore, the embodiments of this application have the following beneficial effects:

[0029] In this embodiment, training samples are first obtained, comprising multiple training images corresponding to the same original image, which contains multiple instances. Each training image has corresponding annotation information, with different annotations for different training images. This annotation information includes pixel-level mask annotations, which reflect the instances included in the training image. Merging the annotation information of multiple training images yields all instances in the original image. The mask annotations of each training image are then superimposed to obtain foreground annotations, which are the set of mask annotations corresponding to all instances in the original image. For any training image, it is input into an initial model to obtain the prediction result output by the initial model. This prediction result includes the predicted mask pairs for each of the N instances and a first foreground prediction result. Based on the prediction result, annotation information, and foreground annotations, a loss function for the initial model is determined. The parameters of the initial model are adjusted to minimize the loss function until the initial model converges, resulting in an instance segmentation model. That is, when training the instance segmentation model, this application trains the prediction ability of the instance segmentation model by obtaining the foreground annotation and the first foreground prediction result, so that the instance segmentation model can segment unannotated instances in the training image, thus providing the segmentation capability of the instance segmentation model. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart of an instance segmentation model training method provided in this application embodiment;

[0032] Figure 2 This is a schematic diagram of an instance segmentation model structure provided in an embodiment of this application;

[0033] Figure 3 A flowchart of an instance segmentation method provided in an embodiment of this application;

[0034] Figure 4 A structural diagram of an instance segmentation model training device provided in an embodiment of this application;

[0035] Figure 5 A structural diagram of an instance segmentation device provided in an embodiment of this application;

[0036] Figure 6 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0037] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0038] Currently, instance segmentation models can only segment objects present in the training set. Objects not present in the training set or unlabeled are considered background. However, in practical applications, instance segmentation models are required to detect all types of objects in the real world, but the training set rarely includes all object types. Ensuring the training set contains all real-world object types requires pixel-level annotation of the objects, which is costly and difficult to implement.

[0039] Based on this, this application proposes an instance segmentation model. First, a training sample set is obtained, comprising multiple training images corresponding to the same original image. Each training image has corresponding annotation information, which differs between different training images. This annotation information includes pixel-level mask annotations that reflect the instances included in the training image. The mask annotations of the multiple training images are superimposed to obtain foreground annotations. For any given training image, it is input into an initial model to obtain the prediction result output by the initial model. This prediction result includes the mask prediction results for each of the N predicted instances and the first foreground prediction result. Based on the prediction results, annotation information, and foreground annotations, a loss function corresponding to the initial model is determined. The parameters of the initial model are optimized to minimize the loss function until the initial model converges, thus obtaining the instance segmentation model. That is, when training the instance segmentation model, this application trains the prediction ability of the instance segmentation model by obtaining the foreground annotation and the first foreground prediction result, so that the instance segmentation model can segment unlabeled instances in the training image, thus providing the segmentation capability of the instance segmentation model.

[0040] To facilitate understanding of the technical solutions provided in this application, the following description will be provided in conjunction with the accompanying drawings.

[0041] See Figure 1 This figure is a flowchart of an instance segmentation model training method provided in an embodiment of this application. This method can be executed by an instance segmentation model training device, which can be an electronic device or a server. The electronic device can include devices with communication functions such as mobile phones, tablets, laptops, desktop computers, in-vehicle terminals, wearable electronic devices, all-in-one machines, and smart home devices, or it can be a device simulated by a virtual machine or simulator. Figure 1 As shown, the method may include the following steps:

[0042] S101: Obtain training samples, which include multiple training images that correspond to the same original image.

[0043] To train the instance segmentation model, training samples are first obtained. These training samples consist of multiple training images, each corresponding to the same original image, and each original image contains multiple instances. Each training image has different annotation information, including pixel-level mask annotations. Specifically, the annotation information includes a mask annotation for each pixel in the training image, indicating whether an instance is present in the image. This mask annotation is independent of the instance category; its value reflects whether an instance exists at that pixel. For example, a mask annotation of 1 indicates the presence of an instance, while a mask annotation of 0 indicates no instance.

[0044] For example, the original image contains four instances: Instance 1, Instance 2, Instance 3, and Instance 4. The original image is copied to obtain three training images. In training image 1, the annotation information includes a mask label of 1 for the pixels in the region containing Instance 1, a mask label of 1 for the pixels in the region containing Instance 2, and a mask label of 0 for all other pixels. Similarly, in training image 2, the annotation information includes a mask label of 1 for the pixels in the region containing Instance 3, and a mask label of 0 for all other pixels. In training image 3, the annotation information includes a mask label of 1 for the pixels in the region containing Instance 4, and a mask label of 0 for all other pixels.

[0045] S102: Overlay the mask annotations corresponding to multiple training images to obtain foreground annotations.

[0046] After obtaining multiple training images, the mask annotations corresponding to each of the multiple training images are superimposed to obtain the foreground annotations corresponding to the foreground region. The foreground annotations can include the mask annotations corresponding to all instances in the original image.

[0047] This involves overlaying the mask annotations corresponding to multiple training images to obtain foreground annotations. This includes: for pixels at the same location in multiple training images, performing a bitwise OR operation on the mask annotations corresponding to those pixels in different training images to obtain the foreground annotations. For example, during training... Figure 1 In training image 2, the mask for pixel (x0, y0) is labeled as 1, and the mask for pixel (x1, y1) is labeled as 1. In training image 2, the mask for pixel (x0, y0) is labeled as 0, and the mask for pixel (x1, y1) is labeled as 1. After superposition, the mask for pixel (x0, y0) will be labeled as 1, and the mask for pixel (x1, y1) will be labeled as 1.

[0048] S103: For any training image among multiple training images, input the training image into the initial model to obtain the prediction result, which includes the mask prediction result corresponding to each of the N predicted instances and the first foreground prediction result.

[0049] For any one of the acquired training images, input it into the initial model. The initial model then predicts the instances included in the training image, obtaining the prediction results. These prediction results include the mask prediction results for each of the N predicted instances and the first foreground prediction result. Here, N is greater than the number of instances labeled in the training image, and its specific value can be determined based on the actual application scenario. For example, if training image 1 only labels 2 instances, then N is 100.

[0050] Specifically, the initial segmentation model includes a foreground prediction module to perform mask prediction on possible instances in the input training image. That is, the first foreground prediction result includes predictions for all instances within the predicted foreground region of the training image. It should be noted that the mask prediction result in the prediction result takes values ​​between 0 and 1.

[0051] S104: Based on the prediction results, annotation information, and foreground annotations, determine the loss function corresponding to the initial model. With the goal of minimizing the loss function, adjust the parameters of the initial model until the initial model converges to obtain the instance segmentation model.

[0052] After obtaining the prediction results for the training images based on the initial model, the loss function of the initial model can be determined according to the prediction results and the corresponding real labels of the training images. Then, by minimizing the loss function, the initial model training is constrained to converge, thereby obtaining the instance segmentation model.

[0053] In one embodiment of this disclosure, the loss function of the initial model can be determined as follows: A first loss function is calculated based on the mask prediction results corresponding to N instances and the mask annotations corresponding to each instance in multiple training images; a second loss function is calculated based on the foreground annotations and the first foreground prediction result; and the first and second loss functions are weighted and summed to obtain the loss function corresponding to the initial model. Here, the first loss function represents the mask loss between the ground truth mask annotations and the predicted mask, and the second loss function represents the foreground loss between the ground truth foreground annotations and the predicted foreground.

[0054] The first and second loss functions can both include the DICE loss function and the cross-entropy loss function. The first loss function is a weighted sum of the DICE loss function and the cross-entropy loss function calculated based on the mask prediction results of N instances and the mask annotations of each instance in multiple training images. The second loss function is a weighted sum of the DICE loss function and the cross-entropy loss function calculated based on the foreground annotations and the first foreground prediction result. The DICE loss function, or dice loss, is a function used to evaluate the similarity between two objects, with a value ranging from 0 to 1; a larger value indicates a higher similarity between the two values. In the cross-entropy loss function, cross-entropy represents the difference between the true probability distribution and the predicted probability distribution; a smaller cross-entropy value indicates a better model prediction result. It is usually used in conjunction with softmax for loss calculation in classification tasks.

[0055] Furthermore, to improve training accuracy, the prediction results can also include the confidence scores of each of the N instances, which reflect the probability that the predicted mask belongs to an instance. When determining the loss function for the initial model, a third loss function can be calculated based on the confidence scores of each of the N instances and the mask annotations of each instance in multiple training images; a second foreground prediction result is determined based on the mask prediction results of each of the N instances, and a fourth loss function is calculated based on the first and second foreground prediction results; a weighted sum is then performed on the first, second, third, and fourth loss functions to obtain the loss function for the initial model. The third loss function can include a binary cross-entropy loss function, and the fourth loss function can include a cross-entropy loss function.

[0056] In one possible implementation, determining the second foreground prediction result based on the mask prediction results corresponding to each of the N instances includes: summing the mask prediction results corresponding to each of the N instances to obtain the second foreground prediction result. That is, the set of mask prediction results composed of the N instances is summed in the third dimension to obtain the second foreground prediction result.

[0057] Typically, the initial model is fed with a training image of (h, w, 3) and outputs a mask prediction result S of (h, w, N), where N is the number of predicted instances. Therefore, S is summed in the third dimension (i.e., the mask prediction results corresponding to the N instances) to obtain the second foreground annotation.

[0058] To improve training accuracy, before accumulating the mask prediction results for each of the N instances, the prediction results are filtered based on the confidence level of each instance. Target instances with a confidence level greater than or equal to a preset confidence threshold are selected from the N instances. The mask prediction results for these multiple target instances are then accumulated to obtain the second foreground prediction result.

[0059] Optionally, when calculating the first loss function based on the mask prediction results and the real mask annotations for each of the N instances, the system can also filter the instances based on their confidence levels. Target instances with confidence levels greater than or equal to a preset confidence threshold can be selected from the N instances. The first loss function can then be calculated based on the mask prediction results and the real mask annotations for these multiple target instances, thereby reducing the impact of instances with lower confidence levels on the loss function and improving training speed.

[0060] In this embodiment, after determining the loss function between the actual labeled results and the predicted results, the parameters of the initial model are adjusted with the goal of minimizing the loss function until the initial model converges, thus obtaining the instance segmentation model.

[0061] As can be seen, when training the instance segmentation model, this application obtains the foreground annotation and the first foreground prediction result to train the prediction ability of the instance segmentation model, so that the instance segmentation model can segment unannotated instances in the training image and provide the segmentation capability of the instance segmentation model.

[0062] For a better understanding of the training framework provided in this application, please refer to [link / reference]. Figure 2 The figure is a schematic diagram of an instance segmentation model structure provided in this application. The instance segmentation model includes an object prediction branch, a mask prediction branch, and a foreground prediction branch. When a training image is input into the instance segmentation model, the confidence level O of the predicted N instances corresponding to the mask belonging to the object can be obtained through the object prediction branch; the mask prediction result S corresponding to the predicted N instances can be obtained through the mask prediction branch; and the first foreground prediction result P can be obtained through the foreground prediction branch.

[0063] Based on this, the predicted mask prediction result S is filtered based on the object confidence O to obtain the filtered result. Based on the filtered result and the ground truth mask annotations (GT) corresponding to each instance included in the training image, the first loss function (maskloss) is calculated. The second loss function (forground loss) is calculated based on the first foreground prediction result P and the foreground annotations obtained by superimposing the ground truth mask annotations. The third loss function (objectness loss) is calculated based on the predicted object confidence O and the object annotations in the training image. The second foreground prediction result obtained by self-accumulation based on the filtered result and passing it through the sigmoid function is combined with the first foreground prediction result to obtain the fourth loss function (align loss), which constrains the cooperative consistency relationship between the two.

[0064] As can be seen, the single-stage instance segmentation model provided in this embodiment predicts a mask for each instance in the main module's mask prediction branch. Within the main module, the mask confidence prediction branch calculates the corresponding quality score for each predicted mask. In the auxiliary module, the foreground prediction branch predicts the foreground region of the image. A cross-task constraint based on the collaborative consistency loss function is applied to the inheritance relationship between each mask and the foreground prediction result image (i.e., the prediction result of each mask superimposed should be consistent with the foreground prediction result). Finally, the mask output by the mask prediction branch is the predicted object instance mask. Therefore, the instance segmentation model in this embodiment can segment objects that did not appear in the training set, demonstrating more robust performance in unknown and challenging scenarios.

[0065] Based on Figure 1 The example shown demonstrates how to train and generate an instance segmentation model, which can then be applied to various scenarios, such as autonomous driving and medical image analysis. The application of the instance segmentation model will be explained below with reference to the accompanying figures.

[0066] See Figure 3 The figure is a flowchart of an instance segmentation method provided in an embodiment of this application, as shown below. Figure 3 As shown, this method can be executed by an instance segmentation device, which is deployed with a system based on... Figure 1 The instance segmentation model trained by the method can be an electronic device or a server. The electronic device can include communication-enabled devices such as mobile phones, tablets, laptops, desktop computers, in-vehicle terminals, wearable electronic devices, all-in-one machines, and smart home devices, or it can be a device simulated by a virtual machine or simulator. Figure 3 As shown, the method may include the following steps:

[0067] S301: Obtain the image to be processed, which includes the instance to be segmented.

[0068] S302: Input the image to be processed into the instance segmentation model and obtain the output result, which includes the mask annotations corresponding to N instances.

[0069] In this embodiment, since the number of predicted instances is set to N when training the instance segmentation model, the output result obtained when using the instance segmentation model for instance segmentation will include the mask prediction results corresponding to N instances. The mask prediction results corresponding to the N instances are values ​​ranging from 0 to 1.

[0070] S303: Determine the instances included in the image to be processed based on the output results and the mask threshold.

[0071] After obtaining the output results, each mask prediction result in the output results is compared with the mask threshold. When the mask prediction result is greater than or equal to the mask threshold, the instance corresponding to the mask prediction result is determined to be an instance in the image to be processed; if the mask prediction result is less than the mask threshold, it is determined that the image to be processed does not include the instance corresponding to the mask prediction result.

[0072] As can be seen, the method provided in this application can segment objects that do not appear in the training set, improve the segmentation capability of the instance segmentation model, and can be applied to more application scenarios.

[0073] Based on the above method embodiments, this application provides an instance segmentation model training device, an instance segmentation device, and an apparatus, which will be described below with reference to the accompanying drawings.

[0074] See Figure 4 This figure is a structural diagram of an instance segmentation model training device provided in an embodiment of this application, as shown below. Figure 4As shown, the device 400 includes: a first acquisition unit 401, a second acquisition unit 402, a third acquisition unit 403, and a fourth acquisition unit 404.

[0075] The first acquisition unit 401 is used to acquire training samples, the training samples include multiple training images, the multiple training images correspond to the same original image, the original image includes multiple instances, the annotation information corresponding to each training image in the multiple training images is different, the annotation information includes pixel-level mask annotation, the mask annotation is used to reflect the instances included in the training image;

[0076] The second acquisition unit 402 is used to superimpose the mask annotations corresponding to the multiple training images to obtain foreground annotations;

[0077] The third acquisition unit 403 is used to input any training image from the plurality of training images into an initial model to obtain the prediction result output by the initial model. The prediction result includes the mask prediction result corresponding to each of the N predicted instances and the first foreground prediction result, where N is greater than the number of instances labeled in the training image.

[0078] The fourth acquisition unit 404 is used to determine the loss function corresponding to the initial model based on the prediction result, the annotation information and the foreground annotation, and to adjust the parameters of the initial model with the goal of minimizing the loss function until the initial model converges to obtain an instance segmentation model.

[0079] In one embodiment of this disclosure, the fourth acquisition unit 404 is specifically used to calculate a first loss function based on the mask prediction results of each of the N instances and the mask annotations corresponding to each instance in the multiple training images; calculate a second loss function based on the foreground annotations and the first foreground prediction results; and perform a weighted summation of the first loss function and the second loss function to obtain the loss function corresponding to the initial model.

[0080] In one embodiment of this disclosure, the prediction result further includes the confidence scores corresponding to each of the N instances. The confidence scores reflect the probability that the mask prediction result belongs to an instance. The fourth acquisition unit 404 is specifically used to calculate a third loss function based on the confidence scores corresponding to each of the N instances and the mask annotations of each instance in the multiple training images; determine a second foreground prediction result based on the mask prediction results corresponding to each of the N instances; calculate a fourth loss function based on the first foreground prediction result and the second foreground prediction result; and perform a weighted summation based on the first loss function, the second loss function, the third loss function, and the fourth loss function to obtain the loss function corresponding to the initial model.

[0081] In one embodiment of this disclosure, the first loss function includes a Dice loss function (DIEC) and / or a cross-entropy loss function, the second loss function includes a DICE loss function and / or a cross-entropy loss function, the third loss function includes a binary cross-entropy loss function, and the fourth loss function includes a cross-entropy loss function.

[0082] In one embodiment of this disclosure, the fourth acquisition unit 404 is specifically used to accumulate the mask prediction results corresponding to each of the N instances to obtain a second foreground prediction result.

[0083] In one embodiment of this disclosure, the second acquisition unit 402 is specifically used to perform an OR operation on the mask annotations corresponding to the pixels at the same position in the multiple training images to obtain the foreground annotation.

[0084] It should be noted that the specific implementation of each unit in this embodiment can be found in the relevant descriptions in the above method embodiments.

[0085] See Figure 5 This figure is a structural diagram of an instance segmentation device provided in an embodiment of this application, as shown below. Figure 5 As shown, the device 500 includes: a first acquisition unit 501, a second acquisition unit 502, and a determination unit 503.

[0086] The first acquisition unit 501 is used to acquire an image to be processed, the image to be processed including an instance to be segmented;

[0087] The second acquisition unit 502 is used to input the image to be processed into an instance segmentation model to obtain an output result. The output result includes mask prediction results corresponding to N instances. The instance segmentation model is based on... Figure 1 The method shown was used to train and generate the data;

[0088] The determining unit 503 is used to determine the instances included in the image to be processed based on the output result and the mask threshold.

[0089] It should be noted that the specific implementation of each unit in this embodiment can be found in the relevant descriptions in the above method embodiments.

[0090] The unit division in this embodiment is illustrative and represents only one logical functional division; in actual implementation, other division methods may be used. The functional units in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. For example, in the above embodiment, the processing unit and the sending unit can be the same unit or different units. The integrated unit can be implemented in hardware or as a software functional unit.

[0091] See Figure 6 This diagram illustrates a structural schematic of an electronic device 600 suitable for implementing embodiments of the present disclosure. The terminal devices in the embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0092] like Figure 6 As shown, electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processor, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from storage device 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of electronic device 600. Processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0093] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0094] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a storage device 608, or installed from a ROM 602. When the computer program is executed by the processing device 601, it performs the functions defined in the methods of embodiments of this disclosure.

[0095] The electronic device provided in this embodiment belongs to the same inventive concept as the method provided in the above embodiments. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0096] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the methods provided in the above embodiments.

[0097] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0098] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0099] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0100] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, enable the electronic device to perform the aforementioned methods.

[0101] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0102] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0103] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units / modules do not necessarily limit the specific unit itself.

[0104] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0105] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0106] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems or apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to the method section.

[0107] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0108] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0109] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0110] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for training an instance segmentation model, characterized in that, The method includes: Obtain training samples, which include multiple training images corresponding to the same original image. The original image includes multiple instances. Each training image has different annotation information, including pixel-level mask annotations, which reflect the instances included in the training image. The mask annotations corresponding to the multiple training images are superimposed to obtain the foreground annotation, which includes the mask annotations corresponding to each instance in the original image; For any one of the multiple training images, the training image is input into an initial model to obtain the prediction result output by the initial model. The prediction result includes the mask prediction result corresponding to each of the N predicted instances and a first foreground prediction result. The N is greater than the number of instances labeled in the training image. The first foreground prediction result includes the mask prediction result of each instance in the foreground region predicted by the training image. Based on the prediction results, the annotation information, and the foreground annotation, the loss function corresponding to the initial model is determined. With the goal of minimizing the loss function, the parameters of the initial model are adjusted until the initial model converges, thereby obtaining an instance segmentation model.

2. The method according to claim 1, characterized in that, The step of determining the loss function corresponding to the initial model based on the prediction result, the annotation information, and the foreground annotation includes: The first loss function is calculated based on the mask prediction results of each of the N instances and the mask annotations corresponding to each instance in the multiple training images; A second loss function is calculated based on the foreground annotation and the first foreground prediction result; The first loss function and the second loss function are weighted and summed to obtain the loss function corresponding to the initial model.

3. The method according to claim 1, characterized in that, The prediction result also includes the confidence score corresponding to each of the N instances, whereby the confidence score reflects the probability that the mask prediction result belongs to an instance. The step of determining the loss function corresponding to the initial model based on the prediction result, the annotation information, and the foreground annotation includes: The third loss function is calculated based on the confidence scores of the N instances and the mask annotations of each instance in the multiple training images. The second foreground prediction result is determined based on the mask prediction results corresponding to each of the N instances, and the fourth loss function is calculated based on the first foreground prediction result and the second foreground prediction result. The loss function corresponding to the initial model is obtained by weighted summation based on the first loss function, the second loss function, the third loss function, and the fourth loss function.

4. The method according to claim 3, characterized in that, The first loss function includes the Dice-Induced Error (DIEC) loss function and / or the cross-entropy loss function; the second loss function includes the Dictation-Induced Error (DICE) loss function and / or the cross-entropy loss function; the third loss function includes the binary cross-entropy loss function; and the fourth loss function includes the cross-entropy loss function.

5. The method according to claim 3, characterized in that, The determination of the second foreground prediction result based on the mask prediction results corresponding to each of the N instances includes: The mask prediction results corresponding to each of the N instances are summed to obtain the second foreground prediction result.

6. The method according to claim 1, characterized in that, The step of superimposing the mask annotations corresponding to the multiple training images to obtain the foreground annotation includes: For pixels at the same position in the multiple training images, the mask annotations corresponding to the pixels at the same position in different training images are ORed to obtain the foreground annotation.

7. An instance segmentation method, characterized in that, The method includes: Obtain the image to be processed, which includes the instance to be segmented; The image to be processed is input into the instance segmentation model to obtain the output result, which includes the mask prediction results corresponding to N instances. The instance segmentation model is trained and generated based on the method described in any one of claims 1-6. The instances included in the image to be processed are determined based on the output results and the mask threshold.

8. An instance segmentation model training device, characterized in that, The device includes: The first acquisition unit is used to acquire training samples, which include multiple training images, each training image corresponding to the same original image, each original image including multiple instances, and each training image having different annotation information, including pixel-level mask annotations, which reflect the instances included in the training images. The second acquisition unit is used to superimpose the mask annotations corresponding to each of the multiple training images to obtain foreground annotations, wherein the foreground annotations include the mask annotations corresponding to each instance in the original image; The third acquisition unit is used to input any training image from the plurality of training images into an initial model to obtain the prediction result output by the initial model. The prediction result includes the mask prediction result corresponding to each of the N predicted instances and a first foreground prediction result. The N is greater than the number of instances labeled in the training image. The first foreground prediction result includes the mask prediction result of each instance in the foreground region predicted by the training image. The fourth acquisition unit is used to determine the loss function corresponding to the initial model based on the prediction result, the annotation information and the foreground annotation, and to adjust the parameters of the initial model with the goal of minimizing the loss function, until the initial model converges to obtain an instance segmentation model.

9. An instance segmentation device, characterized in that, The device includes: The first acquisition unit is used to acquire an image to be processed, the image to be processed including instances to be segmented; The second acquisition unit is used to input the image to be processed into the instance segmentation model and obtain the output result. The output result includes the mask prediction results corresponding to N instances. The instance segmentation model is trained and generated based on the method described in any one of claims 1-6. A determining unit is used to determine the instances included in the image to be processed based on the output result and the mask threshold.

10. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions or computer programs; The processor is configured to execute the instructions or computer program in the memory to cause the electronic device to perform the method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed on the device, cause the device to perform the method according to any one of claims 1-7.

Images