Training method, device, system and storage medium for instance segmentation model

Abstract

The embodiments of the present invention provide a training method, device, system and storage medium for an instance segmentation model. The method includes: obtaining a training image and corresponding annotation data; performing an affine transformation operation on the training image to obtain a transformed training image; inputting the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask; inputting the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask; performing an affine transformation operation on the first prediction mask to obtain a transformed mask; determining the function value of the main loss function based on the true bounding box, the first prediction mask, the transformed mask and the second prediction mask; and training the instance segmentation model based on the function value of the main loss function. This not only improves the autonomous learning ability of the instance segmentation model, but also enhances its generalization, and can obtain more accurate instance segmentation results.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a training method, device, system and storage medium for an instance segmentation model. Background Art

[0002] Instance segmentation is a computer vision task that identifies object masks at the pixel level. The goal is to find objects of interest in an image and give the category and pixel-level segmentation results of the object.

[0003] The training of common instance segmentation models is mostly based on pixel-level annotation. Compared with bounding box annotation at the object box level, pixel-level annotation requires higher manual annotation costs and a longer annotation cycle.

[0004] Therefore, in order to reduce the high cost of pixel-level annotation, some existing technologies use bounding box-based weakly supervised instance segmentation training methods as an alternative. This method enables the instance segmentation model to autonomously learn the segmentation knowledge within the bounding box, achieving results similar to those of supervised instance segmentation methods. The bounding box-based weakly supervised instance segmentation methods in the existing technology can be roughly divided into the following two types: one is based on prior information related to the target such as color, and the other is achieved by generating pseudo segmentation annotations using traditional algorithms.

[0005] However, the above two methods not only lack generalization, but are also easily affected by factors such as the brightness difference of the image or the target, thereby limiting the upper limit capability of the instance segmentation model and making it difficult to obtain a high-precision instance segmentation model. Summary of the invention

[0006] The present invention is proposed in view of the above problems. According to one aspect of the present invention, a training method for an instance segmentation model is provided. It includes: obtaining a training image and corresponding annotation data, wherein the annotation data includes the position information of the real bounding box of the target in the training image. Performing an affine transformation operation on the training image to obtain a transformed training image. Inputting the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask. Inputting the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask. Performing an affine transformation operation on the first prediction mask to obtain a transformed mask. Determining the function value of the main loss function based on the real bounding box, the first prediction mask, the transformed mask and the second prediction mask. Train the instance segmentation model based on the function value of the main loss function.

[0007] Exemplarily, inputting the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first predicted mask includes: inputting the training image into the instance segmentation model, obtaining a first bounding box of the target and a score of the first bounding box. Determining the first predicted mask according to the first bounding box and the score of the first bounding box.

[0008] Exemplarily, the method further includes: determining a function value of a first auxiliary loss function based on the true bounding box, the first bounding box, and the score of the first bounding box. The training instance segmentation model is also based on the function value of the first auxiliary loss function.

[0009] Exemplarily, the transformed training image is input into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask, including: inputting the transformed training image into the instance segmentation model to obtain a second bounding box of the target and a score of the second bounding box. According to the scores of the second bounding box and the second bounding box, the second prediction mask is determined. The method also includes: determining the minimum bounding box of the transformed mask; and determining the function value of the second auxiliary loss function based on the minimum bounding box, the second bounding box, and the scores of the second bounding box. Wherein, the training instance segmentation model is also based on the function value of the second auxiliary loss function at the same time.

[0010] Exemplarily, determining the function value of the main loss function based on the true bounding box, the first prediction mask, the transformed mask, and the second prediction mask includes: calculating the function value of the first dominant loss function based on the true bounding box and the first prediction mask. Calculating the function value of the second dominant loss function based on the transformed mask and the second prediction mask. Calculating the function value of the main loss function based on the function value of the first dominant loss function and the function value of the second dominant loss function.

[0011] Exemplarily, calculating the function value of the second dominant loss function based on the transformed mask and the second predicted mask includes: for each affine transformation operation and for each target in the training image, based on the i-th first predicted mask of the target The transformed mask obtained by this affine transformation operation The jth second prediction mask of the target obtained by performing instance segmentation after performing the affine transformation operation on the training image Calculate the sub-objective function of this goal The function value of the second dominant loss function of the objective is calculated using the following formula based on the function values ​​of all sub-objective functions of the objective: Wherein, N represents the number of first prediction masks of the target, and M represents the number of second prediction masks of the target obtained by performing instance segmentation after performing the affine transformation operation on the training image.

[0012] Exemplarily, the function value of the sub-objective function is calculated according to the following formula:

[0013]

[0014] in, Indicates based on The response function obtained by projecting onto the x-axis and The response function obtained by projecting onto the x-axis The calculated x-axis loss function; Indicates based on The response function obtained by projecting onto the y-axis and The response function obtained by projecting onto the y-axis The calculated y-axis loss function.

[0015] Exemplarily, the second dominant loss function is a multi-instance learning loss function or a pixel-level loss function.

[0016] Exemplarily, calculating the function value of the main loss function based on the function value of the first dominant loss function and the function value of the second dominant loss function includes: performing weighted summation of the function value of the first dominant loss function and the function value of the second dominant loss function to obtain the function value of the main loss function.

[0017] Exemplarily, the affine transformation operation includes multiple different operations.

[0018] Exemplarily, the affine transformation operation includes one or more of the following operations: a rotation operation, a shearing operation, a translation operation, and a scaling operation.

[0019] According to another aspect of the present invention, a training device for an instance segmentation model is also provided. The training device comprises:

[0020] The acquisition module is used to acquire the training image and the corresponding annotation data, wherein the annotation data includes the position information of the real bounding box of the target in the training image.

[0021] The image transformation module is used to perform an affine transformation operation on the training image to obtain a transformed training image.

[0022] The first instance segmentation module is used to input the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask.

[0023] The second instance segmentation module is used to input the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask.

[0024] The mask transformation module is used to perform an affine transformation operation on the first prediction mask to obtain a transformed mask.

[0025] A loss determination module is used to determine a function value of a main loss function based on the true bounding box, the first predicted mask, the transformed mask and the second predicted mask.

[0026] The training module is used to train the instance segmentation model based on the function value of the main loss function.

[0027] According to another aspect of the present invention, a training system for an instance segmentation model is also provided, comprising a processor and a memory, wherein computer program instructions are stored in the memory, and the computer program instructions are used to execute the training method for the instance segmentation model as described above when the processor is executed.

[0028] According to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and the program instructions are used to execute the training method of the instance segmentation model as described above when running.

[0029] In the above technical solution, bounding box annotation is used to replace pixel-level annotation, which greatly reduces the cost of manual annotation and shortens the annotation cycle. At the same time, it does not rely on prior information related to the target, and only uses affine transformation operations to enhance the training of the instance segmentation model. It not only improves the autonomous learning ability of the instance segmentation model, but also enhances its generalization, reduces the degree of influence of factors such as the brightness difference of the image or the target, and can obtain more accurate instance segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0031] Figure 1 A schematic block diagram of an example electronic device for implementing a training method and a training device for an instance segmentation model according to an embodiment of the present invention is shown;

[0032] Figure 2 A schematic flow chart of a training method for an instance segmentation model according to an embodiment of the present invention is shown;

[0033] Figure 3 A schematic diagram showing a training method for an instance segmentation model according to another embodiment of the present invention is shown;

[0034] Figure 4A schematic block diagram showing a training device for an instance segmentation model according to an embodiment of the present invention; and

[0035] Figure 5 A schematic block diagram of a training system for an instance segmentation model according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0036] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0037] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0038] In order to solve the above-mentioned problems and improve the autonomous learning ability of the instance segmentation model, an embodiment of the present invention provides a training method for an instance segmentation model. The method does not rely on manually annotated pixel-level annotation information, nor does it rely on prior information related to the target such as color. Through data enhancement means, the instance segmentation model has a stronger autonomous learning ability for the target body and edges.

[0039] First, refer to Figure 1 An example electronic device 100 for implementing a training method and apparatus for an instance segmentation model according to an embodiment of the present invention is described.

[0040] like Figure 1 As shown, the electronic device 100 includes one or more processors 102 and one or more storage devices 104. Optionally, the electronic device 100 may also include an input device 106 and an output device 108, and these components are interconnected via a bus system 110 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 1 The components and structures of the electronic device 100 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0041] The processor 102 may be implemented in at least one hardware form of a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), or a programmable logic array (PLA). The processor 102 may also be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a combination of one or more of other processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0042] The storage device 104 may include one or more computer program products. The computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 102 may run the program instructions to implement the client functions (implemented by the processor) and / or other desired functions in the embodiments of the present invention described below. Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0043] The input device 106 may be a device used by a user to input instructions or images, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and an image acquisition device. The image acquisition device may acquire images (including static images and video frames) and store the acquired images in the storage device 104 for use by other components. The image acquisition device may be a separate camera, a camera in a mobile terminal, or an image sensor in a capture machine.

[0044] The output device 108 can output various information (such as images and / or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, etc. Optionally, the input device 106 and the output device 108 can be integrated together and implemented using the same interactive device (such as a touch screen).

[0045] Exemplarily, an example electronic device for implementing the training method and apparatus for the instance segmentation model according to an embodiment of the present invention may be implemented on a device such as a personal computer or a remote server.

[0046] The following will refer to Figure 2 A training method for an instance segmentation model according to an embodiment of the present invention is described. Figure 2 FIG. 2 is a schematic flow chart of a training method 200 for an instance segmentation model according to an embodiment of the present invention. Figure 2 As shown, the method 200 includes the following steps.

[0047] Step S210: Acquire a training image and corresponding annotation data. The annotation data includes the position information of the real bounding box of the object in the training image.

[0048] The training image may be any image suitable for instance segmentation. The training image may be an original image captured by an image acquisition device such as a camera or an image obtained after preprocessing the original image. The preprocessing operation may include all operations for performing instance segmentation more clearly. For example, the preprocessing operation may include a denoising operation such as filtering. The training image may be input by the input device 106 of the electronic device 100 and transmitted to the processor 102 for processing.

[0049] The training image includes the target of instance segmentation. For example, a person or a car in the training image can be used as a target. Each target in the training image has corresponding annotation data. The annotation data can be obtained by, for example, manual or machine annotation. The annotation data can include the position information of the target's true bounding box. The annotation data is an accurate annotation of the target in the training image. For example, the true bounding box can be a rectangular box, and its position information can be the position coordinates of any pair of non-adjacent vertices on the rectangular box. In multi-instance learning, for a set of sets, if it contains positive samples, then the set is marked as a positive example; if there is no positive sample, it is marked as a negative example. For a certain target in the training image, for each pixel in the training image, if the pixel belongs to the target, then the pixel is a positive sample of the target; if the pixel does not belong to the target, then the pixel is a negative sample of the target. In the annotated true bounding box, each row and each column in the box contains positive samples, and the samples outside the box must be negative samples for the target. Therefore, the pixel-level instance segmentation results can be supervised by using the true bounding box. It can be understood that, depending on the shape of the target, the real bounding box may also be other shapes such as an ellipse.

[0050] Step S220: performing an affine transformation operation on the training image to obtain a transformed training image.

[0051] Exemplarily, the affine transformation includes one or more of the following operations: rotation operation, shearing operation, translation operation and scaling operation. The rotation operation refers to the operation of rotating all pixels in the training image clockwise or counterclockwise at a certain angle at the same time. The shearing operation refers to the operation of scaling the directed distance of each pixel in the training image to a straight line parallel to the direction along a certain direction according to a certain ratio, so that the training image is deformed. The translation operation refers to the operation of moving all pixels in the training image horizontally, i.e., the x-axis direction and / or vertically, i.e., the y-axis direction, by a certain distance. The scaling operation refers to the operation of scaling the position coordinates of all pixels in the training image, i.e., the x-coordinate and / or the y-coordinate, by the same or different ratios. It can be understood that when the x-coordinate and the y-coordinate are scaled by the same ratio, the resulting transformed training image is not deformed, but the training image as a whole is enlarged or reduced. When the x-coordinate and the y-coordinate are scaled by different ratios, the resulting transformed training image is deformed, and may be elongated or shortened in the horizontal or vertical direction.

[0052] Figure 3 FIG. 4 is a schematic diagram showing a training method for an instance segmentation model according to an embodiment of the present invention. Figure 3 As shown in , a 25 degree clockwise rotation operation is performed on the training image in the upper left corner to obtain the transformed training image in the lower left corner.

[0053] The above affine transformation operation not only has a small amount of calculation but also has high accuracy, thereby ensuring the accuracy of the trained instance segmentation model.

[0054] Exemplarily, the affine transformation operation also includes multiple different operations. The different operations may be different types of operations, such as the above-mentioned rotation operation and translation operation; or they may be the same type of operations with different transformation parameters. In other words, when the same type of affine transformation operation is performed on the training image, changing one or more of the transformation parameters is considered to be performing different operations. For example, a training image is rotated 20 degrees clockwise to obtain a transformed training image A. The same training image is then rotated 50 degrees clockwise to obtain a transformed training image B. Although the types of these two operations have not changed, both are rotation operations, but the rotation angle has changed, and these two operations can be considered to be different operations. It can be understood that when the type of operation remains unchanged and only the transformation parameters of the operation are changed, the multiple transformed training images obtained can form a group of training samples together with the training image. The number of samples in each group of training samples can be increased by changing the transformation parameters. In this example, the transformed training image A, the transformed training image B and the training image can be used as a group of training samples.

[0055] By performing different types of affine transformation operations on the training images, a variety of transformed training images can be obtained. It can be understood that in the process of training the instance segmentation model, the more groups of training samples there are, the greater the data enhancement effect on the training images, the better the effect of the trained instance segmentation model, and the stronger the generalization.

[0056] Step S230: input the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask.

[0057] Exemplarily, the instance segmentation model can be implemented using a neural network, such as a convolutional neural network (CNN), a fully convolutional network (FCN), or a region-based CNN network (R-CNN). These neural networks can respectively constitute one or more networks.

[0058] For the convenience of description, a two-way network is used as an example to illustrate the instance segmentation model. The two-way network shares weights and network structure. Figure 3 As shown in FIG. 1 , the training image can be used as the input of the first network (path 1), and the training image is segmented by the neural network, and then the first prediction mask of the target in the training image is output, which is recorded as M path1The first prediction mask may be an irregularly shaped image or graphic for shielding each target. The first prediction mask may be used to represent all targets in the training image, and different targets may have different prediction masks. For each target in the training image, there may be one or more masks.

[0059] Step S240 , similar to step S230 , also inputs the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask.

[0060] Although as mentioned above, multiple affine transformation operations can be performed on the training image, for simplicity of description, in this embodiment, an affine transformation operation is performed on the training image once as an example for explanation.

[0061] Similarly, the transformed training image can be used as the input of the second network (path2). The second network shares weights and network structure with the first network. The second network is used to perform instance segmentation on the transformed training image to output a second prediction mask of the target in the image, denoted as M path2 . Similarly, the second prediction mask may also be an image or graphic with an irregular shape for occluding each target. The second prediction mask may be used to represent all targets in the transformed training image. For each target in the transformed training image, there may be one or more masks. Since the first network and the second network share weights and network structure, the inference process of the network is not affected, no additional inference calculation is introduced, and the network inference time is not increased.

[0062] Step S250 , performing the same affine transformation operation as that performed on the training image on the first prediction mask to obtain a transformed mask.

[0063] As mentioned above, the affine transformation operation may include one or more of a rotation operation, a shearing operation, a translation operation, and a scaling operation. These operations have been described in the previous text, and for the sake of brevity, they will not be repeated here. In order to align the transformed mask with the transformed training image in space. Therefore, the same affine transformation operation as the training image can be performed on the first prediction mask.

[0064] Step S260, determining a function value of a main loss function based on the true bounding box, the first prediction mask, the transformed mask, and the second prediction mask.

[0065] The true bounding box can be used as a supervisory signal for the prediction result of the first network (i.e., the first prediction mask). The transformed mask can be used as a supervisory signal for the prediction result of the second network (i.e., the second prediction mask). The function value of the main loss function is determined by combining these four parameters. The main loss function is an evaluation indicator of the instance segmentation model. The smaller the function value of the main loss function, the higher the matching degree of the instance segmentation model on the sample prediction. The main loss function not only contains the prediction quality information of the first network, but also contains the prediction quality information of the second network. The prediction of the second network enhances the prediction of the first network. The instance segmentation model trained using the main loss function has better performance not only for the training images, but also for the transformed training images. Therefore, in this scheme, the target edge learning ability of the instance segmentation model is enhanced by data enhancement.

[0066] Step S270: training the instance segmentation model based on the function value of the main loss function.

[0067] The parameters of the instance segmentation model can be adjusted multiple times based on the function value of the main loss function. The function value of the main loss function or the number of training times can be used as the training end condition. Through continuous training, the prediction results of the instance segmentation model can be made more and more accurate. For any image input to the instance segmentation model, a more accurate instance segmentation result can be obtained.

[0068] It is understood that the above technical solution is only for illustration and does not constitute a limitation to the present invention. For example, the order of step S220 and step S230 can be swapped. The order in the above solution is only exemplary and does not limit the order of steps in the actual training process.

[0069] In the above technical solution, bounding box annotation is used to replace pixel-level annotation, which greatly reduces the cost of manual annotation and shortens the annotation cycle. At the same time, it does not rely on prior information related to the target, and only uses affine transformation operations to enhance the training of the instance segmentation model. It not only improves the autonomous learning ability of the instance segmentation model, but also enhances its generalization, reduces the degree of influence of factors such as the brightness difference of the image or the target, and can obtain more accurate instance segmentation results.

[0070] Exemplarily, the function value of the first dominant loss function can be calculated based on the true bounding box and the first predicted mask. The function value of the first dominant loss function can represent the deviation between the first predicted mask and the true bounding box. The true bounding box serves as a supervisory signal for the first predicted mask. The smaller the function value of the first dominant loss function, the smaller the deviation between the first predicted mask and the true bounding box, and the more accurate the first predicted mask; otherwise, vice versa.

[0071] The function value of the second dominant loss function can be calculated based on the transformed mask and the second prediction mask. The function value of the second dominant loss function can represent the deviation between the second prediction mask and the transformed mask. The transformed mask serves as a supervisory signal for the second prediction mask. The smaller the function value of the second dominant loss function, the closer the shape and / or area of ​​the second prediction mask is to the transformed mask, and thus the prediction of the pixels in the second prediction mask is more accurate and the offset is smaller.

[0072] The function value of the main loss function can be calculated based on the function value of the first dominant loss function and the function value of the second dominant loss function. It can be understood that the smaller the function value of the main loss function is, the more accurate the segmentation result of the instance segmentation model is and the better the segmentation effect is. Compared with the true target, the segmentation result has a smaller loss and a smaller offset.

[0073] Exemplarily, the function value of the first dominant loss function and the function value of the second dominant loss function can be simply added and summed to obtain the function value of the main loss function. Alternatively, the function value of the first dominant loss function and the function value of the second dominant loss function can be weighted and summed to obtain the function value of the main loss function. In one example, the function value of the first dominant loss function has a greater impact on the final training result than the function value of the second dominant loss function. Then in the process of calculating the function value of the main loss function, different weight coefficients can be assigned to the function value of the first dominant loss function and the function value of the second dominant loss function, respectively, and then summed. The weight coefficient of the function value of the first dominant loss function is greater than the weight coefficient of the function value of the second dominant loss function. The weight coefficients of the two can be obtained based on multiple repeated experiments and are not limited here.

[0074] In the above technical solution, the instance segmentation effect of the training image and the instance segmentation effect of the transformed training image are considered respectively, and the function value of the first dominant loss function and the function value of the second dominant loss function are weighted summed. This not only comprehensively considers the training image and the transformed training image, but also fully considers the relative importance of the two in general. According to expectations, the function value of the main loss function obtained can represent the training result with some emphasis, thereby making the training effect of the instance segmentation model more in line with expectations.

[0075] Calculating the function value of the first dominant loss function based on the true bounding box and the first predicted mask can be implemented by any existing or future developed technology.

[0076] Exemplarily, first, for each target in the training image, based on the true bounding box of the target and the i-th first predicted mask Calculate the sub-objective function of this goal Then, the first dominant loss function of the target is calculated using the following formula based on the function values ​​of all sub-target functions of the target: The function value of , where N represents the number of the first prediction masks of the target.

[0077] The first dominant loss function may be a multi-instance learning loss function. For example, the multi-instance learning loss function may be implemented in the form of a projection loss function, etc., by taking the weighted responses of the true target box and the first predicted mask on the x-axis and y-axis of the image as projection signals, and further supervising the projection signals on the x-axis and y-axis using a classification loss function. The classification loss function may be a cross entropy loss function (Cross Entropy Loss), a focal loss function (FocalLoss), a similarity loss function (Dice Loss), etc.

[0078] Exemplarily, the function value of the second dominant loss function is calculated based on the transformed mask and the second predicted mask by the following scheme. For each target in the training image, first, based on the i-th first predicted mask of the target The transformed mask obtained by this affine transformation operation And the j-th second prediction mask of the target obtained by performing instance segmentation after performing the affine transformation operation on the training image Calculate the sub-objective function of this goal Then, the second dominant loss function of the target is calculated using the following formula according to the function values ​​of all sub-target functions of the target: , where N represents the number of first prediction masks of the target, and M represents the number of second prediction masks of the target obtained by performing instance segmentation after performing the affine transformation operation on the training image.

[0079] As described above, there may be multiple first or second prediction masks for each object in the training image or the transformed training image. For example, for a certain object in the training image, there are a total of N first prediction masks, where represents the i-th first prediction mask. Perform an affine transformation operation to obtain the transformed mask The same affine transformation operation is performed on the training image to obtain a transformed training image. The transformed training image is then input into the instance segmentation model for instance segmentation to obtain a second prediction mask. For the same target, there are a total of M second prediction masks, where Denotes the jth second prediction mask. Since the affine transformation operation may cause image information loss, M≤N. Represents the loss function between the i-th transformed mask and the j-th second predicted mask of the target. The function values ​​of the loss functions of the N transformed masks and the M second predicted masks of the target can be accumulated, and the sum is used as the function value of the second dominant loss function of the target.

[0080] The above method of calculating the function value of the second dominant loss function has a small amount of calculation and an accurate calculation result, thereby effectively helping the training of the instance segmentation model.

[0081] Exemplarily, the second dominant loss function may be a multi-instance learning loss function or a pixel-level loss function. For example, as described in the relevant description of the first dominant loss function, the multi-instance learning loss function may be implemented in the form of a projection loss function or the like. The pixel-level loss function may also be based on pixel-based calculation of a cross entropy loss function, a focus loss function, or a similarity loss function, etc. These loss functions are more suitable for training instance segmentation models and can obtain more accurate instance segmentation models.

[0082] Exemplarily, the sub-objective function according to which the function value of the second dominant loss function is calculated can be obtained according to the following formula:

[0083]

[0084] Indicates based on The response function obtained by projecting onto the x-axis and The response function obtained by projecting onto the x-axis The calculated x-axis loss function; Indicates based on The response function obtained by projecting onto the y-axis and The response function obtained by projecting onto the y-axis The calculated y-axis loss function. f x and f y They are the response functions that project the parameter mask to the x-axis and y-axis respectively, which can be a maximum function (max) or an activation function (softmax), etc., used to represent the segmentation confidence response of each row or column of pixels.

[0085] The sub-objective function obtained based on the above formula is a projection loss function. The instance segmentation model is trained based on the function value of the projection loss function, which can optimize the edge inaccuracy of the instance segmentation result.

[0086] Reference again Figure 3, taking the dual-path network as an example, the training image is input into the first path network of the instance segmentation model, and before obtaining the first prediction mask, the first bounding box of the target and the score of the first bounding box can be obtained. According to the first bounding box and the score of the first bounding box, the first prediction mask can be determined. Specifically, the first path network can first extract the image features of the training image, and then predict the first bounding box of the target and the score of the first bounding box based on the extracted features. Similar to the first prediction mask, there may be multiple first bounding boxes for each target, and these multiple bounding boxes can correspond to different target categories respectively. The score of the first bounding box is the score that the target in the bounding box belongs to the corresponding category. It can be understood that the higher the score of the first bounding box, the greater the possibility that the target therein belongs to this target category. Exemplarily, the first bounding box, the score of the first bounding box, and the previously extracted image features can be input into the neural network to obtain the first prediction mask.

[0087] The instance segmentation model trained by this technical solution adopts a top-down instance segmentation method. First, the area where the instance is located is predicted by the target detection method, that is, the first bounding box, and then semantic segmentation is performed based on the characteristics of the instance. As a result, the trained instance segmentation model can not only adapt to the situation where there are multiple different targets in the same image, but also provide image regression and classification prediction results without increasing the network inference time, and has more application scenarios.

[0088] Exemplarily, the function value of the first auxiliary loss function can be determined based on the real bounding box, the first bounding box and the score of the first bounding box. The function value of the first auxiliary loss function can represent the deviation of the first bounding box relative to the real bounding box. Here, the real bounding box is used as a supervisory signal. The smaller the function value of the first auxiliary loss function, the more accurate the classification and regression prediction of the pixel points of the target in the first bounding box, and the smaller the offset. For the scores of the first bounding box and the first bounding box, the first auxiliary loss function can respectively adopt the loss function of regression and the loss function of classification. The loss function of regression can adopt the Euclidean distance (L2 distance) loss function, the intersection and union ratio loss function (IOULoss) of the candidate box and the real box, etc. The loss function of classification can adopt the cross entropy loss function, the focus loss function, etc. No restriction is made on the specific loss function here. In addition to the function value of the above-mentioned main loss function, the training instance segmentation model can also be based on the function value of the first auxiliary loss function at the same time. For example, the function value of the first auxiliary loss function can be added to the function value of the main loss function to obtain the function value of a new loss function, and the instance segmentation model is trained based on the function value of this new loss function.

[0089] This provides an extra layer of protection for the accuracy of the instance segmentation model and further improves the performance of the instance segmentation model.

[0090] Exemplarily, the transformed training image is input into the second network of the instance segmentation model, and the second bounding box of the target and the score of the second bounding box can be obtained. According to the second bounding box and the score of the second bounding box, the second prediction mask can be determined. This process is similar to the processing of the first network, and will not be repeated here for brevity.

[0091] Figure 3 The step of determining the minimum bounding box of the transformed mask is also shown. As described above, after performing an affine transformation operation on the first prediction mask, the transformed mask can be obtained. And the transformed mask is spatially aligned with the transformed training image. The operation of extracting and determining the minimum bounding box can be performed on the transformed mask. The minimum bounding box can be the minimum circumscribed rectangular box of the transformed mask, which can be represented in coordinate form. For example, xy coordinates. The function value of the second auxiliary loss function can be determined based on the minimum bounding box, the second bounding box, and the score of the second bounding box. At this time, the minimum bounding box is used as a supervisory signal. The function value of the second auxiliary loss function can represent the deviation of the second bounding box relative to the minimum bounding box. The smaller the function value of the second auxiliary loss function, the more accurate the classification and regression prediction of the pixel points of the target in the second bounding box, and the smaller the offset. Similar to the first auxiliary loss function, for the scores of the second bounding box and the second bounding box, the second auxiliary loss function can respectively adopt a regression loss function and a classification loss function. The loss function for classification can be a cross entropy loss function, a focus loss function, a similarity loss function, etc. The loss function for regression can be a Euclidean distance, an intersection-over-union loss function of a candidate box and a true box, etc. The training instance segmentation model can also be based on the function value of the second auxiliary loss function. Similarly, the function value of the second auxiliary loss function can be added to the function value of the main loss function or the function value of the main loss function and the function value of the first auxiliary loss function to obtain the function value of another new loss function, and the instance segmentation model is trained based on the function value of this new loss function.

[0092] Therefore, the classification and regression prediction of the second prediction mask are supervised by the minimum bounding box, which further enhances the autonomous learning ability of the instance segmentation model, improves the performance, and ensures the accuracy of the instance segmentation model.

[0093] It can be understood that in the above description, for the sake of simplicity, the instance segmentation model is described as a two-way network. As mentioned above, the affine transformation operation may include multiple different operations. For the case of n different operations, the instance segmentation model may include an n-way network with shared weights. A plurality of different transformed training images can be obtained by performing multiple different affine transformation operations on the training image. Each network includes performing similar operations on the corresponding transformed training image. When the function value of the loss function is finally calculated, the function value of the dominant loss function and / or the function value of the auxiliary loss function obtained by each network can be combined together, for example, by adding the data of each network, so that the instance segmentation model can be trained together. Thus, an instance segmentation model with stronger generalization ability and better segmentation effect is obtained.

[0094] According to another aspect of the present invention, a training device for an instance segmentation model is provided. Figure 4 A schematic block diagram of a training device 400 for an instance segmentation model according to an embodiment of the present invention is shown.

[0095] like Figure 4 As shown, the device 400 includes an acquisition module 410, an image transformation module 420, a first instance segmentation module 430, a second instance segmentation module 440, a mask transformation module 450, a loss determination module 460 and a training module 470. The modules can respectively perform the steps / functions of the training method of the instance segmentation model described above. Only the main functions of the components of the device 400 are described below, and the details described above are omitted.

[0096] The acquisition module 410 is used to acquire the training image and the corresponding annotation data. The annotation data includes the position information of the real bounding box of the target in the training image. The acquisition module 410 can be composed of Figure 1 The input device 106 in the electronic device shown is implemented.

[0097] The image transformation module 420 is used to perform an affine transformation operation on the training image to obtain a transformed training image. The image transformation module 420 may be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.

[0098] The first instance segmentation module 430 is used to input the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask. The first instance segmentation module 430 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.

[0099] The second instance segmentation module 440 is used to input the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask. The second instance segmentation module 440 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.

[0100] The mask transformation module 450 is used to perform an affine transformation operation on the first prediction mask to obtain a transformed mask. The mask transformation module 450 may be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.

[0101] The loss determination module 460 is used to determine the function value of the main loss function based on the true bounding box, the first prediction mask, the transformed mask and the second prediction mask. The loss determination module 460 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.

[0102] The training module 470 is used to train the instance segmentation model based on the function value of the main loss function. The training module 470 can be composed of Figure 1 The processor 102 in the electronic device shown executes program instructions stored in the storage device 104 to implement.

[0103] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0104] According to yet another aspect of the present invention, a training system for an instance segmentation model is provided. Figure 5 FIG. 5 is a schematic block diagram of a training system 500 for an instance segmentation model according to an embodiment of the present invention. Figure 5 As shown, the system 500 includes an input device 510 , a memory 520 , and a processor 530 .

[0105] The input device 510 is used to receive the transformation parameters input by the user and collect or acquire images. The input device 510 may include one or more of a keyboard, a mouse, a microphone, a touch screen, and an image acquisition device.

[0106] The memory 520 stores computer program instructions for implementing corresponding steps in the training method of the instance segmentation model according to an embodiment of the present invention.

[0107] The processor 530 is used to run the computer program instructions stored in the memory 520 to execute the corresponding steps of the training method of the instance segmentation model according to the embodiment of the present invention, and is used to implement the acquisition module 410, image transformation module 420, first instance segmentation module 430, second instance segmentation module 440, mask transformation module 450, loss determination module 460 and training module 470 in the training device of the instance segmentation model according to the embodiment of the present invention.

[0108] According to another aspect of the present invention, a storage medium is also provided. Program instructions are stored on the storage medium, and when the program instructions are executed by a computer or a processor, the computer or the processor executes the corresponding steps of the training method of the instance segmentation model of the embodiment of the present invention, and is used to implement the corresponding modules in the training device of the instance segmentation model according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0109] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0110] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0111] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0112] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0113] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0114] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0115] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0116] The various component embodiments of the present invention may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all of the functions of some modules in the training device for the instance segmentation model according to an embodiment of the present invention. The present invention may also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program for implementing the present invention may be stored on a computer-readable medium, or may be in the form of one or more signals. Such a signal may be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0117] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0118] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment. The protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A training method for an instance segmentation model, include: Acquire a training image and corresponding annotation data, wherein the annotation data includes position information of a real bounding box of an object in the training image; Performing an affine transformation operation on the training image to obtain a transformed training image; Inputting the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask; Inputting the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask; performing the affine transformation operation on the first prediction mask to obtain a transformed mask; Determine a function value of a main loss function based on the true bounding box, the first predicted mask, the transformed mask, and the second predicted mask; as well as The instance segmentation model is trained based on the function value of the main loss function.

2. The method according to claim 1, in, Inputting the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask includes: Inputting the training image into the instance segmentation model to obtain a first bounding box of the target and a score of the first bounding box; and The first prediction mask is determined according to the first bounding box and the score of the first bounding box.

3. The method according to claim 2, in, The method further comprises: Determine a function value of a first auxiliary loss function based on the true bounding box, the first bounding box, and the score of the first bounding box; Wherein, the training of the instance segmentation model is also based on the function value of the first auxiliary loss function.

4. The method according to any one of claims 1 to 3, in, The step of inputting the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask comprises: Inputting the transformed training image into the instance segmentation model to obtain a second bounding box of the target and a score of the second bounding box; and Determining the second prediction mask according to the second bounding box and the score of the second bounding box; The method further comprises: determining a minimum bounding box of the transformed mask; and Determine a function value of a second auxiliary loss function based on the minimum bounding box, the second bounding box, and the score of the second bounding box; Wherein, the training of the instance segmentation model is also based on the function value of the second auxiliary loss function.

5. The method according to any one of claims 1 to 3, in, The determining of the function value of the main loss function based on the true bounding box, the first prediction mask, the transformed mask and the second prediction mask comprises: Calculating a function value of a first dominant loss function based on the true bounding box and the first predicted mask; Calculating a function value of a second dominant loss function based on the transformed mask and the second predicted mask; and The function value of the main loss function is calculated based on the function value of the first dominant loss function and the function value of the second dominant loss function.

6. The method according to claim 5, in, The calculating the function value of the second dominant loss function based on the transformed mask and the second predicted mask comprises: for each affine transformation operation and for each target in the training image, The i-th first prediction mask based on the target The transformed mask obtained by this affine transformation operation and the j-th second prediction mask of the target obtained by performing instance segmentation after performing the affine transformation operation on the training image Calculate the sub-objective function of this goal The function value of ; and The function value of the second dominant loss function of the objective is calculated using the following formula based on the function values ​​of all sub-objective functions of the objective: Wherein, N represents the number of first prediction masks of the target, and M represents the number of second prediction masks of the target obtained by performing instance segmentation after performing the affine transformation operation on the training image.

7. The method according to claim 6, in, The function value of the sub-objective function is calculated according to the following formula: in, Indicates based on The response function obtained by projecting onto the x-axis and The response function obtained by projecting onto the x-axis The calculated x-axis loss function; Indicates based on The response function obtained by projecting onto the y-axis and The response function obtained by projecting onto the y-axis The calculated y-axis loss function.

8. The method according to claim 5, in, The second dominant loss function is a multi-instance learning loss function or a pixel-level loss function.

9. The method according to claim 5, in, The calculating the function value of the main loss function based on the function value of the first dominant loss function and the function value of the second dominant loss function comprises: A weighted sum is performed on the function value of the first dominant loss function and the function value of the second dominant loss function to obtain the function value of the main loss function.

10. The method according to any one of claims 1 to 3, in, The affine transformation operation includes a plurality of different operations.

11. The method according to any one of claims 1 to 3, in, The affine transformation operation includes one or more of the following operations: a rotation operation, a shearing operation, a translation operation, and a scaling operation.

12. A training device for an instance segmentation model, include: An acquisition module, used to acquire a training image and corresponding annotation data, wherein the annotation data includes position information of a real bounding box of an object in the training image; An image transformation module, used for performing an affine transformation operation on the training image to obtain a transformed training image; A first instance segmentation module, configured to input the training image into the instance segmentation model to perform instance segmentation on the training image and obtain a first prediction mask; A second instance segmentation module, used for inputting the transformed training image into the instance segmentation model to perform instance segmentation on the transformed training image and obtain a second prediction mask; a mask transformation module, configured to perform the affine transformation operation on the first prediction mask to obtain a transformed mask; A loss determination module, configured to determine a function value of a main loss function based on the true bounding box, the first predicted mask, the transformed mask, and the second predicted mask; as well as A training module is used to train the instance segmentation model based on the function value of the main loss function.

13. A training system for an instance segmentation model, comprising a processor and a memory, in, The memory stores computer program instructions, which are used by the processor to execute the training method of the instance segmentation model as described in any one of claims 1 to 11 when executed.

14. A storage medium having program instructions stored thereon, wherein the program instructions are used to execute the training method for the instance segmentation model as described in any one of claims 1 to 11 when running.

Images