Model training method, instance segmentation method, electronic device, and storage medium

Abstract

Embodiments of the present application provide a model training method, an instance segmentation method, an electronic device and a storage medium. In the embodiments of the present application, the trained instance segmentation model can predict the position information of the predicted detection frame of each instance object in the to-be-detected image, the predicted category information and the predicted mask information of a plurality of object regions in each instance object when performing instance segmentation processing on the to-be-detected image. Such an instance segmentation model has good segmentation accuracy and segmentation efficiency, and has good application effect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a model training method, an instance segmentation method, an electronic device, and a storage medium. BACKGROUND

[0002] Instance segmentation is widely used in various scenarios such as robots, autonomous driving, medical images, etc. The segmentation accuracy and efficiency of instance segmentation directly affect the application effect of instance segmentation. Improving the segmentation accuracy and efficiency of instance segmentation is a research hotspot in the field of computer vision. SUMMARY

[0003] Aspects of the present application provide a model training method, an instance segmentation method, an electronic device, and a storage medium to improve the segmentation accuracy and efficiency of an instance segmentation model.

[0004] The present application embodiment provides a model training method, comprising: obtaining a plurality of sample images, wherein the sample images include at least one instance object, and the annotation information of each instance object includes the position information of the annotation bounding box of the instance object in the sample image, the annotation category information of the instance object, and the annotation mask information of each object region in the instance object; performing model training according to the plurality of sample images and the annotation information of at least one instance object in each sample image to obtain an instance segmentation model, wherein the instance segmentation model is used for instance segmentation processing of a to-be-detected image to obtain an instance segmentation result of the to-be-detected image, and the instance segmentation result at least includes the position information of the predicted bounding box of each instance object in the to-be-detected image, the predicted category information, and the predicted mask information of each object region in the instance object.

[0005] The present application embodiment also provides an instance segmentation method, comprising: obtaining a to-be-detected image; inputting the to-be-detected image into an instance segmentation model to perform instance segmentation processing on the to-be-detected image by the instance segmentation model to obtain an instance segmentation result of the to-be-detected image, wherein the instance segmentation result at least includes the position information of the predicted bounding box of each instance object in the to-be-detected image, the predicted category information, and the predicted mask information of each object region in the instance object.

[0006] The embodiment of the present application further provides a model training device, comprising: an acquisition module, configured to acquire a plurality of sample images, wherein each sample image comprises at least one instance object, and the annotation information of each instance object comprises: the position information of the annotation bounding box of the instance object in the sample image, the annotation category information of the instance object, and the annotation mask information of each object region in the instance object; and a training module, configured to perform model training according to the plurality of sample images and the annotation information of the at least one instance object in each sample image, to obtain an instance segmentation model, wherein the instance segmentation model is configured to perform instance segmentation processing on a to-be-detected image, to obtain an instance segmentation result of the to-be-detected image, and the instance segmentation result at least comprises the position information of the predicted bounding box of each instance object in the to-be-detected image, the predicted category information, and the predicted mask information of each object region in each instance object.

[0007] The embodiment of the present application further provides an electronic device, comprising: a memory and a processor; the memory is configured to store a computer program; and the processor is coupled to the memory and is configured to execute the computer program to perform the steps in the model training method or the instance segmentation method.

[0008] The embodiment of the present application further provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can implement the steps in the model training method or the instance segmentation method.

[0009] In the embodiment of the present application, when the trained instance segmentation model performs instance segmentation processing on a to-be-detected image, the position information of the predicted bounding box of each instance object in the to-be-detected image, the predicted category information, and the predicted mask information of each object region in each instance object can be predicted. Such an instance segmentation model has good segmentation accuracy and segmentation efficiency, and has good application effect. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application, illustrate the exemplary embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings:

[0011] Figure 1 A flowchart of a model training method provided by the embodiment of the present application;

[0012] Figure 2 A flowchart of another model training method provided by the embodiment of the present application;

[0013] Figure 3 A network structure of an exemplary instance segmentation model provided by the embodiment of the present application;

[0014] Figure 4 A flowchart of an example instance segmentation method provided for embodiments of the present application;

[0015] Figure 5 A structural schematic diagram of a model training device provided for embodiments of the present application;

[0016] Figure 6 A structural schematic diagram of an instance segmentation device provided for embodiments of the present application;

[0017] Figure 7 A structural schematic diagram of an electronic device provided for embodiments of the present application. DETAILED DESCRIPTION

[0018] To make the objectives, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described below in detail with the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0019] In the embodiments of the present application, “at least one” means one or more, and “multiple” means two or more. “And / or” describes the access relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In the textual description of the present application, the character “ / ” generally represents an “or” relationship between the associated objects. In addition, in the embodiments of the present application, “first”, “second”, “third”, etc. are only to distinguish the content of different objects, and have no other special meanings.

[0020] The technical solutions provided by the embodiments of the present application will be described in detail below with reference to the drawings.

[0021] Figure 1 A flowchart of a model training method provided for embodiments of the present application. Referring to FIG. 4, Figure 1 The method can include the following steps:

[0022] 101、obtain a plurality of sample images, wherein the sample images include at least one instance object, and the annotation information of each instance object includes: position information of an annotated bounding box of the instance object in the sample image, annotated category information of the instance object, and annotated mask information of a plurality of object regions in the instance object.

[0023] 102. training the model according to the plurality of sample images and the annotation information of at least one instance object in each sample image to obtain an instance segmentation model.

[0024] Specifically, in the model training phase, a plurality of sample images are collected, each of which includes one or more instance objects, which can be various objects such as people, tables, apples, refrigerators, water dispensers, etc., without limitation. For any instance object in the sample image, the position information, category information and mask information of the detection box of the instance object are annotated. Here, the annotated detection box is referred to as an annotated detection box, the annotated category information is referred to as annotated category information, and the annotated mask information is referred to as annotated mask information. The annotated category information is, for example, various object categories such as tables, apples, refrigerators, water dispensers, etc. It is worth noting that when annotating mask information, multiple object regions in the instance object are annotated. For example, the instance object is a table, and the mask information of each object region such as the table top and the table leg is annotated. The instance object is a car, and the mask information of each object region such as the car head, the car wheel, the car body and the car tail is annotated.

[0025] In this embodiment, after completing the information annotation of each instance object in the sample image, the model is trained according to the plurality of sample images and the annotation information of at least one instance object in each sample image to obtain an instance segmentation model. Specifically, the model training is iteratively performed multiple times using the plurality of sample images, after each model training, the instance segmentation model with adjusted model parameters is used as the instance segmentation model to be trained in the next time, and the next model training is performed using the plurality of sample images, until the end of training condition is met, and the instance segmentation model obtained by the last training is used as the final instance segmentation model. The end of training condition can be that the number of model training reaches a specified number, or the instance segmentation model converges, which is not specifically limited here.

[0026] In actual application, in each model training, the sample image is input into the instance segmentation model to be trained, and the prediction result output by the instance segmentation model is obtained, which includes the position information of the prediction detection box of each instance object in the sample image, the prediction category information of each instance object, and the prediction mask information of each object region in each instance object. The prediction detection box is the detection box of the instance object detected by the instance segmentation model, the prediction category information is the category information of the instance object detected by the instance segmentation model, and the prediction mask information is the mask information of the object region in the instance object detected by the instance segmentation model.

[0027] The loss value of the sample image is determined according to the annotation information and the prediction result of each instance object in the sample image. The model parameters of the instance segmentation model are adjusted according to the loss value of the sample image. It is worth noting that the smaller the loss value is, the better the robustness of the model is. Therefore, in the iterative model training process, the model parameters of the instance segmentation model are continuously adjusted to minimize the loss value until the loss value is minimized or the number of model training reaches a specified number. For more information about model training, please refer to related technologies.

[0028] In this embodiment, any loss function can be used to determine the loss value of the training data. For example, the loss function includes but is not limited to: mean square error loss function, L2 loss function (also known as Euclidean distance), L1 loss function (Manhattan distance), cross-entropy loss function, softmax loss function, and Focal loss loss function mainly used to solve the problem of uneven difficult and easy samples, etc.

[0029] It is worth noting that when using a loss function to determine the loss value of the training data, the annotation information of each instance object in the sample image is input into the loss function as the true value, and the prediction result of each instance object in the sample image is input into the loss function as the predicted value. The loss function calculates the loss value of the training data based on the input true value and predicted value. For more information about how the loss function calculates the loss value, please refer to related technologies.

[0030] In this embodiment, for each instance object in the sample image, the loss value between the position information of the annotated bounding box and the position information of the predicted bounding box, the loss value between the annotated category information and the predicted category information, and the loss value between the annotated mask information and the predicted mask information of each object region in the instance object are weighted and summed, averaged, or accumulated. Various operations such as various operations are performed to obtain the loss value of the instance object. The loss values of multiple instance objects in the sample image are weighted and summed, averaged, or accumulated to obtain the loss value of the sample image.

[0031] In this embodiment, after multiple rounds of model training, an instance segmentation model that can be put into operation is obtained. The instance segmentation model is used for instance segmentation processing on the to-be-detected image to obtain an instance segmentation result of the to-be-detected image. The instance segmentation result at least includes the position information of the predicted bounding box, the predicted category information of each instance object in the to-be-detected image, and the predicted mask information of multiple object regions in each instance object.

[0032] The model training method provided in the embodiments of the present application, the instance segmentation model trained by the method can predict the position information of the prediction bounding box of each instance object in the to-be-detected image, the prediction category information, and the prediction mask information of multiple object regions in each instance object when performing instance segmentation processing on the to-be-detected image. Such an instance segmentation model has good segmentation accuracy and segmentation efficiency, and has good application effect.

[0033] The embodiments of the present application do not limit the network structure of the instance segmentation model. The network structure of the instance segmentation model includes, for example, but is not limited to: Convolutional Neural Networks (CNN), Vision Transformer (ViT). Further optionally, in order to improve the segmentation accuracy and segmentation efficiency of the instance segmentation model, the instance segmentation model includes a feature extraction network, an up-sampling network, and a detection network.

[0034] The model training method of another instance segmentation model is introduced below. Figure 2 The flowchart of another model training method provided in the embodiments of the present application is shown in FIG. 2. Figure 2 The method can include the following steps:

[0035] 201. Obtain a plurality of sample images, wherein each sample image includes at least one instance object, and the annotation information of each instance object includes: the position information of the annotation bounding box of the instance object in the sample image, the annotation category information of the instance object, and the annotation mask information of multiple object regions in the instance object.

[0036] For the implementation of step 201, please refer to the related description in the foregoing embodiments, which will not be repeated here.

[0037] 202. For any sample image, use a feature extraction network to perform multi-scale feature extraction on the sample image to obtain a plurality of feature maps of different scales.

[0038] In the present embodiment, the feature extraction network is any neural network structure with feature extraction function, and the number and type of neural network layers in the neural network structure are not limited, and the neural network layers include, for example, but are not limited to: convolution layer, fully connected layer, pooling layer, and activation layer.

[0039] In the present embodiment, the feature extraction network is a neural network structure with multi-scale feature extraction function. Please refer to Figure 3, i.e. can output a plurality of different scale feature maps corresponding to the sample image. For example, a feature map with a scale of 1 / 8 of the sample image, a feature map with a scale of 1 / 16 of the sample image, a feature map with a scale of 1 / 32 of the sample image, and so on. The feature extraction network outputs n different scale feature maps, and n is a positive integer greater than 1.

[0040] 203, using the upsampling network to perform upsampling processing on the largest scale feature map to obtain a mask template image, the mask template image has the same scale as the largest scale feature map, and the mask template image has k channels, k is a positive integer greater than 1.

[0041] In this embodiment, the upsampling network is any neural network structure with upsampling function. Referring to Figure 3 , the largest scale feature map is input into the upsampling network for upsampling processing to obtain

[0042] a mask template image. The mask template image is image data for assisting the generation of mask information. The mask template image has the same scale as the largest scale feature map, and the mask template image has k channels, k is a positive integer greater than 1, i.e. the mask template image has multiple channel data.

[0043] 204, for each feature map in the plurality of different scale feature maps, using the detection network to detect the feature map to obtain a detection result of the feature map, the detection result of the feature map including position information of a predicted bounding box of each instance object, confidence of a predicted class information, and k mask weight coefficients of each object region in each instance object.

[0044] In this embodiment, the detection network is a neural network structure with detection function. Referring to Figure 3 , for any scale feature map, the feature map is input into the detection network for detection, and the detection network outputs a detection result of each instance object in the sample image. The detection result of each instance object includes position information of a predicted bounding box of the instance object, confidence of a predicted class information, and k mask weight coefficients corresponding to each object region in each instance object. The mask weight coefficient has a value range of (0, 1), and the mask weight coefficient is used to assist the generation of mask information of the object region.

[0045] 205, for each object region of each instance object, using the k mask weight coefficients corresponding to the object region to modify the k channel data in the mask template image, and adding the modified k channel data in the mask template image to obtain predicted mask information of the object region.

[0046] Referring to Figure 3The k channel data in the mask template image is corrected by using the k mask weight coefficients corresponding to each object region, the k channel data in the mask template image after correction is added, and prediction mask information of the object region is obtained.

[0047] 206. According to the position information of the prediction bounding box of each instance object, the confidence of the prediction category information, and the prediction mask information of the plurality of object regions in each instance object, and the annotation information of each instance object, the model parameters of the instance segmentation model are adjusted.

[0048] Specifically, the detection network respectively performs feature extraction on the feature maps of multiple scales of the sample image, and can obtain the detection results of each feature map. According to the confidence of the prediction category information of the instance object, the detection results of the same instance object are screened, and the detection result of the instance object with the highest confidence of the prediction category information is retained. According to the position information of the prediction bounding box of each instance object, the confidence of the prediction category information, and the prediction mask information of the plurality of object regions in each instance object, and the annotation information of each instance object, the model parameters of the instance segmentation model are adjusted.

[0049] Further optionally, in order to improve the model performance of the instance segmentation model, when adjusting the model parameters of the instance segmentation model, the loss value of the bounding box dimension can be determined according to the position information of the prediction bounding box of each instance object and the position information of the annotated bounding box in the annotation information of each instance object; the loss value of the category information dimension is determined according to the confidence of the prediction category information of each instance object and the annotated category information in the annotation information of each instance object; the loss value of the mask information dimension is determined according to the prediction mask information of the plurality of object regions of each instance object and the annotated mask information of the plurality of object regions in the annotation information of each instance object; the total loss value is determined according to the loss value of the bounding box dimension, the loss value of the category information dimension and the loss value of the mask information dimension; and the model parameters of the instance segmentation model are adjusted according to the total loss value.

[0050] Specifically, in the determination of the loss value of the bounding box dimension, the loss value corresponding to each instance object is determined according to the position information of the prediction bounding box of each instance object and the position information of the annotated bounding box in the annotation information of each instance object; the loss values corresponding to the plurality of instance objects are weighted and summed, averaged, or accumulated, etc. Various operations are performed to obtain the loss value of the bounding box dimension.

[0051] In determining the loss value of the category information dimension, the confidence of the labeled category information in the label information is set to a specified confidence as needed. The loss value corresponding to each instance object is determined according to the confidence of the predicted category information of each instance object and the labeled category information in the label information of each instance object. The loss values corresponding to the plurality of instance objects are subjected to various operations such as weighted summation, mean operation or accumulation to obtain the loss value of the category information dimension.

[0052] In determining the loss value of the mask information dimension, for each instance object, the loss value of the object region of the instance object is determined according to the predicted mask information of each object region in the instance object and the labeled mask information of the object region. The loss values of the plurality of object regions of the plurality of instance objects are subjected to various operations such as weighted summation, mean operation or accumulation to obtain the loss value of the mask information dimension.

[0053] The loss value of the detection frame dimension, the loss value of the category information dimension and the loss value of the mask information dimension are subjected to various operations such as weighted summation, mean operation or accumulation to determine a total loss value. The model parameters of the instance segmentation model are adjusted according to the total loss value.

[0054] The model training method provided by the embodiments of the present application is used to train an instance segmentation model. The instance segmentation model includes a feature extraction network, an up-sampling network and a detection network. When the instance segmentation model is used to perform instance segmentation processing on a to-be-detected image, the instance segmentation model can predict the position information of the predicted detection frame of each instance object in the to-be-detected image, the predicted category information and the predicted mask information of the plurality of object regions in each instance object. Such an instance segmentation model has good segmentation precision and segmentation efficiency and has good application effect.

[0055] In some optional embodiments, in order to improve the model performance of the instance segmentation model, one implementation of step 204 is that the detection network is used to detect each pixel point in the feature map to obtain the detection result of each pixel point. The detection result of the pixel point includes the position information of the predicted detection frame of the pixel point, the confidence of the predicted category information, the confidence of the pixel point belonging to the foreground target and the k mask weight coefficients of the pixel point for each object region. Each pixel point is screened according to the confidence of the pixel point belonging to the foreground target to obtain a plurality of target pixel points whose confidence of belonging to the foreground target is greater than a pre-set confidence threshold. For each target pixel point, the position information of the predicted detection frame of the target pixel point, the confidence of the predicted category information, the k mask weight coefficients of the target pixel point for each object region are taken as the position information of the predicted detection frame of an instance object, the confidence of the predicted category information and the k mask weight coefficients of each object region in the instance object.

[0056] Specifically, detection is performed in pixel points in the feature map as a granularity, after obtaining the detection result of each pixel point, the plurality of pixel points in the feature map are screened according to the confidence that each pixel point belongs to the foreground target, and the pixel points with a larger probability of being the foreground target are screened out, which are referred to as target pixel points herein, the confidence that the target pixel points belong to the foreground target is greater than a preset confidence threshold, and the preset confidence threshold is flexibly set as needed. After screening out the target pixel points, the position information of the prediction bounding box of the target pixel points, the confidence of the prediction category information, and the k mask weight coefficients of the target pixel points for each object region are taken as the position information of the prediction bounding box, the confidence of the prediction category information, and the k mask weight coefficients of each object region in an instance object.

[0057] Further optionally, during model training, in order to improve the model performance of the instance segmentation model, the anchor point (that is, the pixel point in the feature map) where the center of the prediction bounding box of the instance object falls is taken as a positive sample, and the rest are negative samples. For the negative sample, the cross-entropy of the confidence that the pixel point belongs to the foreground target is taken as the loss function of the negative sample. For the positive sample, in addition to the cross-entropy of the confidence that the pixel point belongs to the foreground target, the loss between the prediction bounding box of the pixel point and the labeled bounding box needs to be determined, the loss information between the confidence of the prediction category information of the pixel point and the labeled category information needs to be determined, and the loss between the prediction mask information of the pixel point for each object region and the standard mask information needs to be determined. Finally, the gradient of the network is calculated through the back propagation of the loss, and the model parameters are updated.

[0058] Based on the instance segmentation model trained above, after the instance segmentation model is put into operation, it can be used for instance segmentation. Next, an instance segmentation method is introduced from the model inference stage. The instance segmentation method can be applied to various scenes, for example, mobile robot execution object grasping scene, object following scene or obstacle avoidance scene.

[0059] Figure 4 A flowchart of an instance segmentation method provided by the embodiments of the present application is shown in FIG. 4. Figure 4 The method can include the following steps:

[0060] 401, obtaining a to-be-detected image.

[0061] 402, inputting the to-be-detected image into an instance segmentation model to perform instance segmentation processing on the to-be-detected image by the instance segmentation model, obtaining an instance segmentation result of the to-be-detected image, the instance segmentation result at least including position information of a prediction bounding box of each instance object in the to-be-detected image, prediction category information, and prediction mask information of a plurality of object regions in each instance object.

[0062] In actual application, the position information of the predicted detection frame, the predicted category information and the predicted mask information of the plurality of object regions in each instance object in the to-be-detected image are obtained. For any target instance object in the plurality of instance objects, the object image of the target instance object is intercepted from the to-be-detected image according to the position information of the predicted detection frame of the target instance object; and the plurality of object regions in the target instance object are labeled in the object image according to the predicted mask information of the plurality of object regions in the target instance object.

[0063] Further, if the instance segmentation model comprises a feature extraction network, an up-sampling network and a detection network, after the to-be-detected image is input into the instance segmentation model, firstly, the feature extraction network is used to perform multi-scale feature extraction on the to-be-detected image to obtain a plurality of feature maps of different scales. Then, the up-sampling network is used to perform up-sampling processing on the feature map of the largest scale to obtain a mask template image, the mask template image has the same scale as the feature map of the largest scale, and the mask template image has k channels, where k is a positive integer greater than 1. Then, for each feature map in the plurality of feature maps of different scales, the detection network is used to detect the feature map to obtain a detection result of the feature map, the detection result of the feature map comprises the position information of the predicted detection frame, the confidence of the predicted category information and the k mask weight coefficients of each object region in each instance object. Then, for the predicted detection frames of the plurality of instance objects corresponding to the plurality of feature maps, the Non-Maximum Suppression (NMS) algorithm is used to screen the predicted detection frames of the plurality of instance objects to obtain the predicted detection frames of the plurality of instance objects that are retained. For each object region of each instance object in the plurality of instance objects that are retained, the k mask weight coefficients corresponding to the object region are used to correct the k channel data in the mask template image, and the k channel data in the mask template image that is corrected is added to obtain the predicted mask information of the object region. In this way, the position information of the predicted detection frame, the predicted category information and the predicted mask information of the plurality of object regions in each instance object in the to-be-detected image can be obtained.

[0064] The instance segmentation method provided by the embodiments of the present application can predict the position information of the predicted detection frame, the predicted category information and the predicted mask information of the plurality of object regions in each instance object in the to-be-detected image when the instance segmentation model performs instance segmentation processing on the to-be-detected image. Such an instance segmentation model has good segmentation accuracy and segmentation efficiency and has good application effect.

[0065] Figure 5 A structural schematic diagram of a model training device provided by the embodiments of the present application is shown in FIG. 1. As shown in FIG. 1, the model training device comprises a processor 100 and a memory 200. Figure 5The apparatus can comprise:

[0066] The acquisition module 51 is configured to acquire a plurality of sample images, wherein each sample image comprises at least one instance object, and the annotation information of each instance object comprises: position information of an annotation bounding box of the instance object in the sample image, annotation category information of the instance object, and annotation mask information of a plurality of object regions in the instance object.

[0067] The training module 52 is configured to train a model according to the plurality of sample images and the annotation information of the at least one instance object in each sample image, to obtain an instance segmentation model, wherein the instance segmentation model is configured to perform instance segmentation processing on a to-be-detected image to obtain an instance segmentation result of the to-be-detected image, and the instance segmentation result comprises at least position information of a predicted bounding box of each instance object in the to-be-detected image, predicted category information of the predicted bounding box, and predicted mask information of a plurality of object regions in each instance object.

[0068] Further, the instance segmentation model comprises a feature extraction network, an up-sampling network, and a detection network, and correspondingly, the training module 52 is specifically configured to: for any sample image, perform multi-scale feature extraction on the sample image by using the feature extraction network to obtain a plurality of feature maps of different scales; perform up-sampling processing on a feature map of the largest scale by using the up-sampling network to obtain a mask template image, wherein the mask template image has the same scale as the feature map of the largest scale, and the mask template image has k channels, and k is a positive integer greater than 1; for each feature map of the plurality of feature maps of different scales, perform detection on the feature map by using the detection network to obtain a detection result of the feature map, wherein the detection result of the feature map comprises position information of a predicted bounding box of each instance object, a confidence of predicted category information of the predicted bounding box, and k mask weight coefficients of each object region in each instance object; for each object region of each instance object, correct k channel data in the mask template image by using the k mask weight coefficients corresponding to the object region, and add the corrected k channel data in the mask template image to obtain predicted mask information of the object region; and adjust model parameters of the instance segmentation model according to the position information of the predicted bounding box of each instance object, the confidence of the predicted category information of the predicted bounding box, the predicted mask information of the plurality of object regions in each instance object, and the annotation information of each instance object.

[0069] Further optionally, when the training module 52 detects the feature map by using the detection network, the method specifically includes: detecting each pixel point in the feature map by using the detection network to obtain a detection result of each pixel point, wherein the detection result of the pixel point includes position information of a predicted bounding box of the pixel point, confidence of predicted category information, confidence of the pixel point belonging to a foreground target, and k mask weight coefficients of the pixel point for each object region; screening each pixel point according to the confidence of the pixel point belonging to the foreground target to obtain a plurality of target pixel points with a confidence greater than a preset confidence threshold; and taking the position information of the predicted bounding box of the target pixel point, the confidence of the predicted category information, and the k mask weight coefficients of the target pixel point for each object region as position information of a predicted bounding box of an instance object, confidence of predicted category information, and k mask weight coefficients of each object region in the instance object.

[0070] Further optionally, when the training module 52 adjusts the model parameters of the instance segmentation model, the method specifically includes: determining a loss value of a bounding box dimension according to the position information of the predicted bounding box of each instance object and the position information of the labeled bounding box in the labeled information of each instance object; determining a loss value of category information dimension according to the confidence of the predicted category information of each instance object and the labeled category information in the labeled information of each instance object; determining a loss value of mask information dimension according to the predicted mask information of each object region of each instance object and the labeled mask information of each object region in the labeled information of each instance object; determining a total loss value according to the loss value of the bounding box dimension, the loss value of the category information dimension, and the loss value of the mask information dimension; and adjusting the model parameters of the instance segmentation model according to the total loss value.

[0071] Figure 5 The apparatus shown can perform Figure 1 or Figure 2 The method shown in the embodiments shown has the same implementation principles and technical effects, which will not be described here. For the Figure 5 The apparatus shown, wherein each module and unit performs operations in the specific manner described in the embodiments related to the method, will not be described in detail here.

[0072] Figure 6 An example structure diagram of an instance segmentation apparatus is provided for the embodiments of the present application. Referring to Figure 6 The apparatus can include:

[0073] The acquisition module 61 is configured to acquire a to-be-detected image.

[0074] The instance segmentation module 62 is configured to input the to-be-detected image into an instance segmentation model, perform instance segmentation processing on the to-be-detected image by the instance segmentation model, and obtain an instance segmentation result of the to-be-detected image. The instance segmentation result at least includes position information of a predicted detection frame of each instance object in the to-be-detected image, predicted category information, and predicted mask information of a plurality of object regions in each instance object.

[0075] Figure 6 The apparatus shown can perform Figure 4 The method shown in the embodiments shown has the same implementation principles and technical effects, which will not be described again. For the steps of the method in the above embodiments Figure 6 The specific manners in which the various modules and units of the apparatus shown perform operations have been described in detail in the embodiments of the method, and will not be described in detail here.

[0076] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 102 can be device A; for another example, the execution subject of step 101 can be device A, and the execution subject of step 102 can be device B; and the like.

[0077] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clear that these operations can be executed or in parallel without the order in which they appear in this document. The serial numbers of the operations, such as 401, 402, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent the order of precedence. Also, "first" and "second" are not different types.

[0078] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0079] Figure 7 A structural schematic diagram of an electronic device is provided in the present application. As shown in Figure 7 The electronic device includes a memory 71 and a processor 72.

[0080] Memory 71 is used to store computer programs and can be configured to store various other data to support operation on the computing platform. Examples of this data include instructions for any application or method operating on the computing platform, contact data, phone book data, messages, pictures, videos, etc.

[0081] The memory 71 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random-access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0082] Processor 72, coupled to memory 71, is used to execute computer programs in memory 71 for steps in: model training methods or instance segmentation methods.

[0083] Furthermore, such as Figure 7 As shown, the electronic device also includes other components such as a communication component 73, a display 74, a power supply component 75, and an audio component 76. Figure 7 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 7 The components shown. Additionally... Figure 7 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the electronic device. The electronic device in this embodiment can be a desktop computer, laptop computer, smartphone, or IoT (Internet of Things) device, or a server-side device such as a conventional server, cloud server, or server array. If the electronic device in this embodiment is a desktop computer, laptop computer, or smartphone, it may include... Figure 7 The components within the dashed box; if the electronic device in this embodiment is implemented as a conventional server, cloud server, or server array, etc., it may be omitted. Figure 7 The component within the dashed box.

[0084] For a detailed description of the implementation process of each action by the processor, please refer to the relevant descriptions in the foregoing method embodiments or device embodiments, which will not be repeated here.

[0085] Accordingly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by an electronic device in the above method embodiments.

[0086] Accordingly, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, enables the processor to perform the steps that can be executed by an electronic device in the above method embodiments.

[0087] The aforementioned communication components are configured to facilitate wired or wireless communication between the device containing the communication components and other devices. The device containing the communication components can access wireless networks based on communication standards, such as WiFi (Wireless Fidelity), 2G (2nd Generation), 3G (3rd Generation), 4G (4th Generation) / LTE (long Term Evolution), 5G (5th Generation), or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wide Band (UWB), Bluetooth, and other technologies.

[0088] The aforementioned display includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from a user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0089] The aforementioned power supply components provide power to various components within the device in which they reside. These power supply components may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device in which they reside.

[0090] The aforementioned audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, or voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0091] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0092] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] In a typical configuration, a computing device includes one or more processors (Central Processing Unit, CPU), input / output interfaces, network interfaces, and memory.

[0096] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0097] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change RAM (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device. As defined in this article, computer-readable media do not include transient media, such as modulated data signals and carrier waves.

[0098] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0099] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A model training method, characterized in that, The method comprises the following steps: obtaining a plurality of sample images, wherein each sample image comprises at least one instance object, and the annotation information of each instance object comprises the position information of the annotation bounding box of the instance object in the sample image, the annotation category information of the instance object, and the annotation mask information of each object region in the instance object; training a model according to the plurality of sample images and the annotation information of the at least one instance object in each sample image to obtain an instance segmentation model, wherein the instance segmentation model is used for performing instance segmentation processing on a to-be-detected image to obtain an instance segmentation result of the to-be-detected image, and the instance segmentation result comprises at least the position information of the predicted bounding box of each instance object in the to-be-detected image, the predicted category information, and the predicted mask information of each object region in each instance object; the instance segmentation model comprises a feature extraction network, an up-sampling network, and a detection network, and the training of the model according to the plurality of sample images and the annotation information of the at least one instance object in each sample image to obtain the instance segmentation model comprises the following steps: for any sample image, performing multi-scale feature extraction on the sample image by using the feature extraction network to obtain a plurality of feature maps of different scales; performing up-sampling processing on the feature map of the largest scale by using the up-sampling network to obtain a mask template image, wherein the mask template image has the same scale as the feature map of the largest scale, and the mask template image has k channels, and k is a positive integer greater than 1; for each feature map of the plurality of feature maps of different scales, performing detection on the feature map by using the detection network to obtain a detection result of the feature map, wherein the detection result of the feature map comprises the position information of the predicted bounding box of each instance object, the confidence of the predicted category information, and k mask weight coefficients of each object region in each instance object; for each object region of each instance object, using the k mask weight coefficients corresponding to the object region to correct the k channel data in the mask template image, and adding the corrected k channel data in the mask template image to obtain the predicted mask information of the object region; adjusting the model parameters of the instance segmentation model according to the position information of the predicted bounding box of each instance object, the confidence of the predicted category information, the predicted mask information of the plurality of object regions in each instance object, and the annotation information of each instance object.

2. The method of claim 1, wherein, for each feature map of the plurality of feature maps of different scales, performing detection on the feature map by using the detection network to obtain a detection result corresponding to the feature map, comprising: performing detection on each pixel point in the feature map by using the detection network to obtain a detection result of each pixel point, wherein the detection result of the pixel point comprises the position information of the predicted bounding box of the pixel point, the confidence of the predicted category information, the confidence of the pixel point belonging to a foreground target, and k mask weight coefficients of the pixel point for each object region; screening each pixel point according to the confidence of the pixel point belonging to the foreground target to obtain a plurality of target pixel points with a confidence greater than a preset confidence threshold. For each target pixel point, the position information of the prediction bounding box of the target pixel point, the confidence of the prediction category information, and the k mask weight coefficients of the target pixel point for each object region are taken as the position information of the prediction bounding box of an example object, the confidence of the prediction category information, and the k mask weight coefficients of each object region in the example object.

3. The method of claim 1, wherein, According to the position information of the prediction bounding box of each example object, the confidence of the prediction category information, and the prediction mask information of multiple object regions in each example object, and the annotation information of each example object, the model parameters of the instance segmentation model are adjusted, including: According to the position information of the prediction bounding box of each example object and the position information of the annotation bounding box in the annotation information of each example object, a loss value in the dimension of the bounding box is determined. According to the confidence of the prediction category information of each example object and the annotation category information in the annotation information of each example object, a loss value in the dimension of the category information is determined. According to the prediction mask information of multiple object regions of each example object and the annotation mask information of multiple object regions in the annotation information of each example object, a loss value in the dimension of the mask information is determined. According to the loss value in the dimension of the bounding box, the loss value in the dimension of the category information, and the loss value in the dimension of the mask information, a total loss value is determined. The model parameters of the instance segmentation model are adjusted according to the total loss value.

4. An instance segmentation method, characterized by, It includes: An image to be detected is obtained. The instance segmentation model is input into the instance segmentation model to perform instance segmentation processing on the image to be detected, and an instance segmentation result of the image to be detected is obtained, the instance segmentation result at least including the position information of the prediction bounding box of each example object in the image to be detected, the prediction category information, and the prediction mask information of multiple object regions in each example object; wherein the instance segmentation model is obtained according to the method of any one of claims 1 to 3.

5. A model training apparatus characterized by comprising: It includes: An acquisition module is configured to acquire a plurality of sample images, wherein the sample images include at least one example object, and the annotation information of each example object includes the position information of the annotation bounding box of the example object in the sample image, the annotation category information of the example object, and the annotation mask information of multiple object regions in the example object. A training module is configured to perform model training according to the plurality of sample images and the annotation information of at least one example object in each sample image, and obtain an instance segmentation model, wherein the instance segmentation model is configured to perform instance segmentation processing on an image to be detected, and obtain an instance segmentation result of the image to be detected, and the instance segmentation result at least includes the position information of the prediction bounding box of each example object in the image to be detected, the prediction category information, and the prediction mask information of multiple object regions in each example object. The instance segmentation model includes a feature extraction network, an up-sampling network, and a detection network, and correspondingly, the training module is specifically configured to: For any sample image, the feature extraction network is used to perform multi-scale feature extraction on the sample image, and a plurality of feature maps with different scales are obtained. The up-sampling network is used for up-sampling processing on the feature map with the largest scale, to obtain a mask template image, the mask template image has the same scale as the feature map with the largest scale, and the mask template image has k channels, k being a positive integer greater than 1. For each feature map in the plurality of feature maps with different scales, the detection network is used for detecting the feature map, to obtain a detection result of the feature map, the detection result of the feature map including position information of a predicted bounding box of each instance object, confidence of predicted category information, and k mask weight coefficients of each object region in each instance object. For each object region of each instance object, the k mask weight coefficients corresponding to the object region are used to correct k channel data in the mask template image, and the corrected k channel data in the mask template image are added, to obtain predicted mask information of the object region. According to the position information of the predicted bounding box of each instance object, the confidence of the predicted category information, the predicted mask information of the plurality of object regions in each instance object, and the labeled information of each instance object, the model parameters of the instance segmentation model are adjusted.

6. An instance segmentation device, comprising: Comprising: An acquisition module is configured to acquire a to-be-detected image. An instance segmentation module is configured to input the to-be-detected image into an instance segmentation model, to perform instance segmentation processing on the to-be-detected image by the instance segmentation model, to obtain an instance segmentation result of the to-be-detected image, the instance segmentation result including at least position information of a predicted bounding box of each instance object in the to-be-detected image, predicted category information, and predicted mask information of a plurality of object regions in each instance object; wherein the instance segmentation model is obtained according to the method of any one of claims 1-3.

7. An electronic device, comprising: Comprising: A memory and a processor; The memory is configured to store a computer program; The processor is coupled to the memory and is configured to execute the computer program to perform the steps in the method of any one of claims 1-4.

8. A computer readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, the processor is enabled to implement the steps in the method of any one of claims 1-4.

Images