Instance segmentation method, device, electronic device and storage medium

Abstract

The present application provides an instance segmentation method, device, electronic device and computer-readable storage medium. The instance segmentation method includes: performing feature extraction processing on an image to be segmented to obtain a first global feature map of the image to be segmented; performing feature prediction based on the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented; obtaining a binary segmentation map of the image to be segmented based on the first instance feature map and the first dynamic convolution feature map; and obtaining an instance segmentation result of the image to be segmented based on the binary segmentation map and the first classification feature map. The present application avoids the problem of position information loss during feature extraction of the ROI area, and improves the prediction accuracy of instance segmentation.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to an instance segmentation method, device, electronic device, and computer-readable storage medium. Background Art

[0002] Computer vision uses various imaging systems to replace visual organs as input sensitive means, and computers replace the brain to complete processing and interpretation. Among them, instance segmentation is a common processing task in computer vision. Instance segmentation is to distinguish and label each object in the image. Instance segmentation has important guiding significance in the fields of behavior recognition, public safety, video surveillance, etc.

[0003] Existing instance segmentation is generally completed based on the target detection algorithm. For example, based on the target bounding box obtained by the target detection algorithm, that is, the ROI (region of interest) of the target, regional features are extracted from the ROI, and the target segmentation result is obtained after a series of mask convolutions.

[0004] However, since the feature layer will be resized and scaled during the feature extraction process of the ROI, this will cause the loss of position information of the feature map, thereby causing the problem of reduced prediction accuracy of instance segmentation. Summary of the invention

[0005] The present application provides an instance segmentation method, device, electronic device and computer-readable storage medium, aiming to solve the problem of low prediction accuracy of instance segmentation caused by the loss of position information of feature maps due to the scale adjustment and scaling of feature layers during feature extraction of ROI areas in existing instance segmentation methods.

[0006] In a first aspect, the present application provides an instance segmentation method, the method comprising:

[0007] Performing feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented;

[0008] Performing feature prediction according to the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented;

[0009] Acquire a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map;

[0010] An instance segmentation result of the image to be segmented is obtained according to the binary segmentation map and the first classification feature map.

[0011] In a second aspect, the present application provides an instance segmentation device, the instance segmentation device comprising:

[0012] A first feature extraction unit is used to perform feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented;

[0013] A second feature extraction unit is used to perform feature prediction according to the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented;

[0014] A segmentation unit, configured to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map;

[0015] A determination unit is used to obtain an instance segmentation result of the image to be segmented according to the binary segmentation map and the first classification feature map.

[0016] In a possible implementation of the present application, the segmentation unit is further configured to:

[0017] According to the first dynamic convolution feature map, obtaining a convolution kernel size of the first instance feature map;

[0018] Performing convolution calculation on the first instance feature map according to the convolution kernel size to obtain a pixel feature map of the image to be segmented;

[0019] According to a preset pixel threshold, the pixel feature map is binarized to obtain a binary segmentation map of the image to be segmented.

[0020] In a possible implementation of the present application, the segmentation unit is further configured to:

[0021] A target segmentation subnetwork in the trained instance segmentation network is used to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map.

[0022] In a possible implementation of the present application, the instance segmentation apparatus further includes a training unit, and the training unit is specifically configured to:

[0023] Acquire a target image, wherein the target image has been annotated with category feature information and binary segmentation information;

[0024] Using a preset instance segmentation network, performing feature extraction processing on the target image to obtain a second global feature map of the target image;

[0025] Using a target segmentation subnetwork in a preset instance segmentation network, determining a binary segmentation map of the target image according to the second global feature map, and obtaining a segmentation prediction result of the target image;

[0026] Using a classification subnetwork in a preset instance segmentation network to classify the second global feature map, to obtain a classification prediction result of the target image;

[0027] The preset instance segmentation network is trained according to the segmentation prediction result and the classification prediction result to obtain a trained instance segmentation network.

[0028] In a possible implementation of the present application, the target segmentation subnetwork includes a feature extraction module and a dynamic convolution module, and the training unit is further specifically used for:

[0029] Using the feature extraction module, performing instance feature extraction processing on the second global feature map to obtain a second instance feature map of the target image;

[0030] Using the dynamic convolution module, performing convolution feature prediction processing on the second global feature map to obtain a second dynamic convolution feature map of the target image;

[0031] The target segmentation subnetwork is used to determine a binary segmentation map of the target image according to the second instance feature map and the second dynamic convolution feature map, so as to obtain a segmentation prediction result of the target image.

[0032] In a possible implementation of the present application, the training unit is further configured to:

[0033] Determining a classification loss value of the classification subnetwork according to the category feature information and the classification prediction result;

[0034] Determining a segmentation loss value of the target segmentation subnetwork according to the binary segmentation information and the segmentation prediction result;

[0035] The preset instance segmentation network is trained according to the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

[0036] In a possible implementation of the present application, the target image is further annotated with target frame information, and the training unit is further specifically used for:

[0037] Using a border regression subnetwork in a preset instance segmentation network, determining a target border of the target image according to the second global feature map, and obtaining a border regression result of the target image;

[0038] Determining a border regression loss value of the border regression subnetwork according to the target border information and the border regression result;

[0039] The preset instance segmentation network is trained according to the border regression loss value, the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

[0040] In a third aspect, the present application further provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, the processor executes the steps of any instance segmentation method provided in the present application.

[0041] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is loaded by a processor to execute the steps in the instance segmentation method.

[0042] The present application obtains the first global feature map of the image to be segmented by performing feature extraction processing on the image to be segmented; according to the first global feature map, obtains the first instance feature map, the first dynamic convolution feature map, and the first classification feature map of the image to be segmented; according to the first instance feature map and the first dynamic convolution feature map, obtains the binary segmentation map of the image to be segmented; according to the binary segmentation map of the image to be segmented and the first classification feature map, obtains the instance segmentation result of the image to be segmented. On the one hand, the instance segmentation of the target can be completed for the image to be segmented. On the other hand, since the binary segmentation map of the image to be segmented is directly obtained after feature extraction based on the first global feature map obtained after full convolution feature extraction of the image to be segmented; therefore, the problem of needing to perform feature extraction based on the ROI region in the previous instance segmentation algorithm is avoided, and then the unequal proportion adjustment and scaling of the feature scale in the feature extraction processing of the ROI region is avoided, resulting in the loss of position information of the feature map, thereby improving the accuracy of instance segmentation. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 is a schematic diagram of a scene of an example segmented image provided in an embodiment of the present application;

[0045] Figure 2 is a schematic diagram of a scenario of an instance segmentation system provided in an embodiment of the present application;

[0046] Figure 3 It is a flowchart of an instance segmentation method provided in an embodiment of the present application;

[0047] Figure 4 It is a schematic diagram of an overall framework of the instance segmentation method provided in the embodiment of the present application;

[0048] Figure 5 This is a schematic diagram of a scene of a binary segmentation map provided in an embodiment of the present application;

[0049] Figure 6 is a scene diagram of the instance segmentation result provided in the example of this application;

[0050] Figure 7 is a schematic diagram of the structure of an embodiment of an instance segmentation device provided in an embodiment of the present application;

[0051] Figure 8 It is a schematic diagram of the structure of an embodiment of an electronic device provided in the embodiments of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0053] In the description of the embodiments of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.

[0054] In order to enable any person skilled in the art to implement and use the present application, the following description is provided. In the following description, details are listed for the purpose of explanation. It should be understood that those of ordinary skill in the art can recognize that the present application can also be implemented without using these specific details. In other examples, the known process will not be elaborated in detail to avoid unnecessary details that make the description of the present application embodiment obscure. Therefore, the present application is not intended to be limited to the embodiments shown, but is consistent with the widest range of principles and features disclosed in accordance with the embodiments of the present application.

[0055] The embodiments of the present application provide an instance segmentation method, device, electronic device and computer-readable storage medium. The instance segmentation device can be integrated in an electronic device, which can be a server or a terminal.

[0056] First of all, before introducing the embodiments of the present application, the relevant content about the application background of the embodiments of the present application is introduced.

[0057] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify, track and measure targets, and further perform graphics processing so that the computer processes the images into images that are more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0058] Instance segmentation is the segmentation of semantic objects in an image at the instance level. Instance segmentation results are used for counting and are widely used in the fields of behavior recognition, public security, video surveillance, etc. For example, please refer to Figure 1 , Figure 1 It is a scene diagram of an instance segmentation image provided in an embodiment of the present application, in which the balloons in the image are used as the target of instance segmentation. The purpose of instance segmentation is to determine which pixel positions in the image are the pixels of the balloons, and to determine the pixels belonging to each balloon (such as balloon 1, balloon 2, balloon 3, balloon 4, balloon 5), so as to count the balloons.

[0059] The general instance segmentation technology is generally based on the target detection algorithm, that is, a two-stage or one-stage target detection algorithm is first performed. Based on the target border obtained by the target detection algorithm, the ROI region is subjected to feature extraction processing, and after a series of mask convolution processing, the target segmentation result is obtained. This type of instance segmentation method will rescale and scale the feature layer during the feature extraction process using the ROI region, which will cause the loss of position information in the full convolution feature map. In addition, the use of a two-stage target detection method will increase the computational complexity, which often means a decrease in computing speed.

[0060] Based on the above-mentioned defects of the existing related technologies, the embodiments of the present application provide an instance segmentation method, which at least overcomes the defects of the existing related technologies to a certain extent.

[0061] The executor of the instance segmentation method of the embodiment of the present application may be the instance segmentation device provided in the embodiment of the present application, or different types of electronic devices such as a server device, a physical host or a user equipment (UE) that integrates the instance segmentation device, wherein the instance segmentation device may be implemented in hardware or software, and the UE may specifically be a terminal device such as a smart phone, a tablet computer, a laptop computer, a PDA, a desktop computer or a personal digital assistant (PDA).

[0062] The electronic device can adopt a working mode of independent operation or a working mode of a device cluster. By applying the instance segmentation method provided in the embodiment of the present application, the instance segmentation of the image can be completed based on dynamic convolution without the need for feature extraction based on the ROI area, thereby avoiding the problem of position information loss during the feature extraction process of the ROI and improving the prediction accuracy of instance segmentation.

[0063] See also Figure 2 , Figure 2 A scene diagram of an instance segmentation system provided in an embodiment of the present application. The instance segmentation system may include an electronic device 100, in which an instance segmentation device is integrated. For example, the electronic device may obtain an image to be segmented, and perform feature extraction processing on the image to be segmented to obtain a first global feature map; according to the first global feature map, obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented; according to the first instance feature map and the first dynamic convolution feature map, obtain a binary segmentation map of the image to be segmented; according to the binary segmentation map and the first classification feature map, obtain an instance segmentation result of the image to be segmented.

[0064] In addition, if Figure 2 As shown, the example segmentation system may also include a memory 200 for storing data, such as image data and video data, for example, image data and video data captured by cameras deployed in specific scene areas (such as sorting sites and subway security inspection areas) over a period of time.

[0065] It should be noted that Figure 2The scenario diagram of the instance segmentation system shown is merely an example. The instance segmentation system and scenario described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the instance segmentation system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is equally applicable to similar technical problems.

[0066] Next, we will introduce the instance segmentation method provided by the embodiment of the present application. In the embodiment of the present application, an electronic device is used as the execution subject. For the sake of simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments. The instance segmentation method includes: performing feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented; performing feature prediction based on the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented; obtaining a binary segmentation map of the image to be segmented based on the first instance feature map and the first dynamic convolution feature map; and obtaining an instance segmentation result of the image to be segmented based on the binary segmentation map and the first classification feature map.

[0067] Reference Figure 3 , Figure 3 A flowchart of an instance segmentation method provided in an embodiment of the present application. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here. The instance segmentation method includes steps S10 to S40, wherein:

[0068] S10, performing feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented.

[0069] The image to be segmented is an image to be segmented, and the first global feature map refers to a feature map obtained after convolution feature extraction is performed on the image to be segmented.

[0070] In the embodiment of the present application, taking instance segmentation performed by a trained instance segmentation network (the specific structure can be referred to the description below) as an example, it is explained how to determine the instance segmentation result of the image to be segmented according to the image to be segmented.

[0071] Before performing instance segmentation on the image to be segmented, it is necessary to obtain the image to be segmented according to the specific application scenario of the instance segmentation.

[0072] For example, in some embodiments, in order to count the number of people in a specific area (such as a subway security area or a supermarket entrance and exit) through computer vision technology, cameras are usually deployed in the specific area in advance so that the camera's field of view covers the specific area. Images can then be captured by the cameras deployed in the specific area, and the captured images can be used as images to be segmented.

[0073] Among them, a fully convolutional network of any depth can be pre-designed as the backbone network of the instance segmentation network for feature extraction of the image to be segmented. For example, a backbone network of a commonly used semantic segmentation network (such as a BiSeNet network) can be used to perform feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented.

[0074] S20. Perform feature prediction according to the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented.

[0075] Among them, the first instance feature map refers to a feature map with a size of M*N and a depth of K; wherein K is a preset value and M*N is the size of the first global feature map.

[0076] The first dynamic convolution feature map refers to a feature map with a size of M*N and a depth of (Q*K); where K is a preset value and M*N is the size of the first global feature map.

[0077] The first classification feature map means that each point in the map can be represented as: i ,y i ) is the feature map of the target classification result (represented by C) of the center point.

[0078] Please refer to Figure 4 , Figure 4 It is a schematic diagram of an overall framework of the instance segmentation method provided in an embodiment of the present application.

[0079] like Figure 4 As shown, the instance segmentation network may include a backbone network, a classification subnetwork, and an object segmentation subnetwork. The specific training process of the instance segmentation network may refer to the following steps A1 to A5, A31 to A33, A51 to A53, and B1 to B2, which will not be repeated here.

[0080] The backbone network is used to output the first global feature map of the image to be segmented. For example, the backbone network takes the image to be segmented (such as a single RGB picture of a supermarket entrance and exit) as input, performs feature extraction on the image to be segmented, and outputs a M*N*D deep feature map (i.e., the first global feature map), where M*N represents the size of the first global feature map and D represents the number of feature channels.

[0081] Among them, it was found through experiments that using the backbone network of the semantic segmentation network as the backbone network of the instance segmentation network to perform feature extraction processing on the segmented image can make the convergence speed of the model (instance segmentation network) training faster; in addition, the first global feature map with richer multi-scale information can be obtained, providing richer scale information for subsequent data processing.

[0082] The classification subnetwork is used to output a first classification feature map of the image to be segmented according to the first global feature map. For example, the classification subnetwork takes the first global feature map of M*N*D as input and outputs the first classification feature map of M*N*C, where C is the target classification type. The size of the first classification feature map is proportional to that of the original image (the image to be segmented), and each point in the first classification feature map can be represented as a target classification result with the point as the center point.

[0083] The target segmentation subnetwork is used to output a binary segmentation map of the image to be segmented according to the first global feature map. The target segmentation subnetwork also includes a feature extraction module and a dynamic convolution module.

[0084] The feature extraction module is used to output a first instance feature map of the image to be segmented according to the first global feature map. For example, the input of the feature extraction module is a first global feature map of M*N*D, and the output is a first instance feature map of M*N*K, where K represents the feature map depth, and K is a preset value that can be set according to specific needs, such as K=512.

[0085] The dynamic convolution module is used to output a first dynamic convolution feature map of the image to be segmented according to the first global feature map. For example, the input of the dynamic convolution module is the first global feature map of M*N*D, and the output is the first dynamic convolution feature map of M*N*(Q*K); wherein Q*K represents the first dynamic convolution feature map of the point (x i ,y i ) is the one-dimensional dynamic convolution kernel of the target at the center point, K is a preset value and can be set according to specific needs, such as K=512.

[0086] S30. Obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map.

[0087] In some embodiments, "obtaining a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map" may specifically include: obtaining the convolution kernel size of the first instance feature map according to the first dynamic convolution feature map; performing convolution calculation on the first instance feature map according to the convolution kernel size to obtain a pixel feature map of the image to be segmented; and binarizing the pixel feature map according to a preset pixel threshold to obtain a binary segmentation map of the image to be segmented.

[0088] For example, the dynamic convolution module outputs a first dynamic convolution feature map of M*N*(Q*K), where Q*K (such as 9*K) represents the point (x i ,y i ) is a one-dimensional dynamic convolution kernel of a target with . ) as the center point. First, the convolution kernel size of the first instance feature map is determined according to the first dynamic convolution feature map. Specifically: the one-dimensional dynamic convolution kernel Q*K is adjusted to the three-dimensional dynamic convolution kernel P*P*K (such as 3*3*K), where Q=P*P, P, P, K are the length, width, and depth of the three-dimensional dynamic convolution kernel respectively; and the three-dimensional dynamic convolution kernel P*P*K is used as the convolution kernel size of the first instance feature map.

[0089] Then, the first instance feature map is convolved to obtain the pixel feature map of the image to be segmented. For example, according to the convolution kernel size of the first instance feature map (such as 3*3*K), the first instance feature map (such as M*N*K) is convolved to obtain an M*N*1 feature map, which is used as the pixel feature map of the image to be segmented. The pixel feature map of the image to be segmented includes the feature value of each pixel.

[0090] Finally, according to the preset pixel threshold, the pixel feature map is binarized to obtain a binary segmentation map of the image to be segmented. The preset pixel threshold can be set to a specific value according to specific needs and is not limited here. For example, if the preset pixel threshold is 0.5, the characteristic values ​​of pixels less than 0.5 in the pixel feature map of the image to be segmented are set to the minimum value 0, and the characteristic values ​​of pixels greater than or equal to 0.5 are set to the maximum value 1, so that a binary segmentation map of the image to be segmented can be obtained. Figure 5 As shown, Figure 5 It is a scene schematic diagram of the binary segmentation map provided in an embodiment of the present application.

[0091] The above-mentioned preset pixel threshold, minimum value of pixel feature value, and maximum value of pixel feature value are only examples, and can be set according to actual conditions, without limitation thereto.

[0092] From the above content, it can be seen that the first dynamic convolution feature map is predicted in real time according to the first global feature map; and the dynamic convolution kernel size is determined according to the first dynamic convolution feature map, and the first instance feature map is convolved to determine the pixel feature map of the image to be segmented; and the pixel feature map is binarized to obtain the binary segmentation map of the image to be segmented. Since the dynamic convolution kernel size can be determined to perform convolution calculation on the first instance feature map to obtain the pixel feature map of the image to be segmented, the corresponding dynamic convolution kernel can be adaptively calculated for different targets, so that the binary segmentation map of the image to be segmented is more accurate, and thus the segmentation result of the target is more accurate.

[0093] In some embodiments, a target segmentation subnetwork in a trained instance segmentation network may be used to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map.

[0094] For example, the target segmentation subnetwork in the instance segmentation network can obtain the convolution kernel size of the first instance feature map according to the first dynamic convolution feature map; perform convolution calculation on the first instance feature map according to the convolution kernel size to obtain the pixel feature map of the image to be segmented; perform binarization processing on the pixel feature map according to the preset pixel threshold to obtain the binary segmentation map of the image to be segmented. For details, please refer to the above examples, which will not be repeated here.

[0095] From the above, it can be seen that, since the corresponding dynamic convolution kernels are adaptively calculated for different targets, the trained instance segmentation network can fully learn and explore the better expression of target segmentation. Therefore, the target segmentation based on the instance segmentation network determines the dynamic convolution kernel size from the network, and performs convolution calculation on the first instance feature map to obtain the pixel feature map of the image to be segmented, which can make the binary segmentation map of the image to be segmented more accurate, and thus make the target segmentation result more accurate.

[0096] S40: Obtain an instance segmentation result of the image to be segmented according to the binary segmentation map and the first classification feature map.

[0097] Specifically, a plurality of continuous pixel points classified as a target (such as a person or a balloon) in the binary segmentation map of the image to be segmented are regarded as instance segmentation results of a target (such as a person or a balloon), thereby obtaining the instance segmentation result of the image to be segmented.

[0098] Furthermore, the number of objects (such as objects being people) in the image can be counted according to the instance segmentation result of the image to be segmented. Furthermore, the action of the object (such as the object being people) in the image can be identified according to the instance segmentation result of the image to be segmented.

[0099] For easier understanding, please refer to Figure 6 , Figure 6 1 is a schematic diagram of the scene of the instance segmentation result provided in the example of this application. Taking the image to be segmented as the regional image of the supermarket entrance and exit and the target as a person as an example, the specific scene is explained.

[0100] For example, the binary segmentation map of the image to be segmented is as follows: Figure 6 As shown in (1), according to the first classification feature map, Figure 6The classification results of each point in (1) (i.e., which positions are the pixel positions of people) can be determined, and then the binary segmentation map of the image to be segmented and the first classification feature map can be used to determine the following: Figure 6 The example segmentation result shown in (2) is that each individual target "person 1", "person 2", and "person 3" are segmented.

[0101] From the above content, it can be seen that the first global feature map of the image to be segmented is obtained by performing feature extraction on the image to be segmented; according to the first global feature map, the first instance feature map, the first dynamic convolution feature map, and the first classification feature map of the image to be segmented are obtained; according to the first instance feature map and the first dynamic convolution feature map, the binary segmentation map of the image to be segmented is obtained; according to the binary segmentation map of the image to be segmented and the first classification feature map, the instance segmentation result of the image to be segmented is obtained. On the one hand, the instance segmentation of the target can be completed for the image to be segmented. On the other hand, since the binary segmentation map of the image to be segmented is directly obtained after feature extraction based on the first global feature map obtained after full convolution feature extraction on the image to be segmented; therefore, the problem of feature extraction based on the ROI region in the previous instance segmentation algorithm is avoided, and the unequal proportion adjustment and scaling of the feature scale in the feature extraction process of the ROI region is avoided, resulting in the loss of position information of the feature map, thereby improving the accuracy of instance segmentation.

[0102] The preset instance segmentation network is trained by following steps A1 to A5 to obtain a trained instance segmentation network, wherein:

[0103] A1. Obtain the target image.

[0104] Among them, the target image has been annotated with category feature information and binary segmentation information.

[0105] The category feature information of the target image refers to the actual classification feature map of the target image. The actual classification feature map of the target image means that each point in the map can be represented as: i ,y i ) is the feature map of the target classification result (represented by C) of the center point.

[0106] The binary segmentation information of the target image refers to the actual binary segmentation map of the target image.

[0107] A2. Using a preset instance segmentation network, perform feature extraction processing on the target image to obtain a second global feature map of the target image.

[0108] The second global feature map refers to a feature map obtained after performing convolution feature extraction on the target image.

[0109] In some implementations, the preset instance segmentation network includes a backbone network, a classification subnetwork, and a target segmentation subnetwork.

[0110] In some embodiments, the preset instance segmentation network includes a backbone network, a classification subnetwork, an object segmentation subnetwork, and a bounding box regression subnetwork.

[0111] The functions of the backbone network, the classification subnetwork, and the target segmentation subnetwork, as well as the input and output of the network can refer to the description in the above step S20, which will not be repeated here. The functions of the bounding box regression subnetwork, as well as the input and output of the network can refer to the description in the following steps B1 to B2, which will not be repeated here.

[0112] Specifically, the target image with labeled category feature information and binary segmentation information is input into a preset instance segmentation network to call the backbone network in the preset instance segmentation network to perform feature extraction processing on the target image to obtain a second global feature map of the target image.

[0113] The backbone network is used to output the first global feature map of the image to be segmented according to the target image. For example, the backbone network takes the target image (such as a single RGB image of the subway security inspection area) as input, performs feature extraction on the target image, and outputs a M*N*D deep feature map (i.e., the second global feature map), where M*N represents the size of the second global feature map and D represents the number of feature channels.

[0114] A3. Using the target segmentation subnetwork in the preset instance segmentation network, determine the binary segmentation map of the target image according to the second global feature map, and obtain the segmentation prediction result of the target image.

[0115] The segmentation prediction result of the target image refers to the binary segmentation map of the target image obtained by prediction.

[0116] In some implementations, the object segmentation subnetwork may include a feature extraction module and a dynamic convolution module.

[0117] Among them, the functions of the feature extraction module and the dynamic convolution module, as well as the input and output of the module can refer to the description in the above step S20, and will not be repeated here.

[0118] At this time, “using the target segmentation subnetwork in the preset instance segmentation network to determine the binary segmentation map of the target image according to the second global feature map to obtain the segmentation prediction result of the target image” may specifically include the following steps A31 to A33, wherein:

[0119] A31. Use the feature extraction module to perform instance feature extraction processing on the second global feature map to obtain a second instance feature map of the target image.

[0120] Among them, the second instance feature map refers to a feature map with a size of M*N and a depth of K; wherein K is a preset value and M*N is the size of the second global feature map.

[0121] Specifically, the second global feature map is input into the feature extraction module, so as to call the feature extraction module to output a second instance feature map of the target image according to the second global feature map. For example, the input of the feature extraction module is a second global feature map of M*N*D, and the output is a second instance feature map of M*N*K, where K represents the feature map depth.

[0122] A32. Use the dynamic convolution module to perform convolution feature prediction processing on the second global feature map to obtain a second dynamic convolution feature map of the target image.

[0123] Among them, the second dynamic convolution feature map refers to a feature map with a size of M*N and a depth of (Q*K); where K is a preset value and M*N is the size of the second global feature map.

[0124] Specifically, the second global feature map is input to the dynamic convolution module, so as to call the dynamic convolution module to output the second dynamic convolution feature map of the target image according to the second global feature map. For example, the input of the dynamic convolution module is the second global feature map of M*N*D, and the prediction and output is the second dynamic convolution feature map of M*N*(Q*K); wherein Q*K represents the point (x i ,y i ) is the one-dimensional dynamic convolution kernel of the target at the center point, K is a preset value and can be set according to specific needs, such as K=512.

[0125] A33. Using the target segmentation subnetwork, determine the binary segmentation map of the target image according to the second instance feature map and the second dynamic convolution feature map, and obtain the segmentation prediction result of the target image.

[0126] The segmentation prediction result of the target image refers to a binary segmentation map of the target image obtained by convolving the second instance feature map.

[0127] Specifically, the target segmentation subnetwork is called to obtain the convolution kernel size of the second instance feature map according to the second dynamic convolution feature map; the second instance feature map is convolutionally calculated according to the convolution kernel size of the second instance feature map to obtain a pixel feature map of the target image; the pixel feature map of the target image is binarized according to a preset pixel threshold to obtain a binary segmentation map of the target image, that is, a segmentation prediction result of the target image is obtained.

[0128] For example, the dynamic convolution module outputs a second dynamic convolution feature map of M*N*(Q*K), where Q*K (such as 9*K) represents the point (x i ,y i ) is a one-dimensional dynamic convolution kernel of a target with . ) as the center point. First, the convolution kernel size of the second instance feature map is determined according to the second dynamic convolution feature map. Specifically: the one-dimensional dynamic convolution kernel Q*K is adjusted to the three-dimensional dynamic convolution kernel P*P*K (such as 3*3*K), where Q=P*P, P, P, K are the length, width, and depth of the three-dimensional dynamic convolution kernel respectively; and the three-dimensional dynamic convolution kernel P*P*K is used as the convolution kernel size of the second instance feature map.

[0129] Then, the second instance feature map is convolved to obtain the pixel feature map of the target image. For example, according to the convolution kernel size of the second instance feature map (such as 3*3*K), the second instance feature map (such as M*N*K) is convolved to obtain an M*N*1 feature map, which is used as the pixel feature map of the target image. The pixel feature map of the target image includes the feature value of each pixel.

[0130] Finally, according to the preset pixel threshold, the pixel feature map of the target image is binarized to obtain a binary segmentation map of the target image. The preset pixel threshold can be set to a specific value according to specific needs and is not limited here. For example, if the preset pixel threshold is 0.5, the characteristic value of the pixel less than 0.5 in the pixel feature map of the target image is set to the minimum value 0, and the characteristic value of the pixel greater than or equal to 0.5 is set to the maximum value 1, so that the binary segmentation map of the target image can be obtained.

[0131] The above-mentioned preset pixel threshold, minimum value of pixel feature value, and maximum value of pixel feature value are only examples, and can be set according to actual conditions, without limitation thereto.

[0132] From the above content, it can be seen that by adopting the feature extraction module, the second global feature map is subjected to instance feature extraction processing to obtain the second instance feature map of the target image, so that the trained instance segmentation network can also determine the first instance feature map of the image to be segmented according to the first global feature map.

[0133] By adopting a dynamic convolution module, convolution feature prediction processing is performed on the second global feature map to obtain a second dynamic convolution feature map of the target image, so that the trained instance segmentation network can also determine the first dynamic convolution feature map according to the first global feature map.

[0134] By using the target segmentation subnetwork, the binary segmentation map of the target image is determined according to the second instance feature map and the second dynamic convolution feature map, and the segmentation prediction result of the target image is obtained. This can improve the accuracy of the binary segmentation map of the image to be segmented determined by the trained instance segmentation network according to the first instance feature map and the first dynamic convolution feature map.

[0135] A4. Using a classification subnetwork in a preset instance segmentation network, classify the second global feature map to obtain a classification prediction result of the target image.

[0136] The classification prediction result of the target image refers to the second classification feature map of the target image obtained by prediction. The second classification feature map means that each point in the map can be represented as: i ,y i ) is the feature map of the target classification result (represented by C) of the center point.

[0137] Specifically, the classification subnetwork in the preset instance segmentation network is called to classify the second global feature map to obtain a classification prediction result of the target image. The classification subnetwork in the preset instance segmentation network is used to predict and output the second classification feature map of the target image based on the second global feature map. For example, the classification subnetwork takes the second global feature map of M*N*D as input and outputs the second classification feature map of M*N*C, where C is the target classification type. The size of the second classification feature map is proportional to that of the original image (target image), and each point in the second classification feature map can be represented as a target classification result with the point as the center point.

[0138] A5. According to the segmentation prediction result and the classification prediction result, the preset instance segmentation network is trained to obtain a trained instance segmentation network.

[0139] At this time, the trained instance segmentation network can be used to perform feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented. It can also be used to determine a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented based on the first global feature map. In addition, it can also be used to obtain a binary segmentation map of the image to be segmented based on the first instance feature map and the first dynamic convolution feature map.

[0140] From the above content, it can be seen that the preset instance segmentation network uses the classification subnetwork to perform classification prediction on the second global feature map of the target image according to the annotated target image, and uses the target segmentation subnetwork to predict the segmentation result of the second global feature map of the target image; and according to the segmentation prediction result and the classification prediction result, the preset instance segmentation network is trained to obtain the trained instance segmentation network. The trained instance segmentation network can be used to determine the first classification feature map and the binary segmentation map of the image to be segmented.

[0141] In some embodiments of the present application, the preset instance segmentation network may include a backbone network, a classification subnetwork, and a target segmentation subnetwork. The preset instance segmentation network may be trained in combination with the classification loss value of the classification subnetwork and the segmentation loss value of the target segmentation subnetwork to obtain a trained instance segmentation network. At this time, "training the preset instance segmentation network according to the segmentation prediction result and the classification prediction result to obtain a trained instance segmentation network" may specifically include the following steps A51 to A53, wherein:

[0142] A51. Determine the classification loss value of the classification subnetwork based on the category feature information and the classification prediction result.

[0143] In some implementations, in order to improve the classification accuracy of the classification subnetwork, the classification loss function may adopt a traditional cross entropy loss function or the like.

[0144] For example, the classification loss value of the classification subnetwork can be calculated by using the following formula (1) as the classification loss function (Focal Loss function):

[0145]

[0146] Among them, Loss class is the classification loss value, p is the classification probability, N represents each point, α and γ are the hyperparameters of Focal Loss. Generally, α is set to 0.25 and γ is set to 2, and label represents the category label (i.e., category feature information).

[0147] A52. Determine a segmentation loss value of the target segmentation subnetwork according to the binary segmentation information and the segmentation prediction result.

[0148] In some implementations, the segmentation loss function may adopt a traditional cross entropy loss function.

[0149] In some embodiments, in order to improve the segmentation accuracy of the target segmentation subnetwork, the segmentation loss function may adopt the Dice Loss function. For example, the segmentation loss value of the target segmentation subnetwork may be calculated by using the following formula (2) as the segmentation loss function:

[0150]

[0151] Among them, Loss mask is the segmentation loss value, X and Y are the actual binary segmentation map of the target image and the binary segmentation map of the target image predicted by the instance segmentation network, respectively.

[0152] The inventors found in practice that the Dice Loss loss function is used for training, and the network converges faster than the traditional pixel-level loss function (such as weighted or unweighted cross entropy function, Focal Loss function), and the final verification set accuracy is also very high; therefore, the accuracy of instance segmentation results can be effectively improved.

[0153] A53. According to the classification loss value and the segmentation loss value, the preset instance segmentation network is trained to obtain a trained instance segmentation network.

[0154] In some implementations, the total loss value of the model training is determined according to the following formula (3):

[0155] Loss = a 1 Loss class +a 2 Loss mask Formula (3)

[0156] Among them, Loss is the total loss value of model training, Loss class is the classification loss value, Loss mask is the segmentation loss value, a 1 、a 2 It is a proportional coefficient, which is used to balance the ratio between the classification loss value and the segmentation loss value.

[0157] Then, the total loss value of the model training is determined by formula (3) as the optimization target, and the preset instance segmentation network is trained to obtain the trained instance segmentation network.

[0158] From the above content, it can be seen that by combining the classification loss value and the segmentation loss value as the total loss value of network training, and taking the total loss value as the optimization target, the preset instance segmentation network is trained, so that the instance segmentation network after training is optimal in classification loss and segmentation loss, thereby improving the instance segmentation accuracy of the instance segmentation network after training.

[0159] In some embodiments of the present application, the preset instance segmentation network may include a backbone network, a classification subnetwork, a target segmentation subnetwork, and a bounding box regression subnetwork. In order to further improve the instance segmentation prediction accuracy of the trained instance segmentation network, the classification loss value of the classification subnetwork, the segmentation loss value of the target segmentation subnetwork, and the bounding box regression loss value of the bounding box regression subnetwork may be combined to train the preset instance segmentation network and obtain the trained instance segmentation network.

[0160] Specifically, the border regression loss value of the border regression subnetwork can be determined by the following steps B1 to B2, where:

[0161] B1. Using a border regression subnetwork in a preset instance segmentation network, determining a target border of the target image according to the second global feature map, and obtaining a border regression result of the target image;

[0162] Specifically, the second global feature map is input into the border regression subnetwork in the preset instance segmentation network to call the border regression subnetwork in the preset instance segmentation network, and the target border of the target image is determined according to the second global feature map to obtain the border regression result of the target image.

[0163] The bounding box regression subnetwork is used to output the bounding box regression result of the target image according to the second global feature map. For example, the bounding box regression subnetwork takes the second global feature map of M*N*D as input and outputs a bounding box regression feature map of M*N*(h*w), where h and w represent the bounding box regression feature map of the point (x i ,y i ) is the length and width of the target with centered point, the position information of the target in the target image can be obtained. i ,y i ) can be expressed as i ,y i ) is the length (h) and width (w) of the target with ) as the center point.

[0164] Among them, the bounding box regression result of the target image refers to the point (x i ,y i ) is the length h and width w of the target with ) as the center point, that is, the location information of the target. The location information of the target can be expressed as the center point (x, y) and the length and width (h, w), or can be expressed as the upper left corner coordinates (x1, y1) and lower right corner coordinates (x2, y2) of the target obtained by further transforming the center point (x, y) and the length and width (h, w). The transformation formula is as follows: Formula (4) to Formula (7):

[0165] x1=xw / 2 Formula (4)

[0166] y1=yh / 2 Formula (5)

[0167] x2=xw / 2 Formula (6)

[0168] y2=yh / 2 formula (7)

[0169] B2. Determine the border regression loss value of the border regression subnetwork according to the target border information and the border regression result.

[0170] In some embodiments, in order to improve the accuracy of the target border predicted by the border regression subnetwork, the border regression loss function may adopt the L1_Loss loss function, etc. The L1_Loss loss function is also called the minimum absolute deviation (LAD) and the minimum absolute error (LAE). In general, it minimizes the sum of the absolute differences between the target value and the estimated value.

[0171] For example, the border regression loss value of the border regression subnetwork can be calculated by using the following formula (8) as the border regression loss function:

[0172]

[0173] Among them, Loss reg is the border regression loss value, N represents each point, h and w are respectively i ,y i ) is the length and width of the actual target at the center point, They are respectively i ,y i ) is the center point of the network to predict the length and width of the target.

[0174] At this time, "training the preset instance segmentation network according to the classification loss value and the segmentation loss value to obtain a trained instance segmentation network" may specifically include: training the preset instance segmentation network according to the border regression loss value, the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

[0175] Specifically, in some implementations, the total loss value of the model training is determined according to the following formula (9):

[0176] Loss = a 1 Loss class +a 2 Loss mask +a 3 Loss reg Formula (9)

[0177] Among them, Loss is the total loss value of model training, Loss class is the classification loss value, Lossmask is the segmentation loss value, Loss reg is the border regression loss value, a 1 、a 2 、a 3 It is a proportional coefficient, which is used to balance the proportions among classification loss, segmentation loss, and bounding box regression loss.

[0178] Then, the total loss value of the model training is determined by formula (9) as the optimization target, and the preset instance segmentation network is trained to obtain the trained instance segmentation network.

[0179] From the above content, it can be seen that by combining the bounding box regression loss value, classification loss value and segmentation loss value as the total loss value of network training, and taking the total loss value as the optimization target, the preset instance segmentation network is trained, so that the trained instance segmentation network is optimal in terms of bounding box regression loss value, classification loss and segmentation loss, thereby improving the instance segmentation accuracy of the trained instance segmentation network.

[0180] In order to better implement the instance segmentation method in the embodiment of the present application, based on the instance segmentation method, the embodiment of the present application also provides an instance segmentation device, such as Figure 7 FIG. 1 is a schematic diagram of the structure of an example segmentation device according to an embodiment of the present application. The example segmentation device 700 includes:

[0181] A first feature extraction unit 701 is used to perform feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented;

[0182] A second feature extraction unit 702 is used to perform feature prediction according to the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented;

[0183] A segmentation unit 703 is used to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map;

[0184] The determining unit 704 is configured to obtain an instance segmentation result of the image to be segmented according to the binary segmentation map and the first classification feature map.

[0185] In some embodiments of the present application, the segmentation unit 703 is further configured to:

[0186] According to the first dynamic convolution feature map, obtaining a convolution kernel size of the first instance feature map;

[0187] Performing convolution calculation on the first instance feature map according to the convolution kernel size to obtain a pixel feature map of the image to be segmented;

[0188] According to a preset pixel threshold, the pixel feature map is binarized to obtain a binary segmentation map of the image to be segmented.

[0189] In some embodiments of the present application, the segmentation unit 703 is further configured to:

[0190] A target segmentation subnetwork in the trained instance segmentation network is used to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map.

[0191] In some embodiments of the present application, the instance segmentation apparatus further includes a training unit (not shown in the figure), and the training unit is specifically used to:

[0192] Acquire a target image, wherein the target image has been annotated with category feature information and binary segmentation information;

[0193] Using a preset instance segmentation network, performing feature extraction processing on the target image to obtain a second global feature map of the target image;

[0194] Using a target segmentation subnetwork in a preset instance segmentation network, determining a binary segmentation map of the target image according to the second global feature map, and obtaining a segmentation prediction result of the target image;

[0195] Using a classification subnetwork in a preset instance segmentation network to classify the second global feature map, to obtain a classification prediction result of the target image;

[0196] The preset instance segmentation network is trained according to the segmentation prediction result and the classification prediction result to obtain a trained instance segmentation network.

[0197] In some embodiments of the present application, the target segmentation subnetwork includes a feature extraction module and a dynamic convolution module, and the training unit is further used to:

[0198] Using the feature extraction module, performing instance feature extraction processing on the second global feature map to obtain a second instance feature map of the target image;

[0199] Using the dynamic convolution module, performing convolution feature prediction processing on the second global feature map to obtain a second dynamic convolution feature map of the target image;

[0200] The target segmentation subnetwork is used to determine a binary segmentation map of the target image according to the second instance feature map and the second dynamic convolution feature map, so as to obtain a segmentation prediction result of the target image.

[0201] In some embodiments of the present application, the training unit is further used to:

[0202] Determining a classification loss value of the classification subnetwork according to the category feature information and the classification prediction result;

[0203] Determining a segmentation loss value of the target segmentation subnetwork according to the binary segmentation information and the segmentation prediction result;

[0204] The preset instance segmentation network is trained according to the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

[0205] In some embodiments of the present application, the target image is also annotated with target frame information, and the training unit is specifically further used for:

[0206] Using a border regression subnetwork in a preset instance segmentation network, determining a target border of the target image according to the second global feature map, and obtaining a border regression result of the target image;

[0207] Determining a border regression loss value of the border regression subnetwork according to the target border information and the border regression result;

[0208] The preset instance segmentation network is trained according to the border regression loss value, the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

[0209] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0210] Since the instance segmentation device can perform the present application as follows Figures 1 to 6 Corresponding to the steps in the instance segmentation method in any embodiment, therefore, the present application can be implemented as follows Figures 1 to 6 The beneficial effects that can be achieved by the instance segmentation method in any embodiment are detailed in the previous description and will not be repeated here.

[0211] In addition, in order to better implement the instance segmentation method in the embodiment of the present application, on the basis of the instance segmentation method, the embodiment of the present application also provides an electronic device, referring to Figure 8 , Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present application is shown. Specifically, the electronic device provided by the embodiment of the present application includes a processor 801, and the processor 801 is used to execute a computer program stored in a memory 802 to implement the following Figures 1 to 6Corresponding to each step of the instance segmentation method in any embodiment; or, the processor 801 is used to execute the computer program stored in the memory 802 to implement the following Figure 7 The functions of each unit in the corresponding embodiment.

[0212] Exemplarily, the computer program may be divided into one or more modules / units, one or more modules / units are stored in the memory 802, and executed by the processor 801 to complete the embodiment of the present application. One or more modules / units may be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device.

[0213] The electronic device may include, but is not limited to, a processor 801 and a memory 802. Those skilled in the art will appreciate that the illustration is merely an example of an electronic device and does not constitute a limitation on the electronic device, and may include more or fewer components than those shown in the illustration, or may combine certain components, or different components, for example, the electronic device may also include input and output devices, network access devices, buses, etc., and the processor 801, memory 802, input and output devices, and network access devices, etc., are connected via a bus.

[0214] The processor 801 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0215] The memory 802 can be used to store computer programs and / or modules. The processor 801 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 802 and calling the data stored in the memory 802. The memory 802 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the electronic device (such as audio data, video data, etc.), etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0216] Those skilled in the art can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described example segmentation device, electronic device and its corresponding units can refer to the following. Figures 1 to 6 The description of the instance segmentation method in any embodiment will not be repeated here in detail.

[0217] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0218] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the present application as follows: Figures 1 to 6 For the steps in the instance segmentation method in any embodiment, the specific operations can be referred to as follows: Figures 1 to 6 The description of the instance segmentation method in any embodiment will not be repeated here.

[0219] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0220] Due to the instructions stored in the computer-readable storage medium, the present application can be executed. Figures 1 to 6 Corresponding to the steps in the instance segmentation method in any embodiment, therefore, the present application can be implemented as follows Figures 1 to 6The beneficial effects that can be achieved by the instance segmentation method in any embodiment are detailed in the previous description and will not be repeated here.

[0221] The above is a detailed introduction to an instance segmentation method, device, electronic device and computer-readable storage medium provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An instance segmentation method, characterized in that: The method comprises: Performing feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented; Performing feature prediction according to the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented; Acquire a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map; Obtaining an instance segmentation result of the image to be segmented according to the binary segmentation map and the first classification feature map; The step of obtaining a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map includes: According to the first dynamic convolution feature map, obtaining a convolution kernel size of the first instance feature map; Performing convolution calculation on the first instance feature map according to the convolution kernel size to obtain a pixel feature map of the image to be segmented; According to a preset pixel threshold, the pixel feature map is binarized to obtain a binary segmentation map of the image to be segmented.

2. The instance segmentation method according to claim 1, characterized in that The step of obtaining a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map includes: A target segmentation subnetwork in the trained instance segmentation network is used to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map.

3. The instance segmentation method according to claim 2, characterized in that: The method further comprises: Acquire a target image, wherein the target image has been annotated with category feature information and binary segmentation information; Using a preset instance segmentation network, performing feature extraction processing on the target image to obtain a second global feature map of the target image; Using a target segmentation subnetwork in a preset instance segmentation network, determining a binary segmentation map of the target image according to the second global feature map, and obtaining a segmentation prediction result of the target image; Using a classification subnetwork in a preset instance segmentation network to classify the second global feature map, to obtain a classification prediction result of the target image; The preset instance segmentation network is trained according to the segmentation prediction result and the classification prediction result to obtain a trained instance segmentation network.

4. The instance segmentation method according to claim 3, characterized in that: The target segmentation subnetwork includes a feature extraction module and a dynamic convolution module. The target segmentation subnetwork in the preset instance segmentation network is used to determine the binary segmentation map of the target image according to the second global feature map to obtain the segmentation prediction result of the target image, including: Using the feature extraction module, performing instance feature extraction processing on the second global feature map to obtain a second instance feature map of the target image; Using the dynamic convolution module, performing convolution feature prediction processing on the second global feature map to obtain a second dynamic convolution feature map of the target image; The target segmentation subnetwork is used to determine a binary segmentation map of the target image according to the second instance feature map and the second dynamic convolution feature map, so as to obtain a segmentation prediction result of the target image.

5. The instance segmentation method according to claim 3, characterized in that: The step of training the preset instance segmentation network according to the segmentation prediction result and the classification prediction result to obtain a trained instance segmentation network comprises: Determining a classification loss value of the classification subnetwork according to the category feature information and the classification prediction result; Determining a segmentation loss value of the target segmentation subnetwork according to the binary segmentation information and the segmentation prediction result; The preset instance segmentation network is trained according to the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

6. The instance segmentation method according to claim 5, characterized in that: The target image is also annotated with target frame information, and the method further includes: Using a border regression subnetwork in a preset instance segmentation network, determining a target border of the target image according to the second global feature map, and obtaining a border regression result of the target image; Determining a border regression loss value of the border regression subnetwork according to the target border information and the border regression result; According to the classification loss value and the segmentation loss value, the preset instance segmentation network is trained to obtain a trained instance segmentation network, including: The preset instance segmentation network is trained according to the border regression loss value, the classification loss value and the segmentation loss value to obtain a trained instance segmentation network.

7. An instance segmentation device, characterized in that: The instance segmentation device comprises: A first feature extraction unit is used to perform feature extraction processing on the image to be segmented to obtain a first global feature map of the image to be segmented; A second feature extraction unit is used to perform feature prediction according to the first global feature map to obtain a first instance feature map, a first dynamic convolution feature map, and a first classification feature map of the image to be segmented; A segmentation unit, configured to obtain a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map; A determination unit, configured to obtain an instance segmentation result of the image to be segmented according to the binary segmentation map and the first classification feature map; The step of obtaining a binary segmentation map of the image to be segmented according to the first instance feature map and the first dynamic convolution feature map includes: According to the first dynamic convolution feature map, obtaining a convolution kernel size of the first instance feature map; Performing convolution calculation on the first instance feature map according to the convolution kernel size to obtain a pixel feature map of the image to be segmented; According to a preset pixel threshold, the pixel feature map is binarized to obtain a binary segmentation map of the image to be segmented.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the instance segmentation method according to any one of claims 1 to 6 is executed.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and the computer program is loaded by a processor to execute the steps in the instance segmentation method according to any one of claims 1 to 6.

Images