Image segmentation method, electronic device, and storage medium

Abstract

The application discloses an image segmentation method, an electronic device and a storage medium. The image segmentation method comprises the following steps: extracting a bottom feature from an image feature, and extracting an interest region from the bottom feature; obtaining an occlusion attention coefficient in a prediction frame through a cascaded graph layer network; fusing the occlusion attention coefficient and the interest region to obtain occlusion boundary data; obtaining an occluded object attention coefficient in the prediction frame through the cascaded graph layer network based on the occlusion boundary data and the occlusion attention coefficient; and fusing the occluded object attention coefficient and the interest region to obtain occluded object boundary data. According to the scheme, two sets of attention coefficients, i.e., the occlusion attention coefficient and the occluded object attention coefficient, are predicted, the occlusion and the occluded object are separated into two layers, and the interest region is extracted from the bottom feature, so that the prediction of the attention coefficients is not affected, the identification of the occluded object is facilitated, and the image segmentation effect is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image segmentation method, an electronic device and a storage medium. BACKGROUND

[0002] When processing image data, there is often a picture with two mutually occluded layers. For the separation of the occluder and the occluded object in the two layers, some existing instance segmentation algorithms are generally difficult to deal with such occlusion, the quality of detection and segmentation of the occluded object will be reduced, and the segmentation of the occluded object may also be affected, and the overall time consumption of the algorithm increases linearly with the increase of the target, which is not conducive to processing scenes with dense targets. SUMMARY

[0003] The present application at least provides an image segmentation method, an electronic device and a storage medium to solve the above problems.

[0004] The first aspect of the present application provides an image segmentation method, the method comprising: acquiring an image to be segmented, and extracting image features of the image to be segmented; performing a detection operation on the image features to predict a prediction box of an instance, and extracting bottom layer features corresponding to instance segmentation from the image features; extracting a region of interest from the bottom layer features according to the prediction box; obtaining an occluder attention coefficient in the prediction box through a cascaded layer network; fusing the occluder attention coefficient and the region of interest to obtain occluder boundary data; obtaining an occluded object attention coefficient in the prediction box through a cascaded layer network based on the occluder boundary data and the occluder attention coefficient; and fusing the occluded object attention coefficient and the region of interest to obtain occluded object boundary data, thereby realizing image segmentation.

[0005] In some embodiments, the detection operation includes regression branch learning to obtain corresponding regression features; the cascaded layer network includes a cascaded first layer network and a second layer network; the obtaining of the occluder attention coefficient in the prediction box through the cascaded layer network includes inputting the regression features into the first layer network to obtain the occluder attention coefficient in the prediction box; and the obtaining of the occluded object attention coefficient in the prediction box through the cascaded layer network based on the occluder boundary data and the occluder attention coefficient includes inputting the occluder boundary data into the second layer network, and multiplying the occluder attention coefficient and the regression features point by point to input the second layer network to obtain the occluded object attention coefficient in the prediction box.

[0006] In some embodiments, the prediction box is obtained by performing convolution operation on the regression features.

[0007] In some embodiments, the extracting the region of interest from the bottom-layer feature according to the prediction box comprises: extracting the region of interest from the bottom-layer feature by a ROI Align network according to the prediction box.

[0008] In some embodiments, the fusing the occluder attention coefficient with the region of interest comprises: multiplying the occluder attention coefficient and the region of interest point by point to obtain occluder boundary data; and the fusing the occluded object attention coefficient with the region of interest comprises: multiplying the occluded object attention coefficient and the region of interest point by point to obtain occluded object boundary data.

[0009] In some embodiments, the occluder boundary data comprises a boundary and / or a mask of the occluder; and the occluded object boundary data comprises a boundary and / or a mask of the occluded object.

[0010] In some embodiments, each of the cascaded graph networks comprises two convolutional networks.

[0011] In some embodiments, the extracting the image feature of the image to be segmented comprises: inputting the image to be segmented into a feature pyramid and a backbone network to obtain the image feature of the image to be segmented.

[0012] The second aspect of the present application provides an electronic device, comprising a memory and a processor coupled with each other, the processor being configured to execute program instructions stored in the memory to implement the image segmentation method in the first aspect.

[0013] The third aspect of the present application provides a non-volatile computer readable storage medium, the computer readable storage medium being configured to store program instructions, the program instructions being configured to implement the image segmentation method in the first aspect when executed by a processor.

[0014] The scheme is characterized in that: the image to be segmented is acquired, image features of the image to be segmented are extracted, a detection operation is performed on the image features to predict a prediction box of an instance, and bottom features corresponding to instance segmentation are extracted from the image features; the prediction box is used to extract a region of interest from the bottom features; an occluder attention coefficient in the prediction box is obtained through a cascaded graph layer network, the occluder attention coefficient is fused with the region of interest for processing to obtain occluder boundary data; the occluder boundary data and the occluder attention coefficient are used to obtain a hidden occluder attention coefficient in the prediction box through the cascaded graph layer network, the hidden occluder attention coefficient is fused with the region of interest for processing to obtain hidden occluder boundary data, so that image segmentation is realized. The scheme in the application separates the occluder and the hidden occluder into two layers by predicting two sets of attention coefficients, i.e., the occluder attention coefficient and the hidden occluder attention coefficient, and the region of interest is extracted from the bottom features, which does not affect the prediction of the attention coefficient, is conducive to the identification of the hidden occluder, and effectively improves the image segmentation effect.

[0015] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, rather than limiting the application. BRIEF DESCRIPTION OF DRAWINGS

[0016] The drawings incorporated into the specification and forming part of the specification, show embodiments consistent with the application, and together with the specification, serve to explain the technical solutions of the application.

[0017] Figure 1 is a flowchart of an image segmentation method in an embodiment of the application;

[0018] Figure 2 is a structural schematic diagram of an electronic device in an embodiment of the application;

[0019] Figure 3 is a structural schematic diagram of a non-volatile computer readable storage medium in an embodiment of the application. DETAILED DESCRIPTION

[0020] The application will be further described in detail below with reference to the drawings and embodiments. It should be particularly pointed out that the following embodiments are only used to illustrate the application, but do not limit the scope of the application. Similarly, the following embodiments are only part of the embodiments of the application, not all embodiments of the application, and all other embodiments obtained by a person of ordinary skill in the art without making creative efforts fall within the scope of protection of the application.

[0021] Reference to an "embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is expressly understood that that the embodiments described herein are merely examples from a multitude of possible embodiments which can be claimed.

[0022] The term "and / or", merely describes an associated relationship between associated objects, and means that there can be three relationships, for example, A and / or B, which can mean that A exists alone, A and B exist together, and B exists alone. In addition, the character " / " in this application generally means that the associated objects before and after are in an "or" relationship. In addition, "multiple" in this application means two or more than two. In addition, the term "at least one" in this application means any one of the multiple or any combination of at least two of the multiple, for example, including at least one of A, B and C, which means including any one or more elements selected from the set consisting of A, B and C. In addition, the terms "first", "second", "third" in this application are only for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features.

[0023] As described above, for the separation of the occluder and the occluded object in the two layers, some existing instance segmentation algorithms are generally difficult to deal with such occlusion, and the quality of detection and segmentation of the occluded object will be reduced, and the segmentation of the occluded object may also be affected, and the overall time consumption of the algorithm increases linearly with the increase of the target, which is not conducive to processing scenes with dense targets.

[0024] Therefore, the application provides an image segmentation method, an electronic device and a storage medium to solve the above problems.

[0025] Please refer to Figure 1 , Figure 1 is a flowchart of the image segmentation method in the embodiments of the application. It should be noted that the method of the application is not limited to the order of the flowchart shown in Figure 1 . The method can be applied to an electronic device with computing functions, and the electronic device can execute the method by receiving information collected by a sensor device, and the sensor device can be a camera. As shown in Figure 1 , the image segmentation method comprises the following steps:

[0026] S1, acquiring an image to be segmented, and extracting image features of the image to be segmented.

[0027] The image to be segmented, i.e., the image in which the objects exist and are mutually occluded, needs to separate the occluded objects and the occluded objects in the image. The image to be segmented can be obtained by using an image sensor to collect image data, for example, the image sensor is installed on a movable device. The movable device can be an automatic moving device, such as a robot, a model training vehicle, etc.

[0028] In some embodiments, the image sensor can be a camera that collects image data at a preset time interval to obtain multiple frames of image data, or a video recorder that continuously records video for a preset time period to collect continuous multiple frames of image data, thereby obtaining the image to be segmented. In an embodiment, the image sensor can be any image device that meets the perception accuracy requirement and can provide image data.

[0029] After obtaining the image to be segmented, the image features of the image to be segmented are extracted by a feature extraction module. The image features can include color features, property features, texture features, and spatial relationship features, or other extractable features, which are not limited.

[0030] S2, performing a detection operation on the image features to predict a prediction box of an instance, and extracting a bottom feature corresponding to instance segmentation from the image features.

[0031] The extracted image features are detected by the detection module to predict the prediction box of the instance on the image features. The instance includes the occluded object and the occluded object. The prediction box can be the circumscribed rectangular box of the instance, i.e., the circumscribed rectangular box of the occluded object and the occluded object. At the same time, the bottom feature module extracts the bottom feature related to the instance segmentation from the image features, i.e., the bottom feature is extracted from the image features.

[0032] For example, the FCOS network can be used for detection operation to obtain the prediction box; the basis module network can be used to extract the corresponding bottom feature from the image features, or other realizable networks can be used, which are not limited.

[0033] S3, extracting a region of interest from the bottom feature according to the prediction box.

[0034] The region of interest (ROI) is extracted from the bottom feature according to the coordinates of the prediction box of the instance.

[0035] S4, obtaining an occluded object attention coefficient in the prediction box by a cascaded layer network.

[0036] The occlusion attention coefficient in the prediction box is obtained through the cascaded layer network, and the occlusion attention coefficient can enable the model to focus attention on the occlusion in the prediction box.

[0037] S5, the occlusion attention coefficient is fused with the interest region to obtain occlusion boundary data.

[0038] The obtained occlusion attention coefficient is fused with the interest region extracted from the bottom layer feature to obtain the occlusion boundary data.

[0039] S6, based on the occlusion boundary data and the occlusion attention coefficient, the occlusion attention coefficient in the prediction box is obtained through the cascaded layer network.

[0040] After obtaining the occlusion boundary data, based on the occlusion boundary data and the occlusion attention coefficient, the occlusion attention coefficient in the prediction box is obtained through the cascaded layer network, and the occlusion attention coefficient can enable the model to focus attention on the occlusion in the prediction box.

[0041] S7, the occlusion attention coefficient is fused with the interest region to obtain the occlusion boundary data, so as to realize image segmentation.

[0042] The obtained occlusion attention coefficient is fused with the interest region extracted from the bottom layer feature to obtain the occlusion boundary data, so as to realize image segmentation.

[0043] The above scheme obtains a to-be-segmented image, extracts image features of the to-be-segmented image, performs a detection operation on the image features to predict a prediction box of an instance, and extracts a bottom layer feature corresponding to instance segmentation from the image features; the interest region is extracted from the bottom layer feature according to the prediction box; the occlusion attention coefficient in the prediction box is obtained through the cascaded layer network, and the occlusion attention coefficient is fused with the interest region to obtain the occlusion boundary data; based on the occlusion boundary data and the occlusion attention coefficient, the occlusion attention coefficient in the prediction box is obtained through the cascaded layer network, and the occlusion attention coefficient is fused with the interest region to obtain the occlusion boundary data, so as to realize image segmentation; the scheme in the present application separates the occlusion and the occluded object into two layers by predicting two sets of attention coefficients, i.e., the occlusion attention coefficient and the occluded object attention coefficient, and the interest region is extracted from the bottom layer feature, which does not affect the prediction of the attention coefficient, is conducive to the identification of the occluded object, and effectively improves the image segmentation effect.

[0044] As described above, the occluder attention coefficient in the prediction box is obtained through the cascaded graph layer network; and the occluded object attention coefficient in the prediction box is obtained through the cascaded graph layer network based on the occluder boundary data and the occluder attention coefficient. In an embodiment of the present application, the detection operation includes regression branch learning to obtain corresponding regression features; the cascaded graph layer network includes a first graph layer network and a second graph layer network; the occluder attention coefficient in the prediction box is obtained through the cascaded graph layer network, including: inputting the regression features into the first graph layer network to obtain the occluder attention coefficient in the prediction box; and the occluded object attention coefficient in the prediction box is obtained through the cascaded graph layer network based on the occluder boundary data and the occluder attention coefficient, including: inputting the occluder boundary data into the second graph layer network, and multiplying the occluder attention coefficient point by point with the regression features to input the second graph layer network to obtain the occluded object attention coefficient in the prediction box.

[0045] The detection operation includes regression branch learning, which is used to realize target positioning, i.e., to locate instances, and to detect image features to obtain corresponding regression features.

[0046] The cascaded graph layer network includes a first graph layer network and a second graph layer network, wherein the first graph layer network is used to predict the occluder attention coefficient, and the second graph layer network is used to predict the occluded object attention coefficient, so that the cascaded graph layer network predicts two sets of attention coefficients for the prediction box.

[0047] The regression features obtained by the detection operation are input into the first graph layer network to predict the occluder attention coefficient of the occluder in the prediction box through the first graph layer network. After obtaining the occluder attention coefficient, the occluder attention coefficient is fused with the region of interest to obtain the occluder boundary data. The obtained occluder boundary data is input into the second graph layer network, so that the second graph layer network can better crack the relevant information of the occluder, and thus better identify the occluder. The occluder attention coefficient is multiplied point by point with the regression features obtained by the detection operation and input into the second graph layer network, so that the second graph layer network predicts the occluded object attention coefficient of the occluded object in the prediction box based on the occluder boundary data and the occluder attention coefficient.

[0048] As described above, the detection operation includes regression branch learning to obtain corresponding regression features. In an embodiment of the present application, the prediction box is obtained by performing convolution operation processing on the regression features.

[0049] The regression features obtained by the detection operation are subjected to convolution operation processing to obtain the prediction box. It can be understood that the regression branch learning is used to realize target positioning, i.e., to locate instances, to obtain corresponding regression features, and to perform convolution operation processing on the obtained regression features to obtain the prediction box of the instance.

[0050] As described above, the region of interest is extracted from the low-level feature according to the prediction box. In an embodiment of the present application, the region of interest is extracted from the low-level feature according to the prediction box, comprising: extracting the region of interest from the low-level feature by the ROI Align network according to the prediction box.

[0051] The regression feature obtained by the detection operation is processed by convolution operation to obtain the prediction box of the instance, and the region of interest is extracted from the low-level feature by the ROI Align network according to the prediction box.

[0052] As described above, the occluder attention coefficient and the region of interest are fused to obtain the occluder boundary data, and the occluded object attention coefficient and the region of interest are fused to obtain the occluded object boundary data. In an embodiment of the present application, the occluder attention coefficient and the region of interest are fused, comprising: multiplying the occluder attention coefficient and the region of interest point by point to obtain the occluder boundary data; and the occluded object attention coefficient and the region of interest are fused, comprising: multiplying the occluded object attention coefficient and the region of interest point by point to obtain the occluded object boundary data.

[0053] The regression feature obtained by the detection operation is input into the first graph layer network, and the occluder attention coefficient of the occluder in the prediction box is predicted by the first graph layer network. After obtaining the occluder attention coefficient, the occluder attention coefficient and the region of interest are multiplied point by point to obtain the occluder boundary data. The obtained occluder boundary data is input into the second graph layer network, and the occluded object attention coefficient of the occluded object in the prediction box is predicted by the second graph layer network by multiplying the occluder attention coefficient and the regression feature obtained by the detection operation point by point. After obtaining the occluded object attention coefficient, the occluded object attention coefficient and the region of interest are multiplied point by point to obtain the occluded object boundary data.

[0054] As described above, the occluder attention coefficient and the region of interest are multiplied point by point to obtain the occluder boundary data, and the occluded object attention coefficient and the region of interest are multiplied point by point to obtain the occluded object boundary data. In an embodiment of the present application, the occluder boundary data comprises the boundary and / or mask of the occluder; and the occluded object boundary data comprises the boundary and / or mask of the occluded object.

[0055] The occluder boundary data comprises the boundary and / or mask of the occluder, and the boundary of the occluder is the contour of the occluder, and the mask of the occluder is an image that can be used to cover the occluder. The occluded object boundary data comprises the boundary and / or mask of the occluded object, and the boundary of the occluded object is the contour of the occluded object, and the mask of the occluded object is an image that can be used to cover the occluded object.

[0056] As described above, the occluder attention coefficient and the occluded attention coefficient in the prediction box can be obtained through the cascaded layer network. In an embodiment of the present application, each layer network in the cascaded layer network includes two convolution networks.

[0057] Each layer network in the cascaded layer network includes two convolution networks. It can be understood that the cascaded layer network includes a first layer network and a second layer network, wherein the first layer network includes two convolution networks, that is, the first layer network is stacked by two convolution networks, and the occluder attention coefficient is predicted through the two convolution networks; the second layer network includes two convolution networks, that is, the second layer network is stacked by two convolution networks, and the occluded attention coefficient is predicted through the two convolution networks.

[0058] As described above, the image feature of the image to be segmented is extracted. In an embodiment of the present application, the image feature of the image to be segmented is extracted by inputting the image to be segmented into a feature pyramid and a backbone network to obtain the image feature of the image to be segmented.

[0059] After obtaining the image to be segmented, the image to be segmented is input into a feature pyramid (Feature Pyramid Networks, FPN) and a backbone network to extract the image feature of the image to be segmented through the feature pyramid and the backbone network.

[0060] Those skilled in the art can understand that in the above method of the specific embodiment, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0061] Please refer to Figure 2 , Figure 2 is a structural schematic diagram of an electronic device in an embodiment of the present application. The electronic device 200 includes a memory 201 and a processor 202 coupled to each other. The processor 202 is used to execute the program instructions stored in the memory 201 to realize the steps in the image segmentation method embodiment described above. In a specific implementation scenario, the electronic device 200 can include but is not limited to a microcomputer, a server, without limitation here.

[0062] Specifically, the processor 202 is configured to control itself and the memory 201 to implement the steps in the above-described image segmentation method embodiments. The processor 202 can also be referred to as a CPU (Central Processing Unit). The processor 202 can be an integrated circuit chip having a processing capability. The processor 202 can also be a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. In addition, the processor 202 can be implemented by an integrated circuit chip together.

[0063] Please refer to Figure 3 , Figure 3 is a structural diagram of a non-volatile computer readable storage medium in the embodiments of the present application. The computer readable storage medium 300 is configured to store program instructions 301, which, when executed by the processor 202, are configured to implement the steps in the above-described image segmentation method embodiments.

[0064] The above description of various embodiments tends to emphasize the differences between various embodiments, and the same or similar parts can be referred to each other, and for brevity, will not be repeated here.

[0065] In several embodiments provided in the present application, it should be understood that the disclosed method and related equipment can be implemented by other ways. For example, the above-described related equipment embodiments are only schematic, for example, the division of modules or units is only a logical function division, and actual implementation can have another division manner, for example, a unit or component can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed coupling or direct coupling or communication disconnecting between each other can be indirect coupling or communication disconnecting through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0066] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0067] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for making a computer device, including a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium that can store program codes.

Claims

1. An image segmentation method characterized by, The method comprises: obtaining an image to be segmented, and extracting image features of the image to be segmented; performing a detection operation on the image features to predict a prediction box of an instance, and extracting bottom features corresponding to instance segmentation from the image features; wherein the detection operation comprises regression branch learning to obtain regression features for positioning the instance; extracting a region of interest from the bottom features according to the prediction box; obtaining an occluder attention coefficient in the prediction box through a cascaded graph layer network; wherein the cascaded graph layer network comprises a cascaded first graph layer network and a second graph layer network; the occluder attention coefficient is obtained by inputting the regression features into the first graph layer network; fusing the occluder attention coefficient and the region of interest to obtain occluder boundary data; inputting the occluder boundary data into the second graph layer network, and multiplying the occluder attention coefficient and the regression features point by point to input the second graph layer network to obtain a concealed object attention coefficient in the prediction box; fusing the concealed object attention coefficient and the region of interest to obtain concealed object boundary data, thereby realizing image segmentation.

2. The method of claim 1, wherein, The prediction box is obtained by performing convolution operation on the regression features.

3. The method of claim 1, wherein, The extracting a region of interest from the bottom features according to the prediction box comprises: extracting a region of interest from the bottom features through a ROI Align network according to the prediction box.

4. The method of claim 1, wherein, The fusing the occluder attention coefficient and the region of interest comprises: multiplying the occluder attention coefficient and the region of interest point by point to obtain the occluder boundary data. The fusing the concealed object attention coefficient and the region of interest comprises: multiplying the concealed object attention coefficient and the region of interest point by point to obtain the concealed object boundary data.

5. The method of claim 4, wherein, The occluder boundary data comprises a boundary and / or a mask of an occluder. The concealed object boundary data comprises a boundary and / or a mask of a concealed object.

6. The method of claim 1, wherein, Each graph layer network in the cascaded graph layer network comprises two convolution networks.

7. The method of claim 1, wherein, The extracting image features of the image to be segmented comprises: inputting the image to be segmented into a feature pyramid and a backbone network to obtain the image features of the image to be segmented.

8. An electronic device, comprising: The computer readable storage medium is used to store program instructions, which, when executed by a processor, are used to implement the image segmentation method according to any one of claims 1-7.

9. A non-transitory computer readable storage medium, comprising: The computer readable storage medium is used to store program instructions, which, when executed by a processor, are used to implement the image segmentation method according to any one of claims 1-7.

Images