Image semantic segmentation method, device, equipment and storage medium

Abstract

An embodiment of the present invention discloses an image semantic segmentation method, apparatus, device and storage medium, the method comprising: acquiring a target depth image and a target color image corresponding to the target depth image; performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, the three-dimensional point cloud data comprising: each visible point generated by the target depth image and each occluded point occluded by the visible point; projecting the target color image into a three-dimensional point cloud space, and determining semantic feature information corresponding to each visible point; inputting the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model for scene completion and semantic segmentation, and determining a semantic label corresponding to each visible point and a semantic label corresponding to each occluded point; and obtaining a semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model, thereby improving the accuracy and efficiency of semantic segmentation.

Description

Technical Field

[0001] The embodiments of the present invention relate to computer processing technology, and in particular to an image semantic segmentation method, apparatus, device and storage medium. Background Art

[0002] With the rapid development of computer technology, we can further develop from image-level classification to pixel-level classification. Image semantic segmentation is to determine the category to which each pixel in the image belongs, thereby segmenting different objects in the image at the pixel level and improving the accuracy of segmentation.

[0003] Currently, fully convolutional network models can be used for image semantic segmentation. However, in the process of implementing the present invention, the inventors found that the existing technology has at least the following problems:

[0004] When performing semantic segmentation on images collected from a single perspective, the segmentation effect is often affected to a certain extent due to the lack of some geometric structures, and the accuracy of semantic segmentation cannot be guaranteed. Summary of the Invention

[0005] Embodiments of the present invention provide an image semantic segmentation method, apparatus, device, and storage medium to improve semantic segmentation accuracy and segmentation efficiency.

[0006] In a first aspect, an embodiment of the present invention provides an image semantic segmentation method, comprising:

[0007] Acquire a target depth image and a target color image corresponding to the target depth image;

[0008] Performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, wherein the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point;

[0009] Projecting the target color image into a three-dimensional point cloud space, and determining semantic feature information corresponding to each visible point;

[0010] Inputting the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model to perform scene completion and semantic segmentation, and determining the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point;

[0011] According to the output of the preset semantic segmentation network model, a semantic segmentation result corresponding to the target depth image is obtained.

[0012] In a second aspect, an embodiment of the present invention further provides an image semantic segmentation apparatus, comprising:

[0013] An image acquisition module, configured to acquire a target depth image and a target color image corresponding to the target depth image;

[0014] a point cloud conversion module, configured to perform point cloud conversion on the target depth image to obtain three-dimensional point cloud data, wherein the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point;

[0015] An image projection module, configured to project the target color image into a three-dimensional point cloud space and determine semantic feature information corresponding to each visible point;

[0016] A data input module is used to input the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model to perform scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point;

[0017] The semantic segmentation result acquisition module is used to obtain the semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model.

[0018] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising:

[0019] one or more processors;

[0020] a memory for storing one or more programs;

[0021] When the one or more programs are executed by the one or more processors, the one or more processors implement the image semantic segmentation method provided by any embodiment of the present invention.

[0022] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image semantic segmentation method provided by any embodiment of the present invention.

[0023] The embodiments of the above invention have the following advantages or beneficial effects:

[0024] By performing point cloud conversion on the target depth image, each visible point generated by the target depth image and each occluded point occluded by the visible points are obtained. The target color image corresponding to the target depth image is then projected into a three-dimensional point cloud space, and semantic feature information corresponding to each visible point is determined. The three-dimensional point cloud data and the semantic feature information corresponding to each visible point are input into a preset semantic segmentation network model for scene completion and semantic segmentation. The semantic label corresponding to each visible point and the semantic label corresponding to each occluded point are determined. Based on the output of the preset semantic segmentation network model, a semantic segmentation result corresponding to the target depth image is obtained. Scene completion can provide more geometric structure information for semantic segmentation, and semantic segmentation is performed by combining the texture information carried by the target color image and the geometric information in the depth image, thereby improving semantic segmentation accuracy. Furthermore, by converting the target depth image into unordered and sparse three-dimensional point cloud data, only the three-dimensional point cloud data consisting of visible points and occluded points occluded by the visible points can be input, eliminating the need to input meaningless points such as visible empty points. This avoids unnecessary calculations for meaningless and negligible points, thereby improving segmentation efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a flowchart of an image semantic segmentation method provided by Example 1 of the present invention;

[0026] Figure 2 This is an example of an image semantic segmentation process involved in the first embodiment of the present invention;

[0027] Figure 3 This is an example of an input point cloud and an output point cloud of a preset semantic segmentation network model involved in the first embodiment of the present invention;

[0028] Figure 4 This is a flowchart of an image semantic segmentation method provided by the second embodiment of the present invention;

[0029] Figure 5 This is an example of a network structure of a preset semantic segmentation network model involved in the second embodiment of the present invention;

[0030] Figure 6 This is a flowchart of an image semantic segmentation method provided by Embodiment 3 of the present invention;

[0031] Figure 7 This is an example of an image semantic segmentation process involved in the third embodiment of the present invention;

[0032] Figure 8 Schematic diagram of the structure of an image semantic segmentation device provided by the fourth embodiment of the present invention;

[0033] Figure 9This is a structural diagram of an electronic device provided in Example 5 of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0035] Example 1

[0036] Figure 1 This is a flowchart of an image semantic segmentation method provided in the first embodiment of the present invention. This embodiment is applicable to the case where semantic segmentation is performed based on depth images and color images collected in the application scenario. The method can be executed by an image semantic segmentation device, which can be implemented by software and / or hardware and integrated into an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0037] S110 : Acquire a target depth image and a target color image corresponding to the target depth image.

[0038] Among them, the target depth image may refer to a depth image from a single perspective captured by a depth camera or other acquisition device placed in the application scene. The application scenario may refer to any business scenario that requires object recognition, such as a robot scene, an unmanned driving scene, etc. The target color image may refer to an image in which each pixel is composed of R (Red), G (Green), and B (Blue) components. The target color image may refer to a color image from a single perspective captured by an RGB camera placed in the application scene.

[0039] Specifically, this embodiment can obtain a target depth image collected in real time for the application scene and a target color image within the same detection area in the application scene, so as to use the texture information carried by the target color image as auxiliary information to distinguish geometrically similar objects and improve the segmentation effect.

[0040] S120 , performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, where the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point.

[0041] Among them, visible points can refer to non-empty points within the preset detection range generated by the target depth image. Occluded points can refer to occluded points that are blocked by visible points in the image viewing angle. Occluded points can include empty points and non-empty points.

[0042] Specifically, Figure 2An example of the image semantic segmentation process is given. Figure 2 As shown, the target depth image can be transformed into coordinates based on the camera's internal calibration parameters to obtain disordered and sparse three-dimensional point cloud data, that is, the three-dimensional coordinate information corresponding to each visible point and the three-dimensional coordinate information corresponding to each occluded point can be obtained. Figure 3 An example of input point cloud and output point cloud of a preset semantic segmentation network model is given. Figure 3 As shown in the input point cloud in , there are three types of points in the three-dimensional grid space: the first type is visible points generated by the target depth image; the second type is occluded points that are occluded by visible points from the image perspective, that is, occluded points behind visible points; and the third type is ignored points, which include visible empty points and points outside the detection range. This embodiment converts the target depth image into three-dimensional point cloud data, focusing only on visible points and occluded points used to determine the geometric structure, without paying attention to meaningless ignored points. Therefore, point cloud coordinates are used instead of the three-dimensional grid, avoiding unnecessary calculations for empty voxels and improving segmentation efficiency.

[0043] S130 , projecting the target color image into a three-dimensional point cloud space, and determining semantic feature information corresponding to each visible point.

[0044] Among them, semantic feature information can be used to represent the texture information of each visible point.

[0045] Specifically, if Figure 2 As shown, based on the camera's intrinsic parameter matrix information and the target depth image, the two-dimensional target color image can be projected into the three-dimensional point cloud space to obtain the semantic feature information corresponding to each visible point.

[0046] S140: Input the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model to perform scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point.

[0047] Among them, the preset semantic segmentation network model can be a preset point cloud segmentation network model with heterogeneous input and output. The preset semantic segmentation network model in this embodiment can be obtained by pre-training based on sample data. The preset semantic segmentation network model can implicitly combine semantic and geometric information by jointly learning scene completion and semantic annotation, thereby obtaining a preset semantic segmentation network model with better segmentation effect. It should be noted that in the process of training the preset semantic segmentation network model, the number of visible points and occluded points in each three-dimensional grid sample is uneven, so that the point cloud can be grouped, each group contains the same number of visible points and occluded points, and the final prediction result is obtained based on the results of all groups, thereby ensuring the training effect of the preset semantic segmentation network model.

[0048] Specifically, if Figure 2 and Figure 3 As shown, by inputting the three-dimensional point cloud data composed of each visible point and each occluded point and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model, for example, the RGB vector and the corresponding point coordinate vector can be spliced and input into the preset semantic segmentation network model, so that the preset semantic segmentation network model can perform input and output heterogeneous semantic scene completion based on the three-dimensional coordinate information and semantic feature information corresponding to each visible point, determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point, thereby effectively fusing the appearance information and geometric information, and based on the visible points, new semantic points can be generated in the occluded area, that is, more geometric information is provided for semantic segmentation through scene completion, so that the preset semantic segmentation network model can accurately identify objects with similar geometric structures, thereby improving the accuracy of semantic segmentation.

[0049] S150. Obtain a semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model.

[0050] Specifically, the preset semantic segmentation network model can display the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point based on the preset display mode. For example, different semantic labels can be displayed in different colors or styles to facilitate user identification and distinction, so that the preset semantic segmentation network model can output the segmentation map after semantic segmentation, that is, the semantic segmentation result corresponding to the target depth image. Figure 3 As shown in the output point cloud in , the preset semantic segmentation network model can infer a semantic label for each visible point and each occluded point. For example, the semantic label of some visible points is table, and the semantic label of some visible points is chair. In addition, the semantic label of some occluded points is table, the semantic label of some occluded points is chair, and the semantic label of some occluded points is empty point, which indicates that there is no object here.

[0051] The technical solution of this embodiment converts the target depth image into a point cloud to obtain the visible points and occluded points generated by the target depth image. The target color image corresponding to the target depth image is then projected into a three-dimensional point cloud space to determine the semantic feature information corresponding to each visible point. The three-dimensional point cloud data and the semantic feature information corresponding to each visible point are input into a preset semantic segmentation network model for scene completion and semantic segmentation. The semantic label corresponding to each visible point and the semantic label corresponding to each occluded point are determined. Based on the output of the preset semantic segmentation network model, the semantic segmentation result corresponding to the target depth image is obtained. Thus, scene completion can provide more geometric structure information for semantic segmentation, and semantic segmentation is performed by combining the texture information carried by the target color image and the geometric information in the depth image, thereby improving the accuracy of semantic segmentation. Furthermore, by converting the target depth image into unordered and sparse three-dimensional point cloud data, only the three-dimensional point cloud data consisting of visible points and occluded points occluded by visible points can be input, eliminating the need to input meaningless points such as empty points. This avoids unnecessary calculations for meaningless and negligible points, thereby improving segmentation efficiency.

[0052] Example 2

[0053] Figure 4 This is a flowchart of a method for image semantic segmentation provided in Example 2 of the present invention. Based on the above examples, this example provides a preset semantic segmentation network model that can include a visible point encoding submodel and an occluded point decoding submodel. The semantic segmentation process of this preset semantic segmentation network model is also described in detail. Explanations of terms that are identical or corresponding to those in the above examples are omitted here.

[0054] Both the visible point encoding sub-model and the occluded point decoding sub-model can utilize the matrix transformation convolution operation XConv for network model design. For example, the visible point encoding sub-model can use the matrix transformation convolution operation XConv to extract features from visible points. The occluded point decoding sub-model can also use the matrix transformation convolution operation XConv to map the extracted visible point features onto the occluded points and perform feature fusion, thereby accurately determining the semantic labels corresponding to each visible point and each occluded point.

[0055] Before introducing the specific structures of the visible point encoding sub-model and the occluded point decoding sub-model, this embodiment first describes the matrix transformation convolution operation XConv. XConv refers to the convolution operation on the point cloud after the X matrix transformation. The input of XConv is a set of points P, each of which is associated with a feature F. The input set of points P can be sampled to select a set of representative points P' to carry the output features. By applying XConv to [PF], a higher-level feature F' associated with the representative point P' can be obtained, that is, the operation process of XConv can be simply summarized as: F' = XConv(P, F, P').

[0056] See also Figure 4 The image semantic segmentation method provided in this embodiment specifically includes the following steps:

[0057] S410: Acquire a target depth image and a target color image corresponding to the target depth image.

[0058] S420 , performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, where the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point.

[0059] S430: Project the target color image into a three-dimensional point cloud space, and determine semantic feature information corresponding to each visible point.

[0060] S440: Input the three-dimensional coordinate information and semantic feature information corresponding to each visible point into the visible point coding sub-model, perform feature extraction on each visible point, and determine the visible feature information corresponding to the visible representative point among each visible point.

[0061] The visible point coding sub-model may include: a preset number of coding layers. The preset number L of coding layers may be set based on business requirements and actual scenarios. For example, Figure 5 An example of a network structure of a preset semantic segmentation network model is given. Figure 5 As shown, the visible point coding sub-model includes 2 coding layers.

[0062] Exemplarily, S440 may include: inputting visible feature information corresponding to visible representative points output by the previous coding layer into the current coding layer, performing a matrix transformation convolution operation, and determining visible feature information corresponding to visible representative points in the current coding layer, wherein the number of visible representative points in the current coding layer is less than the number of visible representative points in the previous coding layer.

[0063] Specifically, each coding layer can be used to: perform matrix transformation convolution operation on the visible feature information corresponding to the visible representative points output by the previous coding layer, determine the visible feature information corresponding to the visible representative points in the current coding layer, and input the visible feature information corresponding to the visible representative points in the current coding layer into the next coding layer and the occlusion point decoding sub-model.

[0064] Specifically, if Figure 5 As shown in the figure, the visible point encoding sub-model can project the 3D coordinate information and semantic feature information of all input visible points onto fewer and fewer representative visible points (low resolution), and input the visible feature information corresponding to the visible representative points output by each coding layer into the next coding layer and the occlusion point decoding sub-model. For example, for each coding layer i (i = 1, 2...L), it can be expressed as follows:

[0065] P b [i]=Represent(P b [i-1])

[0066] F b [i] = XConv(P b [i-1],F b [i-1],P b [i])

[0067] The input of coding layer i is the visible representative point P output by the previous coding layer b [i-1] corresponding visible feature information F b [i-1], the output is the visible representative point P in the current coding layer b [i] Corresponding visible feature information F b [i]. Visible representative point P b [i] is the sampling operation Represent from P b [i-1] is the visible point sampled. It should be noted that each visible representative point is assigned a visible point feature information. When i = 1, that is, the first coding layer, P b [0] and F b [0] represents all visible points of the input and the feature information corresponding to the visible points, which is a fusion of three-dimensional coordinate information and semantic feature information.

[0068] S450. Input the visible feature information corresponding to the visible representative point into the occlusion point decoding sub-model, map the visible feature information corresponding to the visible representative point to the representative occlusion point among the occlusion points and perform feature fusion to determine the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point.

[0069] The occlusion point decoding sub-model may include: a preset number of decoding layers and fully connected layers, the decoding layers correspond to the encoding layers one-to-one, and the decoding order of the decoding layers is opposite to the encoding order of the encoding layers.

[0070] Exemplarily, S450 may include: inputting the visible feature information corresponding to the visible representative point in the current coding layer into the corresponding current decoding layer in the occlusion point decoding submodel to perform a matrix transformation inverse convolution operation to determine the first feature information corresponding to the target representative point in the current decoding layer, wherein the target representative point includes the visible representative point input in the current coding layer and the occlusion representative point in the current decoding layer; inputting the target feature information corresponding to the target representative point in the previous decoding layer into the current decoding layer to perform a matrix transformation inverse convolution operation to determine the second feature information corresponding to the target representative point in the current decoding layer; fusing the first feature information and the second feature information to determine the target feature information corresponding to the target representative point in the current decoding layer, and inputting the target feature information corresponding to the target representative point in the current decoding layer to the next decoding layer; inputting the target feature information corresponding to the target representative point in the last decoding layer into the fully connected layer for processing to determine the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point.

[0071] Specifically, the decoding layers in the occlusion point decoding submodel correspond one-to-one to the encoding layers in the visible point encoding submodel, and the decoding order of the decoding layers is opposite to the encoding order of the encoding layers, that is, the first encoding layer corresponds to the last decoding layer, and the last encoding layer corresponds to the first decoding layer. Each encoding layer in the visible point encoding submodel inputs the visible feature information corresponding to the determined visible representative point into the corresponding decoding layer for decoding from the visible point to the occlusion point. The occlusion point decoding submodel can project the features onto the occlusion representative point and finally predict the semantic labels corresponding to each visible point and occlusion point.

[0072] The number of occlusion representative points in the current decoding layer is greater than the number of occlusion representative points in the previous decoding layer.

[0073] Specifically, the occlusion point decoding sub-model propagates low-resolution information to high-resolution prediction. Figure 5 As shown in Figure 1, each decoding layer in the occlusion point decoding sub-model contains two types of representative points: one is the visible representative point input from the corresponding encoding layer, and the other is the occlusion representative point in the current decoding layer. It should be noted that the occlusion representative point in each decoding layer can be obtained by sampling in the current decoding layer or in the corresponding encoding layer, that is, P c [i]=Represent(P c[i-1]), so that visible points and occluded points can be sampled simultaneously in the coding layer, and then the sampled occluded representative points are input to the corresponding decoding layer to ensure sampling accuracy. The occluded representative points sampled in each coding layer do not participate in the feature extraction process of the coding layer. For example, for decoding layer j (j=1,2...L), decoding layer j corresponds to coding layer i=L-j+1, and the decoding operation of decoding layer j can be expressed as follows:

[0074] P bc [j]=(P b [i],P c [i])

[0075] F e [j] = XDeconv(P b [i],F b [i],P bc [j])

[0076] F d [j] = XDeconv(P bc [j-1],F bc [j-1],P bc [j])

[0077] F bc [j]=F e [j]+F d [j]

[0078] The decoding layer j has two inputs, the first input is the visible representative point P output by the corresponding encoding layer i b [i] Corresponding visible feature information F b [i]; The second input is the previous decoding layer P bc [j-1] The target feature information F corresponding to the target representative point output bc [j-1]. Through the matrix transformation deconvolution operation XDeconv of the decoding layer j, the visible representative point P output by the corresponding encoding layer i can be b [i] Corresponding visible feature information F b [i] Mapped to the target representative point P in the current decoding layer bc [j], that is, the visible representative point P of the corresponding coding layer input b [i] and the occlusion representative point P in the current decoding layer c [i], obtain the target representative point P bc [j] The corresponding first feature information F e [j], and the previous decoding layer P bc [j-1] The target feature information F corresponding to the target representative point output bc[j-1] is mapped to the target representative point P in the current decoding layer bc [j], obtain the target representative point P bc [j] Corresponding second feature information F d [j]. The first feature information F e [j] and second feature information F d [j] performs fusion processing, such as weighted processing, to obtain the target feature information F output by the decoding layer j bc [j]. For example, the first feature information F corresponding to each target representative point in the current decoding layer can be e [j] and second feature information F d [j] is added, and the obtained addition result is used as the target feature information F corresponding to each target representative point in the current decoding layer bc [j]. When j = 1, that is, the first decoding layer, P bc [0] = P b [L] and F bc [0] = F b [L] indicates that the input of the first decoding layer is the visible representative point P output by the last encoding layer b [L] corresponding visible feature information F b [L].

[0079] The last decoding layer in the occlusion point decoding sub-model can input the target feature information corresponding to the final output target representative point, that is, the target feature information corresponding to all visible points and all occluded points, into one or more fully connected layers, so that the fully connected layer can predict a semantic label for each visible point and each occluded point based on the target feature information corresponding to each visible point and each occluded point, thereby realizing semantic scene completion and semantic segmentation, thereby improving the segmentation effect and ensuring the accuracy of semantic segmentation.

[0080] It should be noted that the visible point encoding sub-model and the occluded point decoding sub-model can capture the structural features of visible points, and occluded points do not participate in the feature encoding process. This is because occluded points include both empty and non-empty points, and the object structure is determined only by non-empty points. Therefore, empty points mixed with non-empty points will hide the object structure. Therefore, only visible points with known structure are input into the visible point encoding sub-model for feature extraction, thereby ensuring segmentation accuracy.

[0081] S460: Obtain a semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model.

[0082] The technical solution of this embodiment can realize a point cloud segmentation network model with heterogeneous input and output by utilizing the visible point encoding sub-model and the occluded point decoding sub-model, so that the visible point encoding sub-model and the occluded point decoding sub-model can provide more geometric structure information for semantic segmentation, and combine the texture information carried by the target color image and the geometric information in the depth image to perform semantic segmentation, thereby improving the accuracy and segmentation effect of semantic segmentation.

[0083] Example 3

[0084] Figure 6 This is a flowchart of a method for image semantic segmentation provided in Example 3 of the present invention. This example further optimizes the method for "projecting the target color image into a three-dimensional point cloud space and determining semantic feature information corresponding to each visible point" based on the previous examples. Explanations of terms that are identical or corresponding to those in the previous examples are omitted here.

[0085] See also Figure 6 The image semantic segmentation method provided in this embodiment specifically includes the following steps:

[0086] S610: Acquire a target depth image and a target color image corresponding to the target depth image.

[0087] S620 , performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, where the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point.

[0088] S630: Input the target color image into the image segmentation network model, and obtain the image segmentation result corresponding to the target color image according to the output of the image segmentation network model.

[0089] The image segmentation network model may be an existing network model for performing two-dimensional semantic segmentation on color images. The image segmentation network model may be obtained by pre-training based on sample data.

[0090] Specifically, Figure 7 An example of the image semantic segmentation process is given. Figure 7 As shown in FIG, the target color image is input into the trained image segmentation network model for semantic feature extraction. High-level semantic feature information, that is, the semantic information corresponding to each pixel in the target color image, can be obtained from the obtained image segmentation result.

[0091] It should be noted that since the obtained three-dimensional point cloud data is relatively sparse compared to image pixels, directly projecting the low-level RGB into the point cloud may lose the detailed appearance information of the color image, and since some objects have the same color, such as walls and ceilings are both white, the RGB feature information may add some noise to the input point cloud. Therefore, by utilizing the image segmentation network model, high-level semantic feature information can be obtained, so that objects with the same color can be easily partitioned based on the semantic feature information, further improving the segmentation accuracy.

[0092] S640: Project the image segmentation result corresponding to the target color image into the three-dimensional point cloud space to determine the semantic feature information corresponding to the visible points.

[0093] Specifically, if Figure 7 As shown, based on the camera's intrinsic parameter matrix information and the target depth image, the two-dimensional semantic feature information can be projected into the three-dimensional point cloud space to obtain the semantic feature information corresponding to each visible point.

[0094] S650: Input the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model to perform scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point.

[0095] Specifically, by inputting the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model, for example, the semantic feature vector and the corresponding point coordinate vector can be spliced and input into the preset semantic segmentation network model, so that the preset semantic segmentation network model can input and output heterogeneous semantic scene completion based on the three-dimensional coordinate information and semantic feature information corresponding to each visible point, so that the appearance information and geometric information can be more effectively integrated, so that the preset semantic segmentation network model can more accurately identify objects with similar geometric structures and colors, further improving the accuracy of semantic segmentation.

[0096] S660: Obtain a semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model.

[0097] The technical solution of this embodiment is to input the target color image into the image segmentation network model to obtain the image segmentation result corresponding to the target color image, and project the image segmentation result into the three-dimensional point cloud space to determine the semantic feature information corresponding to the visible points, and input the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model for scene completion and semantic segmentation, so as to more effectively fuse the appearance information and geometric information, so that the preset semantic segmentation network model can more accurately identify objects with similar geometric structure and color, further improving the accuracy and segmentation effect of semantic segmentation.

[0098] The following is an embodiment of the image semantic segmentation device provided by an embodiment of the present invention. The device and the image semantic segmentation methods of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the image semantic segmentation device, please refer to the embodiment of the above-mentioned image semantic segmentation method.

[0099] Example 4

[0100] Figure 8 This is a structural schematic diagram of an image semantic segmentation device provided in Example 4 of the present invention. This embodiment is applicable to the situation where semantic segmentation is performed on depth images collected in application scenarios. The device specifically includes: an image acquisition module 810, a point cloud conversion module 820, an image projection module 830, a data input module 840 and a semantic segmentation result acquisition module 850.

[0101] Among them, the image acquisition module 810 is used to obtain the target depth image and the target color image corresponding to the target depth image; the point cloud conversion module 820 is used to perform point cloud conversion on the target depth image to obtain three-dimensional point cloud data, and the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point; the image projection module 830 is used to project the target color image into the three-dimensional point cloud space and determine the semantic feature information corresponding to each visible point; the data input module 840 is used to input the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model for scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point; the semantic segmentation result acquisition module 850 is used to obtain the semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model.

[0102] Optionally, the preset semantic segmentation network model includes: a visible point encoding sub-model and an occluded point decoding sub-model;

[0103] The data input module 840 includes:

[0104] a visible feature information determination unit, configured to input the three-dimensional coordinate information and semantic feature information corresponding to each visible point into the visible point encoding sub-model, perform feature extraction on each visible point, and determine the visible feature information corresponding to the visible representative point among the visible points;

[0105] The semantic label determination unit is used to input the visible feature information corresponding to the visible representative point into the occlusion point decoding sub-model, map the visible feature information corresponding to the visible representative point to the representative occlusion point among the occlusion points and perform feature fusion to determine the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point.

[0106] Optionally, the visible point coding sub-model includes: a preset number of coding layers;

[0107] The visible feature information determination unit is specifically configured to:

[0108] The visible feature information corresponding to the visible representative point output by the previous coding layer is input into the current coding layer for matrix transformation convolution operation to determine the visible feature information corresponding to the visible representative point in the current coding layer.

[0109] Optionally, the occlusion point decoding sub-model includes: a preset number of decoding layers and fully connected layers, the decoding layers correspond to the encoding layers one-to-one, and the decoding order of the decoding layers is opposite to the encoding order of the encoding layers;

[0110] The semantic label determination unit is specifically used to: input the visible feature information corresponding to the visible representative point in the current coding layer into the corresponding current decoding layer in the occlusion point decoding sub-model to perform a matrix transformation inverse convolution operation to determine the first feature information corresponding to the target representative point in the current decoding layer, wherein the target representative point includes the visible representative point input in the current coding layer and the occlusion representative point in the current decoding layer; input the target feature information corresponding to the target representative point in the previous decoding layer into the current decoding layer to perform a matrix transformation inverse convolution operation to determine the second feature information corresponding to the target representative point in the current decoding layer; fuse the first feature information and the second feature information to determine the target feature information corresponding to the target representative point in the current decoding layer, and input the target feature information corresponding to the target representative point in the current decoding layer to the next decoding layer; input the target feature information corresponding to the target representative point in the last decoding layer into the fully connected layer for processing to determine the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point.

[0111] Optionally, the number of visible representative points in the current coding layer is smaller than the number of visible representative points in the previous coding layer; and the number of occluded representative points in the current decoding layer is larger than the number of occluded representative points in the previous decoding layer.

[0112] Optionally, the image projection module 830 includes:

[0113] An image segmentation result determination unit is used to input the target color image into the image segmentation network model and obtain an image segmentation result corresponding to the target color image according to the output of the image segmentation network model;

[0114] The image segmentation result projection unit is used to project the image segmentation result corresponding to the target color image into the three-dimensional point cloud space to determine the semantic feature information corresponding to the visible points.

[0115] The image semantic segmentation device provided by the embodiment of the present invention can execute the image semantic segmentation method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the image semantic segmentation method.

[0116] It is worth noting that in the embodiment of the above-mentioned image semantic segmentation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0117] Example 5

[0118] Figure 9 This is a structural diagram of an electronic device provided in Example 5 of the present invention. Figure 9 A block diagram of an exemplary electronic device 12 suitable for implementing embodiments of the present invention is shown. Figure 9 The electronic device 12 shown is only an example and should not limit the functionality and scope of use of the embodiments of the present invention.

[0119] like Figure 9 As shown, electronic device 12 is implemented as a general-purpose computing device. Components of electronic device 12 may include, but are not limited to, one or more processors or processing units 16, system memory 28, and a bus 18 that connects various system components (including system memory 28 and processing unit 16).

[0120] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0121] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0122] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be configured to read and write non-removable, non-volatile magnetic media ( Figure 9 Not shown, often called a "hard drive"). Although Figure 9 Not shown, a magnetic disk drive for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of the present invention.

[0123] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0124] The electronic device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, etc.), one or more devices that enable a user to interact with the electronic device 12, and / or any device that enables the electronic device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via an input / output (I / O) interface 22. Furthermore, the electronic device 12 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with other modules of the electronic device 12 via a bus 18. It should be understood that, although not shown, other hardware and / or software modules can be used in conjunction with the electronic device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the steps of an image semantic segmentation method provided by the embodiment of the present invention, which includes:

[0126] Obtain a target depth image and a target color image corresponding to the target depth image;

[0127] Performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, where the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point;

[0128] Project the target color image into the three-dimensional point cloud space and determine the semantic feature information corresponding to each visible point;

[0129] Input the 3D point cloud data and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model for scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point;

[0130] According to the output of the preset semantic segmentation network model, the semantic segmentation result corresponding to the target depth image is obtained.

[0131] Of course, those skilled in the art will appreciate that the processor may also implement the technical solution of the image semantic segmentation method provided by any embodiment of the present invention.

[0132] Example 6

[0133] This embodiment provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the steps of the image semantic segmentation method provided in any embodiment of the present invention are implemented. The method includes:

[0134] Obtain a target depth image and a target color image corresponding to the target depth image;

[0135] Performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, where the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occluded point occluded by the visible point;

[0136] Project the target color image into the three-dimensional point cloud space and determine the semantic feature information corresponding to each visible point;

[0137] Input the 3D point cloud data and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model for scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point;

[0138] According to the output of the preset semantic segmentation network model, the semantic segmentation result corresponding to the target depth image is obtained.

[0139] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0140] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0141] Program code embodied on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0142] Computer program code for performing the operations of the present invention can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0143] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computing device. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Alternatively, they can be implemented using program code executable by a computer device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module. Thus, the present invention is not limited to any specific combination of hardware and software.

[0144] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A method for image semantic segmentation, characterized in that: include: Acquire a target depth image and a target color image corresponding to the target depth image; Performing point cloud conversion on the target depth image to obtain three-dimensional point cloud data, the three-dimensional point cloud data including: each visible point generated by the target depth image and each occlusion point occluded by the visible point; wherein the visible point is a non-empty point within a preset detection range generated by the target depth image, and the occlusion point is an occlusion point occluded by the visible point from the image viewing angle; Projecting the target color image into a three-dimensional point cloud space, and determining semantic feature information corresponding to each visible point; Inputting the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model to perform scene completion and semantic segmentation, and determining the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point; Obtaining a semantic segmentation result corresponding to the target depth image according to the output of the preset semantic segmentation network model; The preset semantic segmentation network model includes: a visible point encoding sub-model and an occlusion point decoding sub-model; the inputting of the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into the preset semantic segmentation network model for scene completion and semantic segmentation, and determining the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point, includes: Inputting the three-dimensional coordinate information and the semantic feature information corresponding to each visible point into the visible point coding sub-model, performing feature extraction on each visible point, and determining visible feature information corresponding to a visible representative point among each visible point; The visible feature information corresponding to the visible representative point is input into the occlusion point decoding sub-model, the visible feature information corresponding to the visible representative point is mapped to the representative occlusion point among the occlusion points and feature fusion is performed to determine the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point.

2. The method according to claim 1, characterized in that The visible point coding sub-model includes: a preset number of coding layers; The step of inputting the three-dimensional coordinate information and the semantic feature information corresponding to each visible point into the visible point coding sub-model, performing feature extraction on each visible point, and determining visible feature information corresponding to a visible representative point among each visible point includes: The visible feature information corresponding to the visible representative point output by the previous coding layer is input into the current coding layer for matrix transformation convolution operation to determine the visible feature information corresponding to the visible representative point in the current coding layer.

3. The method according to claim 2, characterized in that The occlusion point decoding sub-model includes: the preset number of decoding layers and fully connected layers, the decoding layers correspond to the encoding layers one-to-one, and the decoding order of the decoding layers is opposite to the encoding order of the encoding layers; Inputting the visible feature information corresponding to the visible representative point into the occlusion point decoding sub-model, mapping the visible feature information corresponding to the visible representative point to the representative occlusion point among the occlusion points and performing feature fusion, and determining the semantic label corresponding to each visible point and the semantic label corresponding to each occlusion point, includes: Inputting the visible feature information corresponding to the visible representative point in the current coding layer into the corresponding current decoding layer in the occlusion point decoding sub-model to perform a matrix transformation inverse convolution operation, and determining the first feature information corresponding to the target representative point in the current decoding layer, wherein the target representative point includes the visible representative point input in the current coding layer and the occlusion representative point in the current decoding layer; Inputting the target feature information corresponding to the target representative point in the previous decoding layer into the current decoding layer for performing a matrix transformation inverse convolution operation to determine the second feature information corresponding to the target representative point in the current decoding layer; performing a fusion process on the first feature information and the second feature information to determine target feature information corresponding to the target representative point in the current decoding layer, and inputting the target feature information corresponding to the target representative point in the current decoding layer to the next decoding layer; The target feature information corresponding to the target representative point in the last decoding layer is input into the fully connected layer for processing to determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point.

4. The method according to claim 3, characterized in that The number of visible representative points in the current coding layer is less than the number of visible representative points in the previous coding layer; the number of occluded representative points in the current decoding layer is greater than the number of occluded representative points in the previous decoding layer.

5. The method according to claim 3, characterized in that The fusing the first feature information and the second feature information to determine target feature information corresponding to the target representative point in the current decoding layer includes: The first feature information and the second feature information corresponding to each target representative point in the current decoding layer are added together, and the obtained addition result is used as the target feature information corresponding to each target representative point in the current decoding layer.

6. The method according to any one of claims 1 to 5, characterized in that The projecting the target color image into a three-dimensional point cloud space and determining the semantic feature information corresponding to the visible points includes: Inputting the target color image into an image segmentation network model, and obtaining an image segmentation result corresponding to the target color image according to the output of the image segmentation network model; The image segmentation result corresponding to the target color image is projected into a three-dimensional point cloud space to determine the semantic feature information corresponding to the visible points.

7. An image semantic segmentation device, characterized in that: include: An image acquisition module, configured to acquire a target depth image and a target color image corresponding to the target depth image; a point cloud conversion module, configured to perform point cloud conversion on the target depth image to obtain three-dimensional point cloud data, wherein the three-dimensional point cloud data includes: each visible point generated by the target depth image and each occlusion point occluded by the visible point; wherein the visible point is a non-empty point within a preset detection range generated by the target depth image, and the occlusion point is an occlusion point occluded by the visible point from the image perspective; An image projection module, configured to project the target color image into a three-dimensional point cloud space and determine semantic feature information corresponding to each visible point; A data input module is used to input the three-dimensional point cloud data and the semantic feature information corresponding to each visible point into a preset semantic segmentation network model to perform scene completion and semantic segmentation, and determine the semantic label corresponding to each visible point and the semantic label corresponding to each occluded point; a semantic segmentation result acquisition module is used to obtain the semantic segmentation result corresponding to the target depth image based on the output of the preset semantic segmentation network model; The preset semantic segmentation network model includes: a visible point encoding sub-model and an occluded point decoding sub-model; the data input module includes: a visible feature information determining unit, configured to input the three-dimensional coordinate information and the semantic feature information corresponding to each visible point into the visible point coding sub-model, perform feature extraction on each visible point, and determine the visible feature information corresponding to a visible representative point among each visible point; A semantic label determination unit is used to input the visible feature information corresponding to the visible representative point into the occlusion point decoding sub-model, map the visible feature information corresponding to the visible representative point to the representative occlusion point among the occlusion points and perform feature fusion to determine the semantic label corresponding to each of the visible points and the semantic label corresponding to each of the occlusion points.

8. An electronic device, characterized in that: The electronic device comprises: one or more processors; a memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the image semantic segmentation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image semantic segmentation method according to any one of claims 1 to 6 is implemented.

Images