A virtual view selection method and device for indoor scene three-dimensional semantic segmentation

Abstract

The application provides a virtual view selection method and device for three-dimensional semantic segmentation of an indoor scene. A voxel-based three-dimensional deep learning network is used to preliminarily segment a three-dimensional indoor scene to obtain a feature vector and a semantic segmentation result of the three-dimensional scene. An uncertainty scoring network is trained by using the feature vector and the initial segmentation result by using a deep reinforcement learning method to estimate the uncertainty of each part of the scene. A suitable virtual view framing area is selected according to the uncertainty of the scene, and a virtual view is rendered from an appropriate angle. The semantic segmentation result of the three-dimensional indoor scene is obtained by combining the virtual view and the three-dimensional information of the original three-dimensional scene. In the process of optimizing the semantic segmentation result of the three-dimensional indoor scene, the generation of the uncertainty of the scene is supervised by using deep reinforcement learning, and a virtual view rendering method is introduced to ensure that more information is provided on a position with a poor original segmentation result, so that the accuracy of the semantic segmentation of the three-dimensional indoor scene is significantly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional scene understanding, in particular to a semantic segmentation method and device for a three-dimensional scene. BACKGROUND

[0002] In virtual reality, augmented reality, intelligent robots and other applications, accurately performing semantic segmentation on an indoor scene helps an intelligent agent to have a deeper understanding of the scene, and enables the application to interact with the scene better.

[0003] In the prior art, for a three-dimensional semantic segmentation task of an indoor scene, most methods transform three-dimensional geometry into a point cloud, a voxel or a three-dimensional mesh, and then directly perform segmentation through a suitable three-dimensional deep learning network. This method has poor segmentation effect on objects with unobvious geometric shapes due to the lack of two-dimensional picture information. Some methods use a combination of two-dimensional picture information and three-dimensional geometric information, but generally use original scanned pictures. Because the field of view of the original scanned pictures is small and the angle is limited, it is difficult to solve problems such as mutual occlusion between objects and inconsistent lighting. Some methods use virtual views to solve the problem, but the selection of virtual views has no rules, which can lead to many low-quality pictures, increase a large amount of calculation, and reduce the accuracy of the segmentation result.

[0004] In summary, the three-dimensional semantic segmentation technology for an indoor scene needs to be improved. SUMMARY

[0005] The present application provides a virtual view selection method and device for three-dimensional semantic segmentation of an indoor scene. Based on deep reinforcement learning, a scoring network for predicting uncertainty scores of regions of a scene is trained, and a virtual view perspective is selected according to the uncertainty predicted by the network, so that the virtual view covers regions with high uncertainty as much as possible. The semantic segmentation result of the three-dimensional scene is optimized by using these virtual views and geometric features of the scene.

[0006] In a first aspect, the present application provides a three-dimensional semantic segmentation method for an indoor scene based on virtual view selection, which comprises:

[0007] performing preliminary semantic segmentation on a three-dimensional indoor scene by using a three-dimensional deep learning network based on voxels to obtain a feature vector and a semantic segmentation result of the three-dimensional scene;

[0008] training an uncertainty scoring network by using deep reinforcement learning to estimate the uncertainty of each region of the scene by using the feature vector and the initial segmentation result;

[0009] selecting a suitable virtual view framing area according to the uncertainty of the scene, and rendering a virtual view from a proper angle;

[0010] The three-dimensional semantic segmentation result of the three-dimensional indoor scene is optimized by combining the virtual view and the three-dimensional information of the original three-dimensional scene.

[0011] According to the three-dimensional semantic segmentation method for the three-dimensional indoor scene, the three-dimensional indoor scene is preliminarily segmented by using a voxel-based three-dimensional deep learning network, and a feature vector and a semantic segmentation result of the three-dimensional scene are obtained, and the method comprises the following steps:

[0012] The point cloud of the three-dimensional scene is voxelized to obtain a three-dimensional scene geometry representation composed of voxels with a side length of 5 cm.

[0013] The voxels are input into a basic three-dimensional semantic segmentation network skeleton to obtain a geometric feature vector of the scene.

[0014] The geometric feature vector of the scene is input into a classifier of the basic three-dimensional semantic segmentation network to obtain a preliminary three-dimensional semantic segmentation result of the scene.

[0015] According to the three-dimensional semantic segmentation method for the three-dimensional indoor scene, the feature vector and the initial segmentation result are used to train an uncertainty scoring network by using a deep reinforcement learning method to estimate the uncertainty of each part of the scene.

[0016] A three-dimensional uncertainty scoring network with a 3D-UNet architecture is established, and the input of the network is the feature vector and the preliminary segmentation result of the three-dimensional scene, and the output of the network is an uncertainty distribution map of the scene.

[0017] The real label manually labeled by the semantic segmentation of the three-dimensional scene is compared with the preliminary segmentation result of the scene to supervise the uncertainty scoring network.

[0018] According to the three-dimensional semantic segmentation method for the three-dimensional indoor scene, a suitable virtual view framing area is selected according to the uncertainty of the scene, and a virtual view is rendered from a proper angle.

[0019] Based on the input three-dimensional scene, the normal direction vector of each voxel in the scene is calculated.

[0020] Based on the three-dimensional coordinates and the normal vector of each voxel, the k-means algorithm is used for clustering to divide the three-dimensional scene into multiple regions.

[0021] The total uncertainty of each region is sorted, and the regions with high total uncertainty are selected as selected regions.

[0022] The selected regions are subjected to multi-angle virtual view generation and rendering.

[0023] The three-dimensional semantic segmentation method for a three-dimensional indoor scene according to the present application combines virtual views and three-dimensional information of an original three-dimensional scene, optimizes the semantic segmentation result of the three-dimensional indoor scene, and comprises the following steps:

[0024] The generated virtual view is subjected to semantic segmentation using a two-dimensional semantic segmentation network.

[0025] The correspondence between the pixel position of each virtual view and the voxel position of the three-dimensional indoor scene is established.

[0026] The semantic segmentation result of the virtual view and the three-dimensional semantic segmentation result of the voxel corresponding to the virtual view are fused.

[0027] The three-dimensional semantic segmentation method for a three-dimensional indoor scene according to the present application comprises the following steps in the process of supervised training of the uncertainty scoring network by deep reinforcement learning:

[0028] The uncertainty scoring network weight θ is randomly initialized;

[0029] For a scene S, the uncertainty scoring network outputs the uncertainty of each voxel of the scene;

[0030] Each region of the scene is randomly assigned a value of 0 or 1, and all regions assigned a value of 1 are selected, and the selected regions are subjected to virtual view selection and rendering;

[0031] A two-dimensional and three-dimensional joint segmentation result p i is obtained according to the virtual view segmentation and combination method

[0032] The joint segmentation result p i is compared with the joint segmentation result p i-1 of the previous cycle, and a reinforcement learning incentive item Reward is calculated;

[0033] The uncertainty scoring network weight θ is updated using the incentive item Reward;

[0034] The above process is repeated for the scene S.

[0035] In a second aspect, the present application provides an indoor scene three-dimensional semantic segmentation device based on virtual view selection, which comprises:

[0036] A three-dimensional semantic segmentation network module for performing preliminary semantic segmentation on an indoor scene to obtain a feature vector and a preliminary semantic segmentation result of the indoor scene;

[0037] A deep reinforcement learning module for training an uncertainty scoring network to estimate the uncertainty of each region of the scene;

[0038] A virtual view selection module for selecting a suitable virtual view field of view according to the uncertainty of the scene and rendering a virtual view from a proper angle;

[0039] The joint optimization module is used for combining the two-dimensional semantic segmentation result of the virtual view and the original three-dimensional geometric information, and improving the accuracy of the three-dimensional semantic segmentation result of the indoor scene.

[0040] The application provides a virtual view selection method and device for three-dimensional semantic segmentation of an indoor scene. A three-dimensional indoor scene is preliminarily segmented by using a three-dimensional deep learning network based on voxels to obtain a feature vector and a semantic segmentation result of the three-dimensional scene; an uncertainty scoring network is trained by using the feature vector and the initial segmentation result by using a deep reinforcement learning method to estimate the uncertainty of each part of the scene; a suitable virtual view framing area is selected according to the uncertainty of the scene, and a virtual view is rendered from a proper angle; and the semantic segmentation result of the three-dimensional indoor scene is obtained by combining the virtual view and the original three-dimensional scene information. In the process of optimizing the semantic segmentation result of the three-dimensional indoor scene, the generation of the scene uncertainty is supervised by using deep reinforcement learning, a virtual view rendering method is introduced, more information is ensured on positions where the original segmentation result is poor, and the accuracy of the three-dimensional indoor scene semantic segmentation is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0042] Figure 1 is a flowchart of the virtual view selection method for three-dimensional semantic segmentation of a three-dimensional indoor scene provided by the present application;

[0043] Figure 2 is a structural schematic diagram of the virtual view selection device for three-dimensional semantic segmentation of a three-dimensional indoor scene provided by the present application;

[0044] Figure 3 is a structural schematic diagram of an electronic device for realizing the virtual view selection for three-dimensional semantic segmentation of a three-dimensional indoor scene provided by the present application. DETAILED DESCRIPTION

[0045] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the present application.

[0046] The application is described below in combination with Figures 1-3 The application provides a virtual view selection method and device for three-dimensional semantic segmentation of a three-dimensional indoor scene.

[0047] In a first aspect, the application provides a virtual view selection method for three-dimensional semantic segmentation of a three-dimensional indoor scene, which comprises the following steps: Figure 1 As shown in the figure, the method comprises the following steps:

[0048] S11, performing preliminary semantic segmentation on the three-dimensional indoor scene by using a voxel-based three-dimensional deep learning network to obtain a feature vector and a semantic segmentation result of the three-dimensional scene.

[0049] The point cloud of the three-dimensional scene is voxelized to obtain a three-dimensional scene geometry representation composed of voxels with a side length of 5 cm. The voxels are input into a basic three-dimensional semantic segmentation network skeleton to obtain a geometry feature vector of the scene. The geometry feature vector of the scene is input into a classifier of the basic three-dimensional semantic segmentation network to obtain a preliminary result of three-dimensional semantic segmentation of the scene.

[0050] S12, training an uncertainty scoring network by using a deep reinforcement learning method based on the feature vector and the initial segmentation result to estimate the uncertainty of each part of the scene.

[0051] A three-dimensional uncertainty scoring network with a 3D-UNet architecture is established, which takes the feature vector and the preliminary result of three-dimensional semantic segmentation of the scene as input and outputs an uncertainty distribution map of the scene. The uncertainty scoring network is supervised by comparing the real label manually labeled by a semantic segmentation of the three-dimensional scene with the preliminary segmentation result of the scene.

[0052] S13, selecting a suitable virtual view framing area according to the uncertainty of the scene and rendering a virtual view from a proper angle.

[0053] Based on the three-dimensional scene, the normal direction vector of each voxel in the scene is calculated. Based on the three-dimensional coordinates and the normal vector of each voxel, the k-means algorithm is used for clustering to divide the three-dimensional scene into multiple regions. The total uncertainty of each region is sorted, and the region with high total uncertainty is selected as the selected region. The selected region is subjected to multi-angle virtual view generation and rendering.

[0054] S14, combining the virtual view and the three-dimensional information of the original three-dimensional scene to optimize the semantic segmentation result of the three-dimensional indoor scene.

[0055] The generated virtual view is subjected to semantic segmentation by using a two-dimensional semantic segmentation network. The correspondence between the pixel position of each virtual view and the voxel position of the indoor three-dimensional scene is established. The semantic segmentation result of the virtual view and the three-dimensional semantic segmentation result of the corresponding voxel are calculated and combined.

[0056] In the process of optimizing the semantic segmentation result of the three-dimensional indoor scene, the generation of scene uncertainty is supervised by deep reinforcement learning, and a virtual view rendering method is introduced to ensure that more information is available in the position where the original segmentation result is poor, thereby significantly improving the accuracy of semantic segmentation of the three-dimensional indoor scene.

[0057] Based on the above embodiments, as an optional embodiment, the preliminary semantic segmentation of the three-dimensional indoor scene is performed by using a voxel-based three-dimensional deep learning network to obtain a feature vector and a semantic segmentation result of the three-dimensional scene, comprising:

[0058] The point cloud of the three-dimensional scene is voxelized to obtain a three-dimensional scene geometry representation composed of voxels with a side length of 5 cm;

[0059] Specifically, a certain scene in the ScanNet V2 indoor scene dataset is voxelized, the scene is divided into small blocks with a side length of 5 cm, and all voxels containing points in the scene point cloud inside the blocks are marked to obtain a three-dimensional sparse matrix.

[0060] The voxels are input into a basic three-dimensional semantic segmentation network skeleton to obtain a geometric feature vector of the scene;

[0061] Specifically, the three-dimensional sparse matrix composed of voxels is input into the three-dimensional semantic segmentation network skeleton through sparse convolution operation to obtain a high-dimensional geometric feature vector of the scene.

[0062] The three-dimensional semantic segmentation network skeleton is the encoder part of MinkowskiUNet in Minkouski Engine, which is specially used for sparse convolution operation on spatially sparse tensors.

[0063] The geometric feature vector of the scene is input into the classifier of the basic three-dimensional semantic segmentation network to obtain a preliminary result of the three-dimensional semantic segmentation of the scene.

[0064] Specifically, the high-dimensional geometric feature of the scene is input into the decoder part of MinkowskiUNet to perform deconvolution and de-pooling operation to obtain a low-dimensional vector corresponding to each voxel, and each value in the vector represents the probability of the voxel belonging to a certain semantic class. The low-dimensional vector corresponding to each voxel is normalized by the Softmax function to obtain the probability of the voxel corresponding to different semantic categories, and the category with the highest probability is the predicted result of the semantic segmentation of the voxel by the network.

[0065] On the basis of the above embodiments, as an optional embodiment, the method of utilizing the feature vector and the initial segmentation result utilizes the deep reinforcement learning to train the uncertainty scoring network, estimates the uncertainty of each place in the scene, including:

[0066] A three-dimensional uncertainty scoring network of a 3D-UNet architecture is established.

[0067] Specifically, the input of the uncertainty scoring network is a feature tensor spliced from the feature vector of the three-dimensional scene and the three-dimensional semantic segmentation preliminary result, and the network output is the uncertainty distribution map of the scene.

[0068] The uncertainty scoring network is based on Minkowski Engine and includes the encoder part and the decoder part of MinkowsUNet. The network output obtains a numerical value corresponding to each voxel, and the numerical value is the uncertainty score of the voxel.

[0069] The real label of the semantic segmentation of the three-dimensional scene is compared with the preliminary segmentation result of the scene to supervise the uncertainty scoring network.

[0070] Specifically, the indoor scene three-dimensional semantic segmentation network predicts a probability distribution for each voxel, indicating the possibility of the voxel belonging to each semantic; the artificial labeling result of the semantic segmentation of the scene is regarded as the true value; the artificial labeling true value and the predicted probability distribution are compared to obtain the conclusion of whether the prediction of the voxel is correct, and the probability size predicted by the network on the classification of the artificial labeling can be obtained.

[0071] On the basis of the above embodiments, as an optional embodiment, the method of selecting a suitable virtual view framing area according to the uncertainty of the scene and rendering a virtual view from a proper angle includes:

[0072] Based on the three-dimensional scene, the normal direction vector of each voxel in the scene is calculated.

[0073] Specifically, the Vertex normal estimation module in the open3D library can be used to estimate the normal of each point in the indoor scene point cloud. In order to reduce the calculation amount and at the same time ensure the calculation accuracy of the voxel normal direction vector, the voxel-based downsampling operation can be performed on the indoor scene point cloud to obtain a simplified point cloud, and then the normal of the point is estimated.

[0074] For each voxel of the scene, the normals of the points in the simplified point cloud contained therein are averaged to obtain the normal direction vector of each voxel of the scene.

[0075] Based on the three-dimensional coordinates and normal vectors of each voxel, clustering is performed using a k-means algorithm to divide the three-dimensional scene into multiple regions;

[0076] Specifically, the three-dimensional coordinates x y z of the spatial coordinates Pos of the voxel and the three-dimensional components N x N y N z are merged into a 6D space, and the Pos+λN corresponding to the voxel is clustered as a parameter, where λ is the weight of the normal in the clustering process, and the reference value is 10.

[0077] The clustering algorithm uses a k-means algorithm, and the maximum number of clusters of the k-means algorithm is 20 and r cluster *A scene The larger one.

[0078] Wherein, r cluster is the clustering area coefficient, and the reference value is 0.25; A scene is the floor area size of the selected scene.

[0079] The total uncertainty of each region is sorted, and the region with high total uncertainty is selected as the selected region.

[0080] Specifically, the calculation method of the region total uncertainty is to add the uncertainty of all voxels in the region. The higher the region total uncertainty, the more voxels with high uncertainty are contained in the region, and the more inaccurate the preliminary three-dimensional semantic segmentation result of the region is.

[0081] After obtaining the total uncertainty of each region, we select the top k regions with the highest total uncertainty, so that the number of voxels with uncertainty higher than τ0(refer to 0.6) in the k regions accounts for a proportion of the number of voxels with uncertainty higher than τ0in the scene, which is greater than or equal to the coverage τ(refer to 0.7). At this time, we select the k regions as the selected regions.

[0082] Multi-angle virtual view generation and rendering are performed on the selected regions.

[0083] Specifically, the average normal of each selected region is calculated, because the normals of the voxels in the region are roughly in the same direction after clustering, so the average normal of the region can represent the front direction of the object in the region.

[0084] Taking the average normal of the region as the central axis, a conical surface with an angle of 45° with the central axis is made, and the conical surface is evenly divided into three parts by three axes. The three axes are selected as the principal axis directions of the three cameras, and the region is taken along the direction and rendered into a virtual view.

[0085] wherein the camera FOV is selected as 90, the distance between the camera and the region is selected as to make the rendered picture just able to completely cover the region, and the virtual view rendering adopts the Visualization module in the Open3D library.

[0086] On the basis of the above embodiments, as an optional embodiment, the three-dimensional semantic segmentation method for a three-dimensional indoor scene provided by the application combines virtual views and original three-dimensional scene three-dimensional information to optimize the semantic segmentation result of the three-dimensional indoor scene, and comprises:

[0087] performing semantic segmentation on the generated virtual view using a two-dimensional semantic segmentation network;

[0088] Specifically, the virtual view is subjected to two-dimensional semantic segmentation using a CMX (Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers) network.

[0089] wherein the CMX network is a multi-modal semantic segmentation network, the input is an RGB-X type picture, and the output is a semantic classification result of each pixel of the picture. The three-dimensional coordinates of the voxels in the indoor scene corresponding to each pixel in the virtual view and the depth of the corresponding point of each pixel in the virtual view are combined as the X modal input of the network. The backbone network of the CMX network is Mit-B5, and the network parameters are obtained by training the virtual view rendered on the ScanNet V2 scene dataset.

[0090] establishing the correspondence between the pixel positions of each virtual view and the voxel positions of the indoor three-dimensional scene;

[0091] Specifically, the correspondence between the voxel coordinates Pos w and the pixel coordinates Pos uv may be calculated by the following formula:

[0092]

[0093] wherein K is the camera intrinsic matrix, T is the camera extrinsic matrix, f is the camera focal length, c is the camera projection center in the picture coordinate system, R is the camera rotation matrix, and t is the camera translation vector. x f y x c y

[0094] combining the semantic segmentation result of the virtual view and the three-dimensional semantic segmentation result of the corresponding voxel.

[0095] ​​Specifically, for each voxel, we find the corresponding virtual views and calculate the pixel position in the virtual view corresponding to the voxel. For all the virtual views, the cross-entropy value of the classification probability vector of the corresponding pixel points is calculated, and the four virtual views with the smallest cross-entropy are selected as the selected virtual views.

[0096] For the above selected virtual views, the classification probability vectors of the four corresponding pixel points are averaged, and the result is input into a double-layer fully connected network together with the three-dimensional semantic segmentation result probability vector of the voxel, to obtain a total probability vector representing the final semantic segmentation result probability vector of the voxel.

[0097] On the basis of the above embodiments, as an optional embodiment, the depth reinforcement learning process of the supervised training uncertainty scoring network includes:

[0098] Randomly initialize the uncertainty scoring network weight θ;

[0099] For the scene S, the uncertainty scoring network outputs the uncertainty of each voxel of the scene;

[0100] Randomly assign values of 0 or 1 to each region of the scene, select all regions with a value of 1, and perform virtual view selection and rendering on the selected regions;

[0101] According to the virtual view segmentation and combination method, a two-dimensional and three-dimensional joint segmentation result p is obtained i ;

[0102] Compare p i with the joint segmentation result p i-1 of the last cycle to calculate the incentive term Reward of reinforcement learning;

[0103] Update the uncertainty scoring network weight θ using the incentive term Reward;

[0104] Repeat the above process for the scene S.

[0105] Specifically, the Reward calculation method is that for each voxel v, the probability of the correct label in the i-th prediction result is Then

[0106] For each voxel v, the reward function R v is

[0107]

[0108] For the entire scene, the overall reward function R(Reward) is

[0109]

[0110] In a second aspect, the present application provides a device for selecting a virtual view for three-dimensional semantic segmentation of a three-dimensional indoor scene, which can be used in conjunction with the method for selecting a virtual view for three-dimensional semantic segmentation of a three-dimensional indoor scene described above. Figure 2 FIG. 2 is a schematic diagram of the device for selecting a virtual view for three-dimensional semantic segmentation of a three-dimensional indoor scene according to the present application, which shows that the device includes: Figure 2

[0111] an initial semantic segmentation module 21 configured to perform preliminary semantic segmentation on the three-dimensional indoor scene to obtain a feature vector and a semantic segmentation result of the three-dimensional scene;

[0112] an uncertainty estimation module 22 configured to train an uncertainty scoring network to estimate the uncertainty of each part of the scene;

[0113] a virtual view selection module 23 configured to select an appropriate virtual view framing area according to the uncertainty of the scene and render a virtual view from an appropriate angle;

[0114] a semantic segmentation optimization module 24 configured to optimize the semantic segmentation result of the three-dimensional indoor scene in combination with the virtual view and the three-dimensional information of the original three-dimensional scene.

[0115] In the above embodiments, as an optional embodiment, the initial semantic segmentation module includes:

[0116] a scene voxelization unit configured to voxelize the point cloud of the three-dimensional scene to obtain a three-dimensional scene geometry representation composed of voxels with a side length of 5 centimeters;

[0117] a scene feature extraction unit configured to input the voxels into a basic three-dimensional semantic segmentation network skeleton to obtain a geometric feature vector of the scene;

[0118] a scene semantic segmentation unit configured to input the geometric feature vector of the scene into a classifier of the basic three-dimensional semantic segmentation network to obtain a preliminary result of three-dimensional semantic segmentation of the scene.

[0119] In the above embodiments, as an optional embodiment, the uncertainty estimation module includes:

[0120] an uncertainty network unit configured to establish a three-dimensional uncertainty scoring network with a 3D-UNet architecture;

[0121] The input of the uncertainty scoring network is a feature tensor composed of the feature vector of the three-dimensional scene and the preliminary result of three-dimensional semantic segmentation, and the output of the network is an uncertainty distribution map of the scene. ​

[0122] A network supervision unit is configured to compare the real label artificially labeled by semantic segmentation of the three-dimensional scene with the preliminary segmentation result of the scene, and supervise the uncertainty scoring network.

[0123] Based on the above embodiments, as an optional embodiment, the virtual view selection module comprises:

[0124] A voxel normal calculation unit is configured to calculate a normal direction vector of each voxel in the scene;

[0125] A voxel clustering unit is configured to use a k-means algorithm to cluster based on the three-dimensional coordinates and normal vectors of the voxels, and divide the three-dimensional scene into multiple regions;

[0126] A region selection unit is configured to sort the total uncertainty of each region, and select a part of regions with high total uncertainty as selected regions;

[0127] A virtual view generation unit is configured to generate and render multi-angle virtual views for the selected regions.

[0128] Based on the above embodiments, as an optional embodiment, the semantic segmentation optimization module comprises:

[0129] A two-dimensional semantic segmentation unit is configured to use a two-dimensional semantic segmentation network to perform semantic segmentation on the generated virtual views;

[0130] A two-dimensional-three-dimensional mapping unit is configured to establish a correspondence between the pixel positions of the virtual views and the voxel positions of the indoor three-dimensional scene;

[0131] A two-dimensional-three-dimensional semantic fusion unit is configured to fuse the semantic segmentation result of the virtual view and the three-dimensional semantic segmentation result of the voxel corresponding to the virtual view.

[0132] In summary, the present application uses deep reinforcement learning to supervise the generation of scene uncertainty, and introduces a virtual view rendering method and device, which ensures that there is more information in the position with poor original segmentation result, significantly improves the accuracy of three-dimensional indoor scene semantic segmentation, and has important applications in the fields of scene understanding, virtual reality, augmented reality, robots, etc.

[0133] Figure 3 An example of an entity structure diagram of an electronic device is shown in FIG. 1. Figure 3As shown, the electronic device can include a processor 310, a communications interface 320, a memory 330, and a communications bus 340, wherein the processor 310, the communications interface 320, and the memory 330 complete mutual communication through the communications bus 340. The processor 310 can invoke a logical instruction in the memory 330 to execute a virtual view selection method for three-dimensional semantic segmentation of an indoor scene, which includes: performing preliminary semantic segmentation on a three-dimensional indoor scene by using a voxel-based three-dimensional deep learning network to obtain a feature vector and a semantic segmentation result of the three-dimensional scene; training an uncertainty scoring network by using a deep reinforcement learning method to estimate the uncertainty of each part of the scene according to the feature vector and the initial segmentation result; selecting a suitable virtual view framing area according to the uncertainty of the scene to render a virtual view from a proper angle; and combining the virtual view and the original three-dimensional scene three-dimensional information to optimize the semantic segmentation result of the three-dimensional indoor scene.

[0134] In addition, the logical instruction in the memory 330 described above can be implemented in the form of a software functional unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, which is stored in a storage medium, includes several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0135] On the other hand, the present application also provides a computer program product, which includes a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program can be executed by a processor to enable a computer to execute the virtual view selection method for three-dimensional semantic segmentation of an indoor scene provided by the above-mentioned methods, which includes: performing preliminary semantic segmentation on a three-dimensional indoor scene by using a voxel-based three-dimensional deep learning network to obtain a feature vector and a semantic segmentation result of the three-dimensional scene; training an uncertainty scoring network by using a deep reinforcement learning method to estimate the uncertainty of each part of the scene according to the feature vector and the initial segmentation result; selecting a suitable virtual view framing area according to the uncertainty of the scene to render a virtual view from a proper angle; and combining the virtual view and the original three-dimensional scene three-dimensional information to optimize the semantic segmentation result of the three-dimensional indoor scene.

[0136] In yet another aspect, the present application also provides a non-transitory computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the method for virtual view selection for indoor scene three-dimensional semantic segmentation as provided by the above methods, which comprises: performing preliminary semantic segmentation on a three-dimensional indoor scene by using a voxel-based three-dimensional deep learning network to obtain a feature vector and a semantic segmentation result of the three-dimensional scene; training an uncertainty scoring network by using a deep reinforcement learning method to estimate the uncertainty of each part of the scene according to the feature vector and the initial segmentation result; selecting a suitable virtual view framing area according to the uncertainty of the scene to render a virtual view from a proper angle; and combining the virtual view and the original three-dimensional scene three-dimensional information to optimize the semantic segmentation result of the three-dimensional indoor scene.

[0137] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment scheme according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0138] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary universal hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of software products, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and include a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0139] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A virtual view selection method for semantic segmentation of 3D scenes, characterized in that, The method includes: A voxel-based 3D deep learning network is used to perform preliminary semantic segmentation of a 3D indoor scene, resulting in feature vectors and initial semantic segmentation results for the indoor scene. Using the feature vector and the initial semantic segmentation result, an uncertainty scoring network is trained using deep reinforcement learning to estimate the uncertainty at various points in the scene; wherein the input is the feature vector and the initial semantic segmentation result, and the output is an uncertainty distribution map of the indoor scene; Based on the uncertainty of the scenario, a suitable virtual view framing area is selected, and the virtual view is rendered from an appropriate angle; wherein, the total uncertainty of each area is sorted, and the areas with high total uncertainty are selected as suitable virtual view framing areas; By combining the virtual view and the original 3D scene information, the initial semantic segmentation result of the 3D indoor scene is optimized; A voxel-based 3D deep learning network is used to perform preliminary semantic segmentation of a 3D indoor scene, obtaining the feature vector of the 3D scene and the initial semantic segmentation results, including: The point cloud of the three-dimensional scene is voxelized to obtain a three-dimensional geometric representation of the scene composed of voxels with a side length of 5 cm. The voxels are input into a basic 3D semantic segmentation network skeleton to obtain the geometric feature vector of the scene; The geometric feature vector of the scene is input into the classifier of the basic 3D semantic segmentation network to obtain the initial semantic segmentation result of the scene; Based on the feature vectors and the initial semantic segmentation results, an uncertainty scoring network is trained using deep reinforcement learning to estimate the uncertainty at various points in the scene, including: A 3D uncertainty scoring network based on the 3D-UNet architecture is established. Its inputs are the feature vectors of the 3D scene and the initial semantic segmentation results, and its output is the uncertainty distribution map of the scene. The semantic segmentation of the 3D scene is used to generate human-annotated real labels, which are then compared with the initial semantic segmentation results of the scene to supervise the uncertainty scoring network. The semantic segmentation of the 3D scene, using manually labeled real tags, is compared with the initial semantic segmentation result of the scene to supervise the uncertainty scoring network, including: Randomly initialized uncertainty scoring network weights ; For scenario S, the uncertainty scoring network outputs the uncertainty of each voxel in the scenario; Randomly assign 0 or 1 to each area of ​​the scene, select all areas assigned 1, and perform virtual view selection and rendering on the selected areas; The two-dimensional and three-dimensional joint segmentation results are obtained based on the virtual view segmentation and combination method. ; Compare Joint segmentation result with the previous iteration The incentive term Reward for reinforcement learning is calculated. The uncertainty scoring network weights are updated using the incentive term Reward. ; Repeat the above process for scenario S.

2. The virtual view selection method for semantic segmentation of three-dimensional scenes according to claim 1, characterized in that... Selecting a suitable virtual view framing area based on the uncertainty of the scene, and rendering the virtual view from an appropriate angle, includes: Based on the input 3D scene, calculate the normal direction vector of each voxel in the scene; Based on the three-dimensional coordinates and normal vectors of each voxel, the k-means algorithm is used for clustering to divide the three-dimensional scene into multiple regions. The total uncertainty of each region is sorted, and the regions with high total uncertainty are selected as the selected regions. The selected area is used to generate and render a multi-angle virtual view.

3. The virtual view selection method for semantic segmentation of three-dimensional scenes according to claim 1, characterized in that... Combining the virtual view and the original 3D scene information, the initial semantic segmentation result of the 3D indoor scene is optimized, including: The virtual view is semantically segmented using a two-dimensional semantic segmentation network; Establish the correspondence between the pixel positions of each virtual view and the voxel positions of the indoor 3D scene; The initial semantic segmentation result of the virtual view is fused with the three-dimensional semantic segmentation result of its corresponding voxel.

4. A virtual view selection device for semantic segmentation of three-dimensional scenes, characterized in that, The device includes: The 3D semantic segmentation network module is used to perform preliminary semantic segmentation of indoor scenes, and obtain the feature vectors and initial semantic segmentation results of the indoor scenes. The deep reinforcement learning module is used to train an uncertainty scoring network to estimate the magnitude of uncertainty at various points in the scene. The virtual view selection module is used to select a suitable virtual view framing area based on the uncertainty of the scene, and to render the virtual view from an appropriate angle. The joint optimization module is used to combine the two-dimensional semantic segmentation results of the virtual view with the original three-dimensional geometric information to improve the accuracy of the initial semantic segmentation results of the indoor scene. The 3D semantic segmentation network module is specifically used for: The point cloud of the three-dimensional scene is voxelized to obtain a three-dimensional geometric representation of the scene composed of voxels with a side length of 5 cm. The voxels are input into a basic 3D semantic segmentation network skeleton to obtain the geometric feature vector of the scene; The geometric feature vector of the scene is input into the classifier of the basic 3D semantic segmentation network to obtain the initial semantic segmentation result of the scene; The deep reinforcement learning module is specifically used for: A 3D uncertainty scoring network based on the 3D-UNet architecture is established. Its inputs are the feature vectors of the 3D scene and the initial semantic segmentation results, and its output is the uncertainty distribution map of the scene. The semantic segmentation of the 3D scene is used to generate human-annotated real labels, which are then compared with the initial semantic segmentation results of the scene to supervise the uncertainty scoring network. The semantic segmentation of the 3D scene, using manually labeled real tags, is compared with the initial semantic segmentation result of the scene to supervise the uncertainty scoring network, including: Randomly initialized uncertainty scoring network weights ; For scenario S, the uncertainty scoring network outputs the uncertainty of each voxel in the scenario; Randomly assign 0 or 1 to each area of ​​the scene, select all areas assigned 1, and perform virtual view selection and rendering on the selected areas; The two-dimensional and three-dimensional joint segmentation results are obtained based on the virtual view segmentation and combination method. ; Compare Joint segmentation result with the previous iteration The incentive term Reward for reinforcement learning is calculated. The uncertainty scoring network weights are updated using the incentive term Reward. ; Repeat the above process for scenario S.

Images