Method, device and computer readable storage medium for reconstructing a three-dimensional scene

Abstract

Embodiments of the present application relate to the technical field of computer vision, and disclose a three-dimensional scene reconstruction method, device and equipment and a computer readable storage medium. The method comprises: acquiring a target image, wherein the target image comprises scene information to be reconstructed; obtaining a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed through multi-task learning based on the target image; determining the relationship between the points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map, and obtaining a first prediction result for reconstructing the scene to be reconstructed; and fusing a plurality of first prediction results obtained according to a plurality of target images to obtain a second prediction result. Through the above method, the accuracy of the SLAM algorithm for constructing a map is improved.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer vision, and in particular to a three-dimensional scene reconstruction method, device, equipment and computer readable storage medium. BACKGROUND

[0002] Vision-based SLAM (multaneous Localization and Mapping) is a new three-dimensional map construction technology, that is, during the movement of a sensor, the trajectory is estimated and the three-dimensional information of the surrounding environment is constructed. Vision SLAM uses a camera as a sensor for data acquisition, and has the advantages of large amount of information, high flexibility, low cost, high timeliness and easy embedded development.

[0003] In recent years, with the rapid development of AR / VR technology and the landing of the scene, SLAM technology has gradually become a new research hotspot, especially in the fields of autonomous driving and smart cities, and has broad market prospects, but the traditional SLAM method has low precision. How to improve the precision of the existing SLAM method to construct a map is a problem to be solved. SUMMARY

[0004] In view of the above problems, embodiments of the present application provide a three-dimensional scene reconstruction method, device, equipment and computer readable storage medium, which are used to solve the problem of low precision of the SLAM method in the prior art.

[0005] According to a first aspect of the embodiments of the present application, a three-dimensional scene reconstruction method is provided, the method comprising:

[0006] obtaining a target image, the target image comprising scene information to be reconstructed;

[0007] based on the target image, obtaining a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed through multi-task learning, the point feature map comprising point information in the scene to be reconstructed, the line feature map comprising line information in the scene to be reconstructed, the surface feature map comprising surface information in the scene to be reconstructed, and the object feature map comprising object information in the scene to be reconstructed;

[0008] determining the relationship between the points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map, and obtaining a first prediction result for reconstructing the scene to be reconstructed;

[0009] fusing a plurality of first prediction results obtained according to a plurality of target images to obtain a second prediction result.

[0010] In some embodiments, the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed are obtained based on the target image through multi-task learning, comprising:

[0011] inputting the target image into a first feature extraction model;

[0012] determining the convolution kernel of each node in the first feature extraction model, and determining the link path between nodes;

[0013] performing multi-layer convolution operation according to the determined convolution kernel of each node and the link path between nodes to obtain a plurality of feature maps of different scales;

[0014] inputting the feature maps of different scales into a point line detection model, a plane detection model, a depth estimation model, a normal estimation model and a semantic segmentation model to obtain the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed.

[0015] In some embodiments, the method further comprises:

[0016] In the training process of the point line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model, the loss function of the point line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model is jointly optimized. In some embodiments, the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed are obtained based on the target image through multi-task learning, comprising:

[0017] inputting the target image into a second feature extraction model to obtain the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed, wherein the second feature extraction model is obtained through multi-task learning on a plurality of training images labeled with point, line, surface and object features.

[0018] In some embodiments, the relationship between the points, lines, surfaces and objects in each feature map is determined based on the point feature map, the line feature map, the surface feature map and the object feature map, further comprising:

[0019] performing edge detection based on the point features and line features in the point feature map and the line feature map to obtain the edge features of the three-dimensional scene;

[0020] generating a plane layout scheme and an object three-dimensional bounding box scheme according to the two-dimensional object features in the object feature map and the edge features;

[0021] for each object feature, selecting the optimal N object three-dimensional bounding box schemes according to the object three-dimensional bounding box scheme, N is an integer and N>0.

[0022] A new parameter is defined by using an infinite plane and 9-DOF parameter defined object pose;

[0023] According to the plane layout scheme and the optimal N object 3D bounding box scheme, constraints between objects and planes and constraints between points and planes are obtained;

[0024] According to the constraints between objects and planes and the constraints between points and planes, point association, line association, plane association and object association are performed on the point feature map, the line feature map, the plane feature map and the object feature map.

[0025] In some embodiments, for each object feature, the optimal N object 3D bounding box scheme is selected according to the plane layout scheme and the object 3D bounding box scheme, further comprising:

[0026] A binary random variable x is defined for each object 3D bounding box scheme i ∈{0,1}, the binary random variable is used to mark whether the object 3D bounding box scheme is selected;

[0027] The binary random variable is optimized so that the calculation result of the following formula is the minimum value, and the selected N object 3D bounding box scheme is obtained:

[0028]

[0029] Wherein, is the single-point potential energy, is the pair-point potential energy, is the high-order potential energy of c-group.

[0030] In some embodiments, the second prediction result is obtained by fusing the plurality of first prediction results obtained according to the plurality of target images.

[0031] The number of frames of the selected plurality of target images is set according to a time window, and the fusion of the plurality of first prediction results is performed by using a Transformer model or an LSTM long short-term memory network. According to a second aspect of the embodiment of the present application, a three-dimensional scene reconstruction device is provided, and the device comprises:

[0032] An acquisition module is configured to acquire a target image, wherein the target image comprises scene information to be reconstructed;

[0033] obtaining a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed based on the target image by multi-task learning, the point feature map comprising point information in the scene to be reconstructed, the line feature map comprising line information in the scene to be reconstructed, the surface feature map comprising surface information in the scene to be reconstructed, and the object feature map comprising object information in the scene to be reconstructed;

[0034] determining the relationship between the points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map to obtain a first prediction result for reconstructing the scene to be reconstructed;

[0035] fusing a plurality of first prediction results obtained according to a plurality of target images to obtain a second prediction result.

[0036] According to a third aspect of the embodiments of the present application, a three-dimensional scene reconstruction device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus;

[0037] The memory is used to store at least one executable instruction, and the executable instruction causes the processor to perform the operations of the three-dimensional scene reconstruction method as described above.

[0038] According to a fourth aspect of the embodiments of the present application, instructions are provided, which, when executed on a three-dimensional scene reconstruction device, cause the three-dimensional scene reconstruction device to perform the operations of the three-dimensional scene reconstruction method as described above.

[0039] The embodiments of the present application obtain the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed by multi-task learning, obtain the first prediction result for reconstructing the scene to be reconstructed according to the above feature maps, fuse a plurality of first prediction results obtained according to a plurality of target images to obtain the second prediction result, and improve the reconstruction accuracy of the three-dimensional scene.

[0040] The above description is only a summary of the technical solutions of the embodiments of the present application, in order to more clearly understand the technical means of the embodiments of the present application, the content of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0041] The accompanying drawings are only used to illustrate the embodiments and are not considered as limiting the present application. Moreover, the same reference signs are used to represent the same components throughout the drawings. In the drawings:

[0042] Figure 1 A flowchart of a method for reconstructing a three-dimensional scene is shown;

[0043] Figure 2 A schematic diagram of a link path between nodes of a convolution layer is shown;

[0044] Figure 3 A schematic diagram of a link path between nodes of a convolution layer is shown;

[0045] Figure 4 A structural schematic diagram of a method for reconstructing a three-dimensional scene is shown;

[0046] Figure 5 A structural schematic diagram of a method for reconstructing a three-dimensional scene is shown. DETAILED DESCRIPTION

[0047] Exemplary embodiments of the present application will be described herein below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein.

[0048] Figure 1 A flowchart of a method for reconstructing a three-dimensional scene is shown; Figure 1 The method includes the following steps:

[0049] Step 110: Obtain a target image, wherein the target image includes scene information to be reconstructed.

[0050] The target image refers to an image obtained by photographing a scene to be reconstructed, and the target image includes scene information to be reconstructed. The target image can be obtained by an image acquisition device, which can be a camera or other device with image acquisition function. For example, a camera can be installed on a handheld device or a robot, and the robot moves in the scene to be reconstructed while photographing, and can take pictures or videos. Multiple target images can be obtained according to the pictures or videos taken by the robot.

[0051] The scene information to be reconstructed refers to image information of each feature in the scene to be reconstructed, which includes image information of point features, line features, surface features, and object features.

[0052] Step 120: obtaining, based on the target image, a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed by multi-task learning, the point feature map including point information in the scene to be reconstructed, the line feature map including line information in the scene to be reconstructed, the surface feature map including surface information in the scene to be reconstructed, and the object feature map including object information in the scene to be reconstructed.

[0053] Multi-task learning is a machine learning method opposite to single-task learning. Single-task learning means learning one task at a time. Multi-task learning is a kind of joint learning, multiple tasks are learned in parallel, and the learning results of multiple tasks affect each other.

[0054] The feature map generally includes color features, texture features, shape features and spatial relationship features of an image. The feature maps obtained in this step are used for subsequent spatial feature fusion and three-dimensional reconstruction of each target image.

[0055] In this step, step 120 further includes:

[0056] Step a1: inputting the target image into a first feature extraction model; wherein the first feature extraction model can be a convolutional neural network, which can detect features in the input target image. The convolutional neural network is organized by one layer after another, and each node is a neuron, and the nodes in adjacent two layers are connected.

[0057] Step a2: determining a convolution kernel for each node in the first feature extraction model, and determining a link path between nodes;

[0058] This step determines a convolution kernel for each node in the first feature extraction model. The convolution kernel includes different sizes, such as 2*2, 3*3 or 5*5, and different shapes, such as fixed shape convolution kernels of rectangular, prototype, elliptical and deformable shape convolution kernels. Specifically, the convolution kernel can be selected for each node by prior selection. By searching the convolution kernel for each node, the optimal convolution kernel is determined according to the continuous multiple scoring results, and the first feature extraction model using the optimal convolution kernel is saved. For example, the Monte Carlo method can be used for parameter optimization to solve the optimization model. It can be understood that a separate convolution kernel can also be determined for each convolution layer, and the same convolution kernel is used for each node in the layer to improve the calculation efficiency.

[0059] This step also determines the link path between nodes. Please refer to Figure 2As shown, there are multiple possibilities for the link path between the nodes of different convolution layers, and it is necessary to determine which node or nodes in the next convolution layer each node is linked to, that is, to determine the link path between the nodes. Specifically, the link path between the nodes of adjacent convolution layers can also be selected in a priori selection manner. It can be understood that the path for the link between each convolution layer can also be determined, for example, the link between the first convolution layer and the third convolution layer is uniformly determined, and each node in the first convolution layer is linked to a certain node in the third convolution layer, so as to improve the calculation efficiency.

[0060] Please refer to Figure 3 As shown, the link path between the nodes of different convolution layers is determined in the following manner. Figure 2 For example, the link path between the nodes of the three convolution layers in the dashed box in FIG. 1C is determined in the following manner. l-2 The node can be linked to the C l-1 node, array 0, array 1, or array 2. l-1 The node can be linked to array 0, array 1, or array 2, and can also be linked to the C l node through the Concat function. Array 0, array 1, or array 2 can all be linked to the C l node through the Concat function. By searching for different link paths between the nodes, the optimal path is determined according to the scoring results of multiple times, and the first feature extraction model under the optimal path is saved. For example, the Monte Carlo method can be used for parameter optimization to solve the optimization model.

[0061] Step a3: performing multi-layer convolution operation according to the determined convolution kernel of each node and the link path between the nodes, to obtain multiple feature maps of different scales;

[0062] By determining the convolution kernel for each node, multiple combinations of convolution kernels are formed. By searching for the optimal link path between the nodes, multiple feature maps of different scales, that is, multi-scale features, are obtained after multi-layer convolution operation in the first feature extraction model.

[0063] The multi-scale features can extract multiple dimensions of spatial description from spatial picture information, obtain multi-scale, multi-classification, and multi-material data, improve the accuracy and speed of feature extraction, and facilitate the miniaturization of the model.

[0064] The above steps a1-a3 use the first feature extraction model as a general feature model to extract multi-scale features, and the convolution kernel selection, combination, and path selection are performed in a searching manner, so that the model training is automatically realized without human training and path selection, the training efficiency is improved, and the time and cost are saved.

[0065] Step a4: inputting the feature maps of different scales into a point-line detection model, a plane detection model, a depth estimation model, a normal estimation model and a semantic segmentation model to obtain a point feature map, a line feature map, a plane feature map and an object feature map of the scene to be reconstructed.

[0066] The point-line detection model can detect point features and line features, the plane detection model can detect plane features, the point-line detection model, the plane detection model and the semantic segmentation model can be used to detect object features, and the depth estimation model and the normal estimation model are used for auxiliary estimation of the above feature detection.

[0067] In the training process of the point-line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model, the loss functions of the models are jointly optimized. The parameters or features detected by the above models have a corresponding relationship, for example, the relationship between the normal and the plane is that the more accurate the normal detection is, the more accurate the plane detection is. For another example, the relationship between the depth and the plane is that the more accurate the depth detection is, the more accurate the plane detection is. The relationship between the depth and the semantic is that the more accurate the depth detection is, the more accurate the semantic detection is. Therefore, the loss functions of the models can be jointly optimized by adding weights as constraints between the models. The weight adding and optimization process can refer to the general loss function optimization process of a convolutional neural network.

[0068] The point feature map, the line feature map, the plane feature map and the object feature map of the scene to be reconstructed are obtained through the above multi-task learning method, the relationship between the points, lines, planes and objects in the feature maps can be determined, the first prediction result of reconstructing the scene to be reconstructed is more accurate, and the accuracy of the SLAM map construction is improved.

[0069] It can be understood that in some embodiments, step 120 can be implemented by a second feature extraction model. For example, the target image is input into the second feature extraction model to obtain the point feature map, the line feature map, the plane feature map and the object feature map of the scene to be reconstructed, and the feature extraction model is obtained through multi-task learning on a plurality of training images labeled with point, line, plane and object features.

[0070] The second feature extraction model includes the point-line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model.

[0071] Step 130: determining the relationship between the points, lines, planes and objects in the feature maps according to the point feature map, the line feature map, the plane feature map and the object feature map to obtain a first prediction result of reconstructing the scene to be reconstructed.

[0072] wherein, the determining the relationship between the points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map in step 130 further comprises:

[0073] Step b1: performing edge detection based on the point features and line features in the point feature map and the line feature map to obtain edge features of the three-dimensional scene;

[0074] In this step, for the same three-dimensional scene, its edge features (such as the edge of the ground wall) are fixed, so the edge observation is consistent between image frames, and the detected edge features of the three-dimensional scene can be used as landmarks for subsequent three-dimensional reconstruction.

[0075] Step b2: generating a planar layout scheme and an object three-dimensional bounding box scheme according to the two-dimensional object features in the object feature map and the edge features;

[0076] wherein, the planar layout scheme mainly includes representative edge features selected from the edge features detected in step b1. For example, all image edges are first detected by step b1, and then some edges close to the ground wall partition boundary are selected from them. If the edge is located in the object region and is interrupted by the object occlusion, the edge can be extended to intersect with other edges.

[0077] The object three-dimensional bounding box can be a cuboid bounding box. The cuboid bounding box can be generated based on the two-dimensional bounding box detection.

[0078] Step b3: for each object feature, selecting the optimal N object three-dimensional bounding box schemes according to the object three-dimensional bounding box scheme, N is an integer, N>0;

[0079] wherein, step b3 further comprises:

[0080] Step b31: defining a binary random variable x for each object three-dimensional bounding box scheme i ∈{0,1}, the binary random variable is used to mark whether the object three-dimensional bounding box scheme is selected;

[0081] Step b32: optimizing the binary random variable so that the calculation result of the following formula is the minimum value, obtaining the selected N object three-dimensional bounding box schemes:

[0082]

[0083] wherein, is the single-point potential energy, is the pair-point potential energy, is the high-order potential energy of the c-group.

[0084] In this step, E(x|I) is the total potential energy, which is minimized by optimizing x i ∈{0,1} to minimize the total potential energy, so as to obtain the selected object three-dimensional bounding box scheme at this time.

[0085] In step b3, the object three-dimensional bounding box can be scored according to the image features. For each object, the optimal N object three-dimensional bounding box scheme can be selected for subsequent optimization. For example, N can be selected as 10-20. More object three-dimensional bounding box schemes can improve the recognition accuracy, but will increase the amount of calculation.

[0086] Step b4: define a new parameter using the infinite plane and the object pose defined by the 9-DOF parameters;

[0087] Step b5: according to the plane layout scheme and the optimal N object three-dimensional bounding box scheme, obtain the constraint between the object and the plane, and the constraint between the point and the plane;

[0088] Step b6: according to the constraint between the object and the plane and the constraint between the point and the plane, point association, line association, plane association and object association are performed on the point feature map, the line feature map, the plane feature map and the object feature map.

[0089] Through the above steps, the relationship between the points, lines, planes and objects in each feature map is finally determined, thereby obtaining a first prediction result for reconstructing the scene to be reconstructed.

[0090] Step 140: fuse multiple first prediction results obtained according to multiple target images to obtain a second prediction result.

[0091] Among them, multiple target images are generally continuous multiple target images. Here, the number of frames selected can be set according to the time window. For example, a time window t is set, and the target images taken within the time window t are selected. The multiple first prediction results obtained according to these target images are fused to obtain a second prediction result. At the same time, if the parallax of the images within the time window t needs to be within a preset threshold range, the fusion is performed. If it is less than the preset threshold range, it means that the image has basically no change. If it is greater than the preset threshold range, the change between the images is too large. The above situations are not suitable for fusion. The preset threshold range can be set according to experience.

[0092] Specifically, the fusion of multiple first prediction results can be performed by a Transformer model or an LSTM (Long Short-Term Memory) or the like.

[0093] The first prediction results of multiple images in a time window are fused to obtain a second prediction result of a scene photographed by the multiple images, that is, a final three-dimensional reconstruction result.

[0094] The embodiment of the application obtains the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed through multi-task learning, obtains a first prediction result of reconstructing the scene to be reconstructed according to the feature maps, and fuses multiple first prediction results obtained according to multiple target images to obtain a second prediction result, thereby improving the reconstruction accuracy of the three-dimensional scene.

[0095] Figure 4 A structure diagram of a three-dimensional scene reconstruction device provided by an embodiment of the application is shown. Figure 4 As shown in the figure, the device 300 includes:

[0096] The acquisition module 310 is configured to acquire a target image, wherein the target image includes scene information to be reconstructed.

[0097] The obtaining module 320 is configured to obtain, based on the target image, a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed through multi-task learning, wherein the point feature map includes point information in the scene to be reconstructed, the line feature map includes line information in the scene to be reconstructed, the surface feature map includes surface information in the scene to be reconstructed, and the object feature map includes object information in the scene to be reconstructed.

[0098] The determination module 330 is configured to determine the relationship between the points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map, and obtain a first prediction result of reconstructing the scene to be reconstructed.

[0099] The fusion module 340 is configured to fuse multiple first prediction results obtained according to multiple target images to obtain a second prediction result.

[0100] In an optional manner, the obtaining, based on the target image, of the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed through multi-task learning includes:

[0101] inputting the target image into a first feature extraction model;

[0102] determining the convolution kernel of each node in the first feature extraction model, and determining the link path between nodes;

[0103] performing multi-layer convolution operation according to the determined convolution kernel of each node and the link path between nodes to obtain multiple feature maps of different scales;

[0104] inputting the different scale feature maps into a point-line detection model, a plane detection model, a depth estimation model, a normal estimation model, and a semantic segmentation model to obtain a point feature map, a line feature map, a plane feature map, and an object feature map of the scene to be reconstructed.

[0105] In an optional manner, the method further comprises:

[0106] In the training process of the point-line detection model, the plane detection model, the depth estimation model, the normal estimation model, and the semantic segmentation model, the loss functions of the point-line detection model, the plane detection model, the depth estimation model, the normal estimation model, and the semantic segmentation model are jointly optimized.

[0107] In an optional manner, the point feature map, the line feature map, the plane feature map, and the object feature map of the scene to be reconstructed are obtained through multi-task learning based on the target image, comprising:

[0108] inputting the target image into a second feature extraction model to obtain the point feature map, the line feature map, the plane feature map, and the object feature map of the scene to be reconstructed, wherein the second feature extraction model is obtained through multi-task learning on a plurality of training images labeled with point, line, plane, and object features.

[0109] In an optional manner, the relationship between the points, lines, planes, and objects in each feature map is determined based on the point feature map, the line feature map, the plane feature map, and the object feature map, further comprising:

[0110] performing edge detection based on the point features and line features in the point feature map and the line feature map to obtain edge features of the three-dimensional scene;

[0111] generating a plane layout scheme and an object three-dimensional bounding box scheme according to the two-dimensional object features in the object feature map and the edge features;

[0112] for each object feature, selecting the optimal N object three-dimensional bounding box schemes from the object three-dimensional bounding box schemes, N being an integer and N>0;

[0113] adopting an infinite plane and an object pose defined by 9 degrees of freedom parameters to define new parameters;

[0114] obtaining the constraints between objects and planes and the constraints between points and planes according to the plane layout scheme and the optimal N object three-dimensional bounding box schemes;

[0115] According to the constraint between the object and the plane and the constraint between the point and the plane, the point feature map, the line feature map, the plane feature map and the object feature map are subjected to point association, line association, plane association and object association.

[0116] In an alternative manner, for each object feature, the optimal N object bounding box schemes are selected according to the plane layout scheme and the object bounding box scheme, and the method further comprises the following steps:

[0117] A binary random variable x is defined for each object bounding box scheme i ∈{0, 1}, and the binary random variable is used to mark whether the object bounding box scheme is selected;

[0118] The binary random variable is optimized so that the calculation result of the following formula is the minimum value, and the selected N object bounding box schemes are obtained:

[0119]

[0120] Wherein, is a single-point potential energy, is a pair of point potential energy, is a high-order potential energy of a c group.

[0121] In an alternative manner, the multiple first prediction results obtained according to multiple frames of the target image are fused to obtain a second prediction result, and the method further comprises the following steps:

[0122] The number of frames of the selected multiple frames of the target image is set according to a time window, and the multiple first prediction results are fused through a Transformer model or an LSTM long short-term memory network.

[0123] The embodiment of the application obtains the point feature map, the line feature map, the plane feature map and the object feature map of the scene to be reconstructed through multi-task learning, obtains a first prediction result for reconstructing the scene to be reconstructed according to the above feature maps, fuses multiple first prediction results obtained according to multiple frames of the target image to obtain a second prediction result, and improves the reconstruction accuracy of the three-dimensional scene.

[0124] Figure 5 The structure of the three-dimensional scene reconstruction device provided by the embodiment of the application is shown, and the specific implementation of the three-dimensional scene reconstruction device is not limited in the embodiment of the application.

[0125] As Figure 5As shown, the reconstruction device of the three-dimensional scene can include a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0126] The processor 402, the communications interface 404, and the memory 406 can communicate with each other through the communications bus 408. The communications interface 404 is configured to communicate with network elements such as clients or other servers. The processor 402 is configured to execute the program 410, and in particular can execute the related steps in the above-described reconstruction method for a three-dimensional scene.

[0127] In particular, the program 410 can include program code including computer-executable instructions.

[0128] The processor 402 can be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present application. The one or more processors included in the reconstruction device of the three-dimensional scene can be the same type of processor, such as one or more CPUs; or can be different types of processors, such as one or more CPUs and one or more ASICs.

[0129] The memory 406 is configured to store the program 410. The memory 406 can include a high-speed RAM memory, and can also include a non-volatile memory such as at least one disk memory.

[0130] The program 410 can be invoked by the processor 402 to cause the reconstruction device of the three-dimensional scene to perform the following operations:

[0131] Obtain a target image, the target image including scene information to be reconstructed;

[0132] Based on the target image, obtain a point feature map, a line feature map, a surface feature map, and an object feature map of the scene to be reconstructed by multi-task learning, the point feature map including point information in the scene to be reconstructed, the line feature map including line information in the scene to be reconstructed, the surface feature map including surface information in the scene to be reconstructed, and the object feature map including object information in the scene to be reconstructed;

[0133] Determine relationships between the points, lines, surfaces, and objects in the feature maps according to the point feature map, the line feature map, the surface feature map, and the object feature map, to obtain a first prediction result for reconstructing the scene to be reconstructed;

[0134] fuse the plurality of first prediction results obtained according to the plurality of target images to obtain a second prediction result.

[0135] In an optional manner, the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed are obtained based on the target image through multi-task learning, and the method comprises the following steps:

[0136] input the target image into a first feature extraction model;

[0137] determine the convolution kernel of each node in the first feature extraction model, and determine the link path between nodes;

[0138] perform multi-layer convolution operation according to the determined convolution kernel of each node and the link path between nodes to obtain a plurality of feature maps of different scales;

[0139] input the feature maps of different scales into a point-line detection model, a plane detection model, a depth estimation model, a normal estimation model and a semantic segmentation model to obtain the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed.

[0140] In an optional manner, the method further comprises:

[0141] In the training process of the point-line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model, the loss functions of the point-line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model are jointly optimized.

[0142] In an optional manner, the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed are obtained based on the target image through multi-task learning, and the method comprises the following steps:

[0143] input the target image into a second feature extraction model to obtain the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed, wherein the second feature extraction model is obtained through multi-task learning on a plurality of training images labeled with points, lines, surfaces and objects.

[0144] In an optional manner, the relationship between the points, lines, surfaces and objects in each feature map is determined based on the point feature map, the line feature map, the surface feature map and the object feature map, and the method further comprises the following steps:

[0145] perform edge detection based on the point features and line features in the point feature map and the line feature map to obtain edge features of the three-dimensional scene;

[0146] According to the two-dimensional object features in the object feature map and the edge features, a plane layout scheme and an object three-dimensional bounding box scheme are generated;

[0147] For each object feature, the optimal N object three-dimensional bounding box schemes are selected according to the object three-dimensional bounding box scheme, N being an integer and N>0;

[0148] An object pose defined by an infinite plane and a 9-degree-of-freedom parameter is adopted to define a new parameter;

[0149] According to the plane layout scheme and the optimal N object three-dimensional bounding box schemes, the constraints between objects and planes and the constraints between points and planes are obtained;

[0150] According to the constraints between objects and planes and the constraints between points and planes, point association, line association, plane association and object association are performed on the point feature map, the line feature map, the plane feature map and the object feature map.

[0151] In an optional manner, the optimal N object three-dimensional bounding box schemes are selected according to the plane layout scheme and the object three-dimensional bounding box scheme for each object feature, further comprising:

[0152] A binary random variable x is defined for each object three-dimensional bounding box scheme i ∈{0,1}, the binary random variable is used to mark whether the object three-dimensional bounding box scheme is selected;

[0153] The binary random variable is optimized so that the calculation result of the following formula is the minimum value, and the selected N object three-dimensional bounding box schemes are obtained:

[0154]

[0155] Wherein, is a single-point potential energy, is a pair of point potential energy, is a high-order potential energy of a c group.

[0156] In an optional manner, the second prediction result is obtained by fusing the plurality of first prediction results obtained according to the plurality of target images.

[0157] The number of frames of the selected plurality of target images is set according to a time window, and the fusion of the plurality of first prediction results is performed by a Transformer model or an LSTM long short-term memory network.

[0158] The embodiment of the present application obtains the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed through multi-task learning, obtains the first prediction result of reconstructing the scene to be reconstructed according to the above feature maps, and fuses the plurality of first prediction results obtained according to the plurality of target images to obtain the second prediction result, thereby improving the reconstruction accuracy of the three-dimensional scene.

[0159] The embodiment of the present application provides a computer readable storage medium, the storage medium stores at least one executable instruction, the executable instruction enables the three-dimensional scene reconstruction device to perform the three-dimensional scene reconstruction method in any method embodiment when the three-dimensional scene reconstruction device runs.

[0160] The embodiment of the present application provides a three-dimensional scene reconstruction device for performing the three-dimensional scene reconstruction method.

[0161] The embodiment of the present application provides a computer program, the computer program can be called by a processor to enable the three-dimensional scene reconstruction device to perform the three-dimensional scene reconstruction method in any method embodiment.

[0162] The embodiment of the present application provides a computer program product, the computer program product comprises a computer program stored on a computer readable storage medium, the computer program comprises program instructions, when the program instructions run on a computer, enable the computer to perform the three-dimensional scene reconstruction method in any method embodiment.

[0163] The algorithms or displays provided herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with these teachings, based on the description as set forth above. In addition, the present embodiments are not inherently related to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the present teachings as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0164] In the specification provided herein, a large number of specific details are described. However, it can be understood that the embodiments of the present application can be practiced without these specific details. In some examples, well-known methods, structures and techniques are not shown in detail in order not to obscure the understanding of the specification.

[0165] Similarly, it is to be understood that the embodiments of the application can be adapted to any of the various aspects of the application described herein, and that the description of the exemplary embodiments of the application given above, does not limit the scope of the application, but that the scope of the application is to be given that broadest interpretation which will encompass all compatible embodiments including those now known or which will become known in the future.

[0166] Those skilled in the art will appreciate that modules in the apparatuses in the embodiments can be adapted and placed in one or more apparatuses other than the embodiments. Modules or units or components in the embodiments can be combined into one module or unit or component, and can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of such features and / or processes or units are mutually exclusive, all combinations of all features disclosed in this specification (including the accompanying claims, abstract and drawings) and all processes or units of any methods or apparatuses so disclosed can be adopted in any combination. Unless explicitly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstract and drawings) can be replaced by alternative features that provide the same, equivalent or similar functionality.

[0167] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that one skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unit claim, several devices can be listed with a comma. These are regarded as open limits, i.e. a separate unit can consist of one device alone - and a separate unit can consist of several devices. The use of the words 'first','second' and 'third' and the like does not imply any ordering, but rather are used for naming purposes only. The use of the terms 'first','second' and 'third' and the like is merely intended to differentiate between two entities, with the entity referred to as the first being treated as different and distinct from the entity referred to as the second, and so on for the third. The use of the terms 'first','second', 'third' and the like does not imply any ordering. The steps of any of the methods disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

Claims

1. A method of reconstruction of a three-dimensional scene, characterized in that, The method comprises: acquiring a target image, the target image comprising scene information to be reconstructed; based on the target image, obtaining a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed through multi-task learning, the point feature map comprising point information in the scene to be reconstructed, the line feature map comprising line information in the scene to be reconstructed, the surface feature map comprising surface information in the scene to be reconstructed, and the object feature map comprising object information in the scene to be reconstructed; determining the relationship between points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map, to obtain a first prediction result for reconstructing the scene to be reconstructed; fusing a plurality of first prediction results obtained from a plurality of target images to obtain a second prediction result; the method further comprises: in the training process of the point line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model, the loss functions of the point line detection model, the plane detection model, the depth estimation model, the normal estimation model and the semantic segmentation model are jointly optimized. ​ ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ ​ ​ ​ ​ 3. The method of claim 2, wherein, ​ ​ 4. The method of claim 1, wherein, The point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed are obtained through multi-task learning based on the target image, and the method comprises the following steps: The target image is input into a second feature extraction model to obtain the point feature map, the line feature map, the surface feature map and the object feature map of the scene to be reconstructed, wherein the second feature extraction model is obtained through multi-task learning on a plurality of training images with marked point, line, surface and object features.

5. The method of claim 1, wherein, The optimal N object bounding box schemes are selected according to the plane layout scheme and the object bounding box scheme for each object feature, and the method further comprises the following steps: defining a binary random variable for each of the object three-dimensional bounding box schemes , the binary random variable marking whether the object three-dimensional bounding box scheme is selected; The binary random variable is optimized so that the calculation result of the following formula is the minimum value, and the selected N object bounding box schemes are obtained: + wherein, is a single-point potential energy, is a pair-wise point potential energy, is a high-order potential energy of the c-group.

6. The method of claim 1, wherein, The second prediction result is obtained by fusing a plurality of first prediction results obtained according to a plurality of target images, and the method further comprises the following steps: The number of frames of the selected target images is set according to a time window, and the plurality of first prediction results are fused through a Transformer model or an LSTM long short-term memory network.

7. A reconstruction apparatus of a three-dimensional scene, characterized by The device comprises: An acquisition module is configured to acquire a target image, wherein the target image comprises scene information to be reconstructed; A obtaining module is configured to obtain a point feature map, a line feature map, a surface feature map and an object feature map of the scene to be reconstructed through multi-task learning based on the target image, wherein the point feature map comprises point information in the scene to be reconstructed, the line feature map comprises line information in the scene to be reconstructed, the surface feature map comprises surface information in the scene to be reconstructed, and the object feature map comprises object information in the scene to be reconstructed; A determining module is configured to determine the relationship between points, lines, surfaces and objects in each feature map according to the point feature map, the line feature map, the surface feature map and the object feature map, and obtain a first prediction result for reconstructing the scene to be reconstructed; A fusing module is configured to fuse a plurality of first prediction results obtained according to a plurality of target images to obtain a second prediction result. The relationship between the points, lines, surfaces and objects in each feature map is determined according to the point feature map, the line feature map, the surface feature map and the object feature map, and the method further comprises the following steps: Edge features of the three-dimensional scene are obtained through edge detection based on point features and line features in the point feature map and the line feature map; A plane layout scheme and an object bounding box scheme are generated according to two-dimensional object features in the object feature map and the edge features, wherein the edge features are edge features close to a ground wall partition boundary; For each object feature, the optimal N object bounding box schemes are selected according to the object bounding box scheme, and N is 10-20; A new parameter is defined by using an infinite plane and an object pose defined by 9 degrees of freedom parameters; Constraints between objects and planes and constraints between points and planes are obtained according to the plane layout scheme and the optimal N object bounding box schemes; According to the constraint between the object and the plane and the constraint between the point and the plane, the point feature map, the line feature map, the plane feature map and the object feature map are associated in points, lines, planes and objects.

8. A reconstruction device of a three-dimensional scene, characterized in that Comprise: a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete the communication among each other through the communication bus; The memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operation of the three-dimensional scene reconstruction method in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, and the executable instruction runs on the three-dimensional scene reconstruction device, so that the three-dimensional scene reconstruction device executes the operation of the three-dimensional scene reconstruction method in any one of claims 1-6.

Images