Training method for a three-dimensional scene reconstruction apparatus for a multi-camera system

Abstract

The present disclosure provides a training method of a three-dimensional scene reconstruction device for a multi-camera system, comprising: processing a multi-camera image sequence of the multi-camera system using an SFM method to obtain a sparse depth map under each camera perspective, pre-training a depth estimation network based on the sparse depth map to obtain a pre-trained depth estimation network; performing self-supervised training on the depth estimation network and a pose estimation network to output a pose of the multi-camera and a depth map of the multi-camera; and training a multi-view TSDF estimation network based on the multi-camera image sequence of the multi-camera system, the pose of the multi-camera and the depth map of the multi-camera for three-dimensional scene reconstruction. The present disclosure also provides a three-dimensional scene reconstruction device, a three-dimensional scene reconstruction system and a readable storage medium.

Description

Technical Field

[0001] This disclosure relates to the field of 3D reconstruction technology, and more particularly to a training method, a 3D scene reconstruction device, a 3D scene reconstruction system, and a readable storage medium for a 3D scene reconstruction device for a multi-camera system. Background Technology

[0002] 3D reconstruction technology is one of the key technologies for physical environment perception. Generally speaking, image-based 3D reconstruction is a method that reverse-engineers the 3D depth information of a scene or object from multiple images and reconstructs a 3D model of the object or scene with a sense of depth.

[0003] To perceive richer environmental information, multiple cameras are often rigidly coupled to form a multi-camera system, which can simultaneously capture multiple images from different perspectives, such as the surround-view multi-camera group in intelligent vehicles. 3D reconstruction based on multi-camera systems has broad application prospects in remote sensing mapping, robot navigation, autonomous driving, augmented reality, virtual reality, digital twins, smart cities, and other fields.

[0004] Currently, image-based 3D reconstruction methods are mainly divided into three major technical directions: traditional 3D reconstruction based on multi-view geometry, 3D reconstruction based on deep learning, and viewpoint synthesis based on neural radiation fields.

[0005] While traditional 3D reconstruction methods based on multi-view geometry are relatively mature, with numerous open-source algorithms and commercial software available, they often process multiple images from a single camera or scattered images from different sources as input. There are no specific methods for handling image sequences from multi-camera systems, neglecting the inherent spatial constraints between multiple cameras. Existing deep learning-based 3D reconstruction methods mostly incorporate individual or partial processes from traditional methods into deep neural networks, using existing 3D reconstruction models to supervise network training. Although learning can generate 3D models superior to traditional methods, these networks often require users to provide ground truth values ​​of the 3D model to supervise training, which is often costly or even impossible in reality. In recent years, viewpoint synthesis based on neural radiation fields has offered another approach to 3D reconstruction. Instead of reconstructing 3D model entities, it learns radiation field distributions to generate images from arbitrary viewpoints. While viewpoint synthesis offers stunning visual reconstruction results, the lack of 3D model entities limits its application in many scenarios requiring simulation interaction and secondary processing.

[0006] The following are some existing technical solutions.

[0007] Technical Solution 1: Publication number CN108564617A, entitled "3D Reconstruction Method, Apparatus, VR Camera, and Panoramic Camera for Multi-View Cameras," discloses a 3D reconstruction method, apparatus, computer-readable storage medium, VR camera, and panoramic camera for multi-view cameras. The 3D reconstruction method in Technical Solution 1 employs a traditional reconstruction process, performing 3D reconstruction based on keyframes from videos captured by all cameras to generate point clouds and camera poses. Furthermore, it determines the scale of the point cloud based on the known physical distances between adjacent cameras. Although it provides a 3D reconstruction method based on multi-view cameras, its design is relatively simple, merely adding a scale recovery module after the traditional 3D reconstruction module. That is, after generating the point cloud and camera poses in the traditional 3D reconstruction process, it uses the spatial constraints of the multi-view cameras to supplement the physical scale of the point cloud. However, camera pose recovery and multi-view point cloud generation are crucial steps in 3D reconstruction. The 3D reconstruction module in Technical Solution 1 only uses traditional 3D reconstruction steps and does not incorporate the inherent spatiotemporal constraints between multi-view cameras. Therefore, the reconstruction effect of this method depends on the effect of traditional 3D reconstruction methods, and the relationship of multiple cameras is only used to restore scale. When faced with complex large-scale scenes, the reconstruction accuracy may not be high, and the surface of the reconstructed model may show layering or missing parts.

[0008] Technical Solution 2: Published under CN110070615A and entitled "A Panoramic Visual SLAM Method Based on Multi-Camera Collaboration," this invention discloses a multi-camera collaborative SLAM method. Technical Solution 2 combines the motion model of the motion platform containing the multiple cameras to estimate the current pose of the SLAM system in the global coordinate system. It then matches the feature points of the current frame image obtained by each camera with the current local map points to obtain matching point pairs. Furthermore, a machine learning-based loop closure detection method is used to optimize and update the global map. Although Technical Solution 2 utilizes the collaborative relationship of multiple cameras to estimate pose and optimize the map with scale information, this method generates a point cloud based on feature matching points from multiple camera images at the same time, and then uses feature point matching between multiple frames to accurately determine the pose. This requires a sufficiently large overlap of the field of view between the multiple cameras, and a sufficient number of feature matching points within the overlap. This results in poor robustness; it cannot function properly when encountering sparse or repetitive features such as building walls. In addition, the point cloud generated by this method is relatively sparse, and the generated global map is also a discretized sparse point cloud, unable to generate a dense 3D map model.

[0009] Technical Solution 3: Publication number CN114092633A, entitled "A 3D Reconstruction Device and Method Based on Multi-Camera Structure," discloses a vision-wheel velocimeter tightly coupled framework to achieve high-precision and robust state estimation and reconstruction. Technical Solution 3 constrains the camera's degrees of freedom by limiting partial motion consistency of the camera through a 3D reconstruction device, and uses factor graph optimization to more accurately estimate camera motion through the proposed tightly coupled framework, resulting in more accurate reconstruction results. While Technical Solution 3 can accurately reconstruct scenes, it requires multiple depth cameras mounted on a specific 3D reconstruction device, leading to higher costs. Furthermore, the 3D reconstruction device uses wheel velocimeter feedback information to control a servo motor in a closed-loop manner to maintain uniform rotation, limiting its applicability.

[0010] Technical Solution 4: Published under CN113838191A and entitled "A 3D Reconstruction Method Based on Attention Mechanism and Monocular Multi-view," this invention discloses a deep learning-based monocular 3D reconstruction method. Technical Solution 3 reconstructs sparse point clouds from monocular image sequences using feature point extraction and matching. Then, camera pose information, camera intrinsic parameters, the structure information of the scene under test, and the image are input into a pre-defined attention mechanism-based monocular multi-view reconstruction network to obtain several depth estimation maps of the scene under test. Finally, these depth estimation maps are fused to obtain a dense point cloud model. However, Technical Solution 3 lacks scale information in its sparse point cloud reconstruction of monocular image sequences, and the attention mechanism-based depth map estimation network only predicts depth maps from monocular images. The predicted adjacent depth maps lack correlation, and the depths of different views are inconsistent, making the reconstruction results prone to layering or dispersion.

[0011] Technical Solution 5: The journal article "NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video" proposes a deep learning-based TSDF estimation network. Technical Solution 5 directly generates a truncated symbolic field by segmenting the image sequence using a neural network, then uses a TSDF fusion module to guide the network to fuse the segments before and after, updating the global truncated symbolic field, and finally generating a 3D model. This method can capture the local smoothness prior and global shape prior of the 3D surface during sequential surface reconstruction, thereby achieving accurate, consistent, and real-time surface reconstruction. However, this estimation network is a supervised network, requiring the user to provide the ground truth of the 3D model to supervise the network training. In reality, obtaining the ground truth is often costly or even impossible, limiting its practical application scenarios.

[0012] Technical solutions 1-3 are existing multi-camera collaborative mapping methods. All three methods rely on feature point extraction and matching, and point cloud generation in traditional 3D reconstruction processes to recover 3D structural information. Furthermore, the reconstruction results of technical solutions 1 and 2 are merely sparse point clouds. Specifically, technical solution 1 only utilizes the collaborative relationship between multiple cameras to supplement the physical scale of the generated point cloud, without leveraging the inherent spatiotemporal constraints of multi-camera systems during point cloud generation. While technical solution 2 utilizes the collaborative relationship between multiple cameras to estimate pose and optimize maps with scale information, this technique requires a sufficiently large overlap of fields of view among the cameras, and a sufficient number of feature matching points within the overlap. This results in poor robustness and a very sparse reconstructed scene; it cannot function properly when encountering sparse or repetitive features such as building walls. Although technical solution 3 can accurately reconstruct the scene, the hardware cost is high, requiring multiple depth cameras mounted on a specific 3D reconstruction device. Moreover, this technique restricts the motion of the reconstruction device, requiring it to rotate at a constant speed under a given motor drive, limiting the device's application scenarios.

[0013] Technical solutions 4 and 5 represent existing deep learning-based 3D reconstruction methods, both of which rely on monocular vision. Technical solution 4 follows the most common approach in deep learning-based 3D reconstruction: predicting depth estimates of the scene and then fusing these estimates to obtain a dense point cloud model. These methods primarily predict depth maps from monocular images, resulting in a lack of correlation between adjacent predictions and inconsistent depths across different views, leading to layered or scattered reconstruction results. Technical solution 5 proposes a supervised TSDF estimation network based on deep learning. This technique directly predicts the truncated symbolic field through a neural network, capturing local smoothness priors and global shape priors of the 3D surface during sequential reconstruction, thus achieving accurate, consistent, and real-time surface reconstruction. However, this estimation network is a supervised network, which requires users to provide the ground truth of the 3D model to supervise the training of the network. In reality, obtaining the ground truth is often expensive or even impossible, limiting its practical application scenarios. In addition, there are some deep learning-based 3D reconstruction methods on the market that take multi-source scattered images as input, but no training method for 3D scene reconstruction devices for multi-camera systems has been found.

[0014] In summary, existing 3D reconstruction techniques are primarily designed for monocular vision, binocular vision, or scattered images from different sources, and cannot fully utilize the spatiotemporal information of image sequences from multi-camera systems. Furthermore, these methods require additional sensor information (such as LiDAR and GPS positioning) to reconstruct 3D scenes with real-world physical scale. Currently, there is no method specifically for multi-camera systems that can reconstruct dense 3D models with real-world scale simply by inputting multi-camera image sequences. Summary of the Invention

[0015] This disclosure provides a training method, a 3D scene reconstruction device, a 3D scene reconstruction system, and a readable storage medium for a 3D scene reconstruction device for a multi-camera system.

[0016] According to one aspect of this disclosure, a training method for a 3D scene reconstruction apparatus for a multi-camera system is provided, comprising: S100. Use the SFM method to process the multi-camera image sequence of the multi-camera system to obtain sparse depth maps from the perspective of each camera. Based on the sparse depth maps, pre-train the depth estimation network to obtain the pre-trained depth estimation network. S200, a joint depth estimation network, and a pose estimation network are self-supervised to output poses and depth maps for multiple cameras. S300. The multi-camera image sequence, pose of the multi-camera system, and depth map of the multi-camera system are used to train a multi-view TSDF estimation network for 3D scene reconstruction.

[0017] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system disclosed herein, the SFM method is used to process multi-camera image sequences of the multi-camera system to obtain sparse depth maps from each camera's viewpoint, including: S110. Use the SFM method to extract and match feature points in the multi-camera image sequence acquired by the multi-camera system, and use the obtained multiple sets of matched feature points to solve the eigenvalue matrix to obtain the relative pose of each camera. S120. Collect matching feature points at overlapping fields of view in multi-camera image sequences, generate sparse point clouds based on the matching feature points at overlapping fields of view and the relative poses of each camera, and convert them into sparse depth maps under the view of each camera.

[0018] A training method for a 3D scene reconstruction apparatus for a multi-camera system according to at least one embodiment of the present disclosure, comprising pre-training a depth estimation network based on a sparse depth map, including: The sparse depth map is used as supervision to pre-train the depth estimation network.

[0019] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system disclosed herein, a depth estimation network and a pose estimation network are jointly trained under self-supervised supervision to output the poses and depth maps of the multiple cameras, including: S210. Construct the spatiotemporal relationship between the images acquired by each camera in the target frame and the source frame based on the spatiotemporal context relationship between the images of multiple cameras, so as to obtain the reprojection error between frames. S220. Based on the intra-frame multi-camera pose consistency constraint, obtain the intra-frame multi-camera pose consistency error. S230. The depth estimation network and the pose estimation network are trained by minimizing the reprojection error and minimizing the intra-frame multi-camera pose consistency error.

[0020] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system of the present disclosure, S210, constructing the spatiotemporal relationship between the images acquired by each camera in the target frame and the source frame based on the spatiotemporal context relationship between the images of the multi-camera system, in order to obtain the inter-frame reprojection error, including: S211. Based on the spatial constraints between cameras and the temporal constraints between frames of the image sequence acquired by each camera, obtain the spatiotemporal relationship between the image acquired by the i-th camera and the image acquired by the j-th camera in the target frame and the source frame. S212. Based on the aforementioned spatiotemporal relationship, the images in adjacent frames are... Pixels projected onto the target image To obtain a composite image ; S213, Based on the target image and the synthesized image The reprojection error is obtained.

[0021] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system of the present disclosure, S220, based on intra-frame multi-camera pose consistency constraints, to obtain intra-frame multi-camera pose consistency error, including: S221. Based on the intra-frame multi-camera pose consistency constraint, the poses of each camera other than the target camera predicted by the pose estimation network are transformed into the coordinate system of the target camera. S222, Construct a weighted translation function and rotation consistency loss function ; S223, Weighted Translation Function and rotation consistency loss function An intra-frame multi-camera pose consistency function is constructed to balance spatial constraints, so that the rotation and translation predicted by each camera in the same coordinate system tend to be consistent.

[0022] According to a training method for a 3D scene reconstruction apparatus for a multi-camera system according to at least one embodiment of the present disclosure, S230, training a depth estimation network and a pose estimation network by minimizing the reprojection error and minimizing the intra-frame multi-camera pose consistency error includes: S231. Randomly select three consecutive frames of multi-camera images from a multi-camera image sequence; S232. Input the multi-camera images at the intermediate frame time into the depth estimation network and output the depth map of each camera at the intermediate frame time; concatenate three frames from the same camera and input them into the pose estimation network respectively, and output the relative pose of the frame before the intermediate frame and the frame after the intermediate frame. S233. Calculate the reprojection error loss function based on the depth map of each camera at the intermediate frame time to minimize the reprojection error, and calculate the intra-frame multi-camera pose consistency function based on the relative pose of the previous frame and the next frame of the intermediate frame to minimize the intra-frame multi-camera pose consistency error. S234. Backpropagation is used to update the network parameters of the depth estimation network and the pose estimation network. S235. Determine if the number of iterations has been reached. If the number of iterations has not been reached, repeat steps S231 to S234 until the number of iterations is reached.

[0023] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system of the present disclosure, S300, a multi-view TSDF estimation network is trained based on the multi-camera image sequence, the pose of the multi-cameras, and the depth map of the multi-cameras, including: S310. Construct RGB-D image sequences of known poses of each camera based on the poses and depth maps of each camera to obtain the initial truncated symbol field of the multi-view TSDF estimation network. S320. Use the average pose of each camera in a unified coordinate system as the overall pose and filter the overall keyframes based on the translation and rotation of the overall pose. S330. A preset number of overall keyframes are used as an overall keyframe group, and the overall poses corresponding to each overall keyframe are input together into the multi-view TSDF estimation network to predict the global truncated symbol field. S340. The multi-view TSDF estimation network is trained under supervision using the initial truncated symbol field, and finally outputs the global truncated symbol field.

[0024] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system of the present disclosure, S330, a preset number of overall keyframes are used as an overall keyframe group, and the overall poses corresponding to each overall keyframe are input together into the multi-view TSDF estimation network to predict the global truncated symbol field, including: S331. Perform three-level feature extraction on the camera images of the overall keyframe group to obtain the three-level feature maps of each camera. S332. Based on the pose of each camera, the feature maps of each camera at each level are back-projected to aggregate them into three-dimensional feature bodies corresponding to each level in the temporal sequence of the overall keyframe group and between intra-frame cameras. S333. The three-dimensional feature volume of the multi-camera generated based on the current overall keyframe group is recursively fused with the global geometric feature volume generated by the previous overall keyframe group at three levels to obtain the global geometric feature volume of the current overall keyframe group, so as to further predict the global truncated symbol field.

[0025] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system of the present disclosure, S333, the 3D feature volume of the multi-camera system generated based on the current overall keyframe group is recursively locally fused with the global geometric feature volume generated by the previous overall keyframe group at three levels to obtain the global geometric feature volume of the current overall keyframe group, so as to further predict the global truncated symbol field, including: S3331. Perform 3D feature extraction on the three-dimensional feature volume of the current overall keyframe group at each level to extract the local geometric feature volume of the three-dimensional feature volume at each level; S3332. Based on the global geometric feature body of the previous global keyframe group, extract the local geometric feature bodies corresponding to each local geometric feature body of the current global keyframe group. Then, merge each local geometric feature body of the current global keyframe group with each local geometric feature body of the previous global keyframe group according to the hierarchy to generate new local geometric feature bodies of the current global keyframe group, thereby obtaining the global geometric feature body of the current global keyframe group. S3333, Based on the new local feature volumes of the current overall keyframe group at each level, predict the TSDF volume, i.e., the truncated symbol field, at each level. S3334. Concatenate the truncated symbol field of the first level and the truncated symbol field of the second level obtained from the second level and the truncated symbol field obtained from the third level with the truncated symbol field of the third level after nearest neighbor interpolation upsampling, and repeat steps S3331 to S3333. S3335. Replace the corresponding voxels of the global truncated symbol field with the truncated symbol field output from the third level to generate the final global truncated symbol field.

[0026] According to a training method for a 3D scene reconstruction apparatus for a multi-camera system according to at least one embodiment of the present disclosure, the local feature volume is used to predict the TSDF volume of the layer through an MLP layer, wherein each voxel in the TSDF volume contains an occupancy value o and a truncation sign value x.

[0027] According to at least one embodiment of the training method for a 3D scene reconstruction apparatus for a multi-camera system of the present disclosure, S340, the multi-view TSDF estimation network is trained under supervision using the initial truncated symbolic field, and a global truncated symbolic field is finally output, including: S341. On the occupancy value o and truncated symbol value x predicted at the three levels from coarse to fine, supervision is performed based on the occupancy value and truncated symbol value of the region corresponding to the initial truncated symbol field, respectively. S342. For the truncated symbol values ​​in the global truncated symbol field obtained after updating the local truncated symbol field, based on the truncated symbol values ​​of the corresponding region of the initial truncated symbol field, perform a logarithmic transformation on the truncated symbol values ​​of the two to calculate the L1 loss of the two truncated symbol values, so as to update the parameters of the multi-view TSDF estimation network.

[0028] According to another aspect of this disclosure, a three-dimensional scene reconstruction apparatus for a multi-camera system is provided, comprising: A depth estimation network module is trained based on a sparse depth map, which is a sparse depth map of each camera view obtained by processing the multi-camera image sequence of a multi-camera system using the SFM method. The pose estimation network module is a pose estimation network module that is trained in conjunction with the depth estimation network module under self-supervised training. The self-supervised training enables the depth estimation network module and the pose estimation network module to output depth maps and poses of multiple cameras. A multi-view TSDF estimation network module, wherein the multi-view TSDF estimation network module is trained based on the multi-camera image sequence, the pose of the multi-camera and the depth map of the multi-camera system, for use in 3D scene reconstruction.

[0029] According to another aspect of this disclosure, a three-dimensional scene reconstruction system is provided, comprising: A multi-camera system comprising multiple cameras fixedly mounted on a vehicle to acquire image sequences of the vehicle's surrounding environment; A synchronous camera controller, which controls multiple cameras to acquire image sequences synchronously; The on-board memory stores execution instructions; A processor, which processes image sequences acquired by the multi-camera system by executing execution instructions stored in the memory to implement a training method for a three-dimensional scene reconstruction apparatus for a multi-camera system according to any embodiment of the present disclosure.

[0030] According to another aspect of this disclosure, a three-dimensional scene reconstruction system is provided, comprising: A multi-camera system comprising multiple cameras fixedly mounted on a vehicle to acquire image sequences of the vehicle's surrounding environment; A synchronous camera controller, which controls multiple cameras to acquire image sequences synchronously; A memory that stores, in the form of a computer program, the three-dimensional scene reconstruction apparatus of any embodiment of the present disclosure; The processor, when the computer program is executed by the processor, performs 3D scene reconstruction based on the image sequence acquired by the multi-camera system.

[0031] According to at least one embodiment of the three-dimensional scene reconstruction system of this disclosure, the memory is an in-vehicle memory and the processor is an in-vehicle processor.

[0032] According to another aspect of this disclosure, a readable storage medium is provided, wherein executable instructions are stored therein, which, when executed by a processor, are used to implement a training method for a three-dimensional scene reconstruction apparatus for a multi-camera system according to any embodiment of this disclosure.

[0033] According to another aspect of this disclosure, a readable storage medium is provided that stores a three-dimensional scene reconstruction apparatus of any embodiment of this disclosure in the form of a computer program. Attached Figure Description

[0034] The accompanying drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0035] Figure 1 This is a flowchart illustrating a training method for a three-dimensional scene reconstruction apparatus for a multi-camera system according to one embodiment of the present disclosure.

[0036] Figure 2 An embodiment of the present disclosure is shown of an in-vehicle surround-view multi-camera system consisting of four monocular cameras.

[0037] Figure 3 The temporal relationships and spatial constraints of an in-vehicle surround-view multi-camera system according to one embodiment of this disclosure are shown.

[0038] Figure 4 A flowchart illustrating a method for obtaining a sparse depth map according to one embodiment of this disclosure is shown.

[0039] Figure 5A schematic diagram illustrating the process of self-supervised training of a joint depth estimation network and a pose estimation network according to one embodiment of this disclosure is shown.

[0040] Figure 6 A flowchart illustrating a method for obtaining reprojection error according to one embodiment of this disclosure is shown.

[0041] Figure 7 This is a flowchart illustrating a method for obtaining intra-frame multi-camera pose consistency error according to one embodiment of this disclosure.

[0042] Figure 8 The diagram illustrates a flowchart of training a depth estimation network and a pose estimation network by minimizing reprojection error and minimizing intra-frame multi-camera pose consistency error according to one embodiment of the present disclosure.

[0043] Figure 9 A flowchart illustrating a method for training a multi-view TSDF estimation network according to one embodiment of this disclosure is shown.

[0044] Figure 10 A complete flowchart of a self-supervised training method for a multi-view TSDF estimation network according to one embodiment of the present disclosure is shown.

[0045] Figure 11 This is a schematic block diagram of a three-dimensional scene reconstruction device that employs a hardware implementation of a processing system, according to one embodiment of this disclosure.

[0046] Explanation of reference numerals in the attached figures 1000 3D Scene Reconstruction Device 1002 Depth Estimation Network Module 1004 Pose Estimation Network Module 1006 Multi-view TSDF Estimation Network Module 1100 bus 1200 processor 1300 memory 1400 Other circuits. Detailed Implementation

[0047] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the disclosure. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present disclosure are shown in the accompanying drawings.

[0048] It should be noted that, where there is no conflict, the embodiments and features described in this disclosure can be combined with each other. The technical solutions of this disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0049] Unless otherwise stated, the exemplary implementations / embodiments shown are to be understood as providing exemplary features of various details that provide ways in which the technical concepts of this disclosure can be implemented in practice. Therefore, unless otherwise stated, the features of various implementations / embodiments may be additionally combined, separated, interchanged and / or rearranged without departing from the technical concepts of this disclosure.

[0050] The use of crosshairs and / or shading in the accompanying drawings is generally used to clarify the boundaries between adjacent components. Thus, unless otherwise stated, the presence or absence of crosshairs or shading does not convey or indicate any preference or requirement for the specific material, material properties, dimensions, proportions, commonalities between the illustrated components, or any other characteristics, properties, etc., of the components. Furthermore, in the accompanying drawings, the dimensions and relative dimensions of components may be exaggerated for clarity and / or descriptive purposes. When exemplary embodiments can be implemented differently, a specific process sequence may be performed in a different order than that described. For example, two consecutively described processes may be performed substantially simultaneously or in the reverse order of their description. Furthermore, the same reference numerals denote the same components.

[0051] When a component is referred to as being "on" or "above" another component, "connected to," or "joined to" another component, the component may be directly on, directly connected to, or directly joined to the other component, or there may be intermediate components. However, when a component is referred to as being "directly on" another component, "directly connected to," or "directly joined to" another component, there are no intermediate components. Therefore, the term "connection" can refer to a physical connection, an electrical connection, etc., and may or may not have intermediate components.

[0052] The terminology used herein is for the purpose of describing particular embodiments and is not intended to be limiting. As used herein, unless the context clearly indicates otherwise, the singular forms “a” and “the” are intended to include the plural forms as well. Furthermore, when the terms “comprising” and / or “including” and variations thereof are used in this specification, it indicates the presence of the stated features, integrals, steps, operations, parts, components, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, parts, components, and / or groups thereof. It should also be noted that, as used herein, the terms “substantially,” “about,” and other similar terms are used as approximate terms rather than as terms of degree, thus explaining the inherent biases in measurements, calculated values, and / or provided values ​​that would be recognized by one of ordinary skill in the art.

[0053] The following text combines Figures 1 to 11The training method and the 3D scene reconstruction device for a multi-camera system disclosed herein are described in detail.

[0054] Figure 1 This is a flowchart illustrating a training method for a three-dimensional scene reconstruction apparatus for a multi-camera system according to one embodiment of the present disclosure.

[0055] refer to Figure 1 The training method for a 3D scene reconstruction device for a multi-camera system disclosed herein includes: S100. The multi-camera image sequence of the multi-camera system is processed using the SFM method (Structure from motion) to obtain sparse depth maps from the perspective of each camera (for example, four cameras will yield four sparse depth maps). The depth estimation network is pre-trained based on the sparse depth maps to obtain the pre-trained depth estimation network (Monodepth2 is selected as the depth estimation network in this disclosure). S200, a joint depth estimation network, and a pose estimation network are self-supervised to output poses and depth maps for multiple cameras. The S300, multi-camera image sequences based on a multi-camera system, multi-camera poses, and multi-camera depth maps are used to train a multi-view TSDF estimation network for 3D scene reconstruction.

[0056] Among them, SFM (Structure from Motion) described in this disclosure is a three-dimensional reconstruction method, which belongs to the prior art and is used to realize 3D reconstruction from motion, that is, to infer 3D information from 2D images of a time series.

[0057] In some embodiments of this disclosure, the multi-camera system may consist of four monocular cameras fixed to the vehicle and a synchronous camera controller. Figure 2 An embodiment of the present disclosure is shown, comprising an in-vehicle surround-view multi-camera system consisting of four monocular cameras. Figure 3 The temporal relationships and spatial constraints of an in-vehicle surround-view multi-camera system according to one embodiment of this disclosure are shown.

[0058] The driver drives the car, and the synchronous camera controller controls the onboard surround-view multi-camera system to collect image information (image sequence) during the driving process and transmit it to the onboard PC (equipped with a GPU). The PC reconstructs a 3D scene model of the car's driving environment based on the 3D reconstruction apparatus / method disclosed herein.

[0059] Figure 4 A flowchart illustrating a method for obtaining a sparse depth map according to one embodiment of the present disclosure is shown.

[0060] refer to Figure 4 In this disclosure, the SFM (Structure from Motion) method is used to process multi-camera image sequences of a multi-camera system to obtain sparse depth maps from the perspectives of each camera, including: S110. Use the SFM method to extract and match feature points in the multi-camera image sequences acquired by the multi-camera system, and use multiple sets of matched feature points to solve the eigenma matrix to obtain the relative pose of each camera.

[0061] S120. Collect matching feature points at overlapping fields of view in multi-camera image sequences, generate sparse point clouds based on the matching feature points and the poses of each camera, and convert them into sparse depth maps under the viewpoints of each camera.

[0062] In some embodiments of this disclosure, the multi-camera system may include four monocular cameras, i.e., four cameras. In the images with overlapping fields of view among the four cameras, feature points are extracted and matched using the KAZE descriptor to obtain matching feature points. The matching feature points and the poses of each camera obtained by processing the multi-camera images acquired by the multi-camera system using the SFM method are input into the sparse point cloud generation module of Colmap to convert the obtained sparse point cloud into a sparse depth map from the viewpoint of each camera.

[0063] The training method for the 3D scene reconstruction device for a multi-camera system according to the above embodiments includes pre-training the depth estimation network based on a sparse depth map, including: Using sparse depth maps as supervision, the depth estimation network is pre-trained according to the training method of supervised depth estimation networks. Specifically, the obtained sparse depth maps are selected as supervision labels, and the corresponding multi-camera images are used as training data. The training data is fed into the network in batches, performing forward computation layer by layer until the output layer to predict depth values. The depth value output by the current network is compared with the non-zero value at the corresponding position in the sparse depth map, and the loss is calculated using cross-entropy loss. Following the chain rule, the gradient of the loss function with respect to each layer is calculated, and backpropagation is performed to update the weights of the network parameters. These steps are repeated until the loss reaches a set threshold.

[0064] Figure 5 A schematic diagram illustrating the process of self-supervised training of a joint depth estimation network and a pose estimation network according to one embodiment of this disclosure is shown.

[0065] refer to Figure 5 In this disclosure, a joint depth estimation network and a pose estimation network are trained under self-supervised conditions to output multi-camera poses and multi-camera depth maps, including: S210. Construct the spatiotemporal relationship between the images acquired by each camera in the target frame and the source frame based on the spatiotemporal context relationship between the images of multiple cameras, so as to obtain the reprojection error between frames. S220. Based on the intra-frame multi-camera pose consistency constraint, obtain the intra-frame multi-camera pose consistency error. S230. The depth estimation network and pose estimation network are trained by minimizing the reprojection error and minimizing the intra-frame multi-camera pose consistency error.

[0066] For the training method of the 3D scene reconstruction device for a multi-camera system according to the above embodiments, S210, constructing the spatiotemporal relationship between the images acquired by each camera in the target frame and the source frame based on the spatiotemporal context relationship between the images of the multi-camera system, in order to obtain the inter-frame reprojection error, includes: S211. Based on the spatial constraints between cameras and the temporal constraints between frames of the image sequence acquired by each camera, obtain the spatiotemporal relationship between the image acquired by the i-th camera and the image acquired by the j-th camera in the target frame and the source frame. The target frame is the current frame, and the source frame can be multiple frames, such as the frame before and / or the frame after the target frame.

[0067] In some embodiments of this disclosure, the spatiotemporal context relationship between multi-camera images is utilized, that is, based on the inherent spatial constraints (derived from calibrated multi-camera intrinsic and extrinsic parameters) and temporal constraints (which may come from the output of PoseNet) between multiple cameras (this disclosure uses 4 cameras as an example), the spatiotemporal relationship between the image acquired by the i-th camera and the image acquired by the j-th camera in the target frame and the source frame is represented as follows: In the formula, This represents a pixel in the image of camera i at time t. This represents the depth of the image at time t corresponding to camera i (predicted by a depth estimation network). Represents pixels in homogeneous coordinates At a given prediction depth Downward projection as 3D points , and These are the rotation and translation matrices of camera i from time t to time s (t+1 or t-1), respectively (predicted by the pose estimation network). and These are the rotation and translation matrices of camera i relative to camera j (obtained from the calibrated multi-camera extrinsic parameters). Representing 3D points Projection onto the camera's j-plane This represents the composite image of camera i at time t. The pixels in the equation are obtained from the corresponding pixels on the camera's j-plane at time s on the right-hand side.

[0068] In this disclosure, the pose estimation network can be PoseNet from "Unsupervised joint learning of depth and flow using cross-task consistency" to estimate the independent poses of each camera. Following the guidance of the technical solution disclosed herein, those skilled in the art can also use other pose estimation networks, all of which fall within the protection scope of this disclosure.

[0069] S212. Based on spatiotemporal relationships, images from adjacent frames (i.e., adjacent frames in a sequence of images acquired by a multi-camera system, where adjacent frames refer to temporal proximity) are... Pixels projected onto the target image To obtain a composite image ; In some embodiments of this disclosure, utilizing the spatiotemporal context relationship between the multi-camera images in step S210, the STN method of grid sampling and bilinear interpolation can be used to separate images of adjacent frames (set as the previous frame and the next frame). Pixels projected onto the target image To obtain a composite image The view synthesis operation is fully differentiable and supports gradient backpropagation during end-to-end training.

[0070] S213, Based on target image and composite images Obtain the reprojection error.

[0071] In some embodiments of this disclosure, the reprojection error loss of the network is set as a weighted combination of the structuresimilarity (SSIM) metric and the L1 loss (SSIM weights). (Set to 0.85): Figure 6 A flowchart illustrating a method for obtaining reprojection error according to one embodiment of this disclosure is shown.

[0072] Figure 7 This is a flowchart illustrating a method for obtaining intra-frame multi-camera pose consistency error according to one embodiment of this disclosure.

[0073] refer to Figure 7For the training method of the 3D scene reconstruction device for a multi-camera system disclosed herein, preferably, S220, obtaining the intra-frame multi-camera pose consistency error based on intra-frame multi-camera pose consistency constraints, includes: S221. Based on the intra-frame multi-camera pose consistency constraint, the poses of each camera predicted by the pose estimation network, except for the target camera (e.g., camera j as the target camera), are transformed into the coordinate system of the target camera. In this disclosure, preferably, intra-frame multi-camera pose consistency constraints are utilized, meaning the relative poses of the multiple cameras are fixed per unit time, and the extrinsic parameters of camera i and camera j are used. and Transform the poses of all cameras except camera j predicted by the pose estimation network into the camera j coordinate system: Let be the transformation matrix of camera i predicted by the pose estimation network from time t to time t+1. for The transformation matrix in the camera's j-coordinate system.

[0074] S222, Construct a weighted translation function and rotation consistency loss function ; S223, Weighted Translation Function and rotation consistency loss function An intra-frame multi-camera pose consistency function is constructed to balance spatial constraints, so that the rotation and translation predicted by each camera in the same coordinate system tend to be consistent.

[0075] In some embodiments of this disclosure, after unifying the camera poses predicted by the pose estimation network to the camera j coordinate system, a weighted translation consistency loss function is defined to make the rotations and translations predicted by each camera in the same coordinate system tend to be consistent. and rotation consistency loss function To weigh spatial constraints: in, Translation loss function Weighting factors Rotational consistency loss function Weighting factors.

[0076] In some embodiments of this disclosure, the camera j=1 is set, and the translation predicted by the pose estimation network is... The translation obtained by camera i (i=2,…,N) through the transformation method described above is: Calculate the translation consistency loss function : In some embodiments of this disclosure, the rotation matrices of each camera in the same coordinate system are converted into Euler angles. And calculate rotational consistency loss. : Figure 8 The diagram illustrates a flowchart of training a depth estimation network and a pose estimation network by minimizing reprojection error and minimizing intra-frame multi-camera pose consistency error according to one embodiment of the present disclosure.

[0077] refer to Figure 8 In this disclosure, preferably, step S230 involves training the depth estimation network and the pose estimation network by minimizing the reprojection error and the intra-frame multi-camera pose consistency error, including: S231. Randomly select three consecutive frames of multi-camera images from a multi-camera image sequence; S232. Input the multi-camera images at the intermediate frame time into the depth estimation network and output the depth map of each camera at the intermediate frame time; concatenate three frames from the same camera and input them into the pose estimation network respectively, and output the relative pose of the frame before the intermediate frame and the frame after the intermediate frame. S233. Calculate the reprojection error loss function based on the depth map of each camera at the intermediate frame time to minimize the reprojection error. Calculate the intra-frame multi-camera pose consistency function based on the relative pose of the frame before the intermediate frame and the frame after the intermediate frame to minimize the intra-frame multi-camera pose consistency error. S234, Backpropagation to update the network parameters of the depth estimation network and the pose estimation network; and S235. Determine whether the number of iterations has been reached. If the number of iterations has not been reached, repeat steps S231 to S234 until the number of iterations is reached.

[0078] Figure 9 A flowchart illustrating a method for training a multi-view TSDF estimation network according to one embodiment of this disclosure is shown.

[0079] refer to Figure 9 In this disclosure, preferably, S300, the multi-camera image sequence based on the multi-camera system, the poses of the multi-camera system, and the depth maps of the multi-camera system are used to train the multi-view TSDF estimation network, including: S310. Based on the pose and depth map of each camera, construct the known pose RGB-D image sequence of each camera to obtain the initial truncated symbolic field (i.e., truncated symbolic distance function) of the multi-view TSDF estimation network. In some embodiments of this disclosure, based on the poses of each camera and the corresponding depth maps obtained in step S200, a sequence of RGB-D images of known poses of four cameras is constructed, and an initial truncated symbol field is generated using the TSDF method as a supervision signal for network training.

[0080] TSDF (truncated signed distance function) is a truncated signed distance function. The multi-view TSDF estimation network disclosed in this paper can adopt the architecture of existing TSDF estimation networks and be trained by multi-camera image sequences, poses of multiple cameras, and depth maps of multiple cameras to obtain a multi-view (i.e., multi-camera) TSDF estimation network.

[0081] S320. Use the average pose of each camera in a unified coordinate system as the overall pose and filter the overall keyframes based on the translation and rotation of the overall pose. In some embodiments of this disclosure, the average pose of each camera in the unified coordinate system obtained above is regarded as the overall pose, and an overall keyframe is set to ensure that the reconstructed multi-view has sufficient motion parallax and common view. For example, when the overall pose of the four camera frames at a newly introduced moment has a relative translation greater than 0.1m (adjustable) and a relative rotation greater than 15° (adjustable), the four camera images corresponding to that moment are retained as an overall keyframe.

[0082] S330. A preset number of overall keyframes are used as an overall keyframe group, and the overall poses corresponding to each overall keyframe are input into the multi-view TSDF estimation network to predict the global truncated symbol field. In some embodiments of this disclosure, every three overall keyframes can be set as an overall keyframe group, which, together with the corresponding overall pose, can be input into the multi-view TSDF estimation network.

[0083] S340. The multi-view TSDF estimation network is trained under supervision using the initial truncated symbol field, and the global truncated symbol field is finally output.

[0084] That is, step S340 uses the initial truncated symbol field obtained in step S310 to supervise the training of the multi-view TSDF estimation network constructed in step S330, and finally outputs the global truncated symbol field, thereby reconstructing the 3D model of the scene through the ray projection method.

[0085] Figure 10A complete flowchart of a self-supervised training method for a multi-view TSDF estimation network according to one embodiment of the present disclosure is shown.

[0086] In this disclosure, preferably, step S330 involves using a preset number of overall keyframes as an overall keyframe group, and inputting it together with the overall pose corresponding to each overall keyframe into a multi-view TSDF estimation network to predict the global truncated symbol field, including: S331. Perform three-level feature extraction on the camera images of the overall keyframe group to obtain the three-level feature maps of each camera. In some embodiments of this disclosure, a Feature Pyramid Network is used as the feature extraction module. The camera images of a whole keyframe group are used as the input of the feature extraction module. The feature maps of the four cameras at the corresponding levels are obtained on the three feature extraction layers from coarse to fine. That is, the feature maps of each camera are obtained at each level.

[0087] S332. Based on the pose of each camera, the feature maps (2D feature maps) of each camera at each level are back-projected to aggregate them into three-dimensional feature bodies corresponding to each level in the temporal sequence of the overall keyframe group and between intra-frame cameras. In some embodiments of this disclosure, the multi-camera poses obtained above can be used to back-project the 2D feature maps of the four cameras at three levels, and aggregate them into three-dimensional feature bodies corresponding to each level in the temporal sequence of the overall keyframe group and between intra-frame cameras (i.e., each level corresponds to a three-dimensional feature body).

[0088] S333. Based on the multi-camera 3D feature volume generated by the current overall keyframe group, and the global geometric feature volume generated by the previous overall keyframe group, recursively local fusion is performed at three levels to obtain the global geometric feature volume of the current overall keyframe group, so as to further predict the global truncated symbol field.

[0089] In some embodiments of this disclosure, S332, based on the pose of each camera, back-projecting the feature maps (2D feature maps) of each camera at each level to aggregate them into three-dimensional feature volumes corresponding to the three levels in the temporal order of the overall keyframe group and between intra-frame cameras, further includes: Step S3321: Let the image of camera i in the multi-camera image sequence be... The extracted 2D feature map is Where c is the feature size. Using the known intrinsic parameters of each camera in the multi-camera system and the camera poses obtained above, the 2D feature maps of each camera image in the overall keyframe group from the above steps are back-projected onto three-dimensional feature voxels. superior: in, and These are the intrinsic and extrinsic parameters of image i, respectively. For perspective mapping, symbols It is a slicing operator. These are voxel coordinates in world space. If the coordinates are pixel coordinates in the image space, then all voxels along the camera ray are filled with the same features corresponding to that pixel.

[0090] Step S3322: Using a recursive weighted average method, the 3D feature voxels of the overall keyframe group obtained in step 3321 are accumulated and aggregated into 3D feature volumes in the temporal sequence of the overall keyframe group and between intra-frame cameras. The feature volumes obtained at time t from the same camera are then... and the average characteristic volume at time t-1 (Initial time) According to voxel weights) Aggregates into the average feature body at the current time t : The feature volume obtained by camera i at time t Features aggregated to camera j at the same time Get it : The weights are achieved using a binary mask. Storage voxels Is it inside or outside the camera's field of view?

[0091] In this disclosure, preferably, in step S333, the 3D feature volume of the multi-camera generated based on the current overall keyframe group is recursively locally fused with the global geometric feature volume generated by the previous overall keyframe group at three levels to obtain the global geometric feature volume of the current overall keyframe group, in order to further predict the global truncated symbol field, including: S3331. Perform 3D feature extraction on the three-dimensional feature volume of the current overall keyframe group at each level to extract the local geometric feature volume of the three-dimensional feature volume at each level; In some embodiments of this disclosure, the three-dimensional feature volume obtained in step S332 at each level is... First, a 3D sparse convolutional layer is used through the GRU (gated recurrent unit) module ( ), to extract 3D geometric features That is, local geometric features.

[0092] S3332. Based on the global geometric feature body of the previous global keyframe group, extract the local geometric feature bodies corresponding to each local geometric feature body of the current global keyframe group. Then, merge each local geometric feature body of the current global keyframe group with each local geometric feature body of the previous global keyframe group according to the hierarchy to generate new local geometric feature bodies of the current global keyframe group, thereby obtaining the global geometric feature body of the current global keyframe group. More specifically, from the previous global geometric feature volume (Initial moment) (Empty) Extract the local geometric features of each local geometric feature corresponding to the current overall keyframe group. Using the GRU module and The fusion process generates new local feature volumes at each level for the current overall keyframe group. and updated to Local geometric features in (i.e., replacement) ) to obtain .

[0093] In some embodiments of this disclosure, the specific steps of the above-mentioned update are as follows: in, To update the door, For the forget gate, σ is the sigmoid function. As weights for sparse convolution.

[0094] S3333, Based on the new local feature volumes of the current overall keyframe group at each level, predict the TSDF volume, i.e., the truncated symbol field, at each level. In some embodiments of this disclosure, local feature bodies Predict the TSDF volume of this layer using an MLP layer. , Each voxel in the dataset contains an occupancy value o and a truncation sign value x. The occupancy value o represents the confidence level of the voxel within the TSDF truncation distance λ. Voxels with o below the sparsification threshold θ are defined as gap spaces and will be sparsified.

[0095] S3334. Concatenate the truncated symbol field obtained from the second level of the three levels with the truncated symbol field of the first level after nearest neighbor interpolation upsampling, and concatenate the truncated symbol field obtained from the third level with the truncated symbol field of the second level after nearest neighbor interpolation upsampling. Repeat steps S3331 to S3333. S3335. Replace the corresponding voxels of the global truncated symbol field with the truncated symbol field output from the third level (i.e., the last level) to generate the final global truncated symbol field.

[0096] In this disclosure, preferably, step S340 involves supervised training of the multi-view TSDF estimation network using an initial truncated symbol field, and finally outputting a global truncated symbol field, including: S341. On the occupancy value o and truncated symbol value x predicted at three levels from coarse to fine, supervision is performed based on the occupancy value and truncated symbol value of the region corresponding to the initial truncated symbol field, respectively. In some embodiments of this disclosure, Binary Cross Entropy Loss (BCE loss) is used on the predicted occupancy value and the occupancy value of the initial truncated symbol field. The L1 loss is calculated by taking the logarithm of the predicted truncated symbol value and the corresponding initial truncated symbol value.

[0097] S342. For the truncated symbol values ​​in the global truncated symbol field obtained after updating the local truncated symbol field, based on the truncated symbol values ​​of the corresponding region of the initial truncated symbol field, perform a logarithmic transformation on the truncated symbol values ​​of the two to calculate the L1 loss of the two truncated symbol values, so as to update the parameters of the multi-view TSDF estimation network.

[0098] This disclosure also provides a 3D scene reconstruction apparatus 1000 for a multi-camera system, comprising: The depth estimation network module 1002 is trained based on sparse depth maps. The sparse depth maps are obtained by processing the multi-camera image sequences of the multi-camera system using the SFM (Structure from Motion) method under the sparse depth map of each camera (for four cameras, four sparse depth maps are obtained). Pose estimation network module 1004 is a pose estimation network module 1004 that is trained in self-supervised manner after being jointly trained with depth estimation network module 1002. Self-supervised training enables depth estimation network module 1002 and pose estimation network module 1004 to output depth maps and poses of multiple cameras. The multi-view TSDF estimation network module 1006 is a multi-view TSDF estimation network module 1006 trained on multi-camera image sequences, poses and depth maps of a multi-camera system, for use in 3D scene reconstruction.

[0099] The three-dimensional scene reconstruction device 1000 disclosed herein can be implemented through a computer software architecture.

[0100] Figure 11This is a schematic block diagram of a three-dimensional scene reconstruction device 1000 that employs a hardware implementation of a processing system according to one embodiment of the present disclosure.

[0101] The 3D scene reconstruction apparatus 1000 may include corresponding modules that perform one or more steps in the flowchart described above. Therefore, each or more steps in the flowchart can be performed by a corresponding module, and the apparatus may include one or more of these modules. A module may be one or more hardware modules specifically configured to perform a corresponding step, or implemented by a processor configured to perform a corresponding step, or stored in a computer-readable medium for implementation by a processor, or implemented through some combination thereof.

[0102] This hardware architecture can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits, including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400, such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0103] Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, only one connection line is used in this diagram, but this does not imply that there is only one bus or only one type of bus.

[0104] A three-dimensional scene reconstruction system according to one embodiment of the present disclosure includes: A multi-camera system includes multiple cameras that are fixedly mounted on a vehicle to acquire image sequences of the vehicle's surrounding environment; according to a preferred embodiment of this disclosure, the multiple cameras constitute an in-vehicle surround-view multi-camera system. A synchronous camera controller (this disclosure may employ a synchronous camera controller from the prior art), the synchronous camera controller being used to control multiple cameras to synchronously acquire image sequences; The memory, specifically the vehicle-mounted memory, stores execution instructions (computer-executed instructions). The processor processes image sequences acquired by the multi-camera system by executing execution instructions stored in memory to implement the training method for a three-dimensional scene reconstruction apparatus for a multi-camera system according to any embodiment of the present disclosure.

[0105] The processor is preferably a GPU.

[0106] A three-dimensional scene reconstruction system according to another embodiment of the present disclosure includes: A multi-camera system, which includes multiple cameras that are fixedly mounted on the vehicle to capture image sequences of the vehicle’s surroundings; A synchronous camera controller is used to control multiple cameras to acquire image sequences synchronously. The memory stores the three-dimensional scene reconstruction device 1000 disclosed herein in the form of a computer program; A processor is used to reconstruct 3D scenes based on image sequences acquired by a multi-camera system when a computer program is executed by the processor.

[0107] The memory is an in-vehicle memory, and the processor is an in-vehicle processor.

[0108] According to one embodiment of the present disclosure, a readable storage medium stores execution instructions that, when executed by a processor, are used to implement a training method for a three-dimensional scene reconstruction apparatus for a multi-camera system according to any embodiment of the present disclosure.

[0109] According to another embodiment of the present disclosure, a readable storage medium stores a three-dimensional scene reconstruction apparatus of any embodiment of the present disclosure in the form of a computer program.

[0110] As described above, this disclosure provides a method for 3D reconstruction using a multi-camera system based on deep learning. This method can reconstruct a 3D scene with realistic scale based solely on image sequences acquired by the multi-camera system, without requiring prior acquisition of additional supervisory signals (such as camera pose, image depth information, or scene 3D models). This disclosure enables the supervised 3D reconstruction method based on deep learning to self-supervise the temporal relationships, spatial constraints, and scale information of the multi-camera system. It requires no additional sensor information; only the image sequences acquired by the multi-camera system are input, and a 3D model with scale information can be automatically and accurately reconstructed.

[0111] The technical solution disclosed herein utilizes depth maps predicted by a depth estimation network and pose transformations estimated by a pose estimation network to jointly train the network, minimizing inter-frame reprojection errors and intra-frame multi-camera pose consistency errors. This ultimately yields the corresponding poses and depth maps for a multi-camera image sequence, fully leveraging the spatiotemporal context relationships between multiple cameras. Specifically, in the pose estimation part for each camera, the inherent spatial relationships between the multiple cameras are used to establish pose consistency relationships for the corresponding poses of the intra-frame images of each camera in a unified coordinate system, ensuring that the estimated poses have a realistic physical scale and are more accurate. In the depth estimation part, a sparse depth map pre-trained network is obtained based on the SFM method, and the poses of multiple cameras obtained from the multi-camera pose estimation network are used to establish multi-view... Figure 1 Consistency constraints are used to supervise the depth estimation network, which outputs depth maps of each camera image, providing auxiliary information for subsequent TSDF estimation in multi-camera systems.

[0112] The self-supervised TSDF estimation network disclosed herein employs a method of generating local TSDFs based on global TSDFs, and then fusing the generated local models to update the global model, ultimately generating a complete global model. Specifically, the single input for generating local TSDFs is the overall keyframe group of the multi-camera system (including intra-frame multi-camera images and inter-frame images). This disclosure utilizes the temporal mapping relationship between the overall keyframe group and the intra-frame cameras to aggregate features into 3D feature volumes corresponding to each level. These feature volumes are then recursively fused with the previously generated global hidden feature volumes from the overall keyframe group at three levels, from coarse to fine, and the truncated symbol field is predicted using an MLP module.

[0113] The self-supervised TSDF estimation network disclosed herein generates an initial truncated symbol field as a self-supervised signal based on the poses of multiple cameras obtained from a multi-camera pose estimation network and the corresponding depth maps obtained from multi-camera depth estimation, using the TSDF method. It can generate dense 3D models with physical scale without requiring additional sensor information.

[0114] The method disclosed herein for generating a multi-camera system overall keyframe group for the input of a multi-view TSDF estimation network is based on the poses of multiple cameras obtained by the multi-camera system pose estimation method. By setting the overall keyframe number N, the relative translation threshold T and the relative rotation threshold R of the newly introduced frame, the multi-camera keyframes that are greater than the given thresholds are selected to form an overall keyframe. Every N overall keyframes form an overall keyframe group, which is used as the unit input of the multi-view TSDF estimation network.

[0115] The 3D reconstruction method disclosed herein does not require additional sensor information as in prior art solution 3. It only requires an image sequence acquired by a multi-camera system to automatically and accurately reconstruct a dense 3D model with scale information. Addressing the issue in prior art solution 1 that the multi-camera pose estimation process does not utilize the inherent spatiotemporal constraints of multiple cameras, this disclosure proposes a method for self-supervised training of a joint depth estimation network and a pose estimation network. This method utilizes the depth map predicted by the depth estimation network and the pose transformation estimated by the pose estimation network to minimize inter-frame reprojection errors and intra-frame multi-camera pose consistency errors to jointly train the network, ultimately obtaining the corresponding pose and depth map of the multi-camera image sequence. This fully utilizes the spatiotemporal context relationships between multiple cameras. Addressing the problems of prior art solution 2's difficulty in dense reconstruction, the lack of correlation between the predicted depth maps in solution 4, and inconsistent depths across different views, this disclosure proposes a method to first pre-train a depth estimation network using sparse depth maps obtained by the SFM method, and then perform self-supervised training by combining the depth estimation network and the pose estimation network. This method uses images from each camera and their corresponding poses to train the depth estimation network to obtain scale-consistent and interconnected depth maps. To address the issue that the supervised monocular TSDF estimation network in the existing technical solution 5 requires the user to provide the ground truth of the 3D model to supervise the training of the network, while obtaining the ground truth is often costly or even impossible in reality, the self-supervised multi-view TSDF estimation network proposed in this disclosure utilizes the spatiotemporal constraints of a multi-camera system, trains the TSDF estimation network through multi-camera image sequences and predicted depth maps, and generates an accurate, dense, and scale-informed 3D model.

[0116] When using the 3D reconstruction apparatus / system disclosed herein, the training of the relevant network only needs to be performed during the initial use. In subsequent 3D reconstructions, the user only needs to control the multi-camera system to acquire image sequences containing the target to be reconstructed within the field of view. The PC (processor + memory) of the 3D reconstruction system will then process the images according to the specified parameters. Figure 2 The 3D reconstruction structure of the deep learning-based multi-camera system shown automatically executes the 3D reconstruction process and accurately reconstructs a 3D scene model with scale information.

[0117] Any process or method description in the flowcharts of this disclosure or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of preferred embodiments of this disclosure includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of this disclosure pertain. The processor performs the various methods and processes described above. For example, the method embodiments of this disclosure can be implemented as software programs tangibly contained in a machine-readable medium, such as memory. In some embodiments, part or all of the software program may be loaded and / or installed via memory and / or a communication interface. When the software program is loaded into memory and executed by the processor, one or more steps of the methods described above may be performed. Alternatively, in other embodiments, the processor may be configured to perform one of the methods described above by any other suitable means (e.g., by means of firmware).

[0118] The logic and / or steps represented in the flowchart or otherwise described herein may be specifically implemented in any readable storage medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0119] For the purposes of this specification, a "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of readable storage media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Furthermore, a readable storage medium can even be paper or other suitable media on which a program can be printed, since a program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in memory.

[0120] It should be understood that various parts of this disclosure can be implemented in hardware, software, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0121] Those skilled in the art will understand that all or part of the steps of the methods described above can be implemented by a program instructing related hardware. The program can be stored in a readable storage medium, and when executed, the program includes one or a combination of the steps of the method implementation.

[0122] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into a single processing module, or each unit can exist physically separately, or two or more units can be integrated into a single module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0123] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0124] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0125] Those skilled in the art should understand that the above embodiments are merely for illustrating the present disclosure and are not intended to limit the scope of the disclosure. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present disclosure.

Claims

1. A training method for a 3D scene reconstruction device for a multi-camera system, characterized in that, include: S100. Use the SFM method to process the multi-camera image sequence of the multi-camera system to obtain sparse depth maps from the perspective of each camera. Based on the sparse depth maps, pre-train the depth estimation network to obtain the pre-trained depth estimation network. S200, a joint depth estimation network, and a pose estimation network are self-supervised to output poses and depth maps for multiple cameras. as well as S300. The multi-camera image sequence, pose of the multi-camera system, and depth map of the multi-camera system are used to train a multi-view TSDF estimation network for 3D scene reconstruction. The joint depth estimation network and pose estimation network are trained under self-supervised supervision to output multi-camera poses and multi-camera depth maps, including: S210. Construct the spatiotemporal relationship between the images acquired by each camera in the target frame and the source frame based on the spatiotemporal context relationship between the images of multiple cameras, so as to obtain the reprojection error between frames. S220. Based on intra-frame multi-camera pose consistency constraints, obtain intra-frame multi-camera pose consistency error; and S230. The depth estimation network and the pose estimation network are trained by minimizing the reprojection error and minimizing the intra-frame multi-camera pose consistency error. S210 involves constructing the spatiotemporal relationship between the images acquired by each camera in the target frame and the source frame based on the spatiotemporal context relationship between multi-camera images, in order to obtain the inter-frame reprojection error, including: S211. Based on the spatial constraints between cameras and the temporal constraints between frames of the image sequence acquired by each camera, obtain the spatiotemporal relationship between the image acquired by the i-th camera and the image acquired by the j-th camera in the target frame and the source frame; S212. Based on the spatiotemporal relationship, separate the images in adjacent frames... Pixels projected onto the target image To obtain a composite image ; and S213, based on the target image and the synthesized image The reprojection error is obtained; S220, obtaining the intra-frame multi-camera pose consistency error based on intra-frame multi-camera pose consistency constraints, includes: S221, transforming the poses of each camera (excluding the target camera) predicted by the pose estimation network to the target camera's coordinate system based on the intra-frame multi-camera pose consistency constraints; S222, constructing a weighted translation function. and rotation consistency loss function ; and S223, a weighted translation function and rotation consistency loss function An intra-frame multi-camera pose consistency function is constructed to balance spatial constraints, so that the rotation and translation predicted by each camera in the same coordinate system tend to be consistent.

2. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 1, characterized in that, The SFM method is used to process multi-camera image sequences from a multi-camera system to obtain sparse depth maps from each camera's viewpoint, including: S110. Use the SFM method to extract and match feature points in the multi-camera image sequences acquired by the multi-camera system, and use the obtained sets of matched feature points to solve the eigenvalue matrix to obtain the relative pose of each camera; and S120. Collect matching feature points at overlapping fields of view in multi-camera image sequences, generate sparse point clouds based on the matching feature points at overlapping fields of view and the relative poses of each camera, and convert them into sparse depth maps under the view of each camera.

3. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 1 or 2, characterized in that, Pre-training of a depth estimation network based on sparse depth maps includes: The sparse depth map is used as supervision to pre-train the depth estimation network.

4. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 1, characterized in that, S230. Training the depth estimation network and the pose estimation network by minimizing the reprojection error and the intra-frame multi-camera pose consistency error includes: S231. Randomly select three consecutive frames of multi-camera images from a multi-camera image sequence; S232. Input the multi-camera images at the intermediate frame time into the depth estimation network and output the depth map of each camera at the intermediate frame time; concatenate three frames from the same camera and input them into the pose estimation network respectively, and output the relative pose of the frame before the intermediate frame and the frame after the intermediate frame. S233. Calculate the reprojection error loss function based on the depth map of each camera at the intermediate frame time to minimize the reprojection error, and calculate the intra-frame multi-camera pose consistency function based on the relative pose of the previous frame and the next frame of the intermediate frame to minimize the intra-frame multi-camera pose consistency error. S234, Backpropagation to update the network parameters of the depth estimation network and the pose estimation network; and S235. Determine whether the number of iterations has been reached. If the number of iterations has not been reached, repeat steps S231 to S234 until the number of iterations is reached.

5. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 4, characterized in that, S300. The multi-camera image sequence, pose, and depth map of the multi-camera system are used to train the multi-view TSDF estimation network, including: S310. Construct RGB-D image sequences of known poses of each camera based on the poses and depth maps of each camera to obtain the initial truncated symbol field of the multi-view TSDF estimation network. S320. Use the average pose of each camera in a unified coordinate system as the overall pose and filter the overall keyframes based on the translation and rotation of the overall pose. S330. A preset number of overall keyframes are used as an overall keyframe group, and the overall poses corresponding to each overall keyframe are input together into the multi-view TSDF estimation network to predict the global truncated symbol field; and S340. The multi-view TSDF estimation network is trained under supervision using the initial truncated symbol field, and finally outputs the global truncated symbol field.

6. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 5, characterized in that, S330. A preset number of overall keyframes are used as an overall keyframe group, and the overall poses corresponding to each overall keyframe are input together into the multi-view TSDF estimation network to predict the global truncated symbol field, including: S331. Perform three-level feature extraction on the camera images of the overall keyframe group to obtain the three-level feature maps of each camera. S332. Based on the poses of each camera, back-project the feature maps of each camera at each level to aggregate them into 3D feature volumes corresponding to each level in the temporal sequence of the overall keyframe group and between intra-frame cameras; and S333. The three-dimensional feature volume of the multi-camera generated based on the current overall keyframe group is recursively fused with the global geometric feature volume generated by the previous overall keyframe group at three levels to obtain the global geometric feature volume of the current overall keyframe group, so as to further predict the global truncated symbol field.

7. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 6, characterized in that, S333. The 3D feature volume generated by the multi-camera system based on the current overall keyframe group is recursively locally fused with the global geometric feature volume generated by the previous overall keyframe group at three levels to obtain the global geometric feature volume of the current overall keyframe group, in order to further predict the global truncated symbol field, including: S3331. Perform 3D feature extraction on the three-dimensional feature volume of the current overall keyframe group at each level to extract the local geometric feature volume of the three-dimensional feature volume at each level; S3332. Based on the global geometric feature body of the previous global keyframe group, extract the local geometric feature bodies corresponding to each local geometric feature body of the current global keyframe group. Then, merge each local geometric feature body of the current global keyframe group with each local geometric feature body of the previous global keyframe group according to the hierarchy to generate new local geometric feature bodies of the current global keyframe group, thereby obtaining the global geometric feature body of the current global keyframe group. S3333, Based on the new local feature volumes of the current overall keyframe group at each level, predict the TSDF volume, i.e., the truncated symbol field, at each level. S3334. Concatenate the truncated symbol field of the first level and the truncated symbol field of the second level obtained from the second level and the truncated symbol field obtained from the third level with the truncated symbol field of the third level after nearest neighbor interpolation upsampling, and repeat steps S3331 to S3333. S3335. Replace the corresponding voxels of the global truncated symbol field with the truncated symbol field output from the third level to generate the final global truncated symbol field.

8. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 7, characterized in that, The local feature volume is predicted by an MLP layer to determine the TSDF volume of that layer. Each voxel in the TSDF volume contains an occupancy value o and a truncation sign value x.

9. The training method for a 3D scene reconstruction device for a multi-camera system according to claim 8, characterized in that, S340. Supervised training of the multi-view TSDF estimation network is performed using the initial truncated symbol field, and the global truncated symbol field is finally output, including: S341. Supervision is performed on the predicted occupancy value o and truncated symbol value x at the three levels from coarse to fine, based on the occupancy value and truncated symbol value of the region corresponding to the initial truncated symbol field, respectively; and S342. For the truncated symbol values ​​in the global truncated symbol field obtained after updating the local truncated symbol field, based on the truncated symbol values ​​of the corresponding region of the initial truncated symbol field, perform a logarithmic transformation on the truncated symbol values ​​of the two to calculate the L1 loss of the two truncated symbol values, so as to update the parameters of the multi-view TSDF estimation network.

10. A three-dimensional scene reconstruction system, characterized in that, include: A multi-camera system comprising multiple cameras fixedly mounted on a vehicle to acquire image sequences of the vehicle's surrounding environment; A synchronous camera controller, which controls multiple cameras to acquire image sequences synchronously; The memory stores execution instructions; as well as A processor that processes image sequences acquired by the multi-camera system by executing the execution instructions stored in the memory to implement the training method for a three-dimensional scene reconstruction apparatus for a multi-camera system as described in any one of claims 1 to 9.

11. The three-dimensional scene reconstruction system according to claim 10, characterized in that, The memory is an in-vehicle memory, and the processor is an in-vehicle processor.

12. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the training method for a three-dimensional scene reconstruction apparatus for a multi-camera system as described in any one of claims 1 to 9.

Images