Three-dimensional reconstruction model training method, three-dimensional reconstruction method, device and equipment

Abstract

Disclosed are a three-dimensional reconstruction model training method, a three-dimensional reconstruction method, a device and equipment, relating to the technical field of visual perception. The three-dimensional reconstruction model training method comprises: determining a sample image group of a target scene, corresponding reference point clouds and parameters of an image acquisition device for acquiring the sample image group; performing three-dimensional reconstruction based on the parameters, the sample image group and a three-dimensional reconstruction model to obtain predicted point clouds; determining an epipolar geometry constraint value based on the parameters and pixel point matching pairs; the pixel point matching pair is a set of homogeneous pixel coordinates of each projection point corresponding to a same three-dimensional space point in at least two images with different perspectives, and the three-dimensional space point is located in a field of view overlap region covered by the at least two images with different perspectives; determining a three-dimensional distance loss between the predicted point clouds and the reference point clouds; and training the three-dimensional reconstruction model based on the epipolar geometry constraint value and the three-dimensional distance loss to obtain a target three-dimensional reconstruction model. The disclosure can improve the accuracy and efficiency of three-dimensional reconstruction.

Description

Technical Field

[0001] This disclosure relates to the field of visual perception technology, and in particular to a three-dimensional reconstruction model training method, a three-dimensional reconstruction method, an apparatus and equipment. Background Technology

[0002] With the rapid development of deep learning technology, image sensor-based visual perception is gradually emerging. When using binocular cameras for visual perception, it is usually necessary to perform stereo correction on the left and right images captured by the binocular cameras to meet the binocular configuration of horizontal epipolar alignment; then, the corrected images are processed to predict the disparity map; finally, geometric post-processing is performed on the disparity map to obtain a 3D point cloud. However, this reconstruction process, which must include stereo correction and geometric post-processing, is quite complex, and the accuracy of 3D reconstruction is difficult to guarantee.

[0003] Therefore, there is an urgent need for a solution that ensures the accuracy of 3D reconstruction. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides a three-dimensional reconstruction model training method, a three-dimensional reconstruction method, an apparatus, and a device that can jointly supervise model training by combining epipolar geometric constraints and three-dimensional distance loss to obtain a trained neural network model. Based on the trained neural network, a high-precision three-dimensional point cloud with realistic physical scale can be directly generated from at least two perspective images.

[0005] The first aspect of this disclosure provides a method for training a 3D reconstruction model, comprising: determining a set of sample images for a target scene, a reference point cloud corresponding to the sample image set, and parameters of an image acquisition device for acquiring the sample image set; wherein the sample image set includes images acquired at least two different viewpoints at the same time; performing 3D reconstruction of the target scene based on the parameters, the sample image set, and the 3D reconstruction model to obtain a predicted point cloud corresponding to the target scene; determining epipolar geometric constraint values ​​based on the parameters and pixel matching pairs in the sample image set; wherein a pixel matching pair is a set of homogeneous pixel coordinates of projection points corresponding to the same 3D spatial point in images from at least two different viewpoints, and the 3D spatial point is located within a field of view overlap area jointly covered by images from at least two different viewpoints; determining the 3D distance loss between the predicted point cloud and the reference point cloud; and training the 3D reconstruction model based on the epipolar geometric constraint values ​​and the 3D distance loss to obtain a target 3D reconstruction model.

[0006] A second aspect of this disclosure provides a three-dimensional reconstruction method, comprising: determining an image to be processed; processing the image to be processed based on a target three-dimensional reconstruction model to obtain a three-dimensional point cloud corresponding to the image to be processed; wherein the target three-dimensional reconstruction model is trained based on the three-dimensional reconstruction model training method of the first aspect described above.

[0007] A third aspect of this disclosure provides a three-dimensional reconstruction model training apparatus, comprising: The sample determination module is used to determine the sample image group for the target scene, the reference point cloud corresponding to the sample image group, and the parameters of the image acquisition device for acquiring the sample image group; wherein, the sample image group includes at least two images from different perspectives acquired at the same time; The 3D reconstruction module is used to perform 3D reconstruction of the target scene based on parameters, sample image groups, and a 3D reconstruction model to obtain the predicted point cloud corresponding to the target scene. The epipolar geometric constraint value determination module is used to determine the epipolar geometric constraint value based on parameters and pixel matching pairs in the sample image group; wherein, the pixel matching pair is a set of homogeneous pixel coordinates of each projection point corresponding to the same three-dimensional space point in at least two different viewpoints, and the three-dimensional space point is located in the field of view overlap region jointly covered by at least two different viewpoints. The 3D distance loss determination module is used to determine the 3D distance loss between the predicted point cloud and the reference point cloud; The 3D reconstruction model training module is used to train a 3D reconstruction model based on epipolar geometric constraint values ​​and 3D distance loss to obtain the target 3D reconstruction model.

[0008] A fourth aspect of this disclosure provides a three-dimensional reconstruction apparatus, comprising: Image determination module, used to determine the image to be processed; The point cloud generation module is used to process the image to be processed based on the target 3D reconstruction model to obtain the 3D point cloud corresponding to the image to be processed; wherein, the target 3D reconstruction model is trained based on the 3D reconstruction model training method of the first aspect.

[0009] A fifth aspect of this disclosure provides a computer-readable storage medium storing a computer program that is executed by a processor to perform either the three-dimensional reconstruction model training method of the first aspect of this disclosure or the three-dimensional reconstruction method of the second aspect of this disclosure.

[0010] A sixth aspect of this disclosure provides an electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read executable instructions from the memory and execute the instructions to implement the three-dimensional reconstruction model training method of the first aspect of this disclosure or the three-dimensional reconstruction method of the second aspect of this disclosure.

[0011] The seventh aspect of this disclosure provides a computer program product that, when instructions in the computer program product are executed by a processor, performs either the three-dimensional reconstruction model training method of the first aspect of this disclosure or the three-dimensional reconstruction method of the second aspect of this disclosure.

[0012] The technical solution provided in this disclosure can determine a sample image set for a target scene, a reference point cloud corresponding to the sample image set, and parameters of the image acquisition device for acquiring the sample image set. The sample image set includes images from at least two different viewpoints acquired simultaneously. Based on the parameters, the sample image set, and the 3D reconstruction model, the target scene is reconstructed in 3D to obtain a predicted point cloud corresponding to the target scene. Based on the parameters and pixel matching pairs in the sample image set, epipolar geometric constraint values ​​are determined. Each pixel matching pair is a set of homogeneous pixel coordinates of projection points corresponding to the same 3D spatial point in at least two different viewpoints, and the 3D spatial point is located within the overlapping field of view jointly covered by the images from at least two different viewpoints. The 3D distance loss between the predicted point cloud and the reference point cloud is determined. Based on the epipolar geometric constraint values ​​and the 3D distance loss, a 3D reconstruction model is trained to obtain a target 3D reconstruction model.

[0013] This disclosure employs a combined epipolar geometric constraint and 3D distance loss supervised model training, enabling the trained 3D reconstruction model to directly output a high-precision 3D point cloud conforming to the true physical scale based on the input image. Specifically, the epipolar geometric constraint guides the model to learn the inherent geometric relationships between multiple viewpoints at the 2D level, freeing the model from reliance on stereo correction preprocessing. This allows the model to adapt to non-standard multi-view configurations with assembly deviations and sensor heterogeneity, thereby reducing the impact of hardware structural deviations on reconstruction accuracy and minimizing computational overhead caused by preprocessing. The 3D distance loss directly constrains the geometric consistency between the predicted point cloud and the reference point cloud in 3D space, anchoring the reconstruction results to an absolute physical scale. The epipolar geometric constraint and the 3D distance loss work synergistically to improve the accuracy and efficiency of 3D reconstruction. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of a system provided in an exemplary embodiment of this disclosure.

[0015] Figure 2 This is a schematic diagram of a system provided in another exemplary embodiment of this disclosure.

[0016] Figure 3 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in an exemplary embodiment of this disclosure.

[0017] Figure 4 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in another exemplary embodiment of this disclosure.

[0018] Figure 5 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in another exemplary embodiment of this disclosure.

[0019] Figure 6This is a flowchart illustrating a three-dimensional reconstruction model training method provided in yet another exemplary embodiment of this disclosure.

[0020] Figure 7 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in yet another exemplary embodiment of this disclosure.

[0021] Figure 8 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in yet another exemplary embodiment of this disclosure.

[0022] Figure 9 This is a schematic diagram of the structure of a three-dimensional reconstruction model provided in an exemplary embodiment of this disclosure.

[0023] Figure 10 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in yet another exemplary embodiment of this disclosure.

[0024] Figure 11 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in yet another exemplary embodiment of this disclosure.

[0025] Figure 12 This is a flowchart illustrating a three-dimensional reconstruction model training method provided in yet another exemplary embodiment of this disclosure.

[0026] Figure 13 This is a schematic flowchart of a three-dimensional reconstruction method provided in an exemplary embodiment of this disclosure.

[0027] Figure 14 This is a schematic diagram of the structure of a three-dimensional reconstruction model training device provided in an exemplary embodiment of this disclosure.

[0028] Figure 15 This is a schematic diagram of the structure of a three-dimensional reconstruction apparatus provided in an exemplary embodiment of the present disclosure.

[0029] Figure 16 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0030] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0031] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0032] Application Overview In multiple application areas such as intelligent driving (e.g., autonomous driving and assisted driving), robot navigation, and augmented reality, the 3D reconstruction results must have absolute physical scale, that is, the position and size of the object obtained by 3D reconstruction must be consistent with the real physical world, so as to ensure that downstream perception and decision-making tasks such as target detection, path planning, and virtual-real fusion will not make misjudgments.

[0033] In related technologies, the 3D reconstruction process is as follows: acquiring images from at least two viewpoints, performing stereo correction on the images from at least two viewpoints, performing binocular stereo matching on the stereo-corrected images from at least two viewpoints to calculate disparity, and calculating the depth of each pixel in the image based on the disparity to generate a 3D point cloud. However, the following problems exist at both the hardware and algorithm levels.

[0034] The 3D reconstruction process of related technologies includes the aforementioned stereo calibration step, which aims to achieve epipolar horizontal alignment of binocular images. However, the stereo calibration step corrects images from at least two viewpoints, and can only correct errors in the imaging results caused by spatial pose deviations of the binocular cameras at the image level. At the hardware level, binocular cameras cannot avoid assembly errors, making it difficult for the left and right cameras to achieve absolute physical horizontal alignment. At the same time, binocular cameras in some scenarios use heterogeneous sensor configurations, that is, the models, resolutions, or lens parameters of the left and right cameras are different, resulting in systematic differences in brightness, noise, distortion, and other characteristics between the left and right images, further increasing the difficulty of physical horizontal alignment of the binocular cameras.

[0035] It should be noted that physical horizontal alignment refers to the alignment at the hardware structure level of the binocular camera, requiring the optical axes of the left and right cameras to be parallel and their imaging planes to be on the same horizontal plane. This is a configuration requirement at the hardware structure level of the imaging system. Epipolar horizontal alignment, on the other hand, is a stereo correction step at the algorithm level. The accuracy of the physical horizontal alignment of the binocular camera directly determines whether the original left and right views can achieve epipolar horizontal alignment without processing. The greater the deviation in physical horizontal alignment, the further the original left and right views deviate from the epipolar horizontal alignment state, and the greater the processing difficulty of the stereo correction step at the algorithm level. Currently, multi-view cameras have assembly errors in hardware, coupled with structural deviations caused by heterogeneous sensor configurations, further leading to larger deviations in physical horizontal alignment. This makes it difficult for the stereo correction step at the algorithm level to achieve strict epipolar horizontal alignment, resulting in lower parallax accuracy calculated based on at least two viewpoint images after stereo correction, and consequently, lower accuracy of the final generated 3D point cloud.

[0036] At the algorithm level, when performing stereo matching on at least two viewpoint images after stereo correction to calculate disparity, and calculating the depth of each pixel in the image based on the disparity, the current deep learning network completes feature matching of the left and right view images through the cost volume operation module, calculates the disparity based on the matching results, and then predicts the depth of each pixel based on the obtained disparity. Two reasons contribute to the low accuracy of 3D point clouds during this process: First, disparity prediction introduces errors that cannot be corrected. These errors are propagated in subsequent depth map calculations and point cloud predictions, resulting in low accuracy. Second, during the training phase of the aforementioned deep learning network, the physical depth range of the imaging scene is extremely wide (from 0.01m to 100m, spanning four orders of magnitude). When using conventional loss functions (such as L1 or L2 loss) for supervision during training, the loss contribution of nearby pixels is much higher than that of distant pixels. This leads to an imbalance in algorithmic loss, which in turn causes the disappearance of distant gradients (distant gradients refer to the gradient signals generated by the prediction errors of distant pixels and used to update network parameters during the backpropagation process of deep learning network training), ultimately further reducing the accuracy of the 3D point cloud.

[0037] This disclosure provides a three-dimensional reconstruction model training method, a three-dimensional reconstruction method, an apparatus, and a device, which can effectively solve the problem of low accuracy of three-dimensional reconstruction results caused by inherent hardware deviations and algorithm loss imbalances, and obtain high-precision three-dimensional reconstruction results to meet the requirements of various downstream perception and decision-making tasks for three-dimensional reconstruction accuracy.

[0038] Exemplary System Figure 1 This is a schematic diagram of a system provided in an exemplary embodiment of this disclosure. Figure 1 As shown, the system 100 includes an image acquisition device 101 and a three-dimensional reconstruction model training device 102.

[0039] The image acquisition device 101 is used to acquire a sample image group corresponding to the target scene; the sample image group includes at least two images from different perspectives acquired by the image acquisition device 101 at the same time. The images from at least two different perspectives have at least partial field of view overlap, that is, the same three-dimensional spatial point is imaged simultaneously in multiple images from different perspectives, forming a field of view overlap region for three-dimensional reconstruction.

[0040] The image acquisition device 101 can be one of a binocular camera, a trinocular camera, a quad-camera, or other multi-view cameras. Taking a binocular camera as an example, the left and right cameras in the binocular camera acquire images of the same target scene from two different perspectives, obtaining corresponding sample image sets. These sample image sets include the left and right images acquired at the same time.

[0041] The image acquisition device 101 can be a stand-alone image acquisition device, such as a mobile acquisition unit mounted on a data acquisition vehicle or drone; the image acquisition device 101 can also be integrated into a mobile smart device, such as an autonomous driving vehicle (e.g., a vehicle with assisted driving or autonomous driving functions), a mobile robot (e.g., a robotic vacuum cleaner or a navigation robot), an augmented reality (AR) device (e.g., AR glasses or AR headset), a smartphone, or a tablet computer. After acquiring images from at least two perspectives, the image acquisition device in this smart device can send the acquired images from at least two perspectives to the 3D reconstruction model training device 102 via the smart device.

[0042] The 3D reconstruction model training device 102 is used to train the 3D reconstruction model by executing the 3D reconstruction model training method provided in this embodiment, based on at least two viewpoint images (i.e., the above-mentioned sample image group) acquired by the image acquisition device 101, the parameters of the image acquisition device 101 (e.g., the intrinsic parameter matrix of each camera in the image acquisition device 101, the rotation matrix and translation vector between the two cameras, the baseline distance, etc.), and the reference point cloud corresponding to the sample image group (the reference point cloud is the ground truth of the point cloud with real physical scale, used as the scale supervision benchmark for training). This results in a target 3D reconstruction model that can be used to perform 3D reconstruction tasks.

[0043] The specific working principle and beneficial effects of the 3D reconstruction model training device 102 will be described in detail below, and will not be described in detail here.

[0044] The 3D reconstruction model training device 102 can be deployed in electronic devices with computing power, including but not limited to local servers, cloud computing devices, edge computing devices, or terminal devices, depending on the actual computing power requirements and application scenarios. Local servers and cloud computing devices are used for full-scale training of the 3D reconstruction model (i.e., starting from scratch, using training data to train all network parameters of the model until the model converges or the number of iterations reaches a preset number); edge computing devices can be used for incremental training of the 3D reconstruction model to locally optimize the parameters of the 3D reconstruction model; terminal devices have limited computing power and can be used for lightweight, small-scale incremental training of the 3D reconstruction model.

[0045] Local servers include rack-mounted artificial intelligence (AI) training servers equipped with one or more graphics processing units (GPUs), such as enterprise-built computing servers; cloud computing devices include virtualized cloud server instances and cloud computing clusters, which are composed of multiple GPU training servers deployed in the cloud; edge computing devices include multi-access edge computing (MEC) servers and roadside edge computing units; and terminal devices include in-vehicle computing platforms, robot controllers, and AR main control units.

[0046] Figure 2 This is a schematic diagram of a system provided in another exemplary embodiment of this disclosure. Figure 1 Compared to system 100 shown, Figure 2 The system 200 shown, in addition to the image acquisition device 101 and the three-dimensional reconstruction model training device 102, further includes a three-dimensional reconstruction device 103.

[0047] The 3D reconstruction device 103 can be set up independently and communicate with devices that require 3D reconstruction results (such as vehicle-mounted computing platforms, robot controllers, and AR main control units) via a first communication method. The first communication method can be set by those skilled in the art according to actual conditions, and this embodiment does not limit it.

[0048] The 3D reconstruction device 103 can also be integrated into intelligent devices (such as intelligent driving vehicles, mobile robots, AR devices, etc.) that need to perform downstream perception and decision-making tasks based on the 3D reconstruction results. Data interaction is completed based on the data channel inside the intelligent device, thus eliminating the need to build an external communication link.

[0049] The 3D reconstruction device 103 can also communicate with the 3D reconstruction model training device 102 via a second communication method. Based on this communication connection, the 3D reconstruction device 103 can invoke a target 3D reconstruction model pre-trained by the 3D reconstruction model training device 102, and based on the target 3D reconstruction model, process the acquired image to be processed, outputting a high-precision 3D point cloud with realistic physical scale for the corresponding scene. The second communication method can be set by those skilled in the art according to actual conditions, and this embodiment does not limit it.

[0050] Those skilled in the art will understand that the structure of the system 200 described above is merely illustrative, and in practical applications, the integration method of each functional module can be flexibly adjusted according to deployment requirements. For example, the 3D reconstruction model training device 102 can be deployed on a local server or cloud computing device. After the target 3D reconstruction model is trained, the target 3D reconstruction model is sent to the corresponding smart device, or the smart device can download the target 3D reconstruction model from the local server or cloud computing device to its local machine, and then the 3D reconstruction device 103 deployed in the smart device can call the target 3D reconstruction model on the smart device to perform the 3D reconstruction task; alternatively, the 3D reconstruction model training device 102 and the 3D reconstruction device 103 can be integrated into the same electronic device, and the 3D reconstruction device 103 can directly call the target 3D reconstruction model in the electronic device to perform the 3D reconstruction task.

[0051] In this embodiment, the application scenarios of system 100 and system 200 include, but are not limited to, environmental perception of intelligent driving vehicles, scene navigation of mobile robots, virtual-real fusion of AR devices, and 3D dimension detection in industrial scenarios. In practical applications, different scenarios can flexibly adapt the solutions provided in this disclosure according to their own requirements for 3D reconstruction accuracy and scale accuracy. The 3D reconstruction model training method and 3D reconstruction method provided in this disclosure are described in detail below with reference to the accompanying drawings.

[0052] Exemplary methods Figure 3 This is a schematic flowchart of a three-dimensional reconstruction model training method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, and the specific electronic devices can be found in the description of the deployable electronic device part of the three-dimensional reconstruction model training device 102 in the above embodiment, which will not be repeated here.

[0053] like Figure 3 As shown, the training method for the 3D reconstruction model includes the following steps S310~S350.

[0054] S310: Determine the parameters of the sample image group for the target scene, the reference point cloud corresponding to the sample image group, and the image acquisition device for acquiring the sample image group; wherein, the sample image group includes at least two images from different perspectives acquired at the same time.

[0055] The target scenario is any scenario that requires 3D reconstruction, such as road environment scenarios during the driving of intelligent vehicles, indoor or outdoor scenarios for navigation and positioning of mobile robots, interactive scenarios for virtual-real fusion of AR devices, 3D dimension detection scenarios for large workpieces, and cargo volume measurement scenarios. This disclosure does not specifically limit the target scenario. This disclosure is applicable to any scenario that requires high-precision real physical scale 3D reconstruction.

[0056] The sample image set is a collection of at least two viewpoint images acquired by an image acquisition device under the same target scene. It should be noted that there are overlapping fields of view between the at least two viewpoint images included in the sample image set, so as to ensure that the same three-dimensional spatial point in the same scene can be imaged in at least two images at the same time, providing a data basis for the subsequent determination of epipolar geometric constraint values ​​(corresponding to S330).

[0057] The reference point cloud corresponding to the sample image group is a ground truth (GT) point cloud that is temporally and spatially aligned with at least two viewpoint images included in the sample image group and possesses a true physical scale. The reference point cloud is used as a supervisory signal for the predicted point cloud in subsequent steps (corresponding to S340) to supervise whether the predicted point cloud possesses an absolute scale consistent with the real physical world. The method of obtaining the reference point cloud can be set by those skilled in the art according to the actual situation, and this disclosure does not limit it. For example, the reference point cloud can be obtained by scanning the target scene using a lidar.

[0058] The image acquisition device can refer to the description of the image acquisition device 101 above, and can be one of a binocular camera, a trinocular camera, a quad-camera, or other multi-view cameras. The specific number of viewing angles can be selected according to the complexity of the scene and the accuracy requirements. Taking a binocular camera as an example, the sample image group includes the left and right eye images acquired at the same time. Taking a trinocular camera as an example, multiple sets of binocular sample pairs can be formed, such as (left eye image, middle eye image), (middle eye image, right eye image), and (left eye image, right eye image). Alternatively, three images acquired by the trinocular camera at the same time can be used as input for training at one time. This disclosure does not limit this.

[0059] The parameters of the image acquisition device may include the intrinsic parameters (such as the intrinsic parameter matrix) of each camera in the image acquisition device, the extrinsic parameters (such as the rotation matrix and translation vector) between different cameras, and the baseline distance. The intrinsic parameter matrix is ​​used to back-project the depth map corresponding to the target view image to obtain the corresponding 3D point cloud. The target view image is any one of at least two view images. The depth map corresponding to the target view image is obtained based on one of the target view image and other view images. In this disclosure, the predicted point cloud corresponding to the target scene can be obtained based on the intrinsic parameter matrix, the sample image group, and the 3D reconstruction model. The rotation matrix, translation vector, and baseline distance are used to construct the epipolar constraint equation to calculate the epipolar geometric constraint values ​​in subsequent steps (corresponding to S330 and S350). If the image acquisition device is a binocular configuration, the baseline distance is the horizontal distance between the optical centers of the left and right cameras. If the image acquisition device is a trinocular or higher multi-view configuration, any two cameras can form a binocular pair. For each pair of cameras, the distance between their optical centers can be considered as a baseline. In a trinocular or higher multi-view configuration, there will be multiple baselines, each corresponding to a different baseline distance. When the image acquisition device adopts a standard binocular configuration (i.e., an ideal imaging configuration where the optical axes of the left and right cameras are parallel and the epipolar lines are horizontally aligned), the camera's intrinsic parameters can be the camera's intrinsic parameter matrix. The translation vector in the extrinsic parameters between the cameras can be simplified to a translation vector along the baseline direction, and the rotation matrix in the extrinsic parameters is the identity matrix.

[0060] The parameters of the image acquisition device can be obtained through offline calibration (such as laser tracker calibration) or dynamically updated during the acquisition process based on an online calibration method (the online calibration method shall be set by those skilled in the art according to the actual situation). This disclosure does not limit this. The parameters of the image acquisition device must correspond to the sample image group in time to ensure the accuracy of subsequent epipolar geometric constraint value calculation.

[0061] S320: Based on the parameters of the image acquisition device, sample image group and 3D reconstruction model, the target scene is reconstructed in 3D to obtain the predicted point cloud corresponding to the target scene.

[0062] The 3D reconstruction model is a deep learning-based neural network model to be trained. After receiving a group of sample images including images from at least two different viewpoints, the 3D reconstruction model performs depth prediction on each pixel in the at least two viewpoint images to obtain the depth value corresponding to each pixel. Based on the depth value corresponding to each pixel, as well as the intrinsic parameters of each camera in the image acquisition device, the extrinsic parameters between the two different cameras, and the baseline distance, each pixel is projected into 3D space to obtain the predicted point cloud corresponding to the target scene and output the predicted point cloud.

[0063] In this process, each three-dimensional point in the predicted point cloud is represented by its spatial coordinates in physical units (such as meters), and the predicted point cloud and the reference point cloud are in the same three-dimensional coordinate system. This is so that in the subsequent step (S350), the spatial position of the predicted point cloud can be made closer to that of the reference point cloud through supervised constraints, thereby giving the predicted point cloud an absolute physical scale.

[0064] S330: Determine the epipolar geometric constraint value based on the parameters of the image acquisition device and the pixel matching pairs in the sample image group; wherein, the pixel matching pair is a set of homogeneous pixel coordinates of each projection point corresponding to the same three-dimensional space point in at least two different viewpoints, and the three-dimensional space point is located in the field of view overlap area jointly covered by at least two different viewpoints.

[0065] For any 3D spatial point within the overlapping field of view of a sample image group, project this 3D spatial point onto images from various viewpoints in the sample image group to obtain the pixel coordinates of the projected points of this 3D spatial point on each viewpoint image. Extend these pixel coordinates to homogeneous coordinate form to obtain the corresponding homogeneous pixel coordinates. This yields the homogeneous pixel coordinates obtained after projecting the same 3D spatial point onto the pixel coordinate systems of the images from various viewpoints. The set of all homogeneous pixel coordinates obtained from projecting the same 3D spatial point constitutes a set of pixel matching pairs. Extending pixel coordinates to homogeneous coordinate form means that, in the pixel coordinate... Add a 1 at the end to get the 3D vector corresponding to the pixel coordinate. .

[0066] Taking a binocular camera as an example, the left eye camera C1 captures the left eye image I at time T1. 1 The right eye camera C2 captures the right eye image I at time T1. 2 , among which, I 1 I 2 , The height of the left eye image. The width of the left eye image. The height of the right eye image. The width of the right eye image. Image I i It is a three-channel color image. Left eye image I 1 And right eye image I 2 There exists a three-dimensional spatial point X within the overlapping region of the field of view. m (m is a positive integer), a three-dimensional point X m In left eye image I 1 The pixel coordinates corresponding to the projection point are expanded to obtain the corresponding homogeneous pixel coordinates. 3D spatial point X m In the right eye image I2 The pixel coordinates corresponding to the projection point are expanded to obtain the corresponding homogeneous pixel coordinates. ,but and A pixel matching pair is formed, denoted as If the left eye image I 1 And right eye image I 2 The number of three-dimensional spatial points in the overlapping area of ​​the field of view is M, and the set of pixel matching pairs can be obtained as shown in the following expression (1).

[0067] Expression (1): , It is a positive integer.

[0068] When determining the matching point pairs in the pixel matching pair, the feature matching can be performed on two images from different perspectives acquired at the same time based on the binocular matching algorithm; or the pixel matching pair can be determined based on the predicted point cloud, by projecting the three-dimensional spatial points in the predicted point cloud onto the images from each perspective, and then determining the matching pair based on the pixel coordinates of the projected points. This disclosure does not limit the method of determining the pixel matching pair.

[0069] Epipolar geometric constraints between two cameras refer to the geometric correspondence that must be satisfied between the projected points of the same 3D spatial point on images taken from two different viewpoints by these two cameras. This correspondence is such that for any projected point on one viewpoint image, its corresponding matching point on the other viewpoint image must lie on a specific straight line. This straight line can also be called the epipolar line on the other viewpoint image. Epipolar geometric constraints are used to constrain the relative positional relationships of each 3D spatial point in the 3D reconstruction result, ensuring that the 3D reconstruction result is not distorted compared to the corresponding real point cloud, and guaranteeing the accuracy of the reconstructed object's shape and proportions. The epipolar geometric constraint value characterizes the degree of distortion in the relative positions of each 3D spatial point in the 3D reconstruction result; the larger the epipolar geometric constraint value, the greater the degree of distortion, and vice versa. An epipolar geometric constraint value of 0 indicates that the relative positions of each 3D spatial point in the 3D reconstruction result are not distorted, and the accuracy of the 3D reconstruction result is optimal under this condition.

[0070] When determining the epipolar geometric constraint values ​​between two cameras, a fundamental matrix F describing the epipolar geometric relationship between the two cameras can be constructed based on the parameters of the image acquisition device. Then, based on the fundamental matrix F, the epipolar constraint residuals corresponding to each pixel matching pair in the two images taken from different perspectives by the two cameras are calculated. The epipolar constraint residuals characterize the error in the pixel position deviating from the epipolar line. Finally, the epipolar constraint residuals of all pixel matching pairs are aggregated to obtain the epipolar geometric constraint values ​​corresponding to the 3D reconstruction results. This disclosure does not limit the specific calculation method of the above aggregation calculation.

[0071] It should be noted that epipolar geometric constraints can constrain the relative positional relationships of 3D spatial points in the 3D reconstruction result, but cannot constrain the deviation of the 3D reconstruction result from the real 3D scene at the true physical scale. This is because epipolar geometric constraints are determined only by the camera's intrinsic parameters and the extrinsic parameters between cameras. Essentially, they are the geometric correspondence between projection points on the 2D image plane, only constraining the directional consistency of each projection point, without involving the absolute magnitude of the depth value corresponding to each projection point. For example, when a 3D spatial point moves along the projection ray direction (i.e., the ray direction from the camera's optical center to the 3D spatial point), the projection points of that 3D spatial point at each viewpoint can still satisfy the epipolar geometric constraints, but the distance between that 3D spatial point and the optical centers of the cameras at each viewpoint has changed. The epipolar geometric constraints cannot determine the true distance between the 3D spatial point and the optical centers of the cameras at each viewpoint; they can only ensure that the internal structure shape of the reconstructed scene objects is not distorted and that the relative distance ratio between each 3D spatial point remains consistent, but they cannot restore the true physical size of the scene. Therefore, the deviation of the 3D reconstruction result from the real 3D scene at the true physical scale is constrained by a reference point cloud (see S340 below).

[0072] S340: Determine the 3D distance loss between the predicted point cloud and the reference point cloud.

[0073] Based on the pixels in each image of the sample image group, the predicted point cloud and the reference point cloud are mapped pixel-by-pixel. That is, for the same pixel, the pixel corresponds to a first three-dimensional spatial point in the predicted point cloud, and the pixel corresponds to a second three-dimensional spatial point in the reference point cloud. The spatial distance deviation between the first three-dimensional spatial point and the second three-dimensional spatial point is calculated to obtain the spatial distance deviation corresponding to each pixel in the image. The spatial distance deviations corresponding to all pixels in the image are aggregated to obtain the three-dimensional distance loss. The above aggregation process can be, for example, taking an average value, a weighted average value, etc., and the embodiments of this disclosure are not limited thereto.

[0074] S350: Based on epipolar geometric constraints and 3D distance loss, a 3D reconstruction model is trained to obtain the target 3D reconstruction model.

[0075] An objective function can be constructed based on epipolar geometric constraints and 3D distance loss to train the 3D reconstruction model until the objective function converges or a set number of training iterations is reached. Training is then stopped, and the trained target 3D reconstruction model is obtained. The objective function can be a weighted sum of epipolar geometric constraints and 3D distance loss. The weighting coefficients can be determined by those skilled in the art based on the actual situation, and this disclosure does not impose any limitations on this.

[0076] By jointly supervising the training process with epipolar geometric constraints and 3D distance loss, the epipolar geometric constraints can compensate for the errors caused by assembly errors and heterogeneous sensor configurations to the imaging results, and the 3D distance loss can directly constrain the absolute physical scale of the predicted point cloud, ultimately resulting in a target 3D reconstruction model that can directly output a high-precision 3D point cloud with absolute physical scale.

[0077] The 3D reconstruction model training method provided in this disclosure can determine a sample image set for a target scene, a reference point cloud corresponding to the sample image set, and parameters of the image acquisition device for acquiring the sample image set. The sample image set includes images from at least two different viewpoints acquired simultaneously. Based on the parameters, the sample image set, and the 3D reconstruction model, the target scene is reconstructed in 3D to obtain a predicted point cloud corresponding to the target scene. Based on the parameters and pixel matching pairs in the sample image set, epipolar geometric constraint values ​​are determined. Each pixel matching pair is a set of homogeneous pixel coordinates of projection points corresponding to the same 3D spatial point in images from at least two different viewpoints, where the 3D spatial point is located within the overlapping field of view jointly covered by images from at least two different viewpoints. Based on the epipolar geometric constraint values ​​and 3D distance loss, the 3D reconstruction model is trained to obtain the target 3D reconstruction model.

[0078] This disclosure uses a combination of epipolar geometric constraints and 3D distance loss to supervise model training, enabling the trained target 3D reconstruction model to directly output a high-precision 3D point cloud that conforms to the real physical scale based on the input image.

[0079] By training the 3D reconstruction model using epipolar geometric constraints, the trained target 3D reconstruction model can establish pixel-level geometric constraints between images from different viewpoints. These constraints act directly on the image itself, without relying on preprocessing operations such as stereo correction. Therefore, the target 3D reconstruction model can adaptively handle physical misalignment caused by assembly errors during actual installation of the image acquisition device, as well as non-ideal situations such as parameter differences between different cameras included in the image acquisition device. This allows the target 3D reconstruction model to adapt to various binocular or multi-view configurations, including non-standard multi-view systems with limited assembly precision. This significantly reduces the requirements for hardware manufacturing and installation precision, reduces the impact of hardware structural deviations on the accuracy of 3D reconstruction, and, since no preprocessing is required, reduces computational overhead and accuracy loss caused by preprocessing.

[0080] Among them, the 3D distance loss can directly measure the geometric difference between the predicted point cloud output by the model and the real reference point cloud in 3D space. The target 3D reconstruction model is obtained by training through the 3D distance loss, which can constrain the spatial position of the predicted point cloud, ensure the accuracy of the predicted point cloud in 3D space coordinates, reduce the deviation between the predicted point cloud and the geometric shape of the real scene, effectively avoid problems such as scale drift or deformation of the predicted point cloud, and improve the accuracy of the predicted point cloud obtained by the target 3D reconstruction model, thereby further improving the accuracy of 3D reconstruction.

[0081] Furthermore, in this scheme, epipolar geometric constraints and 3D distance loss are used synergistically to train the 3D reconstruction model. Epipolar geometric constraints focus on ensuring the accuracy of the correspondence at the 2D image level and the self-consistency of multi-view geometric relationships, while 3D distance loss focuses on ensuring the geometric accuracy and physical scale accuracy of the 3D output results. The two work together to supervise the training process, enabling the target 3D reconstruction model to generate high-precision 3D point clouds that conform to the true physical scale, effectively improving the overall accuracy and computational efficiency of 3D reconstruction.

[0082] In some embodiments, such as Figure 4 As shown, the execution process of S320 may include the following S410~S420.

[0083] S410: The depth estimation module based on the 3D reconstruction model performs depth estimation on the sample image to obtain a depth map indicating the depth information of each pixel in the sample image; the sample image is any one image in the sample image group.

[0084] The sample image serves as a reference viewpoint image. The depth estimation module takes the reference viewpoint image and one or more corresponding images from other viewpoints as input, and outputs the depth map corresponding to the reference viewpoint image. For example, for a stereo camera, the left eye image can be determined as the sample image, and the left and right eye images can be input into the 3D reconstruction model, outputting the depth map corresponding to the left eye image.

[0085] In this embodiment of the disclosure, the depth map corresponding to the sample image is calculated based on the following formula (1).

[0086] Formula (1): , .

[0087] In formula (1), For the network parameters of the depth estimation module, Indicated by network parameters The defined depth estimation module's depth prediction function, For sample images, Representation: Sample image The height is H, the width is W, and the depth map is... For a single-channel image with the same spatial resolution as the sample image, the depth map The value of each pixel represents the physical distance from the corresponding 3D spatial point to the camera at the reference viewpoint (for example, the unit of this physical distance can be meters).

[0088] It should be noted that, based on formula (1), there is no need to calculate the disparity between the sample image and other viewpoint images. Instead, the depth map corresponding to the sample image is directly output. Therefore, the intermediate step of disparity calculation in the traditional method is skipped, which fundamentally avoids the introduction of errors related to disparity prediction and avoids the cumulative error caused by multi-level operations (i.e., calculating the depth map based on disparity and generating a three-layer point cloud based on the depth map). At the same time, it avoids the intermediate step of deriving the physical depth based on disparity combined with baseline distance and focal length, reduces unnecessary computational overhead, and improves the accuracy of depth estimation in 3D reconstruction.

[0089] S420: Based on the parameters of the image acquisition device, the depth map is transformed through the coordinate transformation module of the 3D reconstruction model to obtain the predicted point cloud corresponding to the sample image.

[0090] The coordinate transformation module can backproject the depth map corresponding to the sample image to the camera coordinate system of the camera that captured the sample image in the image acquisition device, based on the intrinsic parameters of the camera, to obtain the predicted point cloud. The backprojection process can be found in the following formula (2).

[0091] Formula (2): .

[0092] In formula (2), To predict point clouds, the tensor dimension is... The three-dimensional vector at any position inside the tensor is the three-dimensional spatial coordinate obtained by mapping the corresponding pixel point; These are the pixel coordinates of the pixels in the sample image; Indicates pixel coordinates The homogeneous row vector constructed by concatenating it with the constant 1 The intrinsic parameter matrix of the camera that captures the sample images is obtained by using the left multiplication of row vectors into a matrix in formula (2). Therefore, the intrinsic parameter matrix... Transpose the inverse matrix; operator This indicates element-wise multiplication. This indicates a broadcasting operation used to transmit depth maps. The dimension from Expand to To make the depth map With based The dimensions of the obtained coordinate matrix are matched, thus enabling element-wise multiplication.

[0093] It should be noted that for invalid pixels in the depth map or pixels whose depth values ​​exceed a reasonable range, they can be directly removed from the predicted point cloud to reduce invalid calculations. This disclosure does not impose any restrictions on this.

[0094] Based on S410~S420, the depth map corresponding to the reference view image can be directly obtained through the 3D reconstruction model, and then the predicted point cloud at the real physical scale can be directly converted through back projection. The whole process is completed by the model end-to-end operation, which not only ensures the accuracy of the physical scale of the predicted point cloud, but also improves the overall efficiency of 3D reconstruction.

[0095] In some embodiments, such as Figure 5 As shown, the execution process of S330 may include the following S510~S520.

[0096] S510: Based on the parameters of the image acquisition device, determine the epipolar geometric relationship between at least two images from different perspectives.

[0097] In this step, the parameters of the image acquisition device may include the intrinsic parameter matrices of each camera in the image acquisition device, as well as the rotation matrices and translation vectors between different cameras. In this case, based on the intrinsic parameter matrices of each camera in the image acquisition device, as well as the rotation matrices and translation vectors between different cameras, the fundamental matrix F between two corresponding images from different viewpoints is determined. The fundamental matrix F represents the epipolar geometric relationship between the two images from different viewpoints.

[0098] In some examples, the antisymmetric matrix corresponding to the translation vector can be determined based on the translation vector. The essential matrix can be obtained based on the antisymmetric matrix and the rotation matrix. Finally, the fundamental matrix can be determined based on the essential matrix and the intrinsic parameter matrices of the cameras corresponding to the two different viewpoints.

[0099] Taking the binocular configuration in the above example as an example, if the binocular configuration is a standard parallel binocular configuration, then the intrinsic parameter matrices of the left eye camera C1 and the right eye camera C2 in this binocular configuration are the same, both being... Using the left eye camera's 3D coordinate system (where the left eye camera C1 is located) as the world coordinate system, and the rotation matrix of the right eye camera C2 relative to the left eye camera C1 as the identity matrix. Translation vector , Baseline distance, fundamental matrix It can be calculated using the following formula (3).

[0100] Formula (3): ;in, Translation vector antisymmetric matrix, , .

[0101] S520: Determine the epipolar geometric constraint value based on epipolar geometric relationships and pixel matching pairs.

[0102] Based on the fundamental matrix An epipolar constraint equation containing pixel matching pairs is constructed. Substituting the pixel coordinates corresponding to the matching pixel point into this epipolar constraint equation yields the epipolar geometric constraint value for that pixel matching pair. Taking the standard parallel binocular configuration in the example above, the set of pixel matching pairs for the left and right eye images is as follows: For any pixel point matching pair Based on the fundamental matrix Construct the polar constraint equation, which is shown in the following formula (4).

[0103] Formula (4): .

[0104] Formula (4) represents the pixel matching pair under error-free conditions. The theoretical relationships that should be satisfied. Specifically, points in three-dimensional space. The homogeneous pixel coordinates corresponding to the pixel coordinates projected onto the left eye image are: Three-dimensional space point The homogeneous pixel coordinates corresponding to the pixel coordinates projected onto the right eye image are: ,Depend on It can be determined that its corresponding left polar line in the left eye image is the same as... In the matched left eye image It will inevitably fall on this left-hand extreme line; conversely, it will fall on the opposite side. It can be determined that its corresponding right polar line in the right eye image is the same as... Matching It must fall on this right-polar line. Among them, the optical center of the left eye camera, the optical center of the right eye camera, and the three-dimensional spatial point... The plane in question is the polar plane. The left polar line is the straight line formed by the intersection of the polar plane and the imaging plane of the left eye camera (i.e., the plane corresponding to the left eye image). The right polar line is the straight line formed by the intersection of the polar plane and the imaging plane of the right eye camera (i.e., the plane corresponding to the right eye image).

[0105] It should be noted that, due to factors such as assembly errors and heterogeneous sensor configurations, The actual calculated result cannot be strictly equal to 0. That is, G, where G is the pixel matching pair The corresponding epipolar constraint residuals, G, are not zero. The epipolar constraint residuals corresponding to each pixel matching pair in the set of pixel matching pairs are aggregated to obtain the epipolar geometric constraint values ​​corresponding to all pixel matching pairs. The aggregation operation can be performed by taking the absolute value of the epipolar constraint residuals of all pixel matching pairs and calculating the average, or by taking the square of the epipolar constraint residuals of all pixel matching pairs and calculating the average. This disclosure does not limit the specific aggregation method.

[0106] Based on S510~S520, the epipolar geometric constraint value can be determined directly using the inherent hardware parameters of the image acquisition device without the need for additional stereo correction preprocessing of the image. While reducing computational overhead, it can adapt to non-standard multi-view configurations with assembly deviations and sensor heterogeneity, eliminate the influence of hardware structure deviations on reconstruction accuracy, and improve the accuracy of 3D reconstruction results.

[0107] In some embodiments, such as Figure 6 As shown, the execution process of S340 may include the following S610~S640.

[0108] S610: Based on the predicted point cloud, determine the predicted 3D coordinates of each pixel in the sample image, and based on the reference point cloud, determine the reference 3D coordinates of each pixel in the sample image and the edge mask weights corresponding to each pixel position.

[0109] Both the predicted point cloud and the reference point cloud are three-dimensional tensors with the same spatial resolution as the sample image. For both the predicted and reference point clouds, the three-dimensional coordinates corresponding to each pixel position are extracted by combining the pixel arrangement information of the sample image, resulting in the predicted three-dimensional coordinates in the predicted point cloud and the reference three-dimensional coordinates in the reference point cloud. A depth gradient is calculated based on the depth values ​​of each pixel in the reference point cloud. Edge regions with drastic depth changes (e.g., the absolute value of the difference in depth values ​​between adjacent pixels is greater than a preset depth change threshold) are extracted based on the depth gradient. Pixels within these edge regions are assigned a first edge mask weight, and pixels outside these edge regions are assigned a second edge mask weight. The first edge mask weight is greater than the second edge mask weight, thereby enhancing the model's attention to the depth of object edges and alleviating the problem of fuzzy depth prediction in edge regions. The magnitude of the edge mask weights corresponding to each pixel position can be set according to actual needs, and this disclosure does not limit this.

[0110] S620: Calculate the 3D coordinate deviation between the predicted 3D coordinates and the reference 3D coordinates corresponding to the same pixel.

[0111] For example, for each pixel (i.e., a pixel point), the Euclidean distance between its predicted 3D coordinates and the reference 3D coordinates is calculated. This Euclidean distance is used as the deviation of the 3D coordinates corresponding to that pixel (i.e., the 3D coordinate deviation between the predicted 3D coordinates and the reference 3D coordinates), reflecting the degree of geometric deviation between the spatial position predicted by the model at the corresponding pixel position and the actual position.

[0112] In some examples, the L1 norm can be used to measure the deviation of the three-dimensional coordinates between the predicted three-dimensional coordinates and the reference three-dimensional coordinates. The calculation formula is shown in the following formula (5).

[0113] Formula (5): .

[0114] In formula (5), To predict the 3D coordinate deviation between the 3D coordinates and the reference 3D coordinates, To predict three-dimensional coordinates, For reference three-dimensional coordinates, Let L1 norm be represented. The absolute three-dimensional coordinate deviation of each pixel position can be obtained through formula (5), which will serve as the basis input for subsequent aggregation calculations in S640.

[0115] S630: Based on the edge mask weights corresponding to each pixel position, the three-dimensional coordinate deviation of the corresponding pixel position is weighted to obtain the weighted three-dimensional coordinate deviation.

[0116] The 3D coordinate deviation of each pixel is multiplied by the edge mask weight corresponding to that pixel position. Since the edge mask weight of the edge region is greater than that of the non-edge region, the contribution of the edge region deviation to the total loss can be amplified by the edge mask weight, guiding the model to prioritize optimizing the prediction accuracy of the edge region and solving the problem of unclear edge reconstruction.

[0117] S640: Aggregate the weighted 3D coordinate deviations corresponding to each pixel position to obtain the 3D distance loss.

[0118] For example, aggregation can be the average or sum of the weighted 3D coordinate deviations of all pixels, and the final aggregation result can be used as the overall 3D distance loss.

[0119] By calculating the 3D distance loss using the S610~S640 method, the model's prediction accuracy for object contour edges can be specifically improved, resulting in clearer point cloud boundaries, reduced edge blurring and jagged edges, and improved overall visual effect and geometric accuracy of the final 3D reconstruction.

[0120] In some embodiments, such as Figure 7 As shown, the execution process of S350 may include the following S710~S720.

[0121] S710: Weight the epipolar geometric constraint value and the 3D distance loss to obtain the total loss.

[0122] For example, the total loss can be calculated according to the following formula (6). .

[0123] Formula (6): .

[0124] In formula (6), The 3D distance loss is used to supervise the geometric deviation between the predicted point cloud and the reference point cloud in 3D space, ensuring that the model output has a true physical scale. This is the epipolar geometry constraint value, used to constrain the epipolar geometry consistency of pixel matching point pairs between at least two viewpoint images, eliminating the need for stereo correction. As a balancing factor, used for regulation and Relative weight in total loss. The value of can be set by those skilled in the art based on the actual training situation.

[0125] In some examples, if it is desired that the model places greater emphasis on epipolar geometric consistency (e.g., when camera assembly errors are large), the size can be appropriately increased. If you want the model to prioritize absolute scale accuracy (e.g., when the reference point cloud accuracy is high), you can appropriately reduce the size. The embodiments disclosed herein... The specific value is not limited.

[0126] When calculating the total loss using the method shown in formula (6), since the model training process aims to minimize the total loss... As a result, the epipolar geometric constraint value is only added to the total loss as a weighting term. Therefore, the model does not require the epipolar error of each pixel matching pair to be exactly 0 during the optimization process, thus allowing for a certain degree of epipolar offset. To reduce the aforementioned epipolar offset and ensure that the epipolar error of each pixel matching pair strictly approaches 0, in some embodiments, the present disclosure also introduces the Lagrange multiplier method to transform the epipolar geometric constraint into an equality constraint, that is, strictly requiring the epipolar geometric constraint value to approach 0, so that the epipolar geometric constraint can be satisfied more strictly.

[0127] When the polar geometric constraint value is strictly required to approach 0, the total loss can be calculated using the following formula (7).

[0128] Formula (7): st , =1, 2, ..., M.

[0129] In formula (7), the "" after st , =1, 2, ..., M” are merely constraints, not numerical terms that can be directly used in calculations. The model cannot perform gradient descent based on these constraints during training. To enable the model to perform gradient descent based on these constraints, they need to be transformed into a computable, differentiable numerical term and added to the total loss. In this embodiment, the constraints are added to the total loss using the Lagrange multiplier method, resulting in the following formula (8).

[0130] Formula (8): in, It is a Lagrange multiplier.

[0131] By formula (8) When it does not increase the total loss, As the time constraint term increases, the total loss also increases synchronously. The model will make corrections during backpropagation, pulling the pixel matching pairs back to the epipolar lines, so that the epipolar error of each pixel matching pair strictly approaches 0, ensuring that the epipolar geometric constraints are strictly satisfied, and further improving the model's fitting accuracy of the epipolar geometric relationship of multi-view images.

[0132] S720: Update the network parameters of the 3D reconstruction model based on the total loss to iteratively train the 3D reconstruction model and obtain the target 3D reconstruction model.

[0133] The gradient of the total loss with respect to the network parameters can be calculated using the backpropagation algorithm. The network parameters are then updated using a pre-defined optimization algorithm (such as Adam or SGD) to complete the parameter optimization for this training iteration. This iterative training process is repeated until a pre-defined training stopping condition is met, such as the total loss decreasing to a pre-defined threshold or the pre-defined number of training iterations being completed. At this point, training stops, and the trained target 3D reconstruction model is obtained.

[0134] Through S710~S720, the total loss can be obtained by weighted summing of the epipolar geometric constraint value and the 3D distance loss. This allows the 3D reconstruction model to be simultaneously supervised by both epipolar geometric constraints and 3D physical scale accuracy during training, enabling it to directly output high-precision 3D point clouds with true physical scale during the inference phase. Building upon this, the Lagrange multiplier method is introduced to set the epipolar geometric constraint as an equality constraint, ensuring that the epipolar error of each pixel matching pair strictly approaches 0. This further improves the model's fitting accuracy to the epipolar geometric relationships of multi-view images, thereby further enhancing the 3D reconstruction accuracy.

[0135] In some embodiments, such as Figure 8 As shown, the execution process of S410 may include the following S810~S830.

[0136] S810: The sample image is divided into blocks by the embedding network in the 3D reconstruction model, and the image block features corresponding to at least two image blocks obtained after the block division are determined.

[0137] The sample image is input into the embedding network of the 3D reconstruction model. The embedding network first divides the sample image into multiple image patches of the same size. For example, the size of the sample image is... The size of the image patch is Then the number of image patches obtained is: Following this, the embedding network maps each image patch to its corresponding patch embedding through a linear projection layer, obtaining patch features for at least two image patches. The patch features can be represented as... ,in, The number of image patches, is the dimension of the feature vector.

[0138] Figure 9 This is an example diagram of a 3D reconstruction model. In some examples, the binocular view image includes the left eye image I. 1 And right eye image I 2 ,like Figure 9 As shown, left eye image I 1 And right eye image I 2After inputting the 3D reconstruction model, it is first divided into blocks by an embedding network, and then features are extracted from each image block to obtain the left eye image I. 1 And right eye image I 2 Multiple image patch features, each corresponding to a different image patch, are input into an encoding network for feature fusion (see S820 below).

[0139] certainly, Figure 9 The three-dimensional reconstruction model shown is only an example. Other three-dimensional reconstruction models may be used in this disclosure, and this disclosure does not limit them.

[0140] S820: The image patch features are fused through the coding network in the 3D reconstruction model to obtain intermediate features.

[0141] The features of each image patch are input into the encoding network of the 3D reconstruction model. The encoding network employs a Transformer-based encoder architecture (or a CNN encoder architecture), using a multi-head self-attention mechanism and a feedforward network to perform global feature fusion of the image patch features. Each encoder layer models the global context of the input image patch features, allowing for full interaction and fusion of feature information between different image patches. After processing by multiple encoder layers, the encoding network outputs intermediate features that fuse global context information. These intermediate features can be represented as... ,in, This is the feature dimension for encoding the network output.

[0142] In some examples, such as Figure 9As shown, the encoding network can include multiple inter-frame attention modules and multiple global attention modules, which are stacked alternately along the network depth direction. Multiple image patch features are input into the encoding network and then sequentially pass through the alternately stacked inter-frame attention modules and global attention modules. The inter-frame attention modules are used to fuse cross-viewpoint features between corresponding image patches in images from different viewpoints, while the global attention modules are used to fuse global contextual features between image patches within the same viewpoint. The alternating stacking of the inter-frame and global attention modules aims to: establish pixel-level feature correspondences between images from different viewpoints using the inter-frame attention modules, and establish global contextual feature associations within a single viewpoint image; the two complement each other in feature extraction. By alternately setting these two types of modules along the network depth direction, the network can simultaneously obtain cross-viewpoint geometric information and single-viewpoint semantic information at each layer, and gradually fuse them as the network deepens. This provides feature representations for depth prediction that possess both multi-viewpoint geometric consistency and global context awareness, effectively improving the model's depth estimation accuracy in challenging scenarios such as weakly textured regions and regions with repetitive textures. It should be noted that the inter-frame attention module introduces epipolar geometric constraints when performing cross-view feature fusion to improve the accuracy of cross-view feature matching. Specifically, the epipolar geometric constraints require that pixel matching point pairs in images from different viewpoints must lie on the corresponding epipolar lines, enabling the inter-frame attention module to establish accurate cross-view feature correspondences.

[0143] See Figure 9 As shown, the encoding network receives multiple image patch features, processes the image patch features based on multiple inter-frame attention modules and multiple global attention modules, and then outputs intermediate features corresponding to the multiple image patch features that fuse cross-view information and global context information.

[0144] S830: The intermediate features are reconstructed through the decoding network in the 3D reconstruction model to obtain the depth map.

[0145] Intermediate features are input into the decoding network of the 3D reconstruction model. The decoding network employs a Transformer-based decoder architecture or an upsampling structure based on a convolutional neural network. The decoding network progressively processes the intermediate features through multiple decoder layers, restoring the intermediate features from image patch-level representations to pixel-level predictions. Specifically, the decoding network progressively restores the spatial resolution of the feature map through upsampling operations (such as transposed convolution or interpolation upsampling), and finally outputs a depth map with the same spatial resolution as the sample image. .

[0146] The embedding network, encoding network, and decoding network together constitute the depth estimation module of the 3D reconstruction model.

[0147] In some examples, such as Figure 9 As shown, the encoding network outputs multiple intermediate features to the decoding network. The decoding network performs feature recombination and spatial position relationship restoration on the intermediate features corresponding to multiple image blocks, restores the spatial position relationship of each image block in the original image, and predicts the depth value of each pixel in each image block. Finally, it outputs a depth map with the same spatial resolution as the input image.

[0148] like Figure 9 As shown, after obtaining the depth map, the coordinate transformation module of the 3D reconstruction model is based on the camera's intrinsic parameter matrix. Backprojection (PROJ) is performed on the depth map to obtain the predicted point cloud. The output module of the 3D reconstruction model then calculates the 3D distance between the predicted point cloud and the reference point cloud. As a supervisory signal, it constrains the geometric accuracy and physical scale of the model output, ultimately outputting a 3D point cloud with a true physical scale.

[0149] Based on S810~S830, the encoder-decoder architecture can fully mine the global contextual information of different regions in the sample image, thereby improving the depth estimation accuracy of difficult regions such as edge regions and texture-deficient regions, providing a good feature foundation for obtaining high-precision predicted point clouds, and helping to improve the overall quality of the final 3D reconstruction results.

[0150] In some embodiments, such as Figure 10 As shown, the execution process of S420 may include the following S910~S920.

[0151] S910: Perform non-linear scaling on the depth map to obtain a scaled depth map.

[0152] To address the problem of vanishing gradients at long distances in related technologies, this disclosure solves the problem by performing non-linear scaling on the depth map. Non-linear scaling involves applying a non-linear transformation to the original depth values ​​in the depth map, assigning differentiated scaling weights to different depth intervals: for closer intervals with smaller depths, a first scaling ratio is used to retain more depth details; for farther intervals with larger depths, a second scaling ratio is used to mitigate the gradient impact of excessive depth spans, balance the training gradient distribution across different depth intervals, and prevent vanishing gradients at long distances, thereby avoiding a decrease in prediction accuracy due to vanishing gradients at long distances. The first scaling ratio is greater than the second scaling ratio. For example, for the near-distance interval with a smaller depth, the scaling ratio is larger than that for the far-distance interval. For instance, for every unit change in the original depth value, the change in the depth value after transformation in the near-distance interval is the first change, and the change in the depth value after transformation in the far-distance interval is the second change. The first change is greater than the second change, thus preserving the distinguishability of depth details in the near-distance interval and compressing the change in the depth value after transformation in the far-distance interval, thereby reducing the impact of the depth value of that interval on the loss function.

[0153] For example, nonlinear scaling can be cube root scaling, and the calculation formula can be found in the following formula (9).

[0154] Formula (9): .

[0155] In formula (9), pixel coordinates The original depth value at that location, This is the scaled depth value.

[0156] By applying cube root scaling to the depth map, the overall dynamic range of depth values ​​is compressed, reducing the numerical difference between near and far depth values. This balances the gradient contributions across different depth ranges, preventing the gradient from tilting entirely towards near distances, and allowing the model to simultaneously ensure depth estimation accuracy for both near and far distances. For example, if the original depth values ​​are in the range of 0 to 8 meters, after cube root scaling, the depth values ​​will be compressed to the range of 0 to approximately 2 meters. The depth values ​​in the near distance range of 0 to 1 meters will be mapped to 0 to 1, preserving the detail distinction within this range. Meanwhile, the depth values ​​in the far distance range of 2 to 8 meters will be compressed to 1 to 2 meters, reducing the numerical difference between different depth values ​​at far distances. This weakens the dominant effect of the near distance region in the loss function, relatively increasing the gradient contribution of the far distance region, and effectively improving the model's learning performance for depth estimation in far distance regions.

[0157] For example, nonlinear scaling can be logarithmic scaling, and the calculation formula can be found in the following formula (10).

[0158] Formula (10): .

[0159] Logarithmic scaling can compress the depth values ​​from the original range of [0.01, 100] to the range of [-2, 2], effectively weakening the dominant effect of the near-range region in the loss function and relatively increasing the loss contribution of the far-range region. The corresponding gradient signal can be effectively backpropagated to the network parameters, thereby alleviating the problem of gradient vanishing at a distance and improving the depth estimation accuracy across the entire range.

[0160] In the embodiments of this disclosure, nonlinear scaling processing includes, but is not limited to, cube root scaling and logarithmic scaling. Other nonlinear scaling methods that meet the requirements may also be used. Those skilled in the art can select an appropriate nonlinear scaling method according to the actual scenario, and this disclosure does not limit it.

[0161] S920: Based on the parameters of the image acquisition device, the scaled depth map is transformed to obtain the predicted point cloud corresponding to the sample image.

[0162] For the scaled depth map Each pixel in the image has pixel coordinates as follows: The corresponding scaled depth value The following formula (11) can be used to... Convert to 3D coordinates in the camera coordinate system.

[0163] Formula (11): .

[0164] In formula (11), For pixels in the depth map The 3D coordinates of the camera in the camera coordinate system after coordinate transformation. The scaled depth map The middle pixel coordinates are The mid-depth value at that location, This is the intrinsic parameter matrix of the camera that captured the sample images.

[0165] Through the above transformation, the pixel coordinates of each pixel in the scaled depth map and its corresponding scaled depth value The pixel is converted into a 3D point in 3D space, and the corresponding 3D coordinates are obtained. The set of 3D coordinates corresponding to all pixels in the scaled depth map is the predicted point cloud corresponding to the sample image.

[0166] Through the above S910~S920, the dynamic range of the compressed depth value can be non-linearly scaled to balance the gradient contribution of different depth intervals during training, avoid the decrease in prediction accuracy caused by gradient vanishing at long distances, and at the same time retain the detail discrimination of close distances. This can comprehensively improve the prediction accuracy of the entire depth estimation interval, further optimize the accuracy of the final generated prediction point cloud, provide more stable gradient feedback for model training, accelerate the convergence speed of the model, and improve the overall training efficiency.

[0167] In some embodiments, such as Figure 11 As shown, the execution process of S920 may include the following S1010~S1030.

[0168] S1010: Determine the first random disturbance corresponding to the first parameter of the image acquisition device.

[0169] The first parameter includes the intrinsic parameters of the image acquisition device, and the first random perturbation may include the intrinsic parameter perturbation. The intrinsic parameter perturbation may be a pre-set offset of the intrinsic parameters of the camera in the image acquisition device. By adding the intrinsic parameter perturbation, it is possible to simulate the hardware assembly error or parameter deviation caused by the small displacement of the camera in the actual situation, thereby improving the robustness of the model to parameter errors.

[0170] In some examples, the first parameter is the intrinsic parameter matrix of the camera in the image acquisition device. A corresponding random perturbation offset is set for each matrix element in the intrinsic parameter matrix to obtain the random perturbation matrix corresponding to the intrinsic parameter matrix. The specific value of the first random perturbation can be set by those skilled in the art according to actual conditions, and this disclosure does not limit this. The intrinsic parameters in the camera's intrinsic parameter matrix include: , , and , This indicates the focal length of the camera in the horizontal direction. This indicates the focal length of the camera in the vertical direction. Let x be the x-coordinate of the camera's optical center. This represents the ordinate of the camera's optical center. Each intrinsic parameter matrix can be based on a preset random distribution (such as a Gaussian or uniform distribution). The parameters in the table generate corresponding random perturbation offsets.

[0171] For example, the first random perturbation may include a focal length perturbation, which simulates a small zoom or distortion change of an optical lens under actual operating conditions, and may include... Corresponding horizontal focal length random perturbation and / or including Corresponding vertical focal length random perturbation .

[0172] For example, the first random perturbation may include an optical center perturbation, which is an offset of the simulated imaging center and may include optical center coordinates. The corresponding random perturbation of the optical center x-coordinate and / or include optical center coordinates The corresponding random perturbation of the optical center ordinate .

[0173] It should be noted that the first random perturbation can be regenerated in each training iteration or each training batch, so that the model is exposed to a large number of different parameter offsets during training, thereby improving its robustness to actual hardware deviations.

[0174] S1020: Apply a first random perturbation to the first parameter to obtain the first augmented parameter.

[0175] The first parameter is calculated (e.g., added or multiplied) with the offset corresponding to the first random perturbation to obtain the first augmented parameter after applying the first random perturbation to the first parameter. In some examples, the first augmented parameter may include augmented intrinsic parameters (e.g., the augmented intrinsic parameter matrix described below). ).

[0176] For example, the first parameter is an intrinsic parameter matrix. The first random perturbation is the intrinsic parameter perturbation. Apply intrinsic parameter perturbation to the intrinsic parameter matrix The augmented intrinsic parameter matrix was then obtained. , Among them, the augmented intrinsic parameter matrix That is, the first augmentation parameter. Including horizontal focal length random perturbation Vertical focal length random perturbation Random perturbation of the abscissa of the optical center Random perturbation of the optical center's ordinate .

[0177] S1030: Based on the first augmentation parameter, perform coordinate transformation on the scaled depth map to obtain the predicted point cloud.

[0178] During the coordinate transformation, the first augmentation parameter replaces the original first parameter of the image acquisition device, and the transformation from pixel coordinates to three-dimensional spatial coordinates is completed according to the same transformation logic as formula (11), resulting in the predicted point cloud calculated based on the first augmentation parameter. The augmentation intrinsic parameter matrix is ​​used during the coordinate transformation process. Replace the original intrinsic parameter matrix in formula (11) .

[0179] In some embodiments, such as Figure 12As shown, the execution process of S330 may include the following S1110~S1130.

[0180] S1110: Determine the second random perturbation corresponding to the second parameter of the image acquisition device; the second parameter includes at least one of the following: rotation matrix, translation vector, and baseline distance.

[0181] The second random perturbation can be a pre-set offset to the second parameter (e.g., at least one of rotation matrix, translation vector, and baseline distance), which can simulate small offsets related to the second parameter in real-world scenarios, such as small offsets in camera position and pose, to further enhance the model's adaptability to various extrinsic parameter errors.

[0182] In some examples, a second random perturbation can be added to the rotation matrix and translation vector respectively. For example, a small angular perturbation (such as rotating 0.1° to 0.5° around each axis) can be applied to the rotation matrix R, and a small positional offset (such as ±1mm to ±5mm) can be applied to the translation vector t to simulate the pose deviation of the stereo camera during assembly or use.

[0183] In some examples, a corresponding second random perturbation can be applied to the baseline distance b, such as applying a small random offset to the baseline distance b, to simulate the minute changes in the binocular camera baseline under temperature changes or mechanical vibrations.

[0184] The specific value of the second random disturbance corresponding to the second parameter can be set by those skilled in the art based on the actual hardware precision, and this disclosure does not limit this.

[0185] It should be noted that a second random perturbation can be regenerated in each training iteration or each training batch, so that the model is exposed to a large number of different parameter offsets during training, thereby improving its robustness to actual hardware deviations.

[0186] S1120: Apply a second random perturbation to the second parameter to obtain the second augmented parameter.

[0187] The second parameter is calculated (e.g., added or multiplied) with the offset corresponding to the second random perturbation to obtain the second augmented parameter after applying the second random perturbation to the second parameter.

[0188] For example, the second parameter is the rotation matrix R, and a rotation matrix perturbation is applied to the rotation matrix R. Then, the augmented rotation matrix is ​​obtained. .in, , It is a tiny rotation matrix (generated by tiny angular perturbations around each axis).

[0189] For example, the second parameter is the translation vector t, and a translation vector perturbation is applied to the translation vector t. Then, the augmented translation vector is obtained. .in, , It is a tiny translation vector.

[0190] For example, the second parameter is the baseline distance b, and a baseline perturbation is applied to the baseline distance b. The augmented baseline was then obtained. .in, .

[0191] S1130: Determine the epipolar geometric constraint value based on the second augmentation parameter, epipolar geometry relationship, and pixel matching pairs.

[0192] After applying the second random perturbation corresponding to the second parameter based on S1120, the augmented epipolar geometric constraint value is determined based on the obtained second augmented parameter. Specifically, a rotation matrix perturbation is applied. The subsequent rotation matrix Instead of the original rotation matrix R, a translation vector perturbation is applied. The translation vector after Instead of the original translation vector t, a baseline perturbation is applied. The augmented baseline was then obtained. Instead of the original baseline distance b, the augmented foundation matrix is ​​calculated according to formula (3). Substitute the homogeneous pixel coordinates of the matching pixel pair into the left side of equation (4) to calculate... The corresponding values ​​are used to obtain the epipolar constraint residuals for each pixel matching pair; the epipolar constraint residuals for all pixel matching pairs are aggregated (e.g., summed or averaged) to obtain the augmented epipolar geometric constraint values.

[0193] In S350, a 3D reconstruction model can be trained based on the augmented epipolar geometric constraint values ​​and 3D distance loss to obtain the target 3D reconstruction model.

[0194] It should be noted that training the 3D reconstruction model based on the augmented epipolar geometric constraint values ​​aims to improve the model's coverage of hardware bias sources. The random perturbations applied to the external parameters involve perturbations of the rotation matrix, translation vector, and baseline distance. This covers various non-ideal sources that may be encountered in actual deployments, such as rotational deviations during camera module installation, offsets of the optical center baseline due to assembly tolerances, and inherent parameter inconsistencies between heterogeneous sensors due to selection differences. By training the model using the augmented epipolar geometric constraint values, the model can establish generalized responses to these bias sources during parameter updates, thus effectively compatibility with biases caused by various reasons in actual deployments, enabling the model to adapt to more types of hardware configurations.

[0195] By randomly introducing different types of parameter perturbations (such as the first parameter and / or the second parameter mentioned above) during the training process to augment the data, the model can learn to adapt to the feature representation of various parameter errors during the training phase. This reduces the negative impact of parameter deviations on the accuracy of 3D reconstruction in real-world application scenarios, improves the robustness and stability of the model after actual deployment, and prevents a significant decrease in reconstruction accuracy due to hardware processing and assembly errors or small pose shifts during use.

[0196] Figure 13 This is a schematic flowchart of a three-dimensional reconstruction method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, and the specific electronic devices can be found in the description of the deployable electronic device portion of the three-dimensional reconstruction device 103 in the above embodiment, which will not be repeated here.

[0197] like Figure 13 As shown, the three-dimensional reconstruction method includes the following S1310~S1320.

[0198] S1310: Determine the image to be processed.

[0199] Taking a binocular camera as an example, the image to be processed includes a left-eye image and a right-eye image, which are acquired by the left-eye camera and the right-eye camera at the same time. In practical application scenarios, the image to be processed can be a road scene image acquired by an intelligent driving vehicle during driving, or an image of the surrounding environment acquired by a robot during navigation operations, or a virtual-real fusion scene image acquired by an augmented reality device. This disclosure does not make any specific limitations on this.

[0200] It is understandable that the images to be processed can be acquired by the same type of multi-view camera as those used in the training phase, or by multi-view cameras of different models or configurations. Since the target 3D reconstruction model of this scheme has learned the inherent epipolar geometric relationship between images from different viewpoints through epipolar geometric constraints during the training phase, the inference phase does not need to rely on a standard binocular configuration. Therefore, the acquisition equipment for the images to be processed does not need to meet strict binocular alignment conditions, which has significant flexibility advantages in practical engineering deployment.

[0201] S1320: Based on the target 3D reconstruction model, process the image to be processed to obtain the 3D point cloud corresponding to the image to be processed.

[0202] The target 3D reconstruction model is trained based on the 3D reconstruction model training method described in the above embodiments. The image to be processed is input into the target 3D reconstruction model, which processes the image and directly outputs the corresponding 3D point cloud.

[0203] It should be noted that the target 3D reconstruction model does not require independent stereo correction or disparity map post-processing during the inference phase, and can output a 3D point cloud with true physical scale end-to-end. This is because the target 3D reconstruction model has learned the epipolar geometric relationship between at least two viewpoint images and established an accurate mapping from pixel depth to physical scale through joint supervision of epipolar geometric constraints and 3D distance loss during the training phase. Therefore, the inference phase only requires one forward propagation to complete the end-to-end mapping from the input image to the 3D point cloud, simplifying the 3D reconstruction process and avoiding the error accumulation and computational overhead caused by stereo correction and disparity processing in traditional methods.

[0204] Exemplary device Figure 14 This is a schematic diagram of the structure of a three-dimensional reconstruction model training device provided in an exemplary embodiment of this disclosure. Figure 14 As shown, the 3D reconstruction model training device 1400 includes: a sample determination module 1401, a 3D reconstruction module 1402, an epipolar geometric constraint value determination module 1403, a 3D distance loss determination module 1404, and a 3D reconstruction model training module 1405.

[0205] The sample determination module 1401 is used to determine the sample image group for the target scene, the reference point cloud corresponding to the sample image group, and the parameters of the image acquisition device for acquiring the sample image group; wherein, the sample image group includes at least two images from different perspectives acquired at the same time.

[0206] The 3D reconstruction module 1402 is used to perform 3D reconstruction of the target scene based on parameters, sample image groups and 3D reconstruction model to obtain the predicted point cloud corresponding to the target scene.

[0207] The epipolar geometric constraint value determination module 1403 is used to determine the epipolar geometric constraint value based on parameters and pixel matching pairs in the sample image group; wherein, the pixel matching pair is a set of homogeneous pixel coordinates of each projection point corresponding to the same three-dimensional space point in at least two different viewpoints, and the three-dimensional space point is located in the field of view overlap region jointly covered by at least two different viewpoints.

[0208] The 3D distance loss determination module 1404 is used to determine the 3D distance loss between the predicted point cloud and the reference point cloud.

[0209] The 3D reconstruction model training module 1405 is used to train the 3D reconstruction model based on epipolar geometric constraint values ​​and 3D distance loss to obtain the target 3D reconstruction model.

[0210] In some embodiments, the polar geometric constraint value determination module 1403 includes a first determination submodule and a second determination submodule.

[0211] The first determination submodule is used to determine the epipolar geometric relationship between at least two images from different viewpoints based on parameters.

[0212] The second determination submodule is used to determine the epipolar geometric constraint value based on the epipolar geometric relationship and pixel matching pairs.

[0213] In some embodiments, the 3D reconstruction module 1402 includes a depth estimation submodule and a first processing submodule.

[0214] The depth estimation submodule is used to perform depth estimation on the sample image based on the depth estimation module of the 3D reconstruction model, and obtain a depth map indicating the depth information of each pixel in the sample image; the sample image is any image in the sample image group; The first processing submodule is used to transform the depth map based on parameters through the coordinate transformation module of the 3D reconstruction model to obtain the predicted point cloud corresponding to the sample image.

[0215] In some embodiments, the depth estimation module includes: a block segmentation unit, a feature fusion unit, and a reconstruction unit.

[0216] The segmentation unit is used to segment the sample image into blocks through the embedding network in the 3D reconstruction model, and to determine the image block features corresponding to at least two image blocks obtained after segmentation. The feature fusion unit is used to fuse image patch features through the coding network in the 3D reconstruction model to obtain intermediate features; The reconstruction unit is used to reconstruct intermediate features through the decoding network in the 3D reconstruction model to obtain a depth map.

[0217] In some embodiments, the first processing submodule includes: a nonlinear scaling processing unit and a coordinate transformation unit.

[0218] The non-linear scaling processing unit is used to perform non-linear scaling processing on the depth map to obtain a scaled depth map. The coordinate transformation unit is used to perform coordinate transformation on the scaled depth map based on parameters to obtain the predicted point cloud corresponding to the sample image.

[0219] In some embodiments, the coordinate transformation unit is specifically used to: determine a first random perturbation corresponding to a first parameter of the image acquisition device; apply the first random perturbation to the first parameter to obtain a first augmented parameter; and perform coordinate transformation on the scaled depth map based on the first augmented parameter to obtain a predicted point cloud.

[0220] In some embodiments, the second determining submodule is specifically used to: determine a second random perturbation corresponding to a second parameter of the image acquisition device; the second parameter includes at least one of a rotation matrix, a translation vector, and a baseline distance; apply a second random perturbation to the second parameter to obtain a second augmented parameter; and determine an epipolar geometric constraint value based on the second augmented parameter, the epipolar geometric relationship, and the pixel matching pairs.

[0221] In some embodiments, the three-dimensional distance loss determination module 1404 includes: a third determination submodule, a calculation submodule, a weighted processing submodule, and an aggregation processing submodule.

[0222] The third determination submodule is used to determine the predicted 3D coordinates of each pixel in the sample image based on the predicted point cloud, and to determine the reference 3D coordinates of each pixel in the sample image and the edge mask weights corresponding to each pixel position based on the reference point cloud.

[0223] The calculation submodule is used to calculate the 3D coordinate deviation between the predicted 3D coordinates and the reference 3D coordinates corresponding to the same pixel.

[0224] The first weighted processing submodule is used to perform weighted processing on the three-dimensional coordinate deviation of the corresponding pixel position based on the edge mask weight of each pixel position, so as to obtain the weighted three-dimensional coordinate deviation.

[0225] The aggregation processing submodule is used to aggregate the weighted 3D coordinate deviations corresponding to each pixel position to obtain the 3D distance loss.

[0226] In some embodiments, the 3D reconstruction model training module 1405 includes a second weighted processing submodule and a training submodule.

[0227] The second weighted processing submodule is used to weight the epipolar geometric constraint value and the 3D distance loss to obtain the total loss.

[0228] The training submodule is used to update the network parameters of the 3D reconstruction model based on the total loss, so as to iteratively train the 3D reconstruction model and obtain the target 3D reconstruction model.

[0229] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0230] Figure 15 This is a schematic diagram of the structure of a three-dimensional reconstruction apparatus provided in an exemplary embodiment of this disclosure. Figure 15 As shown, the three-dimensional reconstruction device 1500 includes: an image determination module 1501 and a point cloud generation module 1502.

[0231] Image determination module 1501 is used to determine the image to be processed.

[0232] The point cloud generation module 1502 is used to process the image to be processed based on the target 3D reconstruction model to obtain the 3D point cloud corresponding to the image to be processed; wherein, the target 3D reconstruction model is trained by the 3D reconstruction model training method in the above embodiment.

[0233] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0234] Exemplary electronic devices Figure 16 A structural diagram of an electronic device provided in an embodiment of this disclosure includes at least one processor 111 and a memory 112.

[0235] The processor 111 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 11 to perform desired functions.

[0236] The memory 112 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 111 may execute one or more computer program instructions to implement the three-dimensional reconstruction model training method or three-dimensional reconstruction method and / or other desired functions of the various embodiments of this disclosure described above.

[0237] In one example, the electronic device 11 may also include an input device 113 and an output device 114, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0238] The input device 113 may also include, for example, a keyboard, a mouse, etc.

[0239] The output device 114 can output various information to the outside, including, for example, a display, a speaker, a printer, a communication network and its connected remote output devices, etc.

[0240] Of course, for the sake of simplicity, Figure 16 Only some of the components of the electronic device 11 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 11 may include any other suitable components depending on the specific application.

[0241] Exemplary computer program products and computer-readable storage media In addition to the methods and devices described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the three-dimensional reconstruction model training method or three-dimensional reconstruction method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0242] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. These programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0243] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the three-dimensional reconstruction model training method or three-dimensional reconstruction method of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0244] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0245] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0246] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for training a 3D reconstruction model, comprising: Determine the sample image set for the target scene, the reference point cloud corresponding to the sample image set, and the parameters of the image acquisition device for acquiring the sample image set; wherein, the sample image set includes at least two images from different perspectives acquired at the same time; Based on the parameters, the sample image group, and the 3D reconstruction model, the target scene is reconstructed in 3D to obtain the predicted point cloud corresponding to the target scene. Based on the parameters and the pixel matching pairs in the sample image group, the epipolar geometric constraint value is determined; wherein, the pixel matching pair is a set of homogeneous pixel coordinates of each projection point corresponding to the same three-dimensional space point in the at least two different viewpoints, and the three-dimensional space point is located in the field of view overlap region jointly covered by the at least two different viewpoints. Determine the three-dimensional distance loss between the predicted point cloud and the reference point cloud; Based on the epipolar geometric constraint value and the three-dimensional distance loss, the three-dimensional reconstruction model is trained to obtain the target three-dimensional reconstruction model.

2. The method according to claim 1, wherein, The step of determining the epipolar geometric constraint value based on the parameters and pixel matching pairs in the sample image group includes: Based on the parameters, the epipolar geometric relationship between the at least two images from different perspectives is determined; The epipolar geometric constraint value is determined based on the epipolar geometric relationship and the pixel matching pair.

3. The method according to claim 1, wherein, The step of performing 3D reconstruction of the target scene based on the parameters, the sample image group, and the 3D reconstruction model to obtain the predicted point cloud corresponding to the target scene includes: The depth estimation module based on the 3D reconstruction model performs depth estimation on the sample image to obtain a depth map indicating the depth information of each pixel in the sample image; the sample image is any one image in the sample image group; Based on the parameters, the depth map is transformed by the coordinate transformation module of the 3D reconstruction model to obtain the predicted point cloud corresponding to the sample image.

4. The method according to claim 3, wherein, The depth estimation module based on the 3D reconstruction model performs depth estimation on the sample image to obtain a depth map indicating the depth information of each pixel in the sample image, including: The sample image is divided into blocks using the embedding network in the 3D reconstruction model, and the image block features corresponding to at least two image blocks obtained after the block division are determined. The image patch features are fused using the coding network in the 3D reconstruction model to obtain intermediate features; The intermediate features are reconstructed using the decoding network in the 3D reconstruction model to obtain the depth map.

5. The method according to claim 3, wherein, The step of transforming the depth map based on the parameters using the coordinate transformation module of the 3D reconstruction model to obtain the predicted point cloud corresponding to the sample image includes: The depth map is non-linearly scaled to obtain a scaled depth map; Based on the parameters, a coordinate transformation is performed on the scaled depth map to obtain the predicted point cloud corresponding to the sample image.

6. The method according to claim 5, wherein, The step of performing coordinate transformation on the scaled depth map based on the parameters to obtain the predicted point cloud corresponding to the sample image includes: Determine the first random perturbation corresponding to the first parameter of the image acquisition device, wherein the first parameter includes the intrinsic parameter of the image acquisition device; Applying the first random perturbation to the first parameter yields the first augmented parameter; Based on the first augmentation parameter, the scaled depth map is subjected to coordinate transformation to obtain the predicted point cloud.

7. The method according to claim 2, wherein, The step of determining the epipolar geometric constraint value based on the epipolar geometric relationship and the pixel matching pair includes: Determine the second random perturbation corresponding to the second parameter of the image acquisition device, wherein the second parameter includes at least one of the parameters of rotation matrix, translation vector and baseline distance; Applying the second random perturbation to the second parameter yields the second augmented parameter; The epipolar geometric constraint value is determined based on the second augmentation parameter, the epipolar geometry relationship, and the pixel matching pair.

8. The method according to any one of claims 1-7, wherein, Determining the 3D distance loss between the predicted point cloud and the reference point cloud includes: Based on the predicted point cloud, the predicted three-dimensional coordinates corresponding to each pixel in the sample image are determined, and based on the reference point cloud, the reference three-dimensional coordinates corresponding to each pixel in the sample image and the edge mask weights corresponding to each pixel position are determined. Calculate the 3D coordinate deviation between the predicted 3D coordinates and the reference 3D coordinates corresponding to the same pixel; Based on the edge mask weights corresponding to each pixel position, the three-dimensional coordinate deviations of the corresponding pixel positions are weighted to obtain the weighted three-dimensional coordinate deviations. The weighted 3D coordinate deviations corresponding to each pixel position are aggregated to obtain the 3D distance loss.

9. The method according to any one of claims 1-7, wherein, The process of training the 3D reconstruction model based on the epipolar geometric constraint value and the 3D distance loss to obtain the target 3D reconstruction model includes: The total loss is obtained by weighting the epipolar geometric constraint value and the three-dimensional distance loss. The network parameters of the 3D reconstruction model are updated based on the total loss to iteratively train the 3D reconstruction model and obtain the target 3D reconstruction model.

10. A three-dimensional reconstruction method, comprising: Identify the image to be processed; The image to be processed is processed based on the target 3D reconstruction model to obtain the 3D point cloud corresponding to the image to be processed; The target 3D reconstruction model is trained based on the 3D reconstruction model training method described in any one of claims 1-9.

11. A three-dimensional reconstruction model training device, comprising: The sample determination module is used to determine a sample image group for a target scene, a reference point cloud corresponding to the sample image group, and parameters of the image acquisition device for acquiring the sample image group; wherein, the sample image group includes at least two images from different perspectives acquired at the same time; The 3D reconstruction module is used to perform 3D reconstruction of the target scene based on the parameters, the sample image group and the 3D reconstruction model, to obtain the predicted point cloud corresponding to the target scene; The epipolar geometric constraint value determination module is used to determine the epipolar geometric constraint value based on the parameters and the pixel matching pairs in the sample image group; wherein, the pixel matching pair is a set of homogeneous pixel coordinates of each projection point corresponding to the same three-dimensional space point in the images of at least two different viewpoints, and the three-dimensional space point is located in the field of view overlap region jointly covered by the images of at least two different viewpoints; A three-dimensional distance loss determination module is used to determine the three-dimensional distance loss between the predicted point cloud and the reference point cloud; The 3D reconstruction model training module is used to train the 3D reconstruction model based on the epipolar geometric constraint value and the 3D distance loss to obtain the target 3D reconstruction model.

12. A three-dimensional reconstruction device, comprising: Image determination module, used to determine the image to be processed; The point cloud generation module is used to process the image to be processed based on the target 3D reconstruction model to obtain the 3D point cloud corresponding to the image to be processed. The target 3D reconstruction model is trained based on the 3D reconstruction model training method described in any one of claims 1-9.

13. A computer-readable storage medium storing a computer program for executing the three-dimensional reconstruction model training method according to any one of claims 1-9, or the three-dimensional reconstruction method according to claim 10.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the three-dimensional reconstruction model training method according to any one of claims 1-9, or the three-dimensional reconstruction method according to claim 10.

Images