A self-supervised end-to-end visual reconstruction method and system

Abstract

The present application provides a self-supervised end-to-end visual reconstruction method and system, which relates to the field of computer vision processing technology. The method includes: first obtaining multi-camera parameters and image data of different perspectives, including the focal length, lens distortion and principal coordinate point data of each camera, as well as the pixel coordinates of the current frame of the first camera and the reference frame of the second camera, and then building an end-to-end training model, by calculating the reprojection error of the pixel points of the current frame and the reference frame, using the Gauss-Newton iteration method to solve the parameter update amount, iteratively optimizing the camera pose, pixel correspondence and depth data, and after the reprojection error converges, combining the obtained data set to reconstruct the three-dimensional coordinates and transform and splice to achieve three-dimensional scene reconstruction. The implementation of this solution can achieve end-to-end self-supervised training without relying on manual annotation, thereby achieving three-dimensional visual reproduction.

Description

Technical Field

[0001] The present application relates to the field of computer vision processing technology, and in particular to a self-supervised end-to-end visual reconstruction method and system. Background Art

[0002] In today's digital age, visual reconstruction technology plays a vital role in multiple fields. This technology aims to restore three-dimensional scene information from two-dimensional images. However, the requirements for dataset accuracy and manual annotation supervision are increasing, necessitating an end-to-end self-supervised approach.

[0003] In recent years, traditional end-to-end visual reconstruction methods have received increasing research attention. This method directly outputs three-dimensional scene information after inputting images into a deep learning model for processing. The advantage of this method is that the calculation can be accelerated in real time by GPU and has certain generalization performance for low-brightness or weak-texture areas.

[0004] However, traditional visual reconstruction methods typically rely on supervised learning with large amounts of manually annotated data to determine the correspondence between images and the 3D structure of the scene. This approach is not only time-consuming and labor-intensive, but also difficult to ensure the accuracy of annotations, limiting the application of visual reconstruction technology in large-scale scenes. Summary of the Invention

[0005] This application provides a self-supervised end-to-end visual reconstruction method and system for achieving end-to-end self-supervised training without relying on manual labeling, thereby realizing three-dimensional visual reproduction.

[0006] In a first aspect, the present application provides a self-supervised end-to-end visual reconstruction method, which is applied to a visual reconstruction system. The method comprises: S1, obtaining multiple camera parameters and image data captured at different viewing angles covering a target image, wherein the camera parameters include focal length data, lens distortion parameters, and principal coordinate point data, and the image data includes at least current frame pixels captured by a first camera and reference frame pixels captured by a second camera; S2, obtaining the coordinates of the current frame pixels of the target object captured by the first camera, and the coordinates of the reference frame pixels corresponding to the current frame pixels captured by the second camera from different directions; S3, constructing an end-to-end training set. Training model, the end-to-end training model is a model that iteratively trains the pose data of multiple cameras, the correspondence between pixels on the image data, and the initial pixel depth data by calculating the reprojection error between the pixels of the current frame and the pixels of the reference frame; S4, repeating step S3 until the reprojection error tends to converge, and obtaining the final pixel correspondence dataset, the final camera pose dataset, and the final pixel depth dataset; S5, combining the final pixel correspondence dataset and the final pixel depth dataset to reconstruct the three-dimensional coordinates; S6, combining the final camera pose dataset to transform and splice the three-dimensional coordinates to reconstruct the three-dimensional scene.

[0007] By employing the above technical solution and adopting a self-supervised end-to-end training approach, the model eliminates the need for extensive manual data annotation, reducing the workload and cost of data annotation. It also directly learns the features and relationships required for visual reconstruction from raw image data, improving the efficiency and autonomy of model training. It fully utilizes parameters from multiple cameras, including focal length, lens distortion parameters, and principal coordinate data, as well as image data from different viewpoints, comprehensively considering various camera imaging factors to improve the accuracy and stability of visual reconstruction. By obtaining the current frame pixel coordinates and the corresponding reference frame pixel coordinates, and iteratively training the pose data, pixel correspondences, and initial pixel depth data from multiple cameras using reprojection errors, it accurately calculates pixel correspondences between different frames and the depth information for each pixel, providing an accurate data foundation for 3D reconstruction. 3D coordinates are reconstructed based on the final pixel correspondence dataset and the final pixel depth dataset. These coordinates are then transformed and spliced together with the final camera pose dataset to fully reconstruct the 3D scene, achieving accurate reconstruction from multi-view image data.

[0008] In combination with some embodiments of the first aspect, in some embodiments, before the steps of obtaining multiple camera parameters and capturing image data from different perspectives covering the target image, it also includes: constructing a pose mapping model through a deep learning algorithm using multiple historical image data with camera pose annotations; inputting the image data into the pose mapping model to obtain camera pose data corresponding to the image data.

[0009] By adopting the above technical solution and using deep learning algorithms to mine the pose information patterns in historical data, an initial and relatively accurate camera pose reference is provided for subsequent end-to-end training models.

[0010] In combination with some embodiments of the first aspect, in some embodiments, after the steps of obtaining multiple camera parameters and shooting image data covering the target image at different perspectives, it also includes: after extracting visual feature information in the image data through a feature extraction algorithm, determining a high-dimensional feature matrix corresponding to the image data based on the visual feature information; inputting multiple image data into a correspondence determination model to obtain an initial pixel correspondence, the correspondence determination model is pre-obtained through machine learning training on an image set with annotated correspondences between pixel points, and the initial pixel correspondence is the position correspondence between pixel points of the same target part in the current frame and the reference frame.

[0011] By employing these technical solutions, visual feature information helps highlight key image components, a high-dimensional feature matrix can more comprehensively represent image features, and a correspondence determination model, based on machine learning training, can initially determine pixel correspondences. These technical features work together to lay the foundation for subsequent construction of the relationship matrix and acquisition of initial pixel depth data, enhancing the reliability of pixel correlation analysis and improving the accuracy of depth information determination in 3D reconstruction.

[0012] In combination with some embodiments of the first aspect, in some embodiments, after extracting visual feature information from the image data through a feature extraction algorithm, after the step of determining the high-dimensional feature matrix corresponding to the image data based on the visual feature information, it also includes: combining the initial pixel correspondence and the high-dimensional feature matrix to obtain a relationship matrix; and performing operations on the relationship matrix through a pixel association network to obtain initial pixel depth data.

[0013] By adopting the above technical solution, the relationship matrix integrates the pixel correspondence and image feature information, and the pixel association network mines the depth association rules between pixels based on this matrix. The two work together to more accurately infer pixel depth information from multi-view image data, making up for the shortcomings of traditional methods in acquiring depth data, making the 3D reconstruction more accurate in the depth dimension, and enhancing the stereoscopic sense and realism of the entire 3D reconstructed scene.

[0014] In combination with some embodiments of the first aspect, in some embodiments, before inputting multiple image data into the correspondence determination model to obtain the initial pixel correspondence step, it also includes: preprocessing the image data through Gaussian filtering; using the optical flow algorithm to calculate the motion information of pixels between adjacent frames in the image data; and obtaining the motion trend information of dynamic objects in the image and the preliminary division of dynamic areas and static areas through the motion information.

[0015] By adopting the above technical solution, the processed image data can avoid incorrect matching due to noise and interference from dynamic objects in subsequent steps such as determining the initial pixel correspondence, thereby improving the accuracy of pixel correspondence, and thus ensuring the accuracy of image fusion from each perspective during the three-dimensional reconstruction process, making the final reconstructed three-dimensional scene clearer, more complete and without obvious defects.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the step of optimizing the pose data, initial pixel correspondences, and multiple initial pixel depth data of multiple cameras based on the parameter update amount specifically includes: applying the chain rule to calculate the gradient information of the reprojection error to the pose data and the initial pixel depth data respectively; and updating the pose data and the initial pixel depth data using a gradient descent algorithm in combination with the gradient information.

[0017] By adopting the above technical solution, the chain rule accurately analyzes the changing relationship between error and data, providing a precise update direction for the gradient descent algorithm. This gradient-based optimization method can efficiently adjust the camera pose and pixel depth data during iterative training, allowing the model to quickly converge to the optimal solution, effectively improving the efficiency and accuracy of 3D reconstruction.

[0018] In combination with some embodiments of the first aspect, in some embodiments, the step of converting and splicing the three-dimensional coordinates in combination with the final camera pose data set to reconstruct the three-dimensional scene specifically includes: converting the three-dimensional positions of the pixel points captured by multiple cameras in their respective camera coordinate systems into a common coordinate system in combination with the coordinate transformation matrix and the final camera pose data set to determine the common coordinate data of the multiple pixel points in the common coordinate system; combining adjacent pixel points in the three-dimensional space in the common coordinate data, and finally splicing out the three-dimensional reconstruction result of the entire scene.

[0019] By adopting the above technical solution, the coordinate transformation matrix realizes the unified conversion of the coordinate system according to the camera pose data, so that the pixels of different perspectives can be accurately integrated in the same coordinate system. This process effectively integrates multi-perspective information and avoids reconstruction errors caused by coordinate system differences. The final 3D reconstructed scene is complete, continuous, and spatially arranged reasonably, highly restoring the overall appearance of the real scene.

[0020] In a second aspect, the present application provides a visual reconstruction system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the visual reconstruction system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0021] In a third aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a visual reconstruction system, enable the visual reconstruction system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0022] In a fourth aspect, the present application provides a computer program product, which, when executed on a visual reconstruction system, enables the visual reconstruction system to execute the method described in the first aspect and any possible implementation of the first aspect.

[0023] One or more technical solutions provided in the embodiments of this application have at least the following technical effects or advantages:

[0024] 1. By building an end-to-end training model and using reprojection error to iteratively train and optimize camera pose, pixel correspondence, and pixel depth data, this technology effectively solves the problem of low efficiency in existing 3D reconstruction technologies, which relies on manual labeling. It then achieves end-to-end self-supervision and accurately restores the true structure and morphology of the target scene.

[0025] 2. Due to the technical means of extracting visual feature information to construct a high-dimensional feature matrix and combining the correspondence determination model to obtain the initial pixel correspondence, the problems of low reliability of pixel association analysis and inaccurate depth information determination in the existing technology are effectively solved, thereby achieving the technical effect of improving the accuracy of depth information determination in three-dimensional reconstruction and enhancing the reliability of pixel point association analysis.

[0026] 3. Due to the technical means of combining the initial pixel correspondence with the high-dimensional feature matrix to obtain the initial pixel depth data through pixel association network operation, the problem of inaccurate pixel depth data acquisition in the existing technology is effectively solved, thereby achieving the technical effect of making the three-dimensional reconstruction more accurate in the depth dimension and improving the stereoscopic and realism of the reconstructed scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 1 is a flow chart of a self-supervised end-to-end visual reconstruction method in an embodiment of the present application;

[0028] Figure 2 1 is a schematic diagram of a scenario of a self-supervised end-to-end visual reconstruction method in an embodiment of the present application;

[0029] Figure 3 It is a schematic diagram of the physical device structure of the visual reconstruction system in the embodiment of the present application. DETAILED DESCRIPTION

[0030] The terms used in the following examples of the present application are for the purpose of describing specific embodiments only and are not intended to limit the present application. As used in the specification and appended claims of the present application, the singular expressions "a," "an," "said," "above," "the," and "this" are intended to include plural expressions as well, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application refers to and encompasses any or all possible combinations of one or more of the listed items.

[0031] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0032] For ease of understanding, the following describes the process of the method provided by this implementation. Figure 1 , which is a flow chart of the self-supervised end-to-end visual reconstruction method in an embodiment of the present application.

[0033] S1. Acquire multiple camera parameters and capture image data at different viewing angles covering a target image, wherein the camera parameters include focal length data, lens distortion parameters, and principal coordinate point data, and the image data includes at least current frame pixels captured by a first camera and reference frame pixels captured by a second camera;

[0034] The visual reconstruction system first needs to accurately measure the focal length data of each camera in the system in the horizontal and vertical directions. For common camera models, focal length is a key parameter that determines the scaling ratio of the camera imaging.

[0035] Specifically, the camera to be calibrated can be fixed in a stable position to ensure it does not shake during capture. A checkerboard calibration plate is then placed within the camera's field of view. To obtain accurate focal length data, the checkerboard can be placed at different positions and angles, such as directly in front of the camera, diagonally above, or diagonally below it. The checkerboard can also be rotated horizontally at certain angles (e.g., 0 degrees, 30 degrees, 60 degrees), and vertically at certain angles (e.g., 0 degrees, 15 degrees, 30 degrees). A set of images is captured at each position and angle. Images are then analyzed using a specialized calibration algorithm. The algorithm first identifies the checkerboard's corners. Corners are the points where the black and white squares of the checkerboard pattern intersect. These points have distinct characteristics in the image, such as sudden changes in grayscale values. The calibration algorithm locates corners by detecting grayscale changes in the image. Image processing techniques, such as edge detection and threshold segmentation, are typically used to assist in corner identification. After identifying the corner points of the chessboard, the calibration algorithm will establish a correspondence between the coordinates of the corner points in the image plane and the real world. Since the grid size of the chessboard is known, for example, the side length of each grid is 20 mm, the coordinates of each corner point on the chessboard in the real world can be calculated based on the layout of the chessboard. For example, the coordinates of the corner point in the upper left corner of the chessboard can be set to (0, 0), and then the coordinates of other corner points are determined in sequence according to the grid size. At the same time, on the image plane, each corner point also has corresponding pixel coordinates, such as (x, y), where x represents the column position of the corner point in the image and y represents the row position. Through the established corner point correspondence, the calibration algorithm can calculate the camera's intrinsic parameter matrix according to the mathematical model. The camera's intrinsic parameter matrix contains multiple parameters, including the camera's focal length in the horizontal and vertical directions. The calculation of the intrinsic parameter matrix is based on the camera imaging model. Through a series of mathematical derivations and optimization algorithms, the various parameters of the intrinsic parameter matrix are solved from the correspondence between the image coordinates and the real-world coordinates of the corner points. The horizontal direction f is extracted from the calculated intrinsic parameter matrix. u and vertical direction f v focal length data.

[0036] In addition, camera lenses inevitably experience distortion during the manufacturing process, primarily radial and tangential distortion. In actual lenses, radial and tangential distortion often coexist, so a comprehensive distortion model is needed to accurately describe the lens's distortion characteristics. The comprehensive distortion model combines the radial and tangential distortion models to comprehensively consider the various distortion factors of the lens. It is based on a comprehensive analysis of multiple sources of error during the actual imaging process, recognizing that the effects of radial and tangential distortion on the image are superimposed on each other. By adding the two distortion models together, the actual distortion of the lens during imaging can be more accurately described. Therefore, during calibration, the ideal and actual coordinates of the checkerboard corners are substituted into this comprehensive model, and the optimization algorithm is used to solve the various parameters to obtain accurate lens distortion parameters.

[0037] The principal coordinate point is the coordinate of the center of the camera imaging plane in the pixel coordinate system. It is generally regarded as the origin or reference point of the image coordinate system. It is the coordinate position of the intersection of the camera optical axis and the imaging plane on the image plane.

[0038] To obtain image data covering different perspectives of the target image, the visual reconstruction system requires multiple cameras deployed to ensure they can capture the target object from different directions. These cameras require precise spatial positioning and angle adjustment to ensure sufficient overlap between perspectives for subsequent feature matching and 3D reconstruction. The pixels at the target location in the image data captured by the first camera are set as the current frame pixels, while the pixels corresponding to the target location in the image data captured by the second camera are set as the reference frame pixels. The second camera and the first camera capture the target object from different directions, respectively.

[0039] In some embodiments, prior to obtaining multiple camera parameters and capturing image data from different perspectives covering the target image, multiple historical image data sets with camera pose annotations may be collected. This historical image data covers various camera poses in different scenes and under different shooting conditions, providing a rich and diverse sample for model learning. For example, images captured by a camera from different angles and positions in various indoor and outdoor environments are accurately annotated with the camera's position (e.g., 3D spatial coordinates) and pose (e.g., rotation angle, pitch angle, etc.) at the time of capture. A deep learning algorithm is then used to train this historical image data to construct a pose mapping model. Deep learning algorithms (such as convolutional neural networks) can automatically learn the complex relationship between features in image data and camera pose. During training, the network input is historical image data, and the output is the corresponding camera pose annotations. By continuously adjusting the network parameters, the network can accurately predict the camera pose based on the input image features. When new image data needs to be processed, it is input into the trained pose mapping model. The model outputs the camera pose data corresponding to the image data based on the learned mapping relationship. This process enables fast and automatic acquisition of camera pose information without manual measurement or complex calculations.

[0040] S2. Obtaining coordinates of pixel points in a current frame of a target object photographed by a first camera, and coordinates of pixel points in a reference frame corresponding to the pixel points in the current frame of a target object photographed by a second camera from a different direction;

[0041] The first camera captures the current frame of the target object. Its internal imaging sensor converts the optical signal into an electrical signal, which is then converted to digital image data through analog-to-digital conversion. This data is then stored in the system in a specific image format. This process involves the coordinated work of the camera hardware and the system's storage mechanism. The system parses the stored image format and extracts key parameters, such as image width, height, and pixel depth. These parameters are the basis for calculating pixel coordinates. A pixel coordinate system is constructed for the current frame, with the upper-left corner of the image as the origin. Since image data is stored by row, when calculating pixel coordinates, for any pixel in the image, its x-axis coordinate u equals its column position in the image data (counting from 0), and its y-axis coordinate v equals its row position. For example, if the image width is 800 pixels and a pixel is located at column 300 and row 400, its coordinates are (300, 400). This method is used to obtain the coordinates of all pixels in the current frame.

[0042] The second camera captures the target object from a different direction to obtain a reference frame image. This image is also imaged and stored after analog-to-digital conversion. The system then parses the image format to obtain relevant parameters. The pixel coordinates of the reference frame image are obtained in the same way as the current frame: a pixel coordinate system with the same rules is first established. Then, based on the image storage format and the parsed parameters, the coordinates of each pixel in this coordinate system are calculated. For example, for a pixel in the reference frame image, if its column position in the image data is 500 and its row position is 350, and the image width is 1000 pixels, then the pixel coordinates are (500, 350), thus obtaining the coordinates of all pixels in the reference frame image.

[0043] S3. Build an end-to-end training model, where the end-to-end training model is a model that iteratively trains pose data of multiple cameras, correspondences between pixels on image data, and initial pixel depth data by calculating a reprojection error between pixels of the current frame and pixels of the reference frame;

[0044] After obtaining the relevant camera parameters, image data, and pixel coordinates, the visual reconstruction system begins to build an end-to-end training model. The core of this model is to iteratively train the pose data of multiple cameras, the correspondence between pixels on the image data, and the initial pixel depth data by calculating the reprojection error between the pixels of the current frame and the pixels of the reference frame, thereby achieving accurate reconstruction of the three-dimensional scene.

[0045] The specific training steps are as follows:

[0046] S31. Calculate the inverse projection coordinates (x, y, z) of the pixel point on the current frame in the three-dimensional coordinates using the pixel point coordinates (u, v) of the current frame. The expression is as follows:

[0047]

[0048] where f u and f v are the focal lengths of the first camera in the u and v directions respectively, and the coordinates of the first principal point of the first camera are (c u ,c v ), m x and m y Respectively represent the normalized offset of the pixel coordinates of the current frame relative to the coordinates of the first principal point in the u and v directions, α and β are the camera lens distortion parameters, m z It is an intermediate variable considering lens distortion, m d It is a combination of m x 、m y and m z variables;

[0049] S32. Calculate the projection coordinates (U, V) of the inverse projection coordinates on the reference frame. The expression is as follows:

[0050]

[0051] where f U and f V is the focal length data of the second camera in the U and V directions, and the coordinates of the second principal point of the second camera are (c U , c V ), d is an intermediate variable of the inverse projection coordinate under the condition of lens distortion;

[0052] S33. Calculate the reprojection error E:

[0053]

[0054] Among them, P j is the pixel point P of the current frame i The coordinates of the corresponding points on the reference frame, is the pixel point P of the current frame i The second projection coordinate on the reference frame, O represents the set of valid pixel correspondences, w ij Indicates the reliability of the correspondence (i, j); S34, using the Gauss-Newton iteration method to solve and obtain the parameter update amount; according to the parameter update amount, optimizing the pose data, initial pixel correspondence and multiple initial pixel depth data of multiple cameras;

[0055] The Gauss-Newton iteration method is a nonlinear optimization method based on Taylor series expansion, which is used to solve the least squares problem and obtain the parameter update amount by iteratively approximating the optimal solution.

[0056] The camera pose describes its position and orientation in space. When parameter updates are generated, a portion of them is related to the camera pose. The camera position can be imagined as a point in three-dimensional space. The parameter update indicates how much this point needs to move forward, backward, left, right, and up and down. For example, if the update indicates that the camera needs to move forward a certain distance, the camera's position in space is adjusted accordingly, moving it forward. This is done to ensure that the camera's position more accurately reflects its expected position in the scene, thereby ensuring more accurate subsequent calculations and reconstructions based on the captured images. The camera's orientation determines the direction it captures, and the parameter update also involves rotations around different axes (imagine three perpendicular lines passing through the center of the camera). For example, the camera might need to be rotated a certain angle clockwise around the vertical axis or tilted a certain angle upward around the horizontal axis. Based on these updates, the camera's rotation parameters are adjusted to change its orientation. This adjustment allows the camera's orientation to better capture the target object, providing the basis for accurate visual reconstruction.

[0057] Initial pixel correspondences refer to which pixels in images captured by different cameras represent points on the same object in space. When initially determining pixel correspondences, there may be some inaccuracies, and these correspondences can be fine-tuned based on the parameter update amount. For example, a pixel in one image is initially thought to correspond to a pixel in another image, but the calculated update amount indicates that it should more accurately correspond to another adjacent pixel. In this case, the correspondence between these two pixels is adjusted to more accurately match them. The correspondence between some pixels may have significant deviations in the initial judgment, and the parameter update amount can be used to detect and correct these erroneous correspondences. By continuously adjusting the pixel correspondences based on the update amount, the matching of pixels between different images becomes more accurate.

[0058] The initial pixel depth data indicates how far the object surface corresponding to each pixel is from the camera. The parameter update specifies the amount by which each pixel's depth value should be increased or decreased. For example, the initial depth estimate for a pixel may be inaccurate. If the update indicates that the object surface represented by that pixel is actually closer to the camera, the pixel's depth value will be reduced accordingly. By making this adjustment to each pixel's depth value, it is brought closer to the true object depth. The depth data for all pixels is adjusted based on the parameter update, making the depth information of the entire image more consistent with the actual scene. This optimized depth data more accurately reflects the positional relationships of objects in 3D space, providing reliable data support for the ultimate reconstruction of an accurate 3D scene. By performing this optimization on the pose data of multiple cameras, the initial pixel correspondences, and the multiple initial pixel depth data, the accuracy and reliability of the visual reconstruction system can be continuously improved, gradually achieving 3D reconstruction results that are more consistent with the actual scene.

[0059] In some embodiments, in step S3, the reprojection error is a complex function of multiple variables, including the pose data and initial pixel depth data. Using the chain rule, the derivative of the reprojection error with respect to these variables can be decomposed into the product of multiple intermediate derivatives, facilitating computation. This method gradually calculates the gradient of the reprojection error with respect to each variable (i.e., the pose data and initial pixel depth data). The purpose of calculating this gradient information is to understand the rate of change of the reprojection error with respect to the pose data and initial pixel depth data. The direction of the gradient indicates the direction in which the error increases most rapidly, while the opposite direction indicates the direction in which the error decreases most rapidly. The gradient descent algorithm is an iterative optimization algorithm whose basic principle is to gradually update parameters in the opposite direction of the gradient to minimize the objective function (here, the reprojection error). In each iteration, the gradient of the objective function is calculated based on the current parameter values, and then the parameters are updated in the opposite direction of the gradient at a certain learning rate (step size). By continuously iteratively updating the pose data and initial pixel depth data, the reprojection error gradually decreases. During this process, the camera pose is gradually adjusted to a more accurate position and posture, thereby achieving a more accurate projection relationship between images from different perspectives. The initial pixel depth data is also continuously optimized to make it closer to the actual object depth value. In this way, after multiple iterations, the entire visual reconstruction model can more accurately restore the 3D scene, improving the accuracy and quality of the reconstruction.

[0060] In some embodiments, after obtaining multiple camera parameters and capturing image data from different perspectives covering the target image, the image data can be processed using feature extraction algorithms to extract visual feature information. These algorithms can identify key elements in the image, such as edges, textures, and shapes. A high-dimensional feature matrix corresponding to the image data is then determined based on the extracted visual feature information. This matrix comprehensively and comprehensively describes the image's features. It represents the feature information of each pixel or pixel region in vector form and arranges it into a matrix according to certain rules. The multiple image data are then input into a correspondence determination model, which is pre-trained using machine learning using a set of images annotated with pixel correspondences. Finally, the initial pixel correspondence obtained from the correspondence determination model represents the positional correspondence between pixels of the same target part in the current frame and the reference frame. In other words, it determines how the positions of pixels of the same target part in images from different perspectives correspond to each other. This correspondence is crucial for subsequent operations such as 3D reconstruction, helping to determine pixel correspondence between images from different perspectives, thereby better recreating the 3D scene.

[0061] In some embodiments, after obtaining the initial pixel correspondences and the high-dimensional feature matrix, they are combined to construct a relationship matrix. Specifically, the initial pixel correspondences define the positional correspondences of pixels within the same target region in different images (e.g., the current frame and the reference frame), while the high-dimensional feature matrix details the visual feature information of each pixel. The relationship matrix is formed by combining the feature information of these corresponding pixels in the feature matrix according to certain rules. The relationship matrix integrates the positional correspondences and feature information of the pixels, and can more comprehensively reflect the inherent connections between the pixels. Compared to pixel correspondences or high-dimensional feature matrices alone, the relationship matrix provides a richer information framework, facilitating the subsequent exploration of deeper relationships between pixels. The pixel association network is a neural network structure specifically designed to process pixel relationship data. Based on the principles of deep learning, it automatically discovers hidden associations between pixels by learning pixel relationship patterns from large amounts of image data. This network comprises multiple neural layers and is trained using forward and backpropagation algorithms. During training, it learns how to extract useful information from the input relationship matrix to predict pixel depth data. The constructed relationship matrix is then input into the pixel association network for computation. The network processes the information in the relationship matrix based on its learned patterns and algorithms, outputting initial pixel depth data. During this process, the network leverages the pixel correspondences and feature information in the relationship matrix, using complex calculations and reasoning to estimate the depth of each pixel from the object surface to the camera. This provides crucial depth information for 3D reconstruction, making the reconstructed 3D scene more accurate and realistic in depth.

[0062] In some embodiments, before inputting multiple image data into the correspondence determination model and obtaining the initial pixel correspondence, the image data may be preprocessed using Gaussian filtering. An optical flow algorithm is then used to calculate pixel motion information between adjacent frames in the image data. The optical flow algorithm is based on the fundamental assumption that the brightness of pixels in an image remains constant (or changes slowly) between adjacent frames. Pixel motion information is calculated by analyzing changes in pixel brightness between adjacent frames. This algorithm typically formulates an optimization problem with the goal of minimizing the difference in pixel brightness between adjacent frames, while also considering pixel motion constraints (such as smoothness of motion). The pixel motion information calculated by the optical flow algorithm can be used to determine the motion trend of dynamic objects in the image. For example, if multiple adjacent pixels exhibit a displacement trend toward the upper left in consecutive frames at similar speeds, it can be inferred that the corresponding object is moving toward the upper left. This motion trend information is crucial for understanding the behavior of objects in a scene. For example, in video surveillance, it can be used to track the target object's trajectory and predict its future location. Furthermore, pixel motion information can be used to preliminarily delineate dynamic and static regions within an image. Regions consisting of pixels whose movement speed exceeds a certain threshold are classified as dynamic regions. These regions typically correspond to moving objects in the scene. Regions consisting of pixels whose movement speed is below the threshold or with little or no movement are classified as static regions, such as the background. This classification helps to adopt different strategies for different regions in subsequent processing. For example, in visual reconstruction, dynamic regions may require more frequent updates of their pixel correspondences and depth data, while static regions can be updated relatively less frequently, thereby improving processing efficiency and accuracy.

[0063] S4, repeat step S3 until the reprojection error converges, and obtain the final pixel correspondence dataset, the final camera pose dataset, and the final pixel depth dataset;

[0064] After each iteration (repeating step S3), the reprojection error will change. When the value of the reprojection error no longer decreases significantly after multiple iterations, or the decrease is less than a very small pre-set threshold (for example, 0.001), the reprojection error is considered to be converging. This means that the model has fully adjusted parameters such as camera pose, pixel correspondence, and pixel depth, so that the projection result is close enough to the actual situation, and further iterations will not significantly improve the result. By continuously repeating step S3 until the reprojection error tends to converge, an accurate final pixel correspondence dataset, final camera pose dataset, and final pixel depth dataset can be obtained. These datasets are the key data basis for the subsequent reconstruction of three-dimensional coordinates and three-dimensional scenes.

[0065] S5. Reconstructing three-dimensional coordinates by combining the final pixel point correspondence relationship dataset and the final pixel depth dataset;

[0066] Reconstructing 3D coordinates using the final pixel correspondence dataset and the final pixel depth dataset is achieved through the following key steps: The final pixel correspondence dataset provides information about pixels representing the same spatial location in different images. Based on the final pixel depth dataset, the depth value of the object surface from the camera corresponding to each pixel can be obtained. For each camera, a camera model must be established, which describes how points in 3D space are projected onto the 2D image plane. Commonly used camera models include intrinsic parameters (such as focal length and principal point coordinates) and extrinsic parameters (which describe the camera's position and posture in the world coordinate system). The camera model provides a projection equation. With the pixel correspondences, pixel depth values, and the camera model's projection equation, the coordinates of 3D points can be solved using simultaneous equations. Using this method, for all corresponding pixels, using their corresponding depth values and the camera model, the 3D coordinates of each point in space can be gradually reconstructed, thereby constructing the coordinate framework of the entire 3D scene.

[0067] S6. Convert and stitch the three-dimensional coordinates in combination with the final camera pose dataset to reconstruct the three-dimensional scene.

[0068] The final camera pose dataset records the position and angle of each camera in space when taking a picture. The position is the camera's specific location in space, while the angle determines the camera's orientation. First, a large spatial reference must be established, much like a map requires a common orientation and origin. This large spatial reference can be used as the viewpoint of one camera, or a new, independent space independent of all camera viewpoints can be established. The 3D coordinates obtained by different cameras are initially based on their own viewpoints. Now, based on the position and angle of each camera, these coordinates must be uniformly converted to the large reference space just determined. For example, if a camera captures a picture from a certain position and angle and obtains a number of 3D coordinate points, these points can be adjusted to conform to the large reference space based on the camera's position and angle. This way, regardless of the camera's position, the 3D coordinates obtained will have a consistent position within the large reference space. After converting the 3D coordinates from different camera viewpoints to a unified space, some overlap may occur due to overlap between the images captured by different cameras. At this time, some methods are needed to remove these duplicate contents and ensure that the parts can be accurately spliced together, just like piecing together Figure 1In this way, duplicate puzzle pieces are removed and the remaining pieces are accurately assembled. After removing duplicates and aligning them, all the 3D coordinate points are integrated. Together, these points form a set of points representing the 3D scene, just like depicting the entire scene with many small dots. In practice, these points may need further processing, such as converting them into a surface model. This allows for a clearer view of the 3D scene, facilitating observation and subsequent analysis. For example, through some methods, these points can be transformed into a solid model with a shape and surface, ultimately successfully reconstructing the complete 3D scene.

[0069] In some embodiments, each pixel captured by each camera has a corresponding three-dimensional position in its respective camera coordinate system. These three-dimensional positions are transformed using a preset coordinate transformation matrix. The specific calculation process involves mathematical operations such as matrix multiplication. By multiplying the camera coordinate system coordinates of the pixel by the transformation matrix, the coordinates in the common coordinate system are obtained. In this way, the common coordinate data in the common coordinate system can be determined for all pixels captured by multiple cameras. After obtaining the common coordinate data for all pixels, the three-dimensional scene is stitched together. Adjacent pixels in three-dimensional space are combined based on the common coordinate data. Adjacent pixels can be defined based on certain spatial distances or geometric relationships. For example, pixels within a certain threshold are considered adjacent. By connecting these adjacent pixels, the structure of the three-dimensional scene is gradually constructed. For example, for pixels representing the surface of an object, the combination of adjacent pixels can form the object's outline and surface shape. As more and more adjacent pixels are combined, the three-dimensional structure of the entire scene gradually becomes clearer. Ultimately, this method stitches together a complete three-dimensional reconstruction result that accurately reflects the three-dimensional form and spatial layout of the target scene.

[0070] In the embodiment of the present application, due to the technical means of building an end-to-end training model and using reprojection error to iteratively train and optimize camera pose, pixel correspondence and pixel depth data, the problem of low efficiency of three-dimensional reconstruction in the existing technology that relies on manual labeling is effectively solved, thereby achieving end-to-end self-supervision and accurate restoration of the real structure and morphology of the target scene.

[0071] After combining the above content, the following is a more specific scenario description of the method provided by this implementation. Figure 2 , which is a scene diagram of the self-supervised end-to-end visual reconstruction method in an embodiment of the present application.

[0072] exist Figure 2In the scene shown, pixels at specific locations in the current frame have corresponding pixels in the reference frame, meaning there's a correspondence between the pixels at the same locations in the reference frame and the current frame. First, the pixels in the current frame are reverse-projected from the two-dimensional image plane to a three-dimensional coordinate space using a specific algorithm. This yields a series of three-dimensional spatial points, which together form a three-dimensional point cloud. This three-dimensional point cloud is then projected onto the reference frame according to established rules, generating corresponding projection points. The deviation between these projection points and the corresponding pixels in the reference frame is then determined. This deviation is reflected in their positional differences in the reference frame image. Finally, adjustments are made to the relevant parameters or models based on this deviation. These adjustments cover aspects such as camera pose and object position estimation, aiming to ensure that the projection points coincide as closely as possible with the corresponding pixels in the reference frame, thereby improving the accuracy of scene understanding and reconstruction.

[0073] In the embodiment of the present application, due to the use of Figure 2 As shown in the complete technical process from determining pixel correspondence, inverse projection to constructing 3D point cloud, projection calculation deviation to adjusting camera pose and object position estimation based on deviation, it effectively solves the problems of inaccurate pixel correspondence, large projection deviation and difficulty in estimating camera pose and object position in the existing technology, thereby achieving the technical effects of improving scene understanding accuracy, improving 3D reconstruction accuracy, optimizing camera pose and enhancing the authenticity and reliability of reconstructed scene, as well as realizing end-to-end self-supervised efficient visual reconstruction and accurately restoring the target scene structure and morphology.

[0074] The following describes the visual reconstruction system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , is a schematic diagram of a physical device structure of a visual reconstruction system in an embodiment of the present application.

[0075] It should be noted that Figure 3 The structure of the visual reconstruction system shown is only an example and should not limit the functions and scope of use of the embodiments of the present invention.

[0076] like Figure 3As shown, the visual reconstruction system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes based on programs stored in a read-only memory (ROM) 302 or programs loaded from a storage unit 308 into a random access memory (RAM) 303, such as executing the methods described in the above embodiments. RAM 303 also stores various programs and data required for system operation. CPU 301, ROM 302, and RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to bus 304.

[0077] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a push button switch, and the like; an output section 307 including a liquid crystal display (LCD), an audio output device, an indicator light, and the like; a storage section 308 including a hard disk and the like; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. Removable media 311, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 310 as needed, so that computer programs read therefrom can be installed into the storage section 308 as needed.

[0078] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product that includes a computer program carried on a computer-readable medium, the computer program including a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 309 and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the various functions defined in the present invention are performed.

[0079] It should be noted that specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0080] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings.

[0081] Specifically, the visual reconstruction system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the self-supervised end-to-end visual reconstruction method provided by the above embodiment is implemented.

[0082] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the visual reconstruction system described in the above embodiments, or may exist independently and not be incorporated into the visual reconstruction system. The storage medium carries one or more computer programs, which, when executed by a processor of the visual reconstruction system, enable the visual reconstruction system to implement the self-supervised end-to-end visual reconstruction method provided in the above embodiments.

[0083] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0084] As used in the above embodiments, the term “when…” may be interpreted to mean “if…” or “after…” or “in response to determining…” or “in response to detecting…”, depending on the context. Similarly, the phrases “upon determining…” or “if (stated condition or event) is detected” may be interpreted to mean “if determining…” or “in response to determining…” or “upon detecting (stated condition or event)” or “in response to detecting (stated condition or event)”, depending on the context.

[0085] Those skilled in the art will appreciate that all or part of the process steps in the above-described method embodiments can be implemented by a computer program instructing the relevant hardware. The program can be stored in a computer-readable storage medium, and when executed, the program can include the process steps in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as ROM or random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A self-supervised end-to-end visual reconstruction method, applied to a visual reconstruction system, characterized in that: The method comprises: S1. Acquire multiple camera parameters and capture image data at different viewing angles covering a target image, wherein the camera parameters include focal length data, lens distortion parameters, and principal coordinate point data, and the image data includes at least current frame pixels captured by a first camera and reference frame pixels captured by a second camera; S2. Obtaining coordinates of pixel points in a current frame of a target object photographed by a first camera, and coordinates of pixel points in a reference frame corresponding to the pixel points in the current frame of a target object photographed by a second camera from different directions; S3. Construct an end-to-end training model, wherein the end-to-end training model is a model that iteratively trains the pose data of multiple cameras, the correspondence between pixels on the image data, and the initial pixel depth data by calculating the reprojection error between the pixels of the current frame and the pixels of the reference frame; S4, repeating step S3 until the reprojection error converges, obtaining a final pixel point correspondence dataset, a final camera pose dataset, and a final pixel depth dataset; S5. Reconstructing three-dimensional coordinates by combining the final pixel point correspondence relationship dataset and the final pixel depth dataset; S6. Converting and splicing the three-dimensional coordinates in combination with the final camera pose dataset to reconstruct a three-dimensional scene; In constructing an end-to-end training model, the end-to-end training model is a step of iteratively training the pose data of multiple cameras, the correspondence between pixels on the image data, and the initial pixel depth data by calculating the reprojection error between the pixels of the current frame and the pixels of the reference frame, specifically including: S31, calculating the inverse projection coordinates (x, y, z) of the pixel on the current frame in the three-dimensional coordinates using the coordinates of the pixel of the current frame, and the expression is as follows: where f u and f v are the focal lengths of the first camera in the u and v directions respectively, and the coordinates of the first principal point of the first camera are (c u ,c v ), m x and m y Respectively represent the normalized offset of the current frame pixel coordinates relative to the first principal point coordinates in the u and v directions, α and β are the camera lens distortion parameters, m z It is an intermediate variable considering lens distortion, m d It is a combination of m x 、m y and m z variables; S32. Calculate the projection coordinates (U, V) of the inverse projection coordinates on the reference frame. The expression is as follows: where f U and f V is the focal length data of the second camera in the U and V directions, and the coordinates of the second principal point of the second camera are (c U , c V ), d is an intermediate variable of the back-projected coordinates under lens distortion; S33. Calculate the reprojection error E: Among them, P j is the current frame pixel P i The coordinates of the corresponding point on the reference frame, p′ ij is the current frame pixel P i The second projection coordinate on the reference frame, O represents the set of valid pixel correspondences, w ij Indicates the reliability of the corresponding relationship (i, j); S34, using the Gauss-Newton iteration method to solve and obtain the parameter update amount; The pose data, initial pixel correspondences, and multiple initial pixel depth data of multiple cameras are optimized according to the parameter update amount.

2. The method according to claim 1, characterized in that Before the steps of acquiring multiple camera parameters and capturing image data at different viewing angles covering the target image, the following steps are also included: Build a pose mapping model through a deep learning algorithm using multiple historical image data with camera pose annotations; The image data is input into the pose mapping model to obtain camera pose data corresponding to the image data.

3. The method according to claim 1, characterized in that After the steps of acquiring multiple camera parameters and capturing image data at different viewing angles covering the target image, the method further includes: After extracting visual feature information from the image data using a feature extraction algorithm, determining a high-dimensional feature matrix corresponding to the image data based on the visual feature information; Multiple image data are input into a correspondence determination model to obtain an initial pixel correspondence. The correspondence determination model is obtained by pre-training a machine learning method using an image set labeled with correspondences between pixel points. The initial pixel correspondence is the position correspondence between the pixel points of the same target part in the current frame and the reference frame.

4. The method according to claim 3, characterized in that After extracting visual feature information from the image data using a feature extraction algorithm, and determining a high-dimensional feature matrix corresponding to the image data based on the visual feature information, the method further includes: Combining the initial pixel correspondence relationship and the high-dimensional feature matrix to obtain a relationship matrix; Initial pixel depth data is obtained by operating the relationship matrix through a pixel association network.

5. The method according to claim 3, characterized in that Before the step of inputting the plurality of image data into the correspondence determination model to obtain the initial pixel correspondence, the method further includes: Preprocess the image data by Gaussian filtering; Use the optical flow algorithm to calculate the motion information of pixels between adjacent frames in the image data; The motion trend information of the dynamic objects in the image and the preliminary division of the dynamic area and the static area are obtained through the motion information.

6. The method according to claim 1, characterized in that The step of optimizing the pose data, initial pixel correspondences, and multiple initial pixel depth data of multiple cameras according to the parameter update amount specifically includes: Applying the chain rule to calculate the gradient information of the reprojection error with respect to the pose data and the initial pixel depth data respectively; The pose data and the initial pixel depth data are updated using a gradient descent algorithm in combination with the gradient information.

7. A visual reconstruction system, characterized in that: The visual reconstruction system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the visual reconstruction system to execute the method described in any one of claims 1-6.

8. A computer-readable storage medium comprising instructions, characterized in that: When the instructions are executed on a visual reconstruction system, the visual reconstruction system is caused to perform the method according to any one of claims 1 to 6.

9. A computer program product, characterized in that When the computer program product is run on a visual reconstruction system, the visual reconstruction system is caused to perform the method according to any one of claims 1 to 6.

Images