Binocular depth estimation training data generation method

Abstract

The invention provides a binocular depth estimation training data generation method, and relates to the technical field of image processing, and the method comprises the steps: obtaining a plurality of two-dimensional images at a plurality of different visual angles collected by a monocular camera for a target region and camera poses corresponding to the plurality of two-dimensional images, carrying out the neural radiation field training of the target region, and obtaining a neural radiation field training result; obtaining a neural radiation field model of the target area; and according to preset virtual binocular camera parameters, carrying out binocular image rendering by adopting a neural radiation field model to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image and a parallax true value map corresponding to the virtual binocular image, and generating binocular depth estimation training data for carrying out model training to obtain a binocular depth estimation model. According to the invention, the binocular image acquisition cost is reduced, and the training depth truth value range of the binocular depth estimation network is expanded.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a method for generating binocular depth estimation training data. Background Art

[0002] With the rapid development of technologies such as virtual reality, augmented reality, and 3D reconstruction, binocular image depth maps are increasingly used in many fields. Binocular image depth maps simulate the principle of human vision and use the parallax information between the left and right views captured by a binocular camera to calculate the depth information of each point in the image. This technology provides important 3D perception capabilities for machine vision, autonomous driving, robot navigation, and other fields.

[0003] At present, it is necessary to collect a large amount of binocular image data in different environments and conditions. These images are usually taken by two cameras at the same time to form left and right views. In order to ensure the diversity of data, the collection process needs to cover various scenes. In addition to obtaining image data, the corresponding depth information needs to be obtained through depth sensors.

[0004] However, high-precision binocular cameras and depth sensor equipment are expensive and require professional installation and calibration, which increases the cost of acquiring binocular images. In addition, the effective ranging range of the depth sensor limits the acquisition of the true depth value, and its applicability to autonomous driving scenarios is very limited. Summary of the invention

[0005] The purpose of this application is to provide a method for generating binocular depth estimation training data in response to the above-mentioned deficiencies in the prior art, which can generate binocular images without relying on binocular cameras and depth sensors, thereby reducing the cost of acquiring binocular images, expanding the training depth true value range of the binocular depth estimation network, and improving the adaptability of autonomous driving scenarios.

[0006] To achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

[0007] In a first aspect, an embodiment of the present application provides a method for generating binocular depth estimation training data, the method comprising:

[0008] Acquire multiple two-dimensional images of the target area captured by a monocular camera at multiple different viewing angles and the camera poses corresponding to the multiple two-dimensional images;

[0009] Performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain a neural radiation field model of the target area;

[0010] According to preset virtual binocular camera parameters, the neural radiation field model is used to perform binocular image rendering to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image;

[0011] According to the virtual binocular image, the depth map corresponding to the virtual binocular image and the disparity true value map corresponding to the virtual binocular image, binocular depth estimation training data is generated for model training to obtain a binocular depth estimation model.

[0012] Optionally, before performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain the neural radiation field model of the target area, the method further includes:

[0013] Determining positions of two-dimensional key points of feature matching from the plurality of two-dimensional images;

[0014] According to the position of the two-dimensional key point and the camera pose corresponding to the matching two-dimensional image where the two-dimensional key point is located, a preset motion recovery structure algorithm is used to perform three-dimensional reconstruction to map the two-dimensional key point to a preset three-dimensional space, and the initial three-dimensional point corresponding to the two-dimensional key point in the preset three-dimensional space is obtained as the initial three-dimensional point corresponding to the two-dimensional key point;

[0015] According to the initial three-dimensional point and the matching two-dimensional image, the camera posture corresponding to the matching two-dimensional image is corrected.

[0016] Optionally, determining positions of two-dimensional key points of feature matching from the multiple two-dimensional images includes:

[0017] Extracting local feature points from each two-dimensional image to obtain candidate local feature points in each image;

[0018] Feature point matching is performed on the candidate local feature points in different two-dimensional images, and positions of the local feature points with feature matching in the different two-dimensional images are obtained as positions of the two-dimensional key points.

[0019] Optionally, before performing three-dimensional reconstruction using a preset structure-from-motion algorithm according to the positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images, the method further includes:

[0020] According to the position of the two-dimensional key point, feature extraction is performed on different two-dimensional images corresponding to the two-dimensional key point to obtain convolutional neural network features of the two-dimensional key point in the different two-dimensional images;

[0021] Adjusting the position of the two-dimensional key point according to the convolutional neural network features of the two-dimensional key point in the different two-dimensional images;

[0022] The method of performing three-dimensional reconstruction using a preset structure-from-motion algorithm according to the positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images, so as to map the two-dimensional key points to a preset three-dimensional space, includes:

[0023] The algorithm is used to perform three-dimensional reconstruction according to the adjusted positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images.

[0024] Optionally, the correcting the camera pose corresponding to the matching two-dimensional image according to the initial three-dimensional point and the multiple two-dimensional images includes:

[0025] According to the initial three-dimensional point, an initial mapping convolutional neural network feature of an initial mapping two-dimensional point corresponding to the initial three-dimensional point in the matching two-dimensional image is obtained;

[0026] Adjusting a camera pose corresponding to the matching two-dimensional image according to an initial convolutional neural network feature of the two-dimensional key point in the matching two-dimensional image and the initial mapping convolutional neural network feature;

[0027] According to the position of the two-dimensional key point and the adjusted camera pose corresponding to the matched two-dimensional image, the algorithm is used to re-perform three-dimensional reconstruction to obtain a new three-dimensional point corresponding to the two-dimensional key point;

[0028] Re-acquire a new mapping convolutional neural network feature of a new mapping two-dimensional point corresponding to the new three-dimensional point in the matching two-dimensional image;

[0029] According to the initial convolutional neural network features and the new mapping convolutional neural network features, the camera pose corresponding to the matched two-dimensional image is readjusted until the mapping convolutional neural network features of the mapped two-dimensional points corresponding to the two-dimensional key points obtained based on the adjusted camera pose and the initial convolutional neural network features meet the preset convolutional neural network feature consistency conditions;

[0030] Determine that the adjusted camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met as the target camera pose corresponding to the matching two-dimensional image.

[0031] Optionally, obtaining, according to the initial three-dimensional point, an initial mapping convolutional neural network feature of an initial mapping two-dimensional point corresponding to the initial three-dimensional point in the matching two-dimensional image includes:

[0032] Determining the position of the initial mapped two-dimensional point in the matching two-dimensional image according to the camera pose corresponding to the matching two-dimensional image;

[0033] According to the position of the initial mapping two-dimensional point, convolutional neural network feature extraction is performed on the matching two-dimensional image to obtain the initial mapping convolutional neural network feature.

[0034] Optionally, adjusting the camera pose corresponding to the matching two-dimensional image according to the initial convolutional neural network features of the two-dimensional key point in the matching two-dimensional image and the initial mapping convolutional neural network features includes:

[0035] Determining a convolutional neural network feature error value according to the initial convolutional neural network feature and the initial mapping convolutional neural network feature;

[0036] According to the convolutional neural network feature error value, the camera posture corresponding to the matching two-dimensional image is adjusted.

[0037] Optionally, performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain a neural radiation field model of the target area includes:

[0038] Performing spatial division on the target area to obtain a plurality of spatial units;

[0039] Determine pixel data sets corresponding to the plurality of spatial units respectively from the plurality of two-dimensional images;

[0040] According to the pixel data sets corresponding to the multiple spatial units, respectively perform neural radiation field training on the multiple spatial units to obtain neural radiation submodules corresponding to the multiple spatial units;

[0041] A neural radiation field model of the target area is obtained according to the neural radiation submodules corresponding to the multiple spatial units.

[0042] Optionally, the determining respectively from the plurality of two-dimensional images the pixel data sets corresponding to the plurality of spatial units comprises:

[0043] Determine, according to the camera pose corresponding to each two-dimensional image, a ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image;

[0044] The rays from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image are intersected with each spatial unit to obtain a pixel data set corresponding to each spatial unit.

[0045] Optionally, the performing binocular image rendering using the neural radiation field model according to preset virtual binocular camera parameters to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image includes:

[0046] According to the preset posture parameters of the virtual binocular camera, the neural radiation field model is used to perform binocular image rendering to obtain the virtual binocular image and the depth map;

[0047] The disparity true value map is generated according to the pose parameters of the preset virtual binocular camera and the depth map.

[0048] In a second aspect, another embodiment of the present application provides a device for generating binocular depth estimation training data, the device comprising:

[0049] An acquisition module is used to acquire multiple two-dimensional images of a target area captured by a monocular camera at multiple different viewing angles and camera poses corresponding to the multiple two-dimensional images;

[0050] A training module, used for performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images, so as to obtain a neural radiation field model of the target area;

[0051] A rendering module, used to perform binocular image rendering using the neural radiation field model according to preset virtual binocular camera parameters, to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image;

[0052] A generation module is used to generate binocular depth estimation training data according to the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity true value map corresponding to the virtual binocular image, so as to perform model training and obtain a binocular depth estimation model.

[0053] In the third aspect, another embodiment of the present application provides a computer device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the storage medium through the bus, and the processor executes the machine-readable instructions to perform the steps of the method for generating binocular depth residence training data as described in any of the first aspects above.

[0054] In a fourth aspect, another embodiment of the present application provides a storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method for generating in-depth historical residence training data as described in any one of the first aspects above are executed.

[0055] The beneficial effects of this application are:

[0056] The present application provides a method for generating binocular depth estimation training data, obtaining multiple two-dimensional images under multiple different perspectives collected by a monocular camera for a target area and the camera poses corresponding to the multiple two-dimensional images, and performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images, and obtaining a neural radiation field model of the target area. According to the preset virtual binocular camera parameters, the neural radiation field model is used to render the binocular image, obtain the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity truth map corresponding to the virtual binocular image, and generate binocular depth estimation training data according to the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity truth map corresponding to the virtual binocular image, for model training to obtain a binocular depth estimation model. The present application can obtain a neural radiation field model of the target area through the two-dimensional images collected by a monocular camera and the camera poses corresponding to the two-dimensional images, thereby obtaining a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity truth map corresponding to the virtual binocular image, and generate binocular depth estimation training data for model training to obtain a binocular depth estimation model. The virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity truth map corresponding to the virtual binocular image can be obtained without using a binocular camera and a depth sensor, which reduces the cost of obtaining binocular images and simplifies the process of obtaining binocular images. The image review map corresponding to the binocular image can be obtained without relying on a depth sensor, and is not limited by the effective range of the depth sensor. It can generate ultra-large-scale scenes and expand the training depth truth range of the existing binocular depth estimation network. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A flowchart of a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0059] Figure 2 A schematic diagram of a process of posture correction in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0060] Figure 3 A schematic diagram of a process for determining key points in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0061] Figure 4 A schematic diagram of a process of three-dimensional reconstruction in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0062] Figure 5 A schematic diagram of a process of posture correction in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0063] Figure 6 A schematic diagram of a process for determining initial convolutional neural network features in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0064] Figure 7 A flowchart of camera pose in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0065] Figure 8 A schematic diagram of a process for determining a neural radiation field in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0066] Fig. 9 A schematic diagram of a process for determining a pixel data set in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0067] Fig.10 A schematic diagram of a process for determining an image in a method for generating binocular depth estimation training data provided in an embodiment of the present application;

[0068] Fig.11 A schematic diagram of the structure of a device for generating binocular depth estimation training data provided in an embodiment of the present application;

[0069] Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of explanation and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn in real proportion. The flowchart used in this application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can be implemented out of sequence, and the steps without logical context can be reversed in order or implemented simultaneously. In addition, those skilled in the art can add one or more other operations to the flowchart under the guidance of the content of the present application, or remove one or more operations from the flowchart.

[0071] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0072] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0073] The following is a description of the method for generating binocular depth estimation training data provided by the embodiment of the present application in conjunction with the accompanying drawings. The method for generating binocular depth estimation training data in the present application is generally applied to the field of autonomous driving perception, but the existing binocular depth estimation training data collection process is complicated. In the existing method, first, a large amount of binocular image data needs to be collected under different environments and conditions. These images are usually taken by two cameras at the same time to form left and right views. In order to ensure the diversity of data, the collection process needs to cover various scenes, such as urban streets, wild mountains, etc. In addition to image data, it is also necessary to obtain corresponding depth information, which is usually achieved by devices such as laser radar or structured light sensors. Laser radar or structured light sensors can provide high-precision three-dimensional point cloud data as the true depth label of training data. However, high-precision binocular cameras and depth sensor equipment are expensive and require professional installation and calibration, which increases the cost of data collection. In addition, the effective ranging range of the depth sensor limits the acquisition of the true depth value. For example, the depth range of commonly used data sets is between a few meters and tens of meters, and the total scene coverage is reduced.

[0074] However, the present application can obtain a neural radiation field model of the target area through the two-dimensional image collected by the monocular camera and the camera posture training corresponding to the two-dimensional image, thereby obtaining a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image, thereby obtaining binocular depth estimation training data, without installing a binocular camera and a depth sensor device, reducing the cost of obtaining binocular image data, and increasing the scope of obtaining binocular images, which can cover multiple scenes. It is worth noting that before collecting multiple two-dimensional images of the target area at multiple different perspectives, the target area needs to be analyzed in detail to determine the scope and key areas to be covered, including terrain features, building distribution, vegetation conditions, etc. A single flight generally covers several square kilometers of land. It is necessary to use professional path planning tools, such as Pix4D tools, DroneDeploy tools, and other path planning tools to design the flight path of the monocular camera carried by the drone. The path planning tool can generate the optimal path based on information such as the terrain features, building distribution, and vegetation conditions of the target area to ensure that the monocular camera carried by the drone can efficiently cover all angles that need to be shot. Determine the flight altitude and shooting angle of the drone based on the characteristics of the target area and the required image resolution. For example, a lower flight altitude can obtain higher resolution images, but more flight paths are required to cover the entire area. During the flight, the drone can be set to automatic shooting mode to automatically capture images according to preset time intervals or distance intervals. This ensures uniformity and comprehensiveness of image acquisition.

[0075] Figure 1 A flowchart of a method for generating binocular depth estimation training data provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes:

[0076] Step 101: Acquire a plurality of two-dimensional images of a target area captured by a monocular camera at a plurality of different viewing angles and camera poses corresponding to the plurality of two-dimensional images.

[0077] Among them, a monocular camera is a camera that captures two-dimensional images through a lens, and a monocular camera cannot directly obtain depth information. The monocular camera is mounted on a drone, and multiple two-dimensional images from different perspectives are collected through the movement of the drone. The target area can be cities, mountains, plateaus and other areas, and the embodiments of the present application are not limited to this. The drone is also provided with a high-precision attitude sensor to obtain the camera posture corresponding to the two-dimensional image through the attitude sensor on the drone. The attitude sensor can be an inertial measurement module or a global satellite navigation module. The attitude sensor can record the corresponding camera posture of each two-dimensional image. The camera posture is the camera viewpoint and the direction in the world coordinate system.

[0078] Optionally, by using a monocular camera mounted on a drone to collect images of the target area, the drone can be set to an automatic shooting mode to automatically capture images at preset time intervals or distance intervals, and while obtaining the two-dimensional image, the camera pose corresponding to the two-dimensional image corresponding to the attitude sensor on the drone is obtained, and the two-dimensional image and the camera pose corresponding to the two-dimensional image are matched.

[0079] Optionally, the drone is also equipped with a real-time data transmission module to transmit the collected multiple two-dimensional images and the camera positions corresponding to the multiple two-dimensional images to the ground station or cloud server in real time, which can avoid data loss and facilitate subsequent processing. At the same time, the drone is also equipped with a large-capacity storage device, such as an SD (Secure Digital Card) card or a solid-state drive (SSD) to store all captured data during flight. The storage device has high read and write speed and reliability to ensure data integrity.

[0080] Optionally, if the multiple two-dimensional images collected and the camera poses corresponding to the multiple two-dimensional images cannot meet the two-dimensional image acquisition requirements of the target area, the coverage and quality of the two-dimensional images are analyzed based on the multiple two-dimensional images collected initially and the camera poses corresponding to the multiple two-dimensional images, and the flight path of the drone is optimized. The comprehensiveness and quality of two-dimensional image acquisition can be improved by increasing the flight path of the drone, adjusting the flight altitude and angle of the drone, etc. For areas with poor quality or insufficient coverage, repeated acquisition can be performed. The drone can re-fly and collect data according to the optimized flight path to ensure the accuracy of the multiple two-dimensional images of the target area and the camera poses corresponding to the multiple two-dimensional images.

[0081] Step 102: Perform neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain a neural radiation field model of the target area.

[0082] The neural radiation field is used to generate a two-dimensional image under the camera pose.

[0083] Optionally, the camera poses corresponding to the multiple two-dimensional images are used as the input of the neural radiation field, and the loss function of the neural radiation field is determined according to the output of the neural radiation field and the two-dimensional images of the camera poses corresponding to the multiple two-dimensional images. The neural radiation field is optimized according to the loss function of the neural radiation field, so as to obtain a neural radiation field model of the target area.

[0084] Step 103: According to the preset virtual binocular camera parameters, a neural radiation field model is used to render the binocular image to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image.

[0085] Among them, the baseline and internal parameters of the preset virtual binocular camera parameters can be set according to the viewing distance requirements, and the embodiments of the present application do not limit this. The baseline distance is the horizontal distance between the left virtual camera and the right virtual camera in the preset virtual binocular camera. The baseline distance is used to ensure that the virtual binocular image generated by the de-marking has an appropriate parallax. The internal parameters include focal length, principal point, and imaging plane size.

[0086] Optionally, the positions of the left virtual camera and the right virtual camera are determined according to preset virtual binocular camera parameters, and the neural radiation field model is used to perform binocular image rendering to obtain the position rendering image of the left virtual camera and the position rendering image of the right camera. A virtual binocular image and a depth map corresponding to the virtual binocular image are determined according to the position rendering image of the left virtual camera and the position rendering image of the right camera, and a disparity true value map corresponding to the virtual binocular image is determined according to the depth map corresponding to the virtual binocular image.

[0087] Step 104: Generate binocular depth estimation training data according to the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity true value map corresponding to the virtual binocular image for model training to obtain a binocular depth estimation model.

[0088] Optionally, based on the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity truth map corresponding to the virtual binocular image, binocular depth estimation training data is generated, and model training is performed to obtain a binocular depth estimation model. Without setting a binocular camera, a virtual binocular image of the target area, a depth map corresponding to the virtual binocular image, and a disparity truth map corresponding to the virtual binocular image can be obtained by determining preset virtual binocular camera parameters.

[0089] In an embodiment of the present application, a plurality of two-dimensional images under a plurality of different viewing angles captured by a monocular camera for a target area and the camera poses corresponding to the plurality of two-dimensional images are obtained, and a neural radiation field training is performed on the target area according to the plurality of two-dimensional images and the camera poses corresponding to the plurality of two-dimensional images, so as to obtain a neural radiation field model of the target area. According to the preset virtual binocular camera parameters, a neural radiation field model is used to render binocular images, and a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity truth map corresponding to the virtual binocular image are obtained. According to the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity truth map corresponding to the virtual binocular image, binocular depth estimation training data is generated for model training to obtain a binocular depth estimation model. The present application can obtain a neural radiation field model of the target area through training with two-dimensional images captured by a monocular camera and the camera poses corresponding to the two-dimensional images, thereby obtaining a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity truth map corresponding to the virtual binocular image, and generating binocular depth estimation training data for model training to obtain a binocular depth estimation model. The virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity truth map corresponding to the virtual binocular image can be obtained without using a binocular camera and a depth sensor, which reduces the cost of obtaining binocular images and simplifies the process of obtaining binocular images. The image review map corresponding to the binocular image can be obtained without relying on a depth sensor, and is not limited by the effective range of the depth sensor. It can generate ultra-large-scale scenes and expand the training depth truth range of the existing binocular depth estimation network.

[0090] Based on the above embodiments, the present application also provides a process for posture correction in a method for generating binocular depth estimation training data. Figure 2 A flowchart of posture correction in a method for generating binocular depth estimation training data provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, as in the above step 102, before performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain the neural radiation field model of the target area, it also includes:

[0091] Step 201: Determine the positions of two-dimensional key points for feature matching from multiple two-dimensional images.

[0092] Among them, the two-dimensional key points are the salient points in the target area, for example, the corner points or edge points of the objects in the target area. The corner points are usually located on the edges or corners of the objects and can provide the location of the geometric information of the objects in the target area. The edge points are the outlines of the objects or the dividing lines between different areas in the scene. The two-dimensional key points can be obtained by manual calibration or by automatic recognition of the system.

[0093] Optionally, key point matching is performed from multiple two-dimensional images, that is, all two-dimensional images of the target area, and positions of two-dimensional key points of feature matching in the target area are determined from all two-dimensional images of the target area.

[0094] Step 202: Based on the position of the two-dimensional key point and the camera pose corresponding to the matching two-dimensional image where the two-dimensional key point is located, a preset motion recovery structure algorithm is used to perform three-dimensional reconstruction to map the two-dimensional key point to a preset three-dimensional space, and the initial three-dimensional point corresponding to the two-dimensional key point in the preset three-dimensional space is obtained as the initial three-dimensional point corresponding to the two-dimensional key point.

[0095] Among them, a structure from motion (Structure from Motion, referred to as SFM) algorithm is preset.

[0096] Optionally, a preset structure-from-motion algorithm is used to perform 3D reconstruction based on the positions of the 2D key points and the camera pose corresponding to the matching 2D image where the 2D key points are located, that is, a preset structure-from-motion algorithm is used to perform 3D reconstruction based on multiple 2D key points of the target area and the camera pose corresponding to the matching 2D image where the 2D key points of the target area are located, so as to map the 2D key points to the preset 3D space and obtain the initial 3D points corresponding to the 2D key points in the preset 3D space. The initial 3D points corresponding to the 2D key points in the preset 3D space can be determined by triangulation method using the 2D key points and the camera pose corresponding to the matching 2D image where the 2D key points are located.

[0097] Step 203: According to the initial three-dimensional point and the matching two-dimensional image, the camera posture corresponding to the matching two-dimensional image is corrected.

[0098] Optionally, based on the initial three-dimensional point in the target area and the matching two-dimensional image, it is determined whether the camera pose in the matching two-dimensional image is accurate, and if the camera pose is inaccurate, the camera pose corresponding to the matching two-dimensional image is corrected.

[0099] In the embodiment of the present application, the position of the two-dimensional key point of feature matching is determined from multiple two-dimensional images; according to the position of the two-dimensional key point and the camera posture corresponding to the matching two-dimensional image where the two-dimensional key point is located, a preset motion recovery structure algorithm is used to perform three-dimensional reconstruction to map the two-dimensional key point to the preset three-dimensional space, and the initial three-dimensional point corresponding to the two-dimensional key point in the preset three-dimensional space is obtained as the initial three-dimensional point corresponding to the two-dimensional key point; according to the initial three-dimensional point and the matching two-dimensional image, the camera posture corresponding to the matching two-dimensional image is corrected. The accuracy of the camera posture is guaranteed, and the accuracy of the neural radiation field model of the target area is improved.

[0100] Based on the above embodiments, the present application also provides a process for determining key points in a method for generating binocular depth estimation training data. Figure 3 A schematic diagram of a process for determining key points in a method for generating binocular depth estimation training data provided in an embodiment of the present application, such as Figure 3 As shown, in the above step 201, determining the positions of the two-dimensional key points of feature matching from the multiple two-dimensional images includes:

[0101] Step 301: extract local feature points from each two-dimensional image to obtain candidate local feature points in each image.

[0102] The local feature extraction points are corner points or edge points in each two-dimensional image. For example, the eaves of a building in a two-dimensional image are local feature points.

[0103] Optionally, corner points or edge points in each two-dimensional image are determined respectively as candidate local feature points in each image.

[0104] Step 302: Perform feature point matching on candidate local feature points in different two-dimensional images, and obtain the positions of the local feature points with feature matching in different two-dimensional images as the positions of the two-dimensional key points.

[0105] Optionally, feature point matching is performed on candidate local feature points in different two-dimensional images to determine the correspondence between the same candidate local feature points in different images, thereby obtaining the positions of the local feature points with feature matching in different two-dimensional images as the positions of the two-dimensional key points.

[0106] For example, when the eaves in the first two-dimensional image are used as candidate local feature points, the eaves in the second two-dimensional image are also used as candidate local feature points, the correspondence between the candidate local feature points corresponding to the eaves in the first two-dimensional image and the second two-dimensional image is determined, so as to obtain the positions of the local feature points with feature matching in different two-dimensional images as the positions of the two-dimensional key points.

[0107] In the embodiment of the present application, local feature points are extracted from each two-dimensional image to obtain candidate local feature points in each image, and feature point matching is performed on the candidate local feature points in different two-dimensional images to obtain the positions of the local feature points with feature matching in different two-dimensional images as the positions of the two-dimensional key points. The efficiency of determining the positions of the two-dimensional key points is improved, and the influence of detection noise and appearance changes on the positioning of the key points can be significantly reduced, thereby improving the accuracy of the key points.

[0108] Based on the above embodiments, the present application also provides a process of three-dimensional reconstruction in a method for generating binocular depth estimation training data. Figure 4 A schematic diagram of a process of three-dimensional reconstruction in a method for generating binocular depth estimation training data provided in an embodiment of the present application, such as Figure 4As shown, in the above step 202, before using a preset structure-from-motion algorithm to perform three-dimensional reconstruction according to the positions of the two-dimensional key points and the camera poses corresponding to the plurality of two-dimensional images, the method further includes:

[0109] Step 401: According to the position of the two-dimensional key point, feature extraction is performed on different two-dimensional images corresponding to the two-dimensional key point to obtain convolutional neural network features of the two-dimensional key point in different two-dimensional images.

[0110] Optionally, according to the position of the two-dimensional key point, a pre-trained convolutional neural network model is used to extract features from different two-dimensional images corresponding to the two-dimensional key point to obtain the convolutional neural network features of the two-dimensional key point in different two-dimensional images. The pre-trained neural network model is used to extract features from different two-dimensional images corresponding to the two-dimensional key point to obtain the convolutional neural network features of the two-dimensional key point. The convolutional neural network features are the features corresponding to the pre-trained convolutional neural network model, and the convolutional neural network features are composed of multiple vectors of the two-dimensional key point.

[0111] Step 402: Adjust the position of the two-dimensional key point according to the convolutional neural network features of the two-dimensional key point in different two-dimensional images.

[0112] Optionally, the two-dimensional key points are matched according to the convolutional neural network features of the two-dimensional key points in different two-dimensional images, thereby adjusting the positions of the two-dimensional key points. Since the two-dimensional key points are selected manually or by computer, they may not be accurately matched during the selection process of the two-dimensional key points. Therefore, according to the convolutional neural network features of the two-dimensional key points in different two-dimensional images, the positions of the two-dimensional key points are adjusted, thereby optimizing the matching of the two-dimensional key points and improving the matching accuracy.

[0113] As in step 202 above, according to the positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images, a preset structure-from-motion algorithm is used to perform three-dimensional reconstruction to map the two-dimensional key points to a preset three-dimensional space, including:

[0114] Step 403: Perform three-dimensional reconstruction using an algorithm based on the adjusted positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images.

[0115] Among them, the algorithm can be a triangulation method, in which the positions of the adjusted two-dimensional key points and the camera poses corresponding to multiple two-dimensional images are used as known points in the triangulation method, and three-dimensional reconstruction is performed through the triangulation method, so as to map the matching two-dimensional key points to a preset three-dimensional space.

[0116] In an embodiment of the present application, according to the position of the two-dimensional key point, feature extraction is performed on different two-dimensional images corresponding to the two-dimensional key point to obtain the convolutional neural network features of the two-dimensional key point in different two-dimensional images, and according to the convolutional neural network features of the two-dimensional key point in different two-dimensional images, the position of the two-dimensional key point is adjusted, and according to the adjusted position of the two-dimensional key point and the camera pose corresponding to the multiple two-dimensional images, an algorithm is used to perform three-dimensional reconstruction. The present application can ensure the accuracy of the position of the two-dimensional key point, thereby improving the matching efficiency of the two-dimensional key point, and can ensure the accuracy of the three-dimensional reconstruction.

[0117] Based on the above embodiments, the present application also provides a process for posture correction in a method for generating binocular depth estimation training data. Figure 5 A flowchart of posture correction in a method for generating binocular depth estimation training data provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, in step 203, the camera posture corresponding to the matching two-dimensional image is corrected according to the initial three-dimensional point and the multiple two-dimensional images, including:

[0118] Step 501: According to the initial three-dimensional point, an initial mapping convolutional neural network feature of the initial three-dimensional point corresponding to the initial mapping two-dimensional point in the matching two-dimensional image is obtained.

[0119] Optionally, based on the initial three-dimensional point and the camera pose corresponding to the two-dimensional image corresponding to the initial three-dimensional point, the initial mapping two-dimensional point corresponding to the initial three-dimensional point is determined, and based on the initial mapping two-dimensional point and the pre-trained convolutional neural network, the initial mapping convolutional neural network features of the initial mapping two-dimensional point in the matching two-dimensional image are determined.

[0120] Step 502: Adjust the camera pose corresponding to the matching two-dimensional image according to the initial convolutional neural network features of the two-dimensional key points in the matching two-dimensional image and the initial mapping convolutional neural network features.

[0121] Optionally, the two-dimensional key points match the initial convolutional neural network features and the initial mapping convolutional neural network features in the two-dimensional image, determine the feature difference between the initial convolutional neural network features and the initial mapping convolutional neural network features, and then adjust the camera pose corresponding to the matching two-dimensional image according to the feature difference.

[0122] Step 503: According to the positions of the two-dimensional key points and the adjusted camera pose corresponding to the matching two-dimensional image, an algorithm is used to re-perform three-dimensional reconstruction to obtain new three-dimensional points corresponding to the two-dimensional key points.

[0123] Optionally, according to the positions of the two-dimensional key points and the adjusted camera pose corresponding to the matching two-dimensional image, a triangulation algorithm is used to re-reconstruct the three-dimensional image to obtain new three-dimensional points corresponding to the two-dimensional key points.

[0124] Step 504: reacquire new mapping convolutional neural network features of the new three-dimensional point corresponding to the new mapping two-dimensional point in the matching two-dimensional image.

[0125] Optionally, based on the new three-dimensional point and the camera pose corresponding to the two-dimensional image corresponding to the new three-dimensional point, determine the new mapping two-dimensional point corresponding to the new three-dimensional point, and determine the new mapping convolutional neural network features of the new mapping two-dimensional point in the matching two-dimensional image based on the new mapping two-dimensional point and the pre-trained convolutional neural network.

[0126] Step 505: Re-adjust the camera pose corresponding to the matching two-dimensional image according to the initial convolutional neural network features and the new mapping convolutional neural network features, until the mapping convolutional neural network features of the mapped two-dimensional points corresponding to the two-dimensional key points obtained based on the adjusted camera pose and the initial convolutional neural network features meet the preset convolutional neural network feature consistency conditions.

[0127] Among them, the preset convolutional neural network feature consistency condition is that the feature difference between the mapped convolutional neural network feature of the mapped two-dimensional point corresponding to the two-dimensional key point and the initial convolutional neural network feature is the smallest, which can be 0 or other data, depending on the actual scenario.

[0128] Optionally, based on the initial convolutional neural network features and the new mapping convolutional neural network features, the feature difference between the initial convolutional neural network features and the new mapping convolutional neural network features is determined, so that the camera pose corresponding to the matching two-dimensional image is adjusted according to the feature difference through a nonlinear optimization algorithm, until the mapping convolutional neural network features of the mapping two-dimensional points corresponding to the two-dimensional key points obtained based on the adjusted camera pose and the initial convolutional neural network features meet the preset convolutional neural network feature consistency conditions. That is, the difference between the mapping convolutional neural network features of the mapping two-dimensional points corresponding to the two-dimensional key points obtained by the adjusted camera pose and the initial convolutional neural network features is the smallest. Among them, the nonlinear optimization algorithm can be the Levenberg-Marquardt algorithm.

[0129] Step 506: Determine the adjusted camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met as the target camera pose corresponding to the matching two-dimensional image.

[0130] Optionally, determine the adjusted camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met, and use the adjusted camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met as the target camera pose corresponding to the matching two-dimensional image.

[0131] In an embodiment of the present application, the target camera pose is iteratively optimized, the three-dimensional points and camera pose corresponding to the two-dimensional key points are updated each iteration, and the adjusted camera pose corresponding to the matching two-dimensional image is determined as the target camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met, thereby ensuring high accuracy of the three-dimensional points and camera pose, and improving the precision and accuracy of the target camera position.

[0132] Based on the above embodiments, the present application also provides a process for determining initial convolutional neural network features in a method for generating binocular depth estimation training data. Figure 6 A schematic diagram of a process for determining initial convolutional neural network features in a method for generating binocular depth estimation training data provided in an embodiment of the present application, such as Figure 6 As shown, in the above step 502, according to the initial three-dimensional point, obtaining the initial mapping convolutional neural network feature of the initial three-dimensional point corresponding to the initial mapping two-dimensional point in the matching two-dimensional image includes:

[0133] Step 601: Determine the position of the initial mapped two-dimensional point in the matching two-dimensional image according to the camera posture corresponding to the matching two-dimensional image.

[0134] Optionally, the initial three-dimensional point is mapped to the matching two-dimensional image according to the camera pose corresponding to the matching two-dimensional image and the initial three-dimensional point according to a preset triangulation algorithm, so that the initial three-dimensional point is initially mapped to the position of the two-dimensional point in the matching two-dimensional image.

[0135] Step 602: Perform convolutional neural network feature extraction on the matching two-dimensional image according to the position of the initial mapping two-dimensional point to obtain the initial mapping convolutional neural network feature.

[0136] Optionally, convolutional neural network features are extracted based on the positions of the initial mapping two-dimensional points and a pre-trained convolutional neural network model to obtain initial mapping convolutional neural network features.

[0137] In an embodiment of the present application, the position of the initial mapping two-dimensional point in the matching two-dimensional image is determined according to the camera posture corresponding to the matching two-dimensional image, and the convolutional neural network feature extraction is performed on the matching two-dimensional image according to the position of the initial mapping two-dimensional point to obtain the initial mapping convolutional neural network feature. The present application can ensure that the initial mapping convolutional neural network feature obtained is accurate and more closely matches the matching two-dimensional image by capturing the detailed information in the matching two-dimensional image.

[0138] Based on the above embodiments, the present application also provides a process for determining the camera pose in a method for generating binocular depth estimation training data. Figure 7 A flowchart of camera pose in a method for generating binocular depth estimation training data provided in an embodiment of the present application is shown in FIG. Figure 7As shown, as in the above step 506, according to the initial convolutional neural network features of the two-dimensional key points in the matching two-dimensional image, and the initial mapping convolutional neural network features, adjusting the camera pose corresponding to the matching two-dimensional image includes:

[0139] Step 701: Determine a convolutional neural network feature error value based on the initial convolutional neural network features and the initial mapped convolutional neural network features.

[0140] The initial convolutional neural network features are composed of multiple initial vectors, the initial mapping convolutional neural network features are composed of multiple initial mapping vectors, and the convolutional neural network feature error values ​​are composed of multiple error vectors.

[0141] Optionally, a difference is made between multiple initial vectors and multiple initial mapping vectors in the initial convolutional neural network feature and the initial mapping convolutional neural network feature to determine an error value of the convolutional neural network feature.

[0142] Step 702: Adjust the camera pose corresponding to the matching two-dimensional image according to the convolutional neural network feature error value.

[0143] Optionally, according to the convolutional neural network feature error value, the camera pose corresponding to the matching two-dimensional image is adjusted so that the adjusted camera pose makes the new three-dimensional point of the adjusted camera position correspond to the new mapping two-dimensional point in the matching two-dimensional image, and the error value between the new mapping convolutional neural network feature and the initial convolutional neural network feature is reduced.

[0144] In the embodiment of the present application, the convolutional neural network feature error value is determined according to the initial convolutional neural network feature and the initial mapping convolutional neural network feature, and the camera pose corresponding to the matching two-dimensional image is adjusted according to the convolutional neural network feature error value. The influence on the camera pose in the presence of a large amount of detection noise and appearance changes can be reduced, thereby improving the accuracy of the camera pose.

[0145] Based on the above embodiments, the present application also provides a process for determining a neural radiation field in a method for generating binocular depth estimation training data. Figure 8 A schematic diagram of a process for determining a neural radiation field in a method for generating binocular depth estimation training data provided in an embodiment of the present application, such as Figure 8 As shown, in the above step 102, neural radiation field training is performed on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain a neural radiation field model of the target area, including:

[0146] Step 801: spatially divide the target area to obtain multiple spatial units.

[0147] Optionally, the target area is spatially divided to obtain a plurality of spatial units with the same space, and each spatial unit has a corresponding spatial center point.

[0148] Step 802: Determine pixel data sets corresponding to multiple spatial units from multiple two-dimensional images.

[0149] Optionally, pixels are assigned to corresponding submodules according to the intersection of the rays corresponding to the pixels in the multiple two-dimensional images and these spatial units. Each submodule only needs to process the pixel data related to the spatial unit corresponding to it, which significantly reduces the amount of training data for each submodule. The rays corresponding to the pixels in the multiple two-dimensional images are multiple rays composed of the camera postures corresponding to the multiple two-dimensional images as one point and the pixels in the multiple images as another point.

[0150] Step 803: According to the pixel data sets corresponding to the multiple spatial units, neural radiation field training is performed on the multiple spatial units respectively to obtain neural radiation sub-modules corresponding to the multiple spatial units.

[0151] Optionally, based on pixel data sets corresponding to the multiple spatial units and rays corresponding to pixels in the pixel data sets, neural radiation field training is performed on the multiple spatial units respectively to obtain neural radiation sub-modules corresponding to the multiple spatial units.

[0152] Optionally, using data parallelism, the training task of the neural radiation field is distributed to multiple graphics processing units (GPUs) for calculation, which can significantly speed up the training and reduce the training time. Each submodule is trained using the pixel data set assigned to it to learn the radiation field representation of the spatial unit, so that the parameters of each submodule are specialized for processing a specific spatial area, thereby improving the efficiency and accuracy of the model.

[0153] Step 804: Obtain a neural radiation field model of the target area according to the neural radiation submodules corresponding to the multiple spatial units.

[0154] Optionally, according to the neural radiation sub-modules corresponding to the multiple spatial units, a set of neural radiation sub-modules corresponding to the multiple spatial units is used as the neural radiation field model of the target area.

[0155] In an embodiment of the present application, the target area is spatially divided to obtain multiple spatial units, pixel data sets corresponding to the multiple spatial units are respectively determined from multiple two-dimensional images, neural radiation field training is performed on the multiple spatial units according to the pixel data sets corresponding to the multiple spatial units, neural radiation sub-modules corresponding to the multiple spatial units are obtained, and a neural radiation field model of the target area is obtained according to the neural radiation sub-modules corresponding to the multiple spatial units. In the present application, neural radiation field training for multiple spatial units respectively significantly speeds up the training speed and reduces the training time. The parameters of different neural radiation sub-modules are specifically used to process specific spatial areas, thereby improving the efficiency and accuracy of the model.

[0156] Based on the above embodiments, the present application also provides a process for determining a pixel data set in a method for generating binocular depth estimation training data. Fig. 9 A schematic diagram of a process for determining a pixel data set in a method for generating binocular depth estimation training data provided in an embodiment of the present application, such as Fig. 9 As shown, as in the above step 802, respectively determining pixel data sets corresponding to a plurality of spatial units from a plurality of two-dimensional images respectively determining pixel data sets corresponding to a plurality of spatial units from a plurality of two-dimensional images respectively includes:

[0157] Step 901: Determine, according to the camera posture corresponding to each two-dimensional image, a ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image.

[0158] Alternatively, a neural radiance field is a method for representing a 3D scene through a neural network, which represents the scene as an implicit 3D volume function. This function accepts spatial coordinates and viewing direction as input and outputs the color and density of the pixel. The neural radiance field model can be represented as a multi-layer perceptron whose input is the camera pose, color and density, that is: F Θ ∶(x,d)→(c,σ). Among them, F is a multi-layer perceptron, Θ is a weight parameter, x is a three-dimensional space coordinate, that is, any point on the ray, d is the direction from the camera viewpoint to a pixel point, c is the color of the three-dimensional space coordinate, and σ is the density of the three-dimensional space coordinate. The value of σ is only related to x, denoted as σ(x). The value of c is related to x and d, denoted as c(x,d).

[0159] Optionally, all data are normalized before being processed. The camera viewpoint and the coordinates of multiple pixels in the two-dimensional image are determined according to the camera pose corresponding to each two-dimensional image, the camera viewpoint corresponding to each two-dimensional image is taken as a point, and the coordinates of each pixel point are taken as another point. According to the camera viewpoint corresponding to each two-dimensional image, the direction from the camera viewpoint to a pixel point, and the distance from the camera viewpoint to a pixel point, the ray r(t) from the camera viewpoint to a pixel point in each two-dimensional image is determined by formula (1), and the ray corresponding to each pixel point is aggregated into rays from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image.

[0160] r(t)=o+td (1)

[0161] Among them, r(t) is the ray from the camera viewpoint to a pixel point, o is the camera viewpoint, t is the distance from the camera viewpoint to a pixel point, and d is the direction from the camera viewpoint to a pixel point.

[0162] Step 902: Intersect each spatial unit with the ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image to obtain a pixel data set corresponding to each spatial unit.

[0163] Optionally, the ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image is intersected with each spatial unit to determine the pixel corresponding to the ray in each spatial unit, thereby obtaining a pixel data set corresponding to each spatial unit.

[0164] Optionally, determining the ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image also includes determining a density function corresponding to the neural radiation field through the neural radiation field, and according to the density function corresponding to the neural radiation field, determining the cumulative transparency T(t) of a pixel point from the ray from the camera viewpoint to a pixel point in each two-dimensional image through formula (2).

[0165]

[0166] Among them, r(s) is the ray from the camera viewpoint to a pixel in each two-dimensional image, and the distance between the pixel and the camera viewpoint is s. σ(r(t)) is the density function corresponding to the neural radiation field, that is, the density function of the ray r(t). T(t) is the distance between a pixel t and the camera viewpoint. n Cumulative transparency up to t, t n is the distance from the camera viewpoint, which is 0.

[0167] Optionally, the color function corresponding to the neural radiation field is determined by the neural radiation field, and according to a pixel point t nThe accumulated transparency, color function and density function up to t determine the color C(r) of a pixel through formula (3).

[0168]

[0169] Among them, T(t) is a pixel point t n The cumulative transparency to t, σ(r(t) is the density function corresponding to the neural radiation field, that is, the density function of the ray r(t), c(r(t), d) is the color function corresponding to the neural radiation field, that is, the color of the density function of the ray r(t) in the direction d, and C(r) is the color of a pixel.

[0170] Optionally, obtaining the pixel data set corresponding to each spatial unit further includes determining the color of the ray according to the pixel data set corresponding to each spatial unit. Generally, the approximate value of the color of a pixel point is obtained by using a numerical method, and a volume rendering algorithm is used. The ray selects multiple sampling points in the scene, and the coordinates and direction of each sampling point are used as the input of the neural radiation field to determine the color and density of the sampling point. The approximate value of the color of a pixel is determined by the color and density of the multiple sampling points. Multiple sampling points are obtained by sampling on the ray, and the cumulative transparency T of the sampling points is determined according to the density of the multiple sampling points and the distance between the sampling points by formula (4). i .

[0171]

[0172] Among them, i is the i-th sampling point, N is N sampling points, which are determined according to the actual situation. N sampling points are distributed in multiple spatial units, and the neural radiation field of the space is trained at the sampling points in different spaces. i is the density of the i-th sampling point, δ i is δ i =t i+1 -t i , represents the distance between adjacent sampling points, T i is the cumulative transparency of the i-th sampling point

[0173] Optionally, according to the cumulative transparency T of multiple sampling points i , the density of multiple sampling points, the colors of multiple sampling points and the distance between sampling points, and the approximate value of the color of a pixel corresponding to the ray is determined by formula (5)

[0174]

[0175] Among them, c i is the color of the i-th sampling point, σ i is the density of the i-th sampling point, δi is δ i =t i+1 -t i , represents the distance between adjacent sampling points, T i is the cumulative transparency of the i-th sampling point.

[0176] Optionally, the neural radiation field model of the target area is obtained according to the neural radiation submodules corresponding to the multiple spatial units. The loss function of the neural radiation field can be determined by formula (7) based on the approximate value of a pixel color and the actual color of a pixel. The neural radiation submodules corresponding to the multiple spatial units are optimized according to the loss function of the neural radiation field to obtain the neural radiation field model of the target area.

[0177]

[0178] in, is the approximate value of a pixel color, C(r) is the actual color of a pixel, and R is the set of all rays.

[0179] Optionally, the gradient of the loss function with respect to the neural radiation field parameters is calculated using a back-propagation algorithm, and the model parameters are updated using an Adam optimization algorithm, which is repeated multiple times on the entire training dataset until the neural radiation field converges, thereby determining the target neural radiation field.

[0180] In the embodiment of the present application, according to the camera posture corresponding to each two-dimensional image, the ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image is determined; the ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image is intersected with each spatial unit to obtain the pixel data set corresponding to each spatial unit. The processing efficiency of pixel data can be improved, thereby improving the efficiency and accuracy of the model.

[0181] Based on the above embodiments, the present application also provides a process for determining an image in a method for generating binocular depth estimation training data. Fig.10 A schematic diagram of a process for determining an image in a method for generating binocular depth estimation training data provided in an embodiment of the present application, such as Fig.10 As shown, in the above step 103, according to the preset virtual binocular camera parameters, the neural radiation field model is used to perform binocular image rendering to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image, including:

[0182] Step 1001: According to the preset posture parameters of the virtual binocular camera, a neural radiation field model is used to perform binocular image rendering to obtain a virtual binocular image and a depth map.

[0183] Optionally, the pose parameters of the left-eye virtual camera and the pose parameters of the right-eye virtual camera are respectively determined according to the pose parameters of the preset virtual binocular camera, and the pose parameters of the left-eye virtual camera and the pose parameters of the right-eye virtual camera are respectively divided into the left-eye position rendering image and the right-eye position rendering image as data of the neural radiation field model, and binocular image rendering is performed according to the left-eye position rendering image and the right-eye position rendering image to obtain a virtual binocular image.

[0184] Optionally, since the depth map of the left virtual camera is the same as the depth map of the right virtual camera, the depth map of the virtual binocular camera is obtained through the position of the left virtual camera or the position of the right virtual camera and the neural radiation field model.

[0185] Optionally, a plurality of sampling points are obtained by sampling on the ray, and the depth z(r) of a ray is determined by formula (8) according to the density of the plurality of sampling points and the distance between the sampling points, as well as the distance between the sampling points and the left-eye virtual camera. The set of the depths of the rays from the viewpoint of the left-eye virtual camera to each pixel in the rendered image at the left-eye position is taken as the depth map of the left-eye virtual camera.

[0186]

[0187] Among them, i is the i-th sampling point, N is N sampling points, which are determined according to the actual situation. N sampling points are distributed in multiple spatial units, and the neural radiation field of the space is trained at the sampling points in different spaces. i is the density of the i-th sampling point, δ i is δ i =t i+1 -t i , represents the distance between adjacent sampling points, T i is the cumulative transparency of the i-th sampling point, t i is the distance between the ith sampling point and the left virtual camera, and z(r) is the depth of a ray.

[0188] Step 1002: Generate a true disparity map according to the pose parameters and depth map of the preset virtual binocular camera.

[0189] Among them, the pose parameters of the virtual camera also include baseline distance and focal length.

[0190] Optionally, the parallax value D(r) of a ray is determined by formula (10) through the depth of the ray, the baseline distance of the virtual binocular camera, and the focal length of the virtual camera.

[0191]

[0192] Among them, b is the baseline distance of the virtual camera, f is the focal length of the virtual camera, z(r) is the depth of a ray, and D(r) is the parallax value of a ray.

[0193] Optionally, a set of time difference values ​​of rays from the viewpoint of the left-eye virtual camera to each pixel in the left-eye position rendering image is used as the true disparity map of the left-eye virtual camera.

[0194] In the embodiment of the present application, according to the preset virtual binocular camera posture parameters, the neural radiation field model is used to render the binocular image, obtain the virtual binocular image and depth map, and generate the parallax truth map according to the preset virtual binocular camera posture parameters and depth map. The reliability of the generated time difference truth map is guaranteed, rich training data is provided, and it is helpful to improve the generalization ability of the model.

[0195] Based on the same inventive concept, an embodiment of the present application also provides a device for generating binocular depth estimation training data corresponding to the method for generating binocular depth estimation training data. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the method for generating binocular depth estimation training data in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0196] Fig.11 A schematic diagram of a device for generating binocular depth estimation training data provided in an embodiment of the present application is shown in FIG. Fig.11 As shown, the device includes: an acquisition module 1101, which is used to acquire a plurality of two-dimensional images under a plurality of different viewing angles captured by a monocular camera for a target area and camera poses corresponding to the plurality of two-dimensional images;

[0197] A training module 1102 is used to perform neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain a neural radiation field model of the target area;

[0198] A rendering module 1103 is used to perform binocular image rendering using the neural radiation field model according to preset virtual binocular camera parameters to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image;

[0199] The generation module 1104 is used to generate binocular depth estimation training data according to the virtual binocular image, the depth map corresponding to the virtual binocular image, and the disparity true value map corresponding to the virtual binocular image, so as to perform model training and obtain a binocular depth estimation model.

[0200] In a possible implementation, the training module 1102 is further used to: determine positions of two-dimensional key points of feature matching from the plurality of two-dimensional images;

[0201] According to the position of the two-dimensional key point and the camera pose corresponding to the matching two-dimensional image where the two-dimensional key point is located, a preset motion recovery structure algorithm is used to perform three-dimensional reconstruction to map the two-dimensional key point to a preset three-dimensional space, and the initial three-dimensional point corresponding to the two-dimensional key point in the preset three-dimensional space is obtained as the initial three-dimensional point corresponding to the two-dimensional key point;

[0202] According to the initial three-dimensional point and the matching two-dimensional image, the camera posture corresponding to the matching two-dimensional image is corrected.

[0203] In a possible implementation, the training module 1102 is specifically configured to: extract local feature points from each two-dimensional image to obtain candidate local feature points in each image;

[0204] Feature point matching is performed on the candidate local feature points in different two-dimensional images, and positions of the local feature points with feature matching in the different two-dimensional images are obtained as positions of the two-dimensional key points.

[0205] In a possible implementation, the training module 1102 is further used to: extract features of different two-dimensional images corresponding to the two-dimensional key points according to the positions of the two-dimensional key points, to obtain convolutional neural network features of the two-dimensional key points in the different two-dimensional images;

[0206] Adjusting the position of the two-dimensional key point according to the convolutional neural network features of the two-dimensional key point in the different two-dimensional images;

[0207] In a possible implementation, the training module 1102 is specifically used to: perform three-dimensional reconstruction using the algorithm according to the adjusted positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images.

[0208] In a possible implementation, the training module 1102 is specifically configured to: obtain, according to the initial three-dimensional point, an initial mapping convolutional neural network feature of an initial mapping two-dimensional point corresponding to the initial three-dimensional point in the matching two-dimensional image;

[0209] Adjusting a camera pose corresponding to the matching two-dimensional image according to an initial convolutional neural network feature of the two-dimensional key point in the matching two-dimensional image and the initial mapping convolutional neural network feature;

[0210] According to the position of the two-dimensional key point and the adjusted camera pose corresponding to the matched two-dimensional image, the algorithm is used to re-perform three-dimensional reconstruction to obtain a new three-dimensional point corresponding to the two-dimensional key point;

[0211] Re-acquire a new mapping convolutional neural network feature of a new mapping two-dimensional point corresponding to the new three-dimensional point in the matching two-dimensional image;

[0212] According to the initial convolutional neural network features and the new mapping convolutional neural network features, the camera pose corresponding to the matched two-dimensional image is readjusted until the mapping convolutional neural network features of the mapped two-dimensional points corresponding to the two-dimensional key points obtained based on the adjusted camera pose and the initial convolutional neural network features meet the preset convolutional neural network feature consistency conditions;

[0213] Determine that the adjusted camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met as the target camera pose corresponding to the matching two-dimensional image.

[0214] In a possible implementation, the training module 1102 is specifically configured to: determine the position of the initial mapping two-dimensional point in the matching two-dimensional image according to the camera pose corresponding to the matching two-dimensional image;

[0215] According to the position of the initial mapping two-dimensional point, convolutional neural network feature extraction is performed on the matching two-dimensional image to obtain the initial mapping convolutional neural network feature.

[0216] In a possible implementation, the training module 1102 is specifically used to: determine a convolutional neural network feature error value according to the initial convolutional neural network feature and the initial mapping convolutional neural network feature;

[0217] According to the convolutional neural network feature error value, the camera posture corresponding to the matching two-dimensional image is adjusted.

[0218] In a possible implementation, the training module 1102 is specifically configured to: spatially divide the target area to obtain a plurality of spatial units;

[0219] Determine pixel data sets corresponding to the plurality of spatial units respectively from the plurality of two-dimensional images;

[0220] According to the pixel data sets corresponding to the multiple spatial units, respectively perform neural radiation field training on the multiple spatial units to obtain neural radiation submodules corresponding to the multiple spatial units;

[0221] A neural radiation field model of the target area is obtained according to the neural radiation submodules corresponding to the multiple spatial units.

[0222] In a possible implementation, the training module 1102 is specifically configured to: determine, according to the camera pose corresponding to each two-dimensional image, a ray from a camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image;

[0223] The rays from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image are intersected with each spatial unit to obtain a pixel data set corresponding to each spatial unit.

[0224] In a possible implementation, the rendering module 1103 is specifically configured to: perform binocular image rendering using the neural radiation field model according to the preset posture parameters of the virtual binocular camera to obtain the virtual binocular image and the depth map;

[0225] The disparity true value map is generated according to the pose parameters of the preset virtual binocular camera and the depth map.

[0226] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0227] The present application also provides a computer device, Fig.12 A schematic diagram of a computer device structure provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, it includes: a processor 1201, a memory 1202, and optionally, a bus 1203. The memory 1202 stores machine-readable instructions executable by the processor 1201. When the computer device is running, the processor 1201 communicates with the memory 1202 through the bus 1203, and the machine-readable instructions are executed by the processor 1201 to perform the steps of the above-mentioned method for generating binocular depth estimation training data.

[0228] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the method for generating binocular depth estimation training data are executed.

[0229] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0230] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or part of the technical solution that contributes to the prior art or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), disk or optical disk and other media that can store program code.

[0231] The above are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be covered by the protection scope of the present application.

Claims

1. A method for generating binocular depth estimation training data, characterized in that: The method comprises: Obtain multiple two-dimensional images of the target area captured by a monocular camera at multiple different perspectives and the camera poses corresponding to the multiple two-dimensional images; According to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images, performing neural radiation field training on the target area to obtain a neural radiation field model of the target area; According to preset virtual binocular camera parameters, the neural radiation field model is used to perform binocular image rendering to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image; According to the virtual binocular image, the depth map corresponding to the virtual binocular image and the disparity true value map corresponding to the virtual binocular image, binocular depth estimation training data is generated for model training to obtain a binocular depth estimation model.

2. The method according to claim 1, characterized in that Before performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain the neural radiation field model of the target area, the method further includes: Determining positions of two-dimensional key points of feature matching from the plurality of two-dimensional images; According to the position of the two-dimensional key point and the camera pose corresponding to the matching two-dimensional image where the two-dimensional key point is located, a preset motion recovery structure algorithm is used to perform three-dimensional reconstruction to map the two-dimensional key point to a preset three-dimensional space, and the initial three-dimensional point corresponding to the two-dimensional key point in the preset three-dimensional space is obtained as the initial three-dimensional point corresponding to the two-dimensional key point; According to the initial three-dimensional point and the matching two-dimensional image, the camera posture corresponding to the matching two-dimensional image is corrected.

3. The method according to claim 2, characterized in that Determining the positions of the two-dimensional key points of feature matching from the multiple two-dimensional images includes: Extracting local feature points from each two-dimensional image to obtain candidate local feature points in each image; Feature point matching is performed on the candidate local feature points in different two-dimensional images, and positions of the local feature points with feature matching in the different two-dimensional images are obtained as positions of the two-dimensional key points.

4. The method according to claim 2, characterized in that: Before performing three-dimensional reconstruction using a preset structure-from-motion algorithm according to the positions of the two-dimensional key points and the camera poses corresponding to the plurality of two-dimensional images, the method further includes: According to the position of the two-dimensional key point, feature extraction is performed on different two-dimensional images corresponding to the two-dimensional key point to obtain convolutional neural network features of the two-dimensional key point in the different two-dimensional images; Adjusting the position of the two-dimensional key point according to the convolutional neural network features of the two-dimensional key point in the different two-dimensional images; The method of performing three-dimensional reconstruction using a preset structure-from-motion algorithm according to the positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images, so as to map the two-dimensional key points to a preset three-dimensional space, includes: The algorithm is used to perform three-dimensional reconstruction according to the adjusted positions of the two-dimensional key points and the camera poses corresponding to the multiple two-dimensional images.

5. The method according to claim 2, characterized in that: The step of correcting the camera pose corresponding to the matching two-dimensional image according to the initial three-dimensional point and the multiple two-dimensional images includes: According to the initial three-dimensional point, an initial mapping convolutional neural network feature of an initial mapping two-dimensional point corresponding to the initial three-dimensional point in the matching two-dimensional image is obtained; Adjusting a camera pose corresponding to the matching two-dimensional image according to an initial convolutional neural network feature of the two-dimensional key point in the matching two-dimensional image and the initial mapping convolutional neural network feature; According to the position of the two-dimensional key point and the adjusted camera pose corresponding to the matched two-dimensional image, the algorithm is used to re-perform three-dimensional reconstruction to obtain a new three-dimensional point corresponding to the two-dimensional key point; Re-acquire a new mapping convolutional neural network feature of a new mapping two-dimensional point corresponding to the new three-dimensional point in the matching two-dimensional image; According to the initial convolutional neural network features and the new mapping convolutional neural network features, the camera pose corresponding to the matched two-dimensional image is readjusted until the mapping convolutional neural network features of the mapped two-dimensional points corresponding to the two-dimensional key points obtained based on the adjusted camera pose and the initial convolutional neural network features meet the preset convolutional neural network feature consistency conditions; Determine that the adjusted camera pose corresponding to the matching two-dimensional image when the preset convolutional neural network feature consistency condition is met as the target camera pose corresponding to the matching two-dimensional image.

6. The method according to claim 5, characterized in that The step of obtaining, according to the initial three-dimensional point, an initial mapping convolutional neural network feature of an initial mapping two-dimensional point corresponding to the initial three-dimensional point in the matching two-dimensional image comprises: Determining the position of the initial mapped two-dimensional point in the matching two-dimensional image according to the camera pose corresponding to the matching two-dimensional image; According to the position of the initial mapping two-dimensional point, convolutional neural network feature extraction is performed on the matching two-dimensional image to obtain the initial mapping convolutional neural network feature.

7. The method according to claim 5, characterized in that The adjusting the camera pose corresponding to the matching two-dimensional image according to the initial convolutional neural network features of the two-dimensional key points in the matching two-dimensional image and the initial mapping convolutional neural network features comprises: Determining a convolutional neural network feature error value according to the initial convolutional neural network feature and the initial mapping convolutional neural network feature; According to the convolutional neural network feature error value, the camera posture corresponding to the matching two-dimensional image is adjusted.

8. The method according to claim 1, characterized in that The performing neural radiation field training on the target area according to the multiple two-dimensional images and the camera poses corresponding to the multiple two-dimensional images to obtain a neural radiation field model of the target area includes: Performing spatial division on the target area to obtain a plurality of spatial units; Determine pixel data sets corresponding to the plurality of spatial units respectively from the plurality of two-dimensional images; According to the pixel data sets corresponding to the multiple spatial units, respectively perform neural radiation field training on the multiple spatial units to obtain neural radiation submodules corresponding to the multiple spatial units; A neural radiation field model of the target area is obtained according to the neural radiation submodules corresponding to the multiple spatial units.

9. The method according to claim 8, characterized in that The step of respectively determining pixel data sets corresponding to the plurality of spatial units from the plurality of two-dimensional images comprises: Determine, according to the camera pose corresponding to each two-dimensional image, a ray from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image; The rays from the camera viewpoint corresponding to each two-dimensional image to each pixel point in each two-dimensional image are intersected with each spatial unit to obtain a pixel data set corresponding to each spatial unit.

10. The method according to claim 1, characterized in that The method of performing binocular image rendering using the neural radiation field model according to preset virtual binocular camera parameters to obtain a virtual binocular image, a depth map corresponding to the virtual binocular image, and a disparity true value map corresponding to the virtual binocular image includes: According to the preset posture parameters of the virtual binocular camera, the neural radiation field model is used to perform binocular image rendering to obtain the virtual binocular image and the depth map; The disparity true value map is generated according to the pose parameters of the preset virtual binocular camera and the depth map.

Images