Network training and scene reconstruction method, device, machine, system and equipment

Abstract

The application provides a network training and scene reconstruction method, device, machine, system and equipment, and relates to the technical field of image processing. The training method comprises the following steps: acquiring a plurality of groups of sample binocular images of a preset engineering work scene collected by a binocular camera on a preset engineering machine; processing the plurality of groups of sample binocular images to obtain a sparse point cloud of the preset engineering work scene; performing dense mapping on the sparse point cloud according to camera parameters of the binocular camera to obtain a plurality of groups of dense depth maps corresponding to the plurality of groups of sample binocular images; processing each group of dense depth maps to obtain a sample disparity map corresponding to each group of sample binocular images; and performing model training according to the plurality of groups of sample binocular images and the corresponding sample disparity maps to obtain a stereo matching network. The application can reduce the cost of three-dimensional scene reconstruction, improve real-time performance, and achieve three-dimensional scene reconstruction based on high-quality point clouds.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to a method, apparatus, machine, system, and device for network training and scene reconstruction. Background Technology

[0002] Image-based 3D target and scene reconstruction is one of the important research directions in the field of computer vision.

[0003] In many engineering operation scenarios, large-scale engineering machinery is mainly used to carry out the work. Due to the harsh environment of the engineering operation scenarios, the existing engineering operation scenarios mainly rely on the work experience of the operators to operate the engineering machinery, which is highly dangerous. Therefore, three-dimensional scene reconstruction of engineering operation scenarios is carried out so that operators can better understand the situation of the engineering operation scenarios and reduce the danger of the work.

[0004] Existing 3D scene reconstruction solutions mainly fall into three categories: laser point cloud 3D reconstruction, multi-view 3D reconstruction, and binocular camera 3D reconstruction. However, each of these solutions has its own shortcomings. Laser point cloud 3D reconstruction using lidar is costly, and its imaging quality is affected by weather, resulting in poor point cloud quality. While multi-view 3D reconstruction produces good point cloud quality, it requires significant computational resources to construct large amounts of point cloud information for large-scale engineering projects, making it difficult to meet the real-time requirements of 3D scene reconstruction in such environments. Binocular camera 3D reconstruction, although low-cost and fast, heavily relies on the parallax estimation between the two cameras. In open environments with drastic lighting changes, such as engineering projects, the accuracy of the generated point cloud is poor, hindering 3D reconstruction. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of the prior art by providing a network training and scene reconstruction method, apparatus, machinery, system, and equipment to reduce the cost of 3D scene reconstruction, improve real-time performance, and achieve 3D scene reconstruction based on high-quality point clouds.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows:

[0007] In a first aspect, embodiments of this application provide a method for training a stereo matching network, the method comprising:

[0008] Acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery;

[0009] The multiple sets of sample binocular images are processed to obtain the sparse point cloud of the preset engineering operation scene;

[0010] Based on the camera parameters of the binocular camera, a dense mapping is performed on the sparse point cloud to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample binocular images.

[0011] Each group of dense depth maps is processed to obtain a sample disparity map corresponding to each group of sample binocular images;

[0012] The stereo matching network is obtained by training the model based on the multiple sets of sample binocular images and the corresponding sample disparity maps.

[0013] Secondly, embodiments of this application also provide a three-dimensional scene reconstruction method, the method comprising:

[0014] Acquire binocular images of a pre-defined engineering operation scene captured by a binocular camera on a pre-defined engineering machinery;

[0015] The stereo matching network is pre-trained to process the binocular image to obtain the disparity map corresponding to the binocular image. The stereo matching network is obtained by training the stereo matching network as described in any of the first aspects above.

[0016] Based on the configuration parameters of the binocular camera, the disparity map is processed to obtain the three-dimensional point cloud data of the preset engineering operation scene;

[0017] Based on the 3D point cloud data and the preset viewing direction, scene rendering is performed to obtain a 3D scene image of the preset engineering operation scene in the preset viewing direction.

[0018] Thirdly, embodiments of this application also provide a training device for a stereo matching network, the device comprising:

[0019] The sample image acquisition module is used to acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0020] The sparse point cloud generation module is used to process the multiple sets of sample binocular images to obtain the sparse point cloud of the preset engineering operation scene.

[0021] The dense mapping module is used to perform dense mapping on the sparse point cloud according to the camera parameters of the stereo camera, so as to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample stereo images.

[0022] The sample disparity map generation module is used to process each group of dense depth maps to obtain a sample disparity map corresponding to each group of sample binocular images.

[0023] The matching network training module is used to train the model based on the multiple sets of sample binocular images and the corresponding sample disparity maps to obtain the stereo matching network.

[0024] Fourthly, embodiments of this application also provide a three-dimensional scene reconstruction apparatus, the apparatus comprising:

[0025] The binocular image acquisition module is used to acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0026] The disparity map generation module is used to process the binocular image using a pre-trained stereo matching network to obtain the disparity map corresponding to the binocular image. The stereo matching network is trained using any of the stereo matching network training methods described in the first method above.

[0027] A 3D point cloud generation module is used to process the disparity map according to the configuration parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene.

[0028] The rendering module is used to render the scene based on the three-dimensional point cloud data and the preset view direction to obtain a three-dimensional scene image of the preset engineering operation scene in the preset view direction.

[0029] Fifthly, embodiments of this application also provide an engineering machine equipped with a binocular camera and a host terminal communicatively connected to the binocular camera, the host terminal being used to execute the steps of the three-dimensional scene reconstruction method as described in any of the second aspects above.

[0030] Sixthly, embodiments of this application also provide a three-dimensional scene reconstruction system, including: a binocular camera, a host terminal, and a client terminal. The binocular camera and the host terminal are both mounted on engineering machinery, and the host terminal is communicatively connected to the binocular camera and the client terminal respectively.

[0031] The host computer is used to perform the steps of the three-dimensional scene reconstruction method as described in any of the second aspects above.

[0032] In a seventh aspect, embodiments of this application also provide an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores program instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the program instructions to perform the steps of the training method for a stereo matching network as described in any of the first aspects above, or the steps of the three-dimensional scene reconstruction method as described in any of the second methods above.

[0033] Eighthly, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the steps of a training method for a stereo matching network as described in any of the first aspects above, or the steps of a three-dimensional scene reconstruction method as described in any of the second aspects above.

[0034] The beneficial effects of this application are:

[0035] This application provides a network training and scene reconstruction method, apparatus, machine, system, and device. The method involves acquiring multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machine; processing the multiple sets of sample binocular images to obtain a sparse point cloud of the preset engineering operation scene; performing dense mapping on the sparse point cloud based on the camera parameters of the binocular camera to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample binocular images; processing each set of dense depth maps to obtain a sample disparity map corresponding to each set of sample binocular images; and training a model based on the multiple sets of sample binocular images and the corresponding sample disparity maps to obtain a stereo matching network. On the one hand, this application utilizes a trained stereo matching network to infer disparity estimation results from binocular images, which improves the real-time performance of disparity estimation calculations, enabling faster 3D scene reconstruction based on disparity estimation results. On the other hand, this application constructs sparse point clouds from sample binocular images, then uses these sparse point clouds to construct dense depth maps. High-quality dense depth maps are used to generate sample disparity maps, resulting in higher accuracy of disparity estimation results. These high-precision sample disparity maps, along with sample binocular images, form training data for the stereo matching network, improving its performance in inferring disparity estimation results from binocular images. This allows for high-quality point cloud-based 3D scene reconstruction based on disparity estimation results. Furthermore, using a binocular camera to sample a large number of binocular images for model training reduces model training costs. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 A flowchart illustrating a training method for a stereo matching network provided in an embodiment of this application;

[0038] Figure 2 A schematic diagram of a sparse point cloud provided for an embodiment of this application;

[0039] Figure 3 A schematic diagram of a dense point cloud provided for an embodiment of this application;

[0040] Figure 4 A flowchart illustrating another method for training a stereo matching network provided in an embodiment of this application;

[0041] Figure 5 A disparity map provided for an embodiment of this application;

[0042] Figure 6 A flowchart illustrating another method for training a stereo matching network provided in an embodiment of this application;

[0043] Figure 7 This is a schematic diagram of the structure of a three-dimensional scene reconstruction system provided in an embodiment of this application;

[0044] Figure 8 A flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application;

[0045] Figure 9 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment;

[0046] Figure 10 A flowchart illustrating another three-dimensional scene reconstruction method provided in this application embodiment;

[0047] Figure 11 A schematic diagram of the structure of an engineering machine provided in this application embodiment;

[0048] Figure 12 A schematic diagram of the structure of the training device for the stereo matching network provided in the embodiments of this application;

[0049] Figure 13 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application;

[0050] Figure 14 A schematic diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments.

[0052] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0053] Furthermore, the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Additionally, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0054] It should be noted that, where there is no conflict, the features in the embodiments of this application can be combined with each other.

[0055] In many engineering operation scenarios, large-scale engineering machinery is mainly used to carry out the work. Due to the harsh environment of engineering operation scenarios, the existing engineering operation scenarios mainly rely on the work experience of operators to operate the engineering machinery on site, which results in high labor costs and high work risks. Therefore, designing semi-automatic or fully automatic control systems is of great practical significance.

[0056] To achieve effective remote control of construction machinery, the core technology lies in the accurate reconstruction of the construction site. This includes the acquisition of two-dimensional visual information and the generation and rendering of three-dimensional point clouds. Two-dimensional visual information facilitates remote observation of the actual situation at the construction site by operators, while three-dimensional point clouds provide richer scene location information and multi-view rendering information based on two-dimensional visual information. Rendering the three-dimensional point cloud from different perspectives provides operators with more auxiliary viewpoints for observation and judgment, thus avoiding the problem of operators frequently needing to stick their heads out of the cockpit to observe blind spots during actual manual operation, improving operational safety. Furthermore, accurate three-dimensional point cloud information can provide the control system with accurate target orientation and scale. For example, if the construction machinery has a precise working position and loading position, the control system can automatically complete the operation and loading, greatly reducing the workload of manual operation. Therefore, accurate three-dimensional point cloud information is key to achieving semi-automated or even fully automated operation.

[0057] Existing 3D scene reconstruction solutions mainly fall into three categories: laser point cloud 3D reconstruction, multi-view 3D reconstruction, and binocular camera 3D reconstruction. However, each of these solutions has its own shortcomings. Laser point cloud 3D reconstruction, using LiDAR, is costly, generates sparse point clouds that are difficult to render high-quality 3D scenes, and its imaging quality is affected by weather, resulting in poor point cloud quality. While multi-view 3D reconstruction produces good point cloud quality, it requires deploying more cameras for image acquisition in large-scale engineering scenarios. Furthermore, the large amount of point cloud information required consumes significant computational resources, making it difficult to meet the real-time requirements of 3D scene reconstruction in engineering operations. Although binocular camera 3D reconstruction technology has low deployment costs and fast computation speed, its point cloud quality heavily relies on the parallax estimation results between the two cameras. In open environments with drastic lighting changes, such as engineering operations, the accuracy of the generated point cloud is poor, hindering 3D reconstruction.

[0058] Based on this, this application aims to leverage the advantages of binocular cameras—low deployment cost, fast computation speed, and high richness of generated point clouds—to collect a large number of sample binocular images. It also utilizes multi-view 3D reconstruction technology to offline construct accurate point clouds, generating dense depth maps corresponding to the sample binocular images. These dense depth maps are then used to automatically generate sample disparity maps as annotation data to train the stereo matching network. After the high-precision stereo matching network is trained, the disparity maps generated by the stereo matching network are used to calculate 3D point cloud data. Based on this 3D point cloud data, 3D scene rendering from any viewpoint is performed, achieving low-cost, high-real-time, and high-quality 3D scene reconstruction.

[0059] Please refer to Figure 1 This is a flowchart illustrating a training method for a stereo matching network provided in an embodiment of this application. Figure 1 As shown, the method includes:

[0060] S11: Acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0061] Generally, the accuracy of the disparity map estimated by the stereo matching network directly affects the accuracy of the point cloud data used for 3D scene reconstruction. The performance of the stereo matching network largely depends on the quality of the training data. Most mainstream models are trained using publicly available datasets. However, most of the data in these publicly available datasets are artificially synthesized data or laser data, and there is no training specifically for engineering operation scenarios. This makes it difficult for stereo matching networks trained using publicly available datasets to achieve the expected results.

[0062] A binocular camera is a device fixed at two different locations for image acquisition. It captures the same feature point on an object, but the feature point is located in different positions within the binocular images. Construction machinery refers to mechanical equipment used in engineering operations; the type of construction machinery may vary depending on the specific task. In this embodiment, a binocular camera is deployed at a predetermined location on the construction machinery to acquire images of the engineering operation scene, resulting in multiple sets of sample binocular images of the scene. In each set of sample binocular images, the coordinates of a feature point of any object in the scene are different.

[0063] It should be noted that, in order to better reconstruct the preset engineering operation scene in 3D, the multiple sets of sample binocular images obtained by the binocular camera are obtained by taking pictures of the preset engineering operation scene from all angles, that is, the multiple sets of sample binocular images can cover the preset engineering operation scene in 360°.

[0064] S12: Process multiple sets of sample binocular images to obtain a sparse point cloud of the preset engineering operation scene.

[0065] In this embodiment, a preset feature extraction method is used to extract multiple sets of feature points from multiple sets of sample binocular images. An image feature point matching method is then used to match these multiple sets of feature points, resulting in multiple feature point pairs. Each feature point pair represents different imaging of the same feature point in the multiple sets of sample binocular images within an engineering operation scenario. Representative feature points are selected from each set of feature points. Based on the mapping matrix of the target camera in the binocular camera, multiple representative feature points are mapped onto a three-dimensional spatial coordinate system to generate a sparse point cloud. The sample binocular images acquired by the binocular camera are divided into left and right views. Representative feature points are selected from either the left or right views of the multiple sets of sample binocular images. The target camera is the camera in the binocular camera used to generate the left or right view.

[0066] For example, the feature extraction method can be the Scale-Invariant Feature Transform (SIFT) method, which can extract multiple feature points in the target view set; the image feature point matching method can be the Euclidean distance algorithm, which determines multiple pairs of feature points by calculating the Euclidean distance between multiple feature points in the target view set.

[0067] It should be noted that, due to the large range of viewpoints between multiple sets of sample binocular images, a feature in the preset engineering operation scenario may not appear in all sample binocular images. Therefore, matching multiple sets of feature points does not require feature points that match in all sets to form a pair of two-dimensional feature points. When matching multiple sets of feature points, as long as there are a preset number of sets with mutually matching feature points, a pair of two-dimensional feature points can be formed.

[0068] S13: Based on the camera parameters of the stereo camera, perform dense mapping on the sparse point cloud to obtain multiple sets of dense depth maps corresponding to multiple sets of sample stereo images.

[0069] In this embodiment, the camera parameters of the binocular camera include intrinsic and extrinsic parameters. Based on the intrinsic and extrinsic parameters of the binocular camera, the depth values ​​of each pixel projected onto each target view are calculated. The pixels projected onto each target view are used as seed points. Based on the depth values ​​of each seed point, the depth values ​​of other pixels on each target view are calculated. Based on the depth values ​​of all pixels, a dense depth map corresponding to each target view is generated. Each set of sample binocular views corresponds to a set of dense depth maps. Each pixel in the dense depth map is projected onto three-dimensional space to obtain a three-dimensional point cloud. The three-dimensional point clouds corresponding to multiple sets of dense depth maps constitute the dense point cloud of the preset engineering operation scenario.

[0070] S14: Process each group of dense depth maps to obtain a sample disparity map corresponding to each group of sample binocular images.

[0071] Generally, it is difficult to manually annotate the sample disparity maps used to train stereo matching networks as in the training of traditional models. To better train stereo matching networks, this embodiment uses automated annotation to generate sample disparity maps as annotations for sample binocular images. A disparity map refers to the pixel difference between the same pixel in the left and right views of a set of binocular images. In this embodiment, the depth values ​​of dense depth maps constructed from sparse point clouds are more accurate. By projecting the reference depth map in each set of dense depth maps, the difference between the pixel value of the same pixel in the target image of the sample binocular image and the resulting pixel map is calculated. The pixel difference corresponding to each pixel is the sample disparity map corresponding to the sample binocular image. The reference depth map is the depth map corresponding to the left or right view in each set of dense depth maps. If the reference depth map is the depth map corresponding to the left view, the target image is the right view in the sample binocular image; if the reference depth map is the depth map corresponding to the right view, the target image is the left view in the sample binocular image.

[0072] S15: Train the model based on multiple sets of sample binocular images and corresponding sample disparity maps to obtain a stereo matching network.

[0073] In this embodiment, the sample disparity map serves as the annotation for the sample binocular images. Multiple sets of sample binocular images and their corresponding sample disparity maps are used as training data for model training. After training, a stereo matching network is obtained. This stereo matching network can perform stereo matching on any binocular image collected by the binocular camera from the engineering work site to obtain the disparity map corresponding to the binocular image. Based on this disparity map, high-precision point cloud data is constructed so that the high-precision point cloud data can be used to render a three-dimensional reconstructed scene image of the engineering work scene.

[0074] The stereo matching network training method provided in the above embodiments has two advantages. First, by using the trained stereo matching network to infer the disparity estimation results of binocular images, the real-time performance of calculating the disparity estimation results can be improved, enabling faster 3D scene reconstruction based on the disparity estimation results. Second, by constructing sparse point clouds from sample binocular images and then constructing dense depth maps from the sparse point clouds, sample disparity maps are generated using high-quality dense depth maps, resulting in higher accuracy of the disparity estimation results from the sample disparity maps. The high-precision sample disparity maps are used as labeled data and, together with the sample binocular images, constitute training data to train the stereo matching network, making the stereo matching network perform better in inferring the disparity estimation results of binocular images. This enables 3D scene reconstruction based on high-quality point clouds calculated from the disparity estimation results. Furthermore, using a binocular camera to sample a large number of sample binocular images for model training also reduces the model training cost.

[0075] In one optional embodiment, processing multiple sets of sample binocular images in S12 to obtain a sparse point cloud of a preset engineering operation scene may include:

[0076] Using structure-reconstruction-motion technique, multiple sets of sample binocular images are processed to obtain a sparse map of a preset engineering operation scene. The sparse map includes sparse point clouds and camera parameters.

[0077] In this embodiment, Structure From Motion (SFM) refers to the process of calculating camera parameters from images from various angles and restoring the three-dimensional structure of the scene. By using the geometric relationship of feature matching points between images, the position and pose information of the camera and the coordinate positions of feature points in the three-dimensional spatial coordinate system are restored.

[0078] Specifically, a feature extraction method is used to extract multiple sets of feature points from multiple sets of sample stereo images. These feature points are then matched to obtain feature matching points between the multiple sets of sample stereo images. A set of equations is constructed using the two-dimensional coordinates of the feature matching points in the multiple sets of sample stereo images. The equations for the movement of the stereo camera during shooting are then calculated, which means calculating the extrinsic parameter matrix of each camera in the stereo camera. The extrinsic parameter matrix of each camera is the position and pose information of that camera, used to represent the rotation angle and translation vector of that camera during shooting.

[0079] The intrinsic parameter matrix of each camera is the camera's internal parameters, which can be obtained through manual calibration. The intrinsic parameter matrix remains unchanged during camera movement. The intrinsic parameter matrix and extrinsic parameter matrix of each camera together constitute the camera's mapping matrix.

[0080] By performing feature matching on multiple sets of feature points in multiple sets of sample binocular images, multiple sets of feature point pairs are obtained. Representative feature points are then selected from each set of feature point pairs. These representative feature points belong to the view corresponding to the target camera in the multiple sets of sample binocular images. Based on the mapping matrix of the target camera, the three-dimensional coordinates of these representative feature points mapped to the three-dimensional spatial coordinate system are calculated to obtain the sparse point cloud corresponding to the preset engineering operation scenario.

[0081] Please refer to Figure 2 This is a schematic diagram of a sparse point cloud provided in an embodiment of this application, as shown below. Figure 2 As shown, the mapping matrix of the target camera can be used to map multiple sets of feature point pairs to obtain a sparse point cloud of the preset engineering operation scene.

[0082] The stereo matching network training method provided in the above embodiments constructs a high-precision sparse map by using motion recovery structure technology, so as to generate a high-precision dense depth map. This makes the trained stereo matching network perform better on the disparity estimation results of the inferred binocular images, thereby realizing the reconstruction of the three-dimensional scene based on the high-quality point cloud calculated by the disparity estimation results.

[0083] In an optional embodiment, the process of performing dense mapping on the sparse point cloud according to the camera parameters of the stereo camera in S13 above, to obtain multiple sets of dense depth maps corresponding to multiple sets of sample stereo images, may include:

[0084] Based on the camera parameters of the binocular camera, multi-view stereo geometry technology is used to construct dense maps of sparse point clouds, resulting in multiple sets of dense depth maps.

[0085] In this embodiment, the Multi-View Stereo (MVS) technique uses camera parameters and sparse point clouds to reconstruct dense point clouds. During the process of reconstructing dense point clouds using the MVS technique, multiple sets of dense depth maps corresponding to multiple sets of sample binocular images can be generated.

[0086] Specifically, based on the intrinsic and extrinsic parameters of the binocular camera, the depth values ​​of each pixel projected onto each target view of the sparse point cloud are calculated. The pixels projected onto each target view of the sparse point cloud are used as seed points. Based on the depth values ​​of each seed point, the depth values ​​of other pixels on each target view are calculated. Based on the depth values ​​of all pixels, a dense depth map corresponding to each target view is generated. Each set of sample binocular views corresponds to a set of dense depth maps.

[0087] For example, please refer to Figure 3 This is a schematic diagram of a dense point cloud provided in an embodiment of this application, as shown below. Figure 3 As shown, each pixel in the dense depth map is projected into three-dimensional space to obtain a three-dimensional point cloud. Based on the three-dimensional point clouds corresponding to multiple sets of dense depth maps, and the mutual matching and constraint relationships between image blocks in multiple sets of sample binocular images, a dense point cloud of a preset engineering operation scenario is generated.

[0088] The stereo matching network training method provided in the above embodiments uses multi-view stereo geometry technology to perform dense mapping on sparse point clouds based on the camera parameters of the binocular camera, thereby obtaining a high-precision dense depth map. This allows for the generation of accurate sample disparity maps based on the dense depth map, making the stereo matching network trained using the sample disparity map perform better in predicting the disparity estimation results of binocular images. This enables the reconstruction of a 3D scene using high-quality point clouds calculated based on the disparity estimation results.

[0089] Based on the above embodiments, this application also provides another method for training a stereo matching network. Please refer to... Figure 4 This is a flowchart illustrating another method for training a stereo matching network provided in an embodiment of this application, as shown below. Figure 4 As shown, the process of processing each group of dense depth maps in S14 above to obtain a sample disparity map corresponding to each group of sample binocular images may include:

[0090] S141: Calculate the reference point cloud corresponding to the reference depth map based on the reference image in each set of sample binocular images, the reference depth map in each set of dense depth maps, and the camera parameters of the reference camera.

[0091] Each set of sample binocular images includes a reference image and a target image. The reference depth map is the depth map corresponding to the reference image, and the reference camera is the camera that generates the reference image in the binocular camera.

[0092] In this embodiment, the left view is used as the reference image and the right view is used as the target image. The process of calculating the reference point cloud is described in detail. The dense depth map corresponding to the left view is the reference depth map, the dense depth map corresponding to the right view is the target depth map, the camera corresponding to the left view is the reference camera, and the camera corresponding to the right view is the target camera.

[0093] Specifically, based on the pixel coordinates of each pixel in the left view and the intrinsic parameter matrix of the camera corresponding to the left view, the image coordinates of each pixel in the left view are calculated. Based on the image coordinates of each pixel, the depth value of each pixel in the dense depth map, and the extrinsic parameter matrix of the camera corresponding to the left view, the depth pixels are projected onto the spatial coordinate system of the camera corresponding to the left view to obtain the reference point cloud.

[0094] S142: Calculate the projected pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera.

[0095] In this embodiment, in order to calculate the disparity map of the left and right views more accurately, the reference point cloud is projected onto the position and pose of the target camera, and the disparity map is calculated based on the projected pixel map of the reference point cloud at the position and pose of the target camera and the pixel values ​​of the target image.

[0096] Specifically, based on the 3D coordinates of the reference point cloud and the position and pose of the reference camera relative to the target camera, the pixel values ​​of the reference point cloud are calculated to be projected onto the position and pose of the target camera, thus obtaining the projected pixel map of the reference depth map at the position and pose of the target camera.

[0097] S143: Calculate the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image.

[0098] In this embodiment, after generating the projection pixel map of the reference point cloud at the position and pose of the target camera, the sample disparity map is obtained by calculating the difference between the pixel values ​​of each pixel in the projection pixel map and the target image.

[0099] For example, please refer to Figure 5 This is a disparity map provided in an embodiment of this application, such as... Figure 5 As shown, a sample disparity map is obtained by calculating the difference between the pixel values ​​of each pixel in the projected pixel map and the target image.

[0100] In an optional embodiment, before calculating the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image in S143, the method may further include:

[0101] Based on the pixel values ​​of each pixel in the projected pixel map and the pixel values ​​of each pixel in the target image, noise pixels in the projected pixel map are filtered out.

[0102] Generally, the pixel difference between the left and right views of a binocular camera should theoretically be zero in the vertical direction. In this embodiment, in order to avoid the difference between the pixel values ​​of the projected pixel map and the target image in the vertical direction being too large, it is necessary to compare the pixel values ​​of each pixel in the projected pixel map with the pixel values ​​of each pixel in the target image. Pixels with a pixel difference greater than a preset reference pixel difference in the vertical direction are regarded as noise pixels. Noise pixels are filtered out in the projected pixel map. Then, the difference between the pixel values ​​of each pixel in the projected pixel map with the noise pixels filtered out and the target image is calculated to obtain the sample disparity map.

[0103] In one alternative embodiment, pixels that are outside the field of view in the projected pixel map of the target camera's position and pose are filtered out as noise pixels. Then, the difference between the pixel values ​​of each pixel in the projected pixel map with the noise pixels filtered out and the pixel values ​​of each pixel in the target image is calculated to obtain a sample disparity map.

[0104] The stereo matching network training method provided in the above embodiments has a more accurate result because the depth values ​​of each pixel in the reference depth map are very accurate. This makes the results of the projection pixel map obtained by projecting the reference point cloud generated from the reference depth map and the sample disparity map calculated from the target image more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0105] Based on the above embodiments, this application also provides another method for training a stereo matching network. Please refer to... Figure 6 This is a flowchart illustrating another method for training a stereo matching network provided in an embodiment of this application, as shown below. Figure 6 As shown, in step S142 above, calculating the projected pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera may include:

[0106] S1421: Calculate the projected depth map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera.

[0107] In this embodiment, based on the three-dimensional coordinates of the reference point cloud and the position and pose of the reference camera relative to the target camera, a depth map is calculated by projecting the reference point cloud onto the target camera's position and pose, thus obtaining the projected depth map of the reference point cloud at the target camera's position and pose. For example, if the three-dimensional coordinates of a point in the reference point cloud are (x, y, z), where z represents the depth value of that point relative to the target camera's imaging plane, then after projecting the reference point cloud onto the target camera's imaging plane, the image coordinates of the corresponding pixel in the projected depth map are (x, y), and the pixel value is the depth value z.

[0108] S1422: Calculate the projected pixel map based on the projected depth map and the camera parameters of the reference camera.

[0109] In this embodiment, the pixel coordinates and pixel values ​​of each pixel are calculated based on the image coordinates of each pixel in the projection depth map and the intrinsic parameter matrix of the reference camera to obtain the projection pixel map.

[0110] In an optional embodiment, before calculating the projected pixel map based on the projected depth map and the camera parameters of the reference camera in step S1422, the method further includes:

[0111] Based on the depth values ​​of each pixel in the projected depth map and the depth values ​​of each pixel in the target depth map, noise pixels in the projected depth map are filtered out.

[0112] In this embodiment, to avoid a large difference between the depth values ​​of each pixel in the projection depth map of the reference point cloud projected onto the target camera's position and the depth values ​​of each pixel in the target depth map, and to ensure the accuracy of the sample disparity map calculated based on the projection pixel map and the target image, it is necessary to perform depth verification on the projection depth map. Pixels with depth value differences greater than a preset reference depth difference are identified as noise pixels and are filtered out in the projection depth map. Then, the projection pixel map corresponding to the projection depth map with the noise pixels filtered out is calculated.

[0113] The stereo matching network training method provided in the above embodiments has a more accurate result because the depth values ​​of each pixel in the reference depth map are very accurate. This makes the results of the projection pixel map obtained by projecting the reference point cloud generated from the reference depth map and the sample disparity map calculated from the target image more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0114] The stereo matching network trained using the method described in the above embodiments can be used for 3D scene reconstruction. Before providing a detailed introduction to the 3D scene reconstruction method provided in this application, a 3D scene reconstruction system applying this method will be described first.

[0115] Please refer to Figure 7 This is a structural schematic diagram of a three-dimensional scene reconstruction system provided in an embodiment of this application, as shown below. Figure 7 As shown, the three-dimensional scene reconstruction system includes: a binocular camera 11, a host terminal 12, and a client terminal 13. The binocular camera 11 and the host terminal 12 are both mounted on engineering machinery. The host terminal 12 is communicatively connected to the binocular camera 11 and the client terminal 13, respectively. The host terminal is used to execute the steps of the three-dimensional scene reconstruction method provided in the embodiments of this application.

[0116] Specifically, the binocular camera 11 is installed on the exterior of the construction machinery to acquire binocular images of the construction operation scene and send the acquired binocular images to the host terminal 12. After receiving the binocular images, the host terminal 12 generates a disparity map corresponding to the binocular images based on a pre-trained stereo matching network, and performs point cloud computing on the disparity map to generate three-dimensional point cloud data. The host terminal 12 is connected to the client terminal 13 to receive the viewing angle command sent by the client terminal 13, and renders a three-dimensional scene image of the specified viewing angle based on the viewing angle command and the three-dimensional point cloud data, and sends the three-dimensional scene image to the client terminal 13 for display.

[0117] like Figure 7 As shown, the binocular camera 11 includes: an image acquisition module 111, used to acquire binocular images of the engineering operation scene; the host terminal 12 includes: a stereo matching network 121, a point cloud computing module 122, and a scene rendering module 123. The stereo matching network 121 is trained using the training method of the stereo matching network in the above embodiment, and is used to perform disparity estimation on the binocular images sent by the image acquisition module 111 to obtain a disparity map corresponding to the binocular images; the point cloud computing module 122 is used to perform point cloud computing based on the disparity map calculated by the stereo matching network 121 to generate three-dimensional point cloud data; the scene rendering module 123 is used to receive... The system receives viewpoint commands sent by the client, renders a 3D scene image from the specified viewpoint based on the viewpoint commands and the 3D point cloud data sent by the point cloud computing module 122, and sends the 3D scene image to the client 13 for display. The client 13 includes an instruction generation and sending module 131 and a display module 132. The instruction generation and sending module 131 is used to generate viewpoint commands through the graphical display interface of the client 13 and send them to the scene rendering module 124 of the host 12. The display module 132 serves as the graphical display interface of the client 13 and is used to receive and display the 3D scene image sent by the scene rendering module 124.

[0118] The three-dimensional scene reconstruction system provided in the above embodiments acquires binocular images through a binocular camera, processes the binocular images on the host to generate point cloud data, renders a three-dimensional scene image from a specified viewpoint according to the viewpoint command sent by the client, and displays the three-dimensional scene image from the specified viewpoint on the client. This facilitates operators to observe the engineering operation scene based on the three-dimensional scene image, ensuring operational safety. More importantly, this three-dimensional scene reconstruction system can be used to remotely control engineering machinery.

[0119] Based on the 3D scene reconstruction system provided in the above embodiments, embodiments of this application provide a 3D scene reconstruction method applied to the host side of the 3D scene reconstruction system. Please refer to... Figure 8 This is a flowchart illustrating a three-dimensional scene reconstruction method provided in an embodiment of this application. Figure 8 As shown, the method includes:

[0120] S21: Acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0121] In this embodiment, the binocular camera is calibrated and deployed at a preset position on the construction machinery to ensure sufficient field of view coverage. The binocular camera is then driven to acquire images of the construction work scene where the machinery is located. The obtained binocular images are divided into left and right views and sent to the host computer. For example, the preset construction machinery can be an excavator, and the binocular camera can be deployed on the top of the excavator cab near the excavator arm.

[0122] S22: A pre-trained stereo matching network is used to process the binocular images to obtain the disparity map corresponding to the binocular images.

[0123] In this embodiment, the stereo matching network is trained using the method described in the previous embodiment. It generates sparse and dense point clouds of sample binocular images to calculate the disparity map corresponding to the sample binocular images. The model is then trained based on the sample binocular images and the disparity map. The trained stereo matching network is deployed on the host computer to perform stereo matching on the binocular images captured by the binocular camera, obtaining the disparity map corresponding to the binocular images. The disparity map uses one image from the binocular images as the target image and calculates the disparity of the other image relative to a reference image.

[0124] S23: Based on the configuration parameters of the binocular camera, process the disparity map to obtain the 3D point cloud data of the preset engineering operation scene.

[0125] In this embodiment, a two-dimensional Cartesian coordinate system is established for the target image, and a three-dimensional Cartesian coordinate system is established for the target camera. The configuration parameters of the binocular camera include: the baseline distance of the binocular camera, the intrinsic parameters and extrinsic parameters of the target camera. Based on the baseline distance of the binocular camera, the intrinsic parameters and extrinsic parameters of the target camera, and the two-dimensional coordinates of each pixel in the disparity map, the three-dimensional coordinates of each pixel are calculated to obtain the three-dimensional point cloud data of the preset engineering operation scene.

[0126] S24: Based on the 3D point cloud data and the preset viewpoint direction, perform scene rendering to obtain a 3D scene image of the preset engineering operation scene in the preset viewpoint direction.

[0127] In this embodiment, the three-dimensional point cloud data is rendered in the preset viewing direction according to the preset viewing direction to obtain a three-dimensional scene image of the preset engineering operation scene in the preset viewing direction.

[0128] The 3D scene reconstruction method provided in the above embodiments uses a pre-trained stereo matching network to generate a disparity map of binocular images. Based on this disparity map, high-precision 3D point cloud data can be obtained, so as to achieve high-quality 3D scene reconstruction. Furthermore, the pre-trained stereo matching network can be used to perform 3D scene reconstruction in real time, improving the real-time performance of 3D scene reconstruction. In addition, since only the binocular images acquired by the binocular camera need to be sampled for high-quality 3D scene reconstruction, the cost of 3D scene reconstruction is reduced compared to using laser point clouds for 3D scene reconstruction.

[0129] Based on the above embodiments, this application also provides another method for three-dimensional scene reconstruction. Please refer to... Figure 9 This is a flowchart illustrating another three-dimensional scene reconstruction method provided in an embodiment of this application, as shown below. Figure 9 As shown, in step S23 above, the disparity map is processed according to the configuration parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene, which may include:

[0130] S231: Based on the focal length and baseline distance of the binocular camera, process the disparity map to obtain the depth image corresponding to the binocular image.

[0131] In this embodiment, the focal length of the target camera is used as a parameter in the target camera's intrinsic parameter matrix. This intrinsic parameter matrix is ​​obtained through target camera calibration and remains unchanged during camera operation. The baseline distance of the binocular cameras is the distance between the optical centers of the binocular cameras, which can be determined after the binocular cameras are deployed on the engineering machinery. Based on the target camera's focal length and the binocular cameras' baseline distance, the depth value of each pixel in the disparity map is calculated, generating a depth image corresponding to the target image in the binocular image.

[0132] S232: Based on the intrinsic parameters of the binocular camera, process the depth image to obtain the 3D point cloud data of the preset engineering operation scene.

[0133] In this embodiment, based on the pre-established two-dimensional rectangular coordinate system and three-dimensional rectangular coordinate system, and based on the pixel coordinates of each pixel in the parallax map and the intrinsic parameters of the target camera, the two-dimensional coordinates of each pixel in the two-dimensional rectangular coordinate system are determined. Based on the two-dimensional coordinates and depth value of each pixel, the three-dimensional coordinates of each pixel are obtained. The three-dimensional coordinates of multiple pixels in the target image constitute the three-dimensional point cloud data of the preset engineering operation scene.

[0134] The three-dimensional scene reconstruction method provided in the above embodiments processes the disparity map according to the focal length and baseline distance of the binocular camera to obtain the depth image corresponding to the binocular image. Based on the intrinsic and extrinsic parameters of the binocular camera, the depth image is processed to obtain the three-dimensional point cloud data of the preset engineering operation scene, making the three-dimensional scene reconstruction result more accurate.

[0135] Based on the above embodiments, this application also provides another method for three-dimensional scene reconstruction. Please refer to... Figure 10 This is a flowchart illustrating another three-dimensional scene reconstruction method provided in an embodiment of this application, as shown below. Figure 10 As shown, the steps of this 3D scene reconstruction method include: S21-S23, S241, S242, and S243, wherein the implementation of S21-S23 is the same as... Figure 4 The same applies, so I won't go into details here.

[0136] S21: Acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0137] S22: A pre-trained stereo matching network is used to process the binocular images to obtain the disparity map corresponding to the binocular images.

[0138] S23: Based on the configuration parameters of the binocular camera, process the disparity map to obtain the 3D point cloud data of the preset engineering operation scene.

[0139] S241: Receive the preset view direction command sent by the client.

[0140] S242: Based on the 3D point cloud data and the preset viewpoint direction command, perform scene rendering to obtain a 3D scene image of the preset engineering operation scene in the preset viewpoint direction.

[0141] S243: Send a 3D scene image with a preset viewpoint to the client.

[0142] In this embodiment, as Figure 7 As shown, the host and client communicate with each other. The client determines the viewpoint of the 3D scene to be displayed. The client sends the determined viewpoint to the host as a viewpoint direction command. After generating 3D point cloud data, the host renders the 3D scene image of the specified viewpoint according to the viewpoint direction command sent by the client, and sends the 3D scene image of that viewpoint direction to the client for display.

[0143] The three-dimensional scene reconstruction method provided in the above embodiments receives a preset viewpoint direction command sent by the client, performs scene rendering based on the three-dimensional point cloud data and the preset viewpoint direction command, obtains a three-dimensional scene image of the preset engineering operation scene in the preset viewpoint direction, and sends the three-dimensional scene image in the preset viewpoint direction to the client, so that the operator can observe the engineering operation scene from any viewpoint based on the three-dimensional scene image and ensure the safety of the operation; more preferably, the three-dimensional scene reconstruction system can be used to realize remote control of engineering machinery.

[0144] Based on the above embodiments, this application also provides an engineering machinery that applies the three-dimensional scene reconstruction method provided in the above embodiments. Please refer to... Figure 11 This is a structural schematic diagram of an engineering machine provided in an embodiment of this application, such as... Figure 11 As shown, the engineering machinery is equipped with a binocular camera 11 and a host terminal 12 that is communicatively connected to the binocular camera 11. The host terminal is used to execute the steps of the three-dimensional scene reconstruction method provided in the above embodiment.

[0145] Specifically, the binocular camera 11 is installed on the outside of the construction machinery to acquire binocular images of the construction operation scene. The host terminal 12 is a small or micro workstation deployed on the construction machinery. The host terminal 12 can generate disparity maps, calculate 3D point cloud data and render 3D scenes based on the binocular images sent by the binocular camera 11.

[0146] In an optional embodiment, if the construction machinery is operated by an operator inside the construction machinery, the client 13 can also be set on the construction machinery, so that the operator can view the construction operation scene from various perspectives through the client 13 while controlling the construction machinery to perform the operation.

[0147] In another alternative embodiment, if the construction machinery is remotely operated, then client 13 is deployed on a remote control platform.

[0148] Based on the above embodiments, this application also provides a training device for a stereo matching network. Please refer to... Figure 12 This is a schematic diagram of the structure of the training device for the stereo matching network provided in the embodiments of this application, as shown below. Figure 12 As shown, the device includes:

[0149] The sample image acquisition module 101 is used to acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0150] The sparse point cloud generation module 102 is used to process multiple sets of sample binocular images to obtain a sparse point cloud of a preset engineering operation scene.

[0151] The dense mapping module 103 is used to perform dense mapping on sparse point clouds based on the camera parameters of the stereo camera, and obtain multiple sets of dense depth maps corresponding to multiple sets of sample stereo images.

[0152] The sample disparity map generation module 104 is used to process each group of dense depth maps to obtain a sample disparity map corresponding to each group of sample binocular images.

[0153] The matching network training module 105 is used to train the model based on multiple sets of sample binocular images and corresponding sample disparity maps to obtain a stereo matching network.

[0154] The stereo matching network training device provided in the above embodiments, on the one hand, utilizes the trained stereo matching network to infer the disparity estimation results of binocular images, which can improve the real-time performance of calculating the disparity estimation results, so as to complete the 3D scene reconstruction more quickly based on the disparity estimation results; on the other hand, this application utilizes the construction of sparse point clouds from sample binocular images, and then constructs dense depth maps from the sparse point clouds, generating sample disparity maps through high-quality dense depth maps, making the disparity estimation results of the sample disparity maps more accurate. Using high-precision sample disparity maps as labeled data and sample binocular images together to form training data for training the stereo matching network, the stereo matching network performs better in inferring the disparity estimation results of binocular images, thereby realizing the completion of 3D scene reconstruction based on high-quality point clouds calculated from disparity estimation results; and the use of a large number of sample binocular images sampled by a binocular camera for model training also reduces the model training cost.

[0155] Optionally, the sparse point cloud generation module 102 is specifically used to process multiple sets of sample binocular images using motion reconstruction technology to obtain a sparse map of a preset engineering operation scene. The sparse map includes: sparse point cloud and camera parameters.

[0156] The stereo matching network training device provided in the above embodiments constructs a high-precision sparse map by using motion recovery structure technology, so as to generate a high-precision dense depth map. This makes the trained stereo matching network perform better on the disparity estimation results of the inferred binocular images, thereby realizing the reconstruction of the three-dimensional scene based on the high-quality point cloud calculated by the disparity estimation results.

[0157] Optionally, the dense mapping module 103 is specifically used to perform dense mapping on sparse point clouds based on the camera parameters of the binocular camera and using multi-view stereo geometry technology to obtain multiple sets of dense depth maps.

[0158] The stereo matching network training device provided in the above embodiments uses multi-view stereo geometry technology to perform dense mapping on sparse point clouds based on the camera parameters of the binocular camera, thereby obtaining a high-precision dense depth map. This allows for the generation of accurate sample disparity maps based on the dense depth map, making the stereo matching network trained using the sample disparity map perform better in predicting the disparity estimation results of binocular images. This enables the reconstruction of a 3D scene based on high-quality point clouds calculated from the disparity estimation results.

[0159] Optionally, the sample disparity map generation module 104 includes:

[0160] The reference point cloud generation unit is used to calculate the reference point cloud corresponding to the reference depth map based on the reference image in each set of sample stereo images, the reference depth map in each set of dense depth maps, and the camera parameters of the reference camera. Each set of sample stereo images includes a reference image and a target image. The reference depth map is the depth map corresponding to the reference image, and the reference camera is the camera corresponding to the generated reference image in the stereo camera.

[0161] The projection pixel map generation unit is used to calculate the projection pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera.

[0162] The sample disparity map generation unit is used to calculate the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image.

[0163] Optionally, before the sample disparity map generation unit calculates the sample disparity map, the noise filtering subunit is also used to filter out noise pixels in the projected pixel map based on the pixel values ​​of each pixel in the projected pixel map and the pixel values ​​of each pixel in the target image.

[0164] The stereo matching network training device provided in the above embodiments has a very accurate depth value for each pixel in the reference depth map, which makes the results of the projection pixel map obtained by the reference point cloud projection generated from the reference depth map and the sample disparity map calculated from the target image more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0165] Optionally, the projected pixel map generation unit includes:

[0166] The projection depth map generation subunit is used to calculate the projection depth map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera.

[0167] The projection pixel map generation subunit is used to calculate the projection pixel map based on the projection depth map and the camera parameters of the reference camera.

[0168] Optionally, before the projected pixel map generation subunit calculates the projected pixel map, the apparatus further includes:

[0169] The noise filtering subunit is used to filter out noisy pixels in the projection depth map based on the depth values ​​of each pixel in the projection depth map and the depth values ​​of each pixel in the target depth map.

[0170] The stereo matching network training device provided in the above embodiments has a very accurate depth value for each pixel in the reference depth map, which makes the results of the projection pixel map obtained by the reference point cloud projection generated from the reference depth map and the sample disparity map calculated from the target image more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0171] Based on the above embodiments, this application also provides a three-dimensional scene reconstruction device. Please refer to... Figure 13 This is a schematic diagram of the structure of a three-dimensional scene reconstruction device provided in an embodiment of this application, as shown below. Figure 13 As shown, the device includes:

[0172] The binocular image acquisition module 201 is used to acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery.

[0173] The disparity map generation module 202 is used to process the binocular image using a pre-trained stereo matching network to obtain the disparity map corresponding to the binocular image. The stereo matching network is trained using any of the stereo matching networks trained by the first method described above.

[0174] The 3D point cloud generation module 203 is used to process the disparity map according to the configuration parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene.

[0175] The rendering module 204 is used to render the scene based on the 3D point cloud data and the preset view direction to obtain a 3D scene image of the preset engineering operation scene in the preset view direction.

[0176] The 3D scene reconstruction device provided in the above embodiments uses a pre-trained stereo matching network to generate a disparity map of binocular images. Based on this disparity map, high-precision 3D point cloud data can be obtained, so as to achieve high-quality 3D scene reconstruction. Moreover, the pre-trained stereo matching network can be used to perform 3D scene reconstruction in real time, improving the real-time performance of 3D scene reconstruction. In addition, since only the binocular images acquired by the binocular camera need to be sampled for high-quality 3D scene reconstruction, the cost of 3D scene reconstruction is reduced compared to using laser point clouds for 3D scene reconstruction.

[0177] Optionally, the 3D point cloud generation module 203 includes:

[0178] The depth image generation unit is used to process the disparity map based on the focal length and baseline distance of the binocular camera to obtain the depth image corresponding to the binocular image.

[0179] The 3D point cloud generation unit is used to process the depth image based on the intrinsic and extrinsic parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene.

[0180] The three-dimensional scene reconstruction device provided in the above embodiments processes the disparity map according to the focal length and baseline distance of the binocular camera to obtain the depth image corresponding to the binocular image. It processes the depth image according to the intrinsic and extrinsic parameters of the binocular camera to obtain the three-dimensional point cloud data of the preset engineering operation scene, making the result of three-dimensional scene reconstruction more accurate.

[0181] Optionally, the device may also include: an instruction receiving module and an image sending module;

[0182] The instruction receiving module is used to receive preset view direction instructions sent by the client;

[0183] The rendering module 204 is specifically used to render the scene based on the 3D point cloud data and the preset view direction instruction, so as to obtain the 3D scene image of the preset engineering operation scene in the preset view direction.

[0184] The image sending module is used to send 3D scene images with a preset viewpoint to the client.

[0185] The three-dimensional scene reconstruction device provided in the above embodiments receives a preset viewpoint direction command sent by the client, performs scene rendering based on the three-dimensional point cloud data and the preset viewpoint direction command, obtains a three-dimensional scene image of the preset engineering operation scene in the preset viewpoint direction, and sends the three-dimensional scene image in the preset viewpoint direction to the client, so that the operator can observe the engineering operation scene from any viewpoint based on the three-dimensional scene image and ensure the safety of the operation; more preferably, the three-dimensional scene reconstruction system can be used to realize the remote control of engineering machinery.

[0186] The above-described device is used to execute the method provided in the foregoing embodiments, and its implementation principle and technical effect are similar, so they will not be described again here.

[0187] These modules can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more microprocessors, or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when a module is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these modules can be integrated together as a system-on-a-chip (SOC).

[0188] Please refer to Figure 14 This is a schematic diagram of the electronic device provided in the embodiments of this application, such as... Figure 14 As shown, the electronic device 300 includes a processor 301, a storage medium 302, and a bus. The storage medium 302 stores program instructions that can be executed by the processor 301. When the electronic device 300 is running, the processor 301 and the storage medium 302 communicate through the bus. The processor 301 executes the program instructions to perform the steps of the above-mentioned stereo matching network training method or the steps of the three-dimensional scene reconstruction method.

[0189] Specifically, the steps of the processor executing the above-described training method for the stereo matching network include:

[0190] Acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery; process the multiple sets of sample binocular images to obtain a sparse point cloud of the preset engineering operation scene; perform dense mapping on the sparse point cloud according to the camera parameters of the binocular camera to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample binocular images; process each set of dense depth maps to obtain a sample disparity map corresponding to each set of sample binocular images; train the model based on the multiple sets of sample binocular images and the corresponding sample disparity maps to obtain a stereo matching network.

[0191] The stereo matching network training method executed by the processor in the above embodiments has two main advantages. First, by using the trained stereo matching network to infer the disparity estimation results of binocular images, the real-time performance of the disparity estimation results can be improved, enabling faster 3D scene reconstruction based on the disparity estimation results. Second, by constructing sparse point clouds from sample binocular images and then constructing dense depth maps from the sparse point clouds, a sample disparity map is generated using the high-quality dense depth map. This results in higher accuracy of the disparity estimation results from the sample disparity map. The high-precision sample disparity map is used as labeled data and, together with the sample binocular images, constitutes the training data for training the stereo matching network. This improves the stereo matching network's performance in inferring the disparity estimation results of binocular images, thereby enabling 3D scene reconstruction based on high-quality point clouds calculated from the disparity estimation results. Furthermore, using a binocular camera to sample a large number of sample binocular images for model training also reduces the model training cost.

[0192] In one possible implementation, the processor performs the step of processing multiple sets of sample binocular images to obtain a sparse point cloud of a preset engineering task scene, which may include:

[0193] Using structure-reconstruction-motion technique, multiple sets of sample binocular images are processed to obtain a sparse map of a preset engineering operation scene. The sparse map includes sparse point clouds and camera parameters.

[0194] The stereo matching network training method executed by the processor in the above embodiments uses motion recovery structure technology to construct a high-precision sparse map in order to generate a high-precision dense depth map. This makes the trained stereo matching network perform better on the disparity estimation results of the inferred binocular images, thereby realizing the reconstruction of the 3D scene based on the high-quality point cloud calculated from the disparity estimation results.

[0195] In one possible implementation, the processor performing the steps described above—namely, performing dense mapping on the sparse point cloud based on the camera parameters of the stereo camera to obtain multiple sets of dense depth maps corresponding to multiple sets of sample stereo images—may include:

[0196] Based on the camera parameters of the binocular camera, multi-view stereo geometry technology is used to construct dense maps of sparse point clouds, resulting in multiple sets of dense depth maps.

[0197] The stereo matching network training method executed by the processor in the above embodiments uses multi-view stereo geometry technology to perform dense mapping on sparse point clouds based on the camera parameters of the stereo camera, thereby obtaining a high-precision dense depth map. This allows for the generation of accurate sample disparity maps based on the dense depth map, making the stereo matching network trained using the sample disparity map perform better in inferring the disparity estimation results of the stereo image. This enables the high-quality point cloud calculated based on the disparity estimation results to complete the 3D scene reconstruction.

[0198] In one possible implementation, the processor performing the steps described above—processing each set of dense depth maps to obtain a sample disparity map corresponding to each set of sample binocular images—may include:

[0199] Based on the reference image in each set of sample binocular images, the reference depth map in each set of dense depth maps, and the camera parameters of the reference camera, calculate the reference point cloud corresponding to the reference depth map; based on the reference point cloud and the camera parameters of the reference camera, calculate the projected pixel map of the reference point cloud; based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image, calculate the sample disparity map.

[0200] In an optional embodiment, before performing the step of calculating the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image, the processor may further include the following steps:

[0201] Based on the pixel values ​​of each pixel in the projected pixel map and the pixel values ​​of each pixel in the target image, noise pixels in the projected pixel map are filtered out.

[0202] In the training method of the stereo matching network executed by the processor in the above embodiments, since the depth values ​​of each pixel in the reference depth map are very accurate, the results of the projection pixel map obtained by the reference point cloud generated from the reference depth map and the sample disparity map calculated from the target image are more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0203] In one possible implementation, the processor performing the steps described above—calculating the projected pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera—may include:

[0204] Calculate the projected depth map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera; calculate the projected pixel map based on the projected depth map and the camera parameters of the reference camera.

[0205] In an alternative embodiment, before performing the step of calculating the projected pixel map based on the projected depth map and the camera parameters of the reference camera, the processor may further include the following steps:

[0206] Based on the depth values ​​of each pixel in the projected depth map and the depth values ​​of each pixel in the target depth map, noise pixels in the projected depth map are filtered out.

[0207] In the training method of the stereo matching network executed by the processor in the above embodiments, since the depth values ​​of each pixel in the reference depth map are very accurate, the results of the projection pixel map obtained by the reference point cloud generated from the reference depth map and the sample disparity map calculated from the target image are more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0208] Specifically, the steps by which the processor executes the above-mentioned 3D scene reconstruction method include:

[0209] Acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery; process the binocular images using a pre-trained stereo matching network to obtain a disparity map corresponding to the binocular images; process the disparity map according to the configuration parameters of the binocular camera to obtain 3D point cloud data of the preset engineering operation scene; perform scene rendering based on the 3D point cloud data and the preset viewing direction to obtain a 3D scene image of the preset engineering operation scene in the preset viewing direction.

[0210] The 3D scene reconstruction method executed by the processor in the above embodiments uses a pre-trained stereo matching network to generate a disparity map of binocular images. Based on this disparity map, high-precision 3D point cloud data can be obtained, so as to achieve high-quality 3D scene reconstruction based on the high-precision 3D point cloud data. Moreover, the pre-trained stereo matching network can be used to perform 3D scene reconstruction in real time, improving the real-time performance of 3D scene reconstruction. In addition, since only the binocular images acquired by the binocular camera need to be sampled for high-quality 3D scene reconstruction, the cost of 3D scene reconstruction is reduced compared to using laser point clouds for 3D scene reconstruction.

[0211] In one possible implementation, the processor performing the steps described above—processing the disparity map according to the configuration parameters of the binocular camera to obtain 3D point cloud data of the preset engineering operation scene—may include:

[0212] Based on the focal length and baseline distance of the binocular camera, the disparity map is processed to obtain the depth image corresponding to the binocular image; based on the intrinsic and extrinsic parameters of the binocular camera, the depth image is processed to obtain the 3D point cloud data of the preset engineering operation scene.

[0213] The 3D scene reconstruction method executed by the processor in the above embodiments processes the disparity map according to the focal length and baseline distance of the binocular camera to obtain the depth image corresponding to the binocular image. The depth image is then processed according to the intrinsic and extrinsic parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene, making the 3D scene reconstruction result more accurate.

[0214] In one possible implementation, the steps of the processor executing the above-described 3D reconstruction method further include:

[0215] The system acquires binocular images of a pre-defined engineering operation scene captured by a binocular camera on a pre-defined engineering machinery; it processes the binocular images using a pre-trained stereo matching network to obtain a disparity map corresponding to the binocular images; it processes the disparity map according to the configuration parameters of the binocular camera to obtain 3D point cloud data of the pre-defined engineering operation scene; it receives a pre-defined viewing direction command sent by the client; it performs scene rendering based on the 3D point cloud data and the pre-defined viewing direction command to obtain a 3D scene image of the pre-defined engineering operation scene in the pre-defined viewing direction; and it sends the 3D scene image in the pre-defined viewing direction to the client.

[0216] The three-dimensional scene reconstruction method executed by the processor in the above embodiments receives a preset viewpoint direction command sent by the client, performs scene rendering based on the three-dimensional point cloud data and the preset viewpoint direction command, obtains a three-dimensional scene image of the preset engineering operation scene in the preset viewpoint direction, and sends the three-dimensional scene image in the preset viewpoint direction to the client, so that the operator can observe the engineering operation scene from any viewpoint based on the three-dimensional scene image and ensure the safety of the operation; more preferably, the three-dimensional scene reconstruction system can be used to realize remote control of engineering machinery.

[0217] Optionally, embodiments of this application also provide a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described stereo matching network training method or the steps of the three-dimensional scene reconstruction method.

[0218] Specifically, the steps of the computer program executing the above-mentioned training method for the stereo matching network include:

[0219] Acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery; process the multiple sets of sample binocular images to obtain a sparse point cloud of the preset engineering operation scene; perform dense mapping on the sparse point cloud according to the camera parameters of the binocular camera to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample binocular images; process each set of dense depth maps to obtain a sample disparity map corresponding to each set of sample binocular images; train the model based on the multiple sets of sample binocular images and the corresponding sample disparity maps to obtain a stereo matching network.

[0220] The training method of the stereo matching network executed by the computer program in the above embodiments has two advantages. First, this application uses the trained stereo matching network to infer the disparity estimation results of the binocular images, which can improve the real-time performance of calculating the disparity estimation results, so as to complete the 3D scene reconstruction more quickly based on the disparity estimation results. Second, this application uses sparse point clouds to construct sample binocular images, and then uses sparse point clouds to construct dense depth maps. Sample disparity maps are generated through high-quality dense depth maps, which makes the disparity estimation results of the sample disparity maps more accurate. The high-precision sample disparity maps are used as labeled data and together with sample binocular images to form training data for training the stereo matching network, which makes the stereo matching network perform better in inferring the disparity estimation results of the binocular images, thereby realizing the 3D scene reconstruction based on high-quality point clouds calculated from the disparity estimation results. Furthermore, the use of a large number of sample binocular images sampled by a binocular camera for model training also reduces the model training cost.

[0221] In one possible implementation, the computer program performs the step of processing multiple sets of sample binocular images to obtain a sparse point cloud of a preset engineering operation scene, which may include:

[0222] Using structure-reconstruction-motion technique, multiple sets of sample binocular images are processed to obtain a sparse map of a preset engineering operation scene. The sparse map includes sparse point clouds and camera parameters.

[0223] The training method of the stereo matching network executed by the computer program in the above embodiments constructs a high-precision sparse map by using motion recovery structure technology in order to generate a high-precision dense depth map. This makes the trained stereo matching network perform better on the disparity estimation results of the inferred binocular images, thereby realizing the reconstruction of the three-dimensional scene based on the high-quality point cloud calculated by the disparity estimation results.

[0224] In one possible implementation, the computer program performing the steps described above—namely, constructing a dense map of the sparse point cloud based on the camera parameters of the stereo camera to obtain multiple sets of dense depth maps corresponding to multiple sets of sample stereo images—may include:

[0225] Based on the camera parameters of the binocular camera, multi-view stereo geometry technology is used to construct dense maps of sparse point clouds, resulting in multiple sets of dense depth maps.

[0226] The training method of the stereo matching network executed by the computer program in the above embodiments uses multi-view stereo geometry technology to perform dense mapping on sparse point clouds based on the camera parameters of the binocular camera, so as to obtain a high-precision dense depth map. This allows for the generation of accurate sample disparity maps based on the dense depth maps, making the stereo matching network trained with sample disparity maps perform better in predicting the disparity estimation results of binocular images. This enables the reconstruction of a 3D scene based on high-quality point clouds calculated from the disparity estimation results.

[0227] In one possible implementation, the computer program performing the above steps of processing each set of dense depth maps to obtain a sample disparity map corresponding to each set of sample binocular images may include:

[0228] Based on the reference image in each set of sample binocular images, the reference depth map in each set of dense depth maps, and the camera parameters of the reference camera, calculate the reference point cloud corresponding to the reference depth map; based on the reference point cloud and the camera parameters of the reference camera, calculate the projected pixel map of the reference point cloud; based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image, calculate the sample disparity map.

[0229] In an optional embodiment, before the computer program performs the step of calculating the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image, the step may further include:

[0230] Based on the pixel values ​​of each pixel in the projected pixel map and the pixel values ​​of each pixel in the target image, noise pixels in the projected pixel map are filtered out.

[0231] In the above embodiments, the training method of the stereo matching network executed by the computer program has a more accurate result because the depth values ​​of each pixel in the reference depth map are very accurate. This makes the results of the projection pixel map obtained by the reference point cloud generated from the reference depth map and the sample disparity map calculated from the target image more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0232] In one possible implementation, the computer program performing the steps described above—calculating the projected pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera—may include:

[0233] Calculate the projected depth map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera; calculate the projected pixel map based on the projected depth map and the camera parameters of the reference camera.

[0234] In an alternative embodiment, before performing the step of calculating the projected pixel map based on the projected depth map and the camera parameters of the reference camera, the computer program may further include the following steps:

[0235] Based on the depth values ​​of each pixel in the projected depth map and the depth values ​​of each pixel in the target depth map, noise pixels in the projected depth map are filtered out.

[0236] In the above embodiments, the training method of the stereo matching network executed by the computer program has a more accurate result because the depth values ​​of each pixel in the reference depth map are very accurate. This makes the results of the projection pixel map obtained by the reference point cloud generated from the reference depth map and the sample disparity map calculated from the target image more accurate. This makes the stereo matching network trained with the sample disparity map perform better on the disparity estimation results of the inferred binocular image, thereby realizing the reconstruction of the three-dimensional scene by using a high-quality point cloud calculated based on the disparity estimation results.

[0237] Specifically, the steps for a computer program to execute the above-mentioned 3D scene reconstruction method include:

[0238] Acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery; process the binocular images using a pre-trained stereo matching network to obtain a disparity map corresponding to the binocular images; process the disparity map according to the configuration parameters of the binocular camera to obtain 3D point cloud data of the preset engineering operation scene; perform scene rendering based on the 3D point cloud data and the preset viewing direction to obtain a 3D scene image of the preset engineering operation scene in the preset viewing direction.

[0239] The 3D scene reconstruction method executed by the computer program in the above embodiments uses a pre-trained stereo matching network to generate a disparity map of binocular images. Based on this disparity map, high-precision 3D point cloud data can be obtained, so as to achieve high-quality 3D scene reconstruction. Moreover, the pre-trained stereo matching network can be used to perform 3D scene reconstruction in real time, improving the real-time performance of 3D scene reconstruction. In addition, since only the binocular images acquired by the binocular camera need to be sampled for high-quality 3D scene reconstruction, the cost of 3D scene reconstruction is reduced compared to using laser point clouds for 3D scene reconstruction.

[0240] In one possible implementation, the computer program performing the steps described above—processing the disparity map according to the configuration parameters of the binocular camera to obtain 3D point cloud data of the preset engineering operation scene—may include:

[0241] Based on the focal length and baseline distance of the binocular camera, the disparity map is processed to obtain the depth image corresponding to the binocular image; based on the intrinsic and extrinsic parameters of the binocular camera, the depth image is processed to obtain the 3D point cloud data of the preset engineering operation scene.

[0242] The three-dimensional scene reconstruction method executed by the computer program in the above embodiments processes the disparity map according to the focal length and baseline distance of the binocular camera to obtain the depth image corresponding to the binocular image. Based on the intrinsic and extrinsic parameters of the binocular camera, the depth image is processed to obtain the three-dimensional point cloud data of the preset engineering operation scene, making the result of three-dimensional scene reconstruction more accurate.

[0243] In one possible implementation, the steps of the computer program executing the above-described 3D reconstruction method further include:

[0244] The system acquires binocular images of a pre-defined engineering operation scene captured by a binocular camera on a pre-defined engineering machinery; it processes the binocular images using a pre-trained stereo matching network to obtain a disparity map corresponding to the binocular images; it processes the disparity map according to the configuration parameters of the binocular camera to obtain 3D point cloud data of the pre-defined engineering operation scene; it receives a pre-defined viewing direction command sent by the client; it performs scene rendering based on the 3D point cloud data and the pre-defined viewing direction command to obtain a 3D scene image of the pre-defined engineering operation scene in the pre-defined viewing direction; and it sends the 3D scene image in the pre-defined viewing direction to the client.

[0245] The three-dimensional scene reconstruction method executed by the computer program in the above embodiments receives a preset viewpoint direction command sent by the client, performs scene rendering based on the three-dimensional point cloud data and the preset viewpoint direction command, obtains a three-dimensional scene image of the preset engineering operation scene in the preset viewpoint direction, and sends the three-dimensional scene image in the preset viewpoint direction to the client, so that the operator can observe the engineering operation scene from any viewpoint based on the three-dimensional scene image and ensure the safety of the operation; more preferably, the three-dimensional scene reconstruction system can be used to realize remote control of engineering machinery.

[0246] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0247] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0248] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0249] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0250] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A training method for a stereo matching network, characterized in that, The method includes: Acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery; The multiple sets of sample binocular images are processed to obtain the sparse point cloud of the preset engineering operation scene; Based on the camera parameters of the binocular camera, a dense mapping is performed on the sparse point cloud to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample binocular images. Each set of dense depth maps is processed to obtain a sample disparity map corresponding to each set of binocular images; The stereo matching network is obtained by training the model based on the multiple sets of sample binocular images and the corresponding sample disparity maps. The process of processing each group of dense depth maps to obtain a sample disparity map corresponding to each group of sample binocular images includes: Based on the reference image in each set of sample binocular images, the reference depth map in each set of dense depth maps, and the camera parameters of the reference camera, a reference point cloud corresponding to the reference depth map is calculated. Each set of sample binocular images includes the reference image and the target image. The reference depth map is the depth map corresponding to the reference image, and the reference camera is the camera in the binocular camera that generates the reference image. Calculate the projected pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera; The sample disparity map is calculated based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image.

2. The method according to claim 1, characterized in that, The process of processing the multiple sets of sample binocular images to obtain the sparse point cloud of the preset engineering operation scene includes: Using structure-of-motion reconstruction (SCOR) technology, the multiple sets of sample binocular images are processed to obtain a sparse map of the preset engineering operation scene. The sparse map includes the sparse point cloud and the camera parameters.

3. The method according to claim 1, characterized in that, The step of performing dense mapping on the sparse point cloud based on the camera parameters of the stereo camera to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample stereo images includes: Based on the camera parameters of the binocular camera, multi-view stereo geometry technology is used to perform dense mapping on the sparse point cloud to obtain the multiple sets of dense depth maps.

4. The method as described in claim 1, characterized in that, The step of calculating the projected pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera includes: Calculate the projection depth map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera; The projected pixel map is calculated based on the projected depth map and the camera parameters of the reference camera.

5. The method as described in claim 4, characterized in that, Before calculating the projected pixel map based on the projected depth map and the camera parameters of the reference camera, the method further includes: Based on the depth values ​​of each pixel in the projected depth map and the depth values ​​of each pixel in the target depth map, noise pixels in the projected depth map are filtered out, and the target depth map is the depth map corresponding to the target image.

6. The method as described in claim 1, characterized in that, Before calculating the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image, the method further includes: Based on the pixel values ​​of each pixel in the projected pixel map and the pixel values ​​of each pixel in the target image, noise pixels in the projected pixel map are filtered out.

7. A method for reconstructing a three-dimensional scene, characterized in that, The method includes: Acquire binocular images of a pre-defined engineering operation scene captured by a binocular camera on a pre-defined engineering machinery; The stereo matching network is pre-trained to process the binocular image to obtain the disparity map corresponding to the binocular image. The stereo matching network is obtained by training the stereo matching network as described in any one of claims 1-6. Based on the configuration parameters of the binocular camera, the disparity map is processed to obtain the three-dimensional point cloud data of the preset engineering operation scene; Based on the 3D point cloud data and the preset viewing direction, scene rendering is performed to obtain a 3D scene image of the preset engineering operation scene in the preset viewing direction.

8. The method according to claim 7, characterized in that, The step of processing the disparity map according to the configuration parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene includes: Based on the focal length and baseline distance of the binocular camera, the disparity map is processed to obtain the depth image corresponding to the binocular image; Based on the intrinsic parameters of the binocular camera, the depth image is processed to obtain the three-dimensional point cloud data of the preset engineering operation scene.

9. The method according to claim 7, characterized in that, Before performing scene rendering based on the 3D point cloud data and a preset viewing direction to obtain the 3D scene image of the preset engineering operation scene in the preset viewing direction, the method further includes: Receive preset viewpoint direction commands sent by the client; The step of rendering the scene based on the 3D point cloud data and a preset viewing direction to obtain a 3D scene image of the preset engineering operation scene in the preset viewing direction includes: Based on the three-dimensional point cloud data and the preset viewpoint direction command, scene rendering is performed to obtain a three-dimensional scene image of the preset engineering operation scene in the preset viewpoint direction; After rendering the scene based on the 3D point cloud data and the preset viewing direction to obtain a 3D scene image of the preset engineering operation scene in the preset viewing direction, the method further includes: Send a 3D scene image with the preset viewpoint direction to the client.

10. A training device for a stereo matching network, characterized in that, The device includes: The sample image acquisition module is used to acquire multiple sets of sample binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery. The sparse point cloud generation module is used to process the multiple sets of sample binocular images to obtain the sparse point cloud of the preset engineering operation scene. The dense mapping module is used to perform dense mapping on the sparse point cloud according to the camera parameters of the stereo camera, so as to obtain multiple sets of dense depth maps corresponding to the multiple sets of sample stereo images. The sample disparity map generation module is used to process each group of dense depth maps to obtain a sample disparity map corresponding to each group of sample binocular images. The matching network training module is used to train the model based on the multiple sets of sample binocular images and the corresponding sample disparity maps to obtain the stereo matching network. The sample disparity map generation module includes: The reference point cloud generation unit is used to calculate the reference point cloud corresponding to the reference depth map based on the reference image in each set of sample binocular images, the reference depth map in each set of dense depth maps, and the camera parameters of the reference camera. The set of sample binocular images includes the reference image and the target image. The reference depth map is the depth map corresponding to the reference image, and the reference camera is the camera in the binocular camera that generates the reference image. The projection pixel map generation unit is used to calculate the projection pixel map of the reference point cloud based on the reference point cloud and the camera parameters of the reference camera. The sample disparity map generation unit is used to calculate the sample disparity map based on the pixel values ​​of the projected pixel map and the pixel values ​​of the target image.

11. A three-dimensional scene reconstruction device, characterized in that, The device includes: The binocular image acquisition module is used to acquire binocular images of a preset engineering operation scene captured by a binocular camera on a preset engineering machinery. The disparity map generation module is used to process the binocular image using a pre-trained stereo matching network to obtain a disparity map corresponding to the binocular image. The stereo matching network is obtained by training the stereo matching network as described in any one of claims 1-6. A 3D point cloud generation module is used to process the disparity map according to the configuration parameters of the binocular camera to obtain the 3D point cloud data of the preset engineering operation scene. The rendering module is used to render the scene based on the three-dimensional point cloud data and the preset view direction to obtain a three-dimensional scene image of the preset engineering operation scene in the preset view direction.

12. An engineering machinery, characterized in that, The engineering machinery is equipped with a binocular camera and a host terminal that is communicatively connected to the binocular camera. The host terminal is used to perform the steps of the three-dimensional scene reconstruction method as described in any one of claims 7 to 9.

13. A three-dimensional scene reconstruction system, characterized in that, include: The system includes a binocular camera, a host device, and a client device. The binocular camera and the host device are both mounted on the engineering machinery. The host device is communicatively connected to both the binocular camera and the client device. The host computer is used to perform the steps of the three-dimensional scene reconstruction method as described in any one of claims 7 to 9.

14. An electronic device, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores program instructions executable by the processor. When the electronic device is running, the processor communicates with the storage medium via the bus. The processor executes the program instructions to perform the steps of the training method for a stereo matching network as described in any one of claims 1 to 6, or the steps of the three-dimensional scene reconstruction method as described in any one of claims 7 to 9.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, performs the steps of the training method for the stereo matching network as described in any one of claims 1 to 6, or the steps of the three-dimensional scene reconstruction method as described in any one of claims 7 to 9.

Images