A three-dimensional point cloud reconstruction method, device and electronic equipment

Abstract

The disclosure provides a three-dimensional point cloud reconstruction method, device and electronic equipment, the method comprising: acquiring a first multi-view image associated with a first scene and corresponding first camera parameters; extracting a first feature map of N stages corresponding to each first image, the first feature map comprising a first depth feature of each pixel point of the corresponding stage; determining a matching cost of a first source image in the first multi-view image and a first reference image in the first multi-view image at each depth value of the Nth stage according to the first feature map of N stages corresponding to the first image and the first camera parameters; based on a regularization network of the Nth stage, regularizing the matching cost at each depth value of the Nth stage to obtain a first depth map of the first source image; and performing three-dimensional point cloud reconstruction on the first scene according to the first source image and the first depth map.

Description

Technical Field

[0001] This disclosure relates to the field of three-dimensional point cloud reconstruction technology, and more specifically, to a three-dimensional point cloud reconstruction method, apparatus, and electronic device. Background Technology

[0002] 3D point cloud reconstruction is a computer graphics and computer vision technique that constructs a 3D model of an object or scene by processing point cloud data.

[0003] Multi-image stereo matching (MVS) is a key technology in computer vision used to recover the three-dimensional structure of a scene from images taken from multiple different perspectives. It reconstructs the three-dimensional model of a scene by analyzing multiple images taken from different perspectives.

[0004] The multi-image stereo matching method based on deep learning models follows the traditional planar scanning algorithm, discretizes the depth range into a limited number of depth value candidates, and constructs a matching cost to measure the difference between the image features of multiple views and each depth value, resulting in high memory usage and high computational cost. Summary of the Invention

[0005] One object of the present disclosure is to provide a three-dimensional point cloud reconstruction method, apparatus, and electronic device.

[0006] According to a first aspect of the present disclosure, a three-dimensional point cloud reconstruction method is provided, comprising:

[0007] Obtain a first multi-view image associated with a first scene and corresponding first camera parameters. The first multi-view image is a first image obtained by capturing the first scene from multiple perspectives using multiple first cameras with different poses. The first camera parameters include the pose information of each first camera.

[0008] Extract the first feature map corresponding to N stages for each first image. The first feature map includes the first depth feature of each pixel in the corresponding stage; N is a positive integer.

[0009] Based on the first feature map corresponding to the first image in N stages and the first camera parameters, determine the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage.

[0010] Based on the regularization network of the Nth stage, the matching cost under each depth value in the Nth stage is regularized to obtain the first depth map of the first source image.

[0011] The first scene is reconstructed into a 3D point cloud based on the first source image and the first depth map.

[0012] Optionally, the method further includes:

[0013] The first image in the first multi-view image is traversed, and the first image currently traversed is taken as the first source image. At least one first image in the first multi-view image other than the first source image is taken as the first reference image.

[0014] Determine the first depth map of the first source image;

[0015] Determine if there is an untraversed first image in the first multi-view image. If so, continue to traverse multiple first images in the first multi-view image; otherwise, end the traversal and obtain the first depth map corresponding to each first image in the first multi-view image.

[0016] Optionally, determining the matching cost between the first source image in the first multi-view image and the first reference image in the first multi-view image at each depth value in the Nth stage, based on the first feature map corresponding to the first image in N stages and the first camera parameters, includes:

[0017] Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the depth value of the Nth stage;

[0018] Based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters, the matching cost for each depth value in the Nth stage is obtained.

[0019] Optionally, determining the depth value of the Nth stage based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters includes:

[0020] Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the matching cost between the first source image and the first reference image at each depth value in the N-1 stages;

[0021] Based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the predicted depth map of the first source image in the N-1th stage.

[0022] The depth value for the Nth stage is determined based on the predicted depth map of the first source image at the (N-1)th stage.

[0023] Optionally, obtaining the matching cost for each depth value in the Nth stage based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters includes:

[0024] For each depth value in the Nth stage, according to the first camera parameters, each pixel in the first source image corresponding to the Nth stage is projected onto the first reference image according to the corresponding depth value to obtain the corresponding projected pixel.

[0025] By comparing the difference in the first depth features between each pixel of the first source image at each depth value in the Nth stage and the corresponding projected pixel, the matching cost at each depth value in the Nth stage is obtained.

[0026] Optionally, the regularization network based on the Nth stage performs regularization processing on the matching cost at each depth value in the Nth stage to obtain the first depth map of the first source image, including:

[0027] Based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the probability distribution of the first source image at each depth value in the Nth stage.

[0028] Based on the probability distribution of the first source image at each depth value in the Nth stage, the predicted depth map of the first source image in the Nth stage is obtained, which is used as the first depth map.

[0029] Optionally, the method further includes the step of training an N-stage regularization network, including:

[0030] Acquire a second multi-view image associated with the second scene and corresponding second camera parameters. The second multi-view image is a second image obtained by capturing the second scene from multiple perspectives using multiple second cameras with different poses. The second camera parameters include the pose information of each second camera.

[0031] Extract the second feature map corresponding to N stages of the second image. The second feature map includes the second depth feature of each pixel in the corresponding stage.

[0032] Based on the second feature map corresponding to the second image in N stages and the second camera parameters, determine the matching cost between the second source image and the second reference image in the second multi-view image at each depth value in the N stages;

[0033] Obtain the true depth maps of the second source image at N stages;

[0034] The N-stage regularization network is trained based on the matching cost at each depth value in the N stages. The training objective is to minimize the loss between the second depth map output by the N-stage regularization network and the corresponding real depth map, and the network parameters of the N-stage regularization network are updated accordingly.

[0035] Optionally, training the regularized network for the N stages based on the matching cost at each depth value in the N stages includes:

[0036] Using the network parameters of the regularized network with N stages as variables, the second depth map of the second source image in N stages is determined based on the matching cost at each depth value in the N stages.

[0037] For each stage of the regularized network, the first loss function of the regularized network for the corresponding stage is constructed based on the second depth map and the true depth map of the corresponding stage.

[0038] Based on the weights corresponding to the N stages, the first loss function of each stage is weighted and summed to obtain the second loss function.

[0039] Find the network parameters of the N-stage regularized network when the value of the second loss function is minimized, and complete the training of the N-stage regularized network.

[0040] According to a second aspect of this disclosure, a three-dimensional point cloud reconstruction apparatus is provided, comprising:

[0041] The first acquisition module is used to acquire a first multi-view image associated with a first scene and corresponding first camera parameters. The first multi-view image is a first image obtained by capturing the first scene from multiple perspectives using multiple first cameras with different poses. The first camera parameters include the pose information of each first camera.

[0042] The feature extraction module is used to extract the first feature map corresponding to N stages for each first image. The first feature map includes the first depth feature of each pixel in the corresponding stage; N is a positive integer.

[0043] The cost determination module is used to determine the matching cost between the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage based on the first feature map corresponding to the Nth stage of the first image and the first camera parameters.

[0044] The depth estimation module obtains a regularized network based on the Nth stage, and performs regularization processing on the matching cost under each depth value in the Nth stage to obtain the first depth map of the first source image.

[0045] The point cloud reconstruction module is used to reconstruct a three-dimensional point cloud of the first scene based on the first source image and the first depth map.

[0046] According to a third aspect of this disclosure, an electronic device is provided, including a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the method as described in the first aspect of this disclosure under the control of the computer program.

[0047] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described in the first aspect of this disclosure.

[0048] Through the embodiments of this disclosure, a first multi-view image associated with a first scene and corresponding first camera parameters are obtained, with each first image corresponding to a first feature map of N stages; based on the first feature map of the first image corresponding to the N stages and the first camera parameters, the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage is determined; based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the first depth map of the first source image; the first scene is reconstructed into a 3D point cloud based on the first source image and the first depth map, which can reduce the occupation of video memory and computational cost, and improve the speed of 3D point cloud reconstruction.

[0049] Other features and advantages of the invention will become clear from the following detailed description of exemplary embodiments of the invention with reference to the accompanying drawings. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments of the invention and, together with their description, serve to explain the principles of the invention.

[0051] Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device that can implement embodiments of the present disclosure;

[0052] Figure 2 This is a flowchart of a three-dimensional point cloud reconstruction method according to an embodiment of the present disclosure;

[0053] Figure 3 This is a schematic diagram illustrating an example of a three-dimensional point cloud reconstruction method according to an embodiment of the present disclosure;

[0054] Figure 4 This is a block diagram of a three-dimensional point cloud reconstruction apparatus according to an embodiment of the present disclosure;

[0055] Figure 5 This is a block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0056] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the invention.

[0057] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the invention or its application or use.

[0058] Techniques, methods, and apparatus known to those skilled in the art in the relevant field may not be discussed in detail, but where appropriate, such techniques, methods, and apparatus should be considered part of the specification.

[0059] In all the examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values.

[0060] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0061] <Hardware Configuration>

[0062] Figure 1 This is a block diagram illustrating the hardware configuration of an electronic device 1000 that can implement embodiments of the present disclosure.

[0063] Electronic device 1000 can be a portable computer, desktop computer, mobile phone, tablet computer, etc. For example... Figure 1 As shown, the electronic device 1000 may include a processor 1100, a memory 1200, an interface device 1300, a communication device 1400, a display device 1500, an input device 1600, a speaker 1700, a microphone 1800, etc. The processor 1100 may be a CPU, a microprocessor (MCU), etc. The memory 1200 may include, for example, ROM (Read-Only Memory), RAM (Random Access Memory), or non-volatile memory such as a hard disk. The interface device 1300 may include, for example, a USB interface, a headphone jack, etc. The communication device 1400 may be capable of wired or wireless communication, specifically including Wi-Fi communication, Bluetooth communication, 2G / 3G / 4G / 5G communication, etc. The display device 1500 may be, for example, an LCD screen, a touch screen, etc. The input device 1600 may include, for example, a touch screen, a keyboard, motion input, etc. Users can input / output voice information through the speaker 1700 and the microphone 1800.

[0064] Figure 1 The electronic devices shown are merely illustrative and in no way intended to limit this disclosure, its application, or use. In embodiments applied to this disclosure, the memory 1200 of the electronic device 1000 is used to store instructions for controlling the processor 1100 to operate to perform any of the methods provided in the embodiments of this disclosure. Those skilled in the art will understand that, although... Figure 1 The electronic device 1000 is shown with multiple devices shown; however, this disclosure may relate only to some of these devices. For example, electronic device 1000 may only relate to processor 1100 and memory 1200. Those skilled in the art can design instructions based on the schemes disclosed herein. How the instructions control the processor to operate is well known in the art and will not be described in detail here.

[0065] <Method Implementation>

[0066] This disclosure provides a method for reconstructing three-dimensional point clouds, which can be implemented by an electronic device. For example, the electronic device can be the aforementioned electronic device 1000.

[0067] Figure 2 This is a flowchart of a three-dimensional point cloud reconstruction method according to an embodiment of the present disclosure.

[0068] like Figure 2 As shown, the 3D point cloud reconstruction method includes the following steps S2100 to S2500:

[0069] Step S2100: Obtain the first multi-view image associated with the first scene and the corresponding first camera parameters. The first multi-view image is a first image obtained by capturing the first scene from multiple perspectives using multiple first cameras with different poses. The first camera parameters include the pose information of each first camera.

[0070] In this embodiment, multiple cameras can take pictures of the first scene to be modeled from multiple perspectives to obtain images corresponding to each camera.

[0071] To ensure the accuracy of the modeling, a high degree of overlap between each image is required, meaning that the difference in viewpoint between different images cannot be too large, in order to reflect the smoothness of viewpoint changes.

[0072] To ensure the accuracy of the modeling, each first image can have the same resolution.

[0073] Step S2200: Extract the first feature map corresponding to N stages for each first image.

[0074] In this embodiment, each first feature map may include the first depth feature of each pixel in the corresponding stage.

[0075] In one embodiment, a feature extraction network may be used to perform feature extraction processing on each first image to obtain a first feature map corresponding to the first image.

[0076] In this embodiment, the feature extraction network can be any neural network capable of extracting features from an image, and the specific structure of the feature extraction network is not limited here.

[0077] In one embodiment, the feature extraction network may be a feature pyramid network to extract multi-scale depth image features.

[0078] In one embodiment, a feature pyramid network can be used to obtain first feature maps for each first image at N stages, where N is a positive integer. That is, for each first image, first feature maps corresponding to N different stages can be obtained.

[0079] Furthermore, the size of the first feature map in the j-th stage is smaller than the size of the first feature map in the (j+1)-th stage, where j is a positive integer less than or equal to N, and the size of the first feature map in the N-th stage is the same as the size of the first image.

[0080] N can be pre-set based on the number of layers in the cascade structure of the subsequent regularization network, and can be the same as the number of layers in the cascade structure of the regularization network. For example, N can be 3.

[0081] In the example where N is 3, each first image corresponds to three stages of first feature maps. The size of the first feature map in the first stage is smaller than the size of the first feature map in the second stage, the size of the first feature map in the second stage is smaller than the size of the first feature map in the third stage, and the size of the first feature map in the third stage is the same as the size of the first image. For example, the size of the first feature map in the second stage can be half the size of the first feature map in the third stage, and the size of the first feature map in the first stage can be half the size of the first feature map in the second stage. As another example, the size of the first feature map in the second stage can be one-quarter the size of the first feature map in the third stage, and the size of the first feature map in the first stage can be one-quarter the size of the first feature map in the second stage.

[0082] In one example, based on obtaining the first feature maps of N stages corresponding to each first image, it is also possible to perform fusion processing on the first feature maps of N stages corresponding to each first image to obtain a fused feature map for the corresponding first image, so as to capture image details and contextual information at different levels.

[0083] Step S2300: Based on the first feature map and the first camera parameters corresponding to the N stages of the first image, determine the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage.

[0084] In one embodiment, the first source image can be any one of the first images in the first multi-view images.

[0085] In one embodiment of this disclosure, the method further includes: traversing a plurality of first images in a first multi-view image, and taking the currently traversed first image as a first source image, and taking at least one first image in the first multi-view image other than the first source image as a first reference image; determining a first depth map of the first source image; determining whether there is an untraversed first image in the first multi-view image, and if so, continuing to perform the step of traversing a plurality of first images in the first multi-view image; if not, ending the traversal, and obtaining a first depth map corresponding to each first image in the first multi-view image.

[0086] In this embodiment, the first depth map of the first source image can be determined through steps S2300 to S2400 of this embodiment.

[0087] By traversing multiple first images in the first multi-view image, the first depth map of each first image in the first multi-view image can be obtained.

[0088] In one embodiment of this disclosure, N is greater than 1. Therefore, determining the matching cost between the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage, based on the first feature map corresponding to the first image in N stages and the first camera parameters, may include the following steps S2310 to S2320:

[0089] Step S2310: Determine the depth value of the Nth stage based on the first feature map and the first camera parameters corresponding to the first N-1 stages of the first image.

[0090] In this embodiment, determining the depth value of the Nth stage based on the first feature map corresponding to the first N-1 stages and the first camera parameters can include the following steps S2311 to S2313:

[0091] Step S2311: Based on the first feature map and first camera parameters corresponding to the first image in the first N-1 stages, determine the matching cost of the first source image and the first reference image at each depth value in the N-1 stages.

[0092] When N=2, the depth value of the (N-1)th stage can be a set of depth values ​​set according to the application scenario or specific requirements. For example, it can be an integer value from 1 to M.

[0093] When N is greater than 2, the depth value of the (N-1)th stage can be determined based on the first feature map and the first camera parameters of the previous N-2 stages. For details, please refer to steps S2311 to S2313 in this embodiment, which will not be repeated here.

[0094] Furthermore, the step of determining the matching cost of the first source image and the first reference image at each depth value in the (N-1)th stage can refer to steps S2321 to S2322 in this embodiment, and will not be repeated here.

[0095] Step S2312: Based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the predicted depth map of the first source image in the N-1th stage.

[0096] In this embodiment, based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the predicted depth map of the first source image in the N-1th stage, which may include the following steps S23121 to S23122:

[0097] Step S23121: Based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the probability distribution of the first source image under each depth value in the N-1th stage.

[0098] In this embodiment, N cascaded regularization networks can be pre-configured as a hybrid regularization network, combining the recurrent regularization method of recurrent neural networks (RNNs) with a multi-stage cascaded structure from coarse to fine. In each regularization stage, the input matching cost is sliced ​​along the depth dimension, with each slice sequentially fed into a 2D TransConv module. The output of each layer of the 2D TransConv is connected to a ConvLSTM module, enabling the network to efficiently process sequential information while maintaining spatial continuity. As the network depth gradually increases, the number and range of depth candidates gradually decrease, thus achieving progressively refined depth estimation. This is achieved through progressive refinement at each stage of the cascaded structure, allowing the network to optimize and adjust the depth estimation at higher levels. This design not only utilizes spatial constraints to reduce noise and enhance the fragmented smoothness of the depth map but also ensures optimization in computational efficiency and memory usage.

[0099] In the embodiment where N is 3, the three cascaded regularization networks correspond to three stages from coarse to fine: a coarse depth estimation stage, an intermediate depth refinement stage, and a detailed depth refinement stage. Each stage focuses on further refining the depth estimation through iterative and cascaded regularization methods.

[0100] The main task of the coarse depth estimation stage corresponding to the first regularization network is to quickly locate possible depth candidate intervals starting from a relatively coarse depth range. This stage does not need to be very precise, but it should be able to determine the approximate depth distribution area, providing a basis for subsequent stages.

[0101] Specifically, the matching cost can be processed using 2D TransConv and connected ConvLSTM modules. 2D TransConv helps capture spatial features, while ConvLSTM processes sequence information in the depth dimension, providing temporal memory for depth slicing. At this stage, the number of depth candidates is relatively large, and the depth range is wide, resulting in coarse-grained depth estimation.

[0102] The second regularization network, corresponding to the intermediate depth refinement stage, further narrows the depth range and increases the accuracy of depth estimation based on the coarse depth estimation stage. The goal of this stage is to explore and adjust the potential depth intervals more finely based on the coarse localization. The regularized depth prediction output from the coarse depth estimation stage is used as input for further refinement. Similarly, each depth slice continues to be processed by modified 2D TransConv and ConvLSTM modules, which now handle finer depth slices, reducing the number of depth candidates and narrowing the intervals. The output of this stage is a more accurate depth candidate, ready to enter the final refinement stage.

[0103] The task of the third regularization network, corresponding to the detail refinement stage, is to perform the most meticulous depth adjustments, ensuring that the final depth estimate is as close as possible to the actual value. This stage emphasizes the detail accuracy of the depth and the sensitive capture of depth variations within small ranges. The output of the intermediate depth refinement stage is continued, but now each depth slice is processed more finely, the depth interval is further narrowed, and the number of depth candidates is further reduced. Depth slices are processed through further optimized 2DTransConv and ConvLSTM modules, which are now configured to focus on higher resolution detail information. The final output depth prediction is highly accurate and can be directly used to generate high-quality point cloud data.

[0104] In this embodiment, a recurrent network structure is applied in each regularized network, deploying a combination of convolutional layers and LSTM units.

[0105] The input to the regularization network in the (N-1)th stage can be the matching cost C between the first source image and the first reference image at each depth value in the (N-1)th stage. N-1 (d1), C N-1 (d2), ..., C N-1 (dM).

[0106] The matching cost between the first source image and the first reference image at each depth value in the (N-1)th stage is processed by a series of convolutional LSTM units. These units are designed to process temporal data, but here they are used to process the sequence information of depth. Specifically, the matching cost C between the first source image and the first reference image at each depth value in the (N-1)th stage is... N-1 (d1), C N-1 (d2), ..., C N-1 (dM) passes through a series of ConvLSTM units. After each unit finishes processing, the output result is passed to the next unit until all depth values ​​of the N-1th stage have been processed.

[0107] The internal structure of a ConvLSTM unit includes Hadamard products, Sigmoid and Tanh activation functions, addition, and 2D convolution operations. These components work together to enable LSTM gating of temporal data, but here it is applied to the spatial relationships of depth data. Hadamard products are used for direct element-wise multiplication, typically applied with gating signals. Sigmoid activation generates the gating signal, controlling the flow of information. Tanh activation produces new candidate values. Addition and 2D convolution are used to merge information and generate new state outputs.

[0108] The processing of the i-th depth value in the (N-1)th stage depends not only on its own input C N-1 (di) also depends on the output of the processing of the (i-1)th depth value in N-1 stages. This design helps the model capture the contextual relationships between depths and improves the continuity and accuracy of depth estimation.

[0109] After the (N-1)th stage of the regularization network has processed all depth layers, each layer outputs a probability distribution Y. N-1 (d1), Y N-1 (d2), ..., Y N-1 (dM). These probability distributions represent the probability of each pixel of the first source image at the (j-1)th resolution at each depth value in the (N-1)th stage, which is the probability distribution of the first source image at each depth value in the (N-1)th stage.

[0110] Step S23122: Based on the probability distribution of the first source image at each depth value in the N-1 stage, obtain the predicted depth map of the first source image in the N-1 stage.

[0111] The final predicted depth map is estimated using a winner-take-all strategy based on a regularized probability volume.

[0112] Specifically, the shape of the probability distribution output by the regularization network in the (N-1)th stage is H. N-1 *W N-1 *M, iterate through H N-1 *W N-1 Each pixel location corresponds to the output Y for each depth value. N-1 (di) is converted into a probability distribution by applying the Softmax function. This step converts the cost of each pixel in the 1st to Mth depth layers into a probability, representing the likelihood that the pixel is located at that depth.

[0113] The Winner-Take-All strategy selects the depth value with the highest probability from the probability volume as the final depth estimate.

[0114] Step S2313: Determine the depth value of the Nth stage based on the predicted depth map of the first source image in the (N-1)th stage.

[0115] In the predicted depth map of the first source image at the (N-1)th stage, the depth value of the i-th pixel is D. Then, the depth value of the corresponding N-th stage can include M values ​​belonging to [D-△D, D+△D], and the difference between two adjacent values ​​among the M values ​​is △D*2 / (M-1).

[0116] Step S2320: Based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters, obtain the matching cost for each depth value in the Nth stage.

[0117] In one embodiment of this disclosure, the matching cost for each depth value in the Nth stage is obtained based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters. This may include the following steps S2321 to S2322:

[0118] Step S2321: For each depth value in the Nth stage, according to the first camera parameters, project each pixel in the first source image corresponding to the Nth stage onto the first reference image according to the corresponding depth value to obtain the corresponding projected pixel.

[0119] Step S2322: Compare the difference of the first depth features between each pixel of the first source image at each depth value in the Nth stage and the corresponding projected pixel to obtain the matching cost at each depth value in the Nth stage.

[0120] In this embodiment, the statistical value of the first depth feature of the first source image corresponding to the projection pixel at the i-th depth value in the N-th stage can be determined first, and then the difference between the statistical value and the first depth feature of the k-th pixel can be determined. This difference is the first depth feature difference between the first source image at the i-th depth value in the N-th stage and the k-th pixel and its corresponding projection pixel.

[0121] The first depth feature difference between all pixels of the first source image and the corresponding projected pixels at each depth value in the Nth stage is the matching cost between the first source image and the first reference image at each depth value in the Nth stage.

[0122] The matching cost for each depth value in the Nth stage can be expressed as H. N *W N The tensor of H, where H N *W N This refers to the number of pixels in the Nth stage.

[0123] In another embodiment of this disclosure, when N is 1, determining the matching cost of the first source image in the first multi-view image and the first reference image in the first multi-view image at each depth value in the Nth stage based on the first feature map corresponding to the Nth stage of the first image and the first camera parameters may include: obtaining the depth value of the Nth stage, and obtaining the matching cost at each depth value in the Nth stage based on the depth value of the Nth stage, the first feature map corresponding to the Nth stage of the first image and the first camera parameters.

[0124] Each depth value in the Nth stage can be a set of depth values ​​set according to the application scenario or specific requirements, for example, it can be an integer value from 1 to M.

[0125] Step S2400: Based on the regularization network of the Nth stage, the matching cost under each depth value in the Nth stage is regularized to obtain the first depth map of the first source image.

[0126] In one embodiment of this disclosure, based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the first depth map of the first source image, which may include the following steps S2410 to S2420:

[0127] Step S2410: Based on the regularization network of the Nth stage, the matching cost under each depth value in the Nth stage is regularized to obtain the probability distribution of the first source image under each depth value in the Nth stage.

[0128] In this embodiment, N cascaded regularization networks can be pre-configured as a hybrid regularization network, combining the recurrent regularization method of recurrent neural networks (RNNs) with a multi-stage cascaded structure from coarse to fine. In each regularization stage, the input matching cost is sliced ​​along the depth dimension, with each slice sequentially fed into a 2D TransConv module. The output of each layer of the 2D TransConv is connected to a ConvLSTM module, enabling the network to efficiently process sequential information while maintaining spatial continuity. As the network depth gradually increases, the number and range of depth candidates gradually decrease, thus achieving progressively refined depth estimation. This is achieved through progressive refinement at each stage of the cascaded structure, allowing the network to optimize and adjust the depth estimation at higher levels. This design not only utilizes spatial constraints to reduce noise and enhance the fragmented smoothness of the depth map but also ensures optimization in computational efficiency and memory usage.

[0129] In the embodiment where N is 3, the three cascaded regularization networks correspond to three stages from coarse to fine: a coarse depth estimation stage, an intermediate depth refinement stage, and a detailed depth refinement stage. Each stage focuses on further refining the depth estimation through iterative and cascaded regularization methods.

[0130] The main task of the coarse depth estimation stage corresponding to the first regularization network is to quickly locate possible depth candidate intervals starting from a relatively coarse depth range. This stage does not need to be very precise, but it should be able to determine the approximate depth distribution area, providing a basis for subsequent stages.

[0131] Specifically, the matching cost can be processed using 2D TransConv and connected ConvLSTM modules. 2D TransConv helps capture spatial features, while ConvLSTM processes sequence information in the depth dimension, providing temporal memory for depth slicing. At this stage, the number of depth candidates is relatively large, and the depth range is wide, resulting in coarse-grained depth estimation.

[0132] The second regularization network, corresponding to the intermediate depth refinement stage, further narrows the depth range and increases the accuracy of depth estimation based on the coarse depth estimation stage. The goal of this stage is to explore and adjust the potential depth intervals more finely based on the coarse localization. The regularized depth prediction output from the coarse depth estimation stage is used as input for further refinement. Similarly, each depth slice continues to be processed by modified 2D TransConv and ConvLSTM modules, which now handle finer depth slices, reducing the number of depth candidates and narrowing the intervals. The output of this stage is a more accurate depth candidate, ready to enter the final refinement stage.

[0133] The task of the third regularization network, corresponding to the detail refinement stage, is to perform the most meticulous depth adjustments, ensuring that the final depth estimate is as close as possible to the actual value. This stage emphasizes the detail accuracy of the depth and the sensitive capture of depth variations within small ranges. The output of the intermediate depth refinement stage is continued, but now each depth slice is processed more finely, the depth interval is further narrowed, and the number of depth candidates is further reduced. Depth slices are processed through further optimized 2DTransConv and ConvLSTM modules, which are now configured to focus on higher resolution detail information. The final output depth prediction is highly accurate and can be directly used to generate high-quality point cloud data.

[0134] In this embodiment, a recurrent network structure is applied in each regularized network, deploying a combination of convolutional layers and LSTM units.

[0135] The input to the regularization network in the Nth stage can be the matching cost C between the first source image and the first reference image at each depth value in the Nth stage. N (d1), C N (d2), ..., C N (dM).

[0136] The matching cost between the first source image and the first reference image at each depth value in the Nth stage is processed by a series of convolutional LSTM units. These units are designed to process temporal data, but here they are used to process the sequential information of depth. Specifically, the matching cost C between the first source image and the first reference image at each depth value in the Nth stage is... N (d1), C N (d2), ..., C N (dM) passes through a series of ConvLSTM units. After each unit finishes processing, the output result is passed to the next unit until all depth values ​​of the Nth stage have been processed.

[0137] The internal structure of a ConvLSTM unit includes Hadamard products, Sigmoid and Tanh activation functions, addition, and 2D convolution operations. These components work together to enable LSTM gating of temporal data, but here it is applied to the spatial relationships of depth data. Hadamard products are used for direct element-wise multiplication, typically applied with gating signals. Sigmoid activation generates the gating signal, controlling the flow of information. Tanh activation produces new candidate values. Addition and 2D convolution are used to merge information and generate new state outputs.

[0138] The processing of the i-th depth value in the Nth stage depends not only on its own input C N (di) also depends on the output of the processing of the (i-1)th depth value in N stages. This design helps the model capture the contextual relationships between depths and improves the continuity and accuracy of depth estimation.

[0139] After the Nth stage of the regularization network has processed all depth layers, each layer outputs a probability distribution Y. N (d1), Y N (d2), ..., Y N (dM). These probability distributions represent the probability of each pixel in the first source image at the (j-1)th resolution at each depth value in the Nth stage, which is the probability distribution of the first source image at each depth value in the Nth stage.

[0140] Step S2420: Based on the probability distribution of the first source image at each depth value in the Nth stage, obtain the predicted depth map of the first source image in the Nth stage, which is used as the first depth map.

[0141] The final predicted depth map is estimated using a winner-take-all strategy based on a regularized probability volume.

[0142] Specifically, the shape of the probability distribution output by the regularization network in the Nth stage is H. N *W N *M, iterate through H N *W N Each pixel location corresponds to the output Y for each depth value. N (di) is converted into a probability distribution by applying the Softmax function. This step converts the cost of each pixel in the 1st to Mth depth layers into a probability, representing the likelihood that the pixel is located at that depth.

[0143] Step S2500: Reconstruct the 3D point cloud of the first scene based on the first source image and the first depth map.

[0144] In this embodiment, the first scene can be reconstructed into a 3D point cloud based on the first image and the corresponding first depth map when a first depth map of at least one first image is obtained; or the first scene can be reconstructed into a 3D point cloud based on all the first images and the corresponding first depth maps when a first depth map corresponding to all the first images in the first multi-view image is obtained.

[0145] In this embodiment, for each pixel of the first image corresponding to the first depth map, its coordinates can be transformed from the image coordinate system to the three-dimensional world coordinate system based on its depth value and the first camera parameters to obtain the corresponding three-dimensional world coordinates. All three-dimensional world coordinates are then aggregated to obtain a complete three-dimensional point cloud.

[0146] In one embodiment of this disclosure, before reconstructing a three-dimensional point cloud of a first scene based on a first source image and a first depth map, point cloud construction parameters can be set, including the required precision and range of the point cloud to be generated.

[0147] In one embodiment of this disclosure, after reconstructing a three-dimensional point cloud of a first scene based on a first multi-view image and a first depth map, the method may further include: filtering the three-dimensional point cloud; and / or optimizing the three-dimensional point cloud.

[0148] Filtering can be used to remove noise from 3D point clouds, improving their quality. Optimization can be achieved through point cloud trimming, downsampling, and other methods to refine the point cloud data, making it more suitable for subsequent applications.

[0149] In one embodiment of this disclosure, the final point cloud data may be saved in a common format, such as PLY or OBJ, for subsequent use and analysis.

[0150] In one embodiment of this disclosure, a three-dimensional visualization tool can be used to display the reconstructed point cloud to visually demonstrate the reconstruction effect.

[0151] Through the embodiments of this disclosure, a first multi-view image associated with a first scene and corresponding first camera parameters are obtained, with each first image corresponding to a first feature map of N stages; based on the first feature map of the first image corresponding to the N stages and the first camera parameters, the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage is determined; based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the first depth map of the first source image; the first scene is reconstructed into a 3D point cloud based on the first source image and the first depth map, which can reduce the occupation of video memory and computational cost, and improve the speed of 3D point cloud reconstruction.

[0152] In one embodiment of this disclosure, the method further includes the step of training an N-stage regularization network, comprising steps S3100 to S3500 as shown below:

[0153] Step S3100: Obtain the second multi-view image associated with the second scene and the corresponding second camera parameters.

[0154] The second multi-view image is a second image obtained by taking multiple photos of the second scene from multiple perspectives using second cameras in different poses. The second camera parameters include the pose information of each second camera.

[0155] In this embodiment, the parameters of the second camera may be the same as or different from the parameters of the first camera, which will not be elaborated here.

[0156] Step S3200: Extract the second feature map corresponding to N stages of the second image. The second feature map includes the second depth feature of each pixel in the corresponding stage.

[0157] For details, please refer to the method of extracting the first feature map in step S2200 above, which will not be elaborated here.

[0158] Step S3300: Based on the second feature map and the second camera parameters corresponding to the N stages of the second image, determine the matching cost of the second source image and the second reference image in the second multi-view image at each depth value in the N stages.

[0159] For details, please refer to the method for determining the matching cost in step S2300 above, which will not be elaborated here.

[0160] Step S3400: Obtain the true depth map of the N stages corresponding to the second source image.

[0161] Step S3500: Train the regularization network of N stages according to the matching cost at each depth value in N stages. Take minimizing the loss between the second depth map output by the regularization network of N stages and the corresponding real depth map as the training objective, and update the network parameters of the regularization network of N stages.

[0162] In one embodiment of this disclosure, a regularized network for N stages is trained based on the matching cost at each depth value in the N stages, including steps S3510 to S3540 as shown below:

[0163] Step S3510: Using the network parameters of the regularized network in N stages as variables, determine the second depth map of the second source image in N stages based on the matching cost at each depth value in N stages.

[0164] Step S3520: For the regularization network of each stage, construct the first loss function of the regularization network of the corresponding stage based on the second depth map and the true depth map of the corresponding stage.

[0165] In this embodiment, a first loss function for a regularized network corresponding to N stages can be constructed using a classification-based cross-entropy loss function, where each depth value of a stage is considered a predefined class.

[0166] Step S3530: Based on the weights corresponding to the N stages, the first loss function of each stage is weighted and summed to obtain the second loss function.

[0167] Step S3540: Solve for the network parameters of the regularized network in N stages when the value of the second loss function is minimized, and complete the training of the regularized network in N stages.

[0168] In this embodiment, the loss of the regularization network at each stage is weighted according to its contribution to the final depth estimation in order to achieve the optimal training effect of the regularization network with N stages.

[0169] In this embodiment, the backpropagation method can be used to optimize the network parameters of the regularized network in N stages.

[0170] <Example>

[0171] Figure 3 This is a flowchart of an example of a three-dimensional point cloud reconstruction method according to an embodiment of the present disclosure.

[0172] exist Figure 3The diagram illustrates the process of determining a first depth map of a first source image based on three cascaded regularization networks. Specifically, it includes: obtaining a pre-set depth value M1 for the first stage; obtaining matching costs C1(d1), C1(d2), ..., C1(dM) for each depth value in the first stage based on the depth value of the first stage, the first feature map 11 of the first source image corresponding to the first stage, the first feature map 12 of the first reference image corresponding to the first stage, and the first camera parameters; performing regularization processing on the matching costs C1(d1), C1(d2), ..., C1(dM) for each depth value in the first stage based on the first stage regularization network 1 to obtain the predicted depth map Y1 of the first source image in the first stage; and determining the depth value M2 for the second stage based on the predicted depth map of the first source image in the first stage. Based on the depth value M2 of the second stage, the first feature map 21 of the first source image corresponding to the second stage, the first feature map 22 of the first reference image corresponding to the second stage, and the first camera parameters, the matching costs C2(d1), C2(d2), ..., C2(dM) for each depth value in the second stage are obtained. Based on the regularization network 2 of the second stage, the matching costs C2(d1), C2(d2), ..., C2(dM) for each depth value in the second stage are regularized to obtain the predicted depth map Y2 of the first source image in the second stage. The depth value M3 of the third stage is determined based on the predicted depth map of the first source image in the second stage. Based on the depth value M3 of the third stage, the first feature map 31 of the first source image corresponding to the third stage, the first feature map 32 of the first reference image corresponding to the third stage, and the first camera parameters, the matching costs C3(d1), C3(d2), ..., C3(dM) for each depth value in the third stage are obtained. Based on the regularization network 3 of the third stage, the matching costs C3(d1), C3(d2), ..., C3(dM) under each depth value in the third stage are regularized to obtain the predicted depth map Y3 of the first source image in the third stage, which is used as the first depth map of the first source image.

[0173] <Device Embodiment>

[0174] This disclosure provides a three-dimensional point cloud reconstruction device, such as Figure 4 As shown, the 3D point cloud reconstruction device 4000 includes a first acquisition module 4100, a feature extraction module 4200, a cost determination module 4300, a depth estimation module 4400, and a point cloud reconstruction module 4500.

[0175] The first acquisition module 4100 is used to acquire a first multi-view image associated with a first scene and corresponding first camera parameters. The first multi-view image is a first image obtained by capturing the first scene from multiple perspectives using multiple first cameras with different poses. The first camera parameters include the pose information of each first camera.

[0176] The feature extraction module 4200 is used to extract the first feature map corresponding to N stages for each first image. The first feature map includes the first depth feature of each pixel in the corresponding stage; N is a positive integer.

[0177] The cost determination module 4300 is used to determine the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage based on the first feature map corresponding to the first image in the Nth stage and the first camera parameters.

[0178] The depth estimation module 4400 obtains a regularized network based on the Nth stage, and performs regularization processing on the matching cost under each depth value in the Nth stage to obtain the first depth map of the first source image.

[0179] The point cloud reconstruction module 4500 is used to reconstruct a three-dimensional point cloud of the first scene based on the first source image and the first depth map.

[0180] In one embodiment of this disclosure, the three-dimensional point cloud reconstruction device 4000 further includes:

[0181] The traversal module is used to traverse multiple first images in the first multi-view image, and take the currently traversed first image as the first source image, and take at least one first image in the first multi-view image other than the first source image as the first reference image.

[0182] The depth estimation module 4400 is used to determine a first depth map of the first source image;

[0183] The module is used to determine whether there is an untraversed first image in the first multi-view image. If so, the traversal module continues to execute the step of traversing multiple first images in the first multi-view image; if not, the traversal module ends the traversal and obtains the first depth map corresponding to each first image in the first multi-view image.

[0184] In one embodiment of this disclosure, the cost determination module 4300 is used to:

[0185] Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the depth value of the Nth stage;

[0186] Based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters, the matching cost for each depth value in the Nth stage is obtained.

[0187] In one embodiment of this disclosure, determining the depth value of the Nth stage based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters includes:

[0188] Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the matching cost between the first source image and the first reference image at each depth value in the N-1 stages;

[0189] Based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the predicted depth map of the first source image in the N-1th stage.

[0190] The depth value for the Nth stage is determined based on the predicted depth map of the first source image at the (N-1)th stage.

[0191] In one embodiment of this disclosure, obtaining the matching cost for each depth value in the Nth stage based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters includes:

[0192] For each depth value in the Nth stage, according to the first camera parameters, each pixel in the first source image corresponding to the Nth stage is projected onto the first reference image according to the corresponding depth value to obtain the corresponding projected pixel.

[0193] By comparing the difference in the first depth features between each pixel of the first source image at each depth value in the Nth stage and the corresponding projected pixel, the matching cost at each depth value in the Nth stage is obtained.

[0194] In one embodiment of this disclosure, the depth estimation module 4400 is used for:

[0195] Based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the probability distribution of the first source image at each depth value in the Nth stage.

[0196] Based on the probability distribution of the first source image at each depth value in the Nth stage, the predicted depth map of the first source image in the Nth stage is obtained, which is used as the first depth map.

[0197] In one embodiment of this disclosure, the 3D point cloud reconstruction device 4000 further includes a training module for:

[0198] Acquire a second multi-view image associated with the second scene and corresponding second camera parameters. The second multi-view image is a second image obtained by capturing the second scene from multiple perspectives using multiple second cameras with different poses. The second camera parameters include the pose information of each second camera.

[0199] Extract the second feature map corresponding to N stages of the second image. The second feature map includes the second depth feature of each pixel in the corresponding stage.

[0200] Based on the second feature map corresponding to the second image in N stages and the second camera parameters, determine the matching cost between the second source image and the second reference image in the second multi-view image at each depth value in the N stages;

[0201] Obtain the true depth maps of the second source image at N stages;

[0202] The N-stage regularization network is trained based on the matching cost at each depth value in the N stages. The training objective is to minimize the loss between the second depth map output by the N-stage regularization network and the corresponding real depth map, and the network parameters of the N-stage regularization network are updated accordingly.

[0203] In one embodiment of this disclosure, training the N-stage regularized network based on the matching cost at each depth value in the N stages includes:

[0204] Using the network parameters of the regularized network with N stages as variables, the second depth map of the second source image in N stages is determined based on the matching cost at each depth value in the N stages.

[0205] For each stage of the regularized network, the first loss function of the regularized network for the corresponding stage is constructed based on the second depth map and the true depth map of the corresponding stage.

[0206] Based on the weights corresponding to the N stages, the first loss function of each stage is weighted and summed to obtain the second loss function.

[0207] Find the network parameters of the N-stage regularized network when the value of the second loss function is minimized, and complete the training of the N-stage regularized network.

[0208] <Electronic Device Examples>

[0209] This embodiment provides an electronic device, which may include the aforementioned three-dimensional point cloud reconstruction device 4000.

[0210] On the other hand, such as Figure 5 As shown, the electronic device 5000 may include a processor 5100 and a memory 5200. The memory 5200 is used to store computer programs, and the processor 5100 is used to control the electronic device to execute the methods of any embodiment of this disclosure under the control of the computer programs.

[0211] <Example of a readable storage medium>

[0212] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the methods described in any of the method embodiments of this disclosure.

[0213] This invention can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0214] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0215] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0216] The computer program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions. This electronic circuitry can execute the computer-readable program instructions to implement various aspects of the invention.

[0217] Various aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0218] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0219] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0220] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions. It will be known to those skilled in the art that implementation in hardware, implementation in software, and implementation using a combination of software and hardware are equivalent.

[0221] The various embodiments of the present invention have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the invention is defined by the appended claims.

Claims

1. A method for reconstructing three-dimensional point clouds, characterized in that, include: Obtain a first multi-view image associated with a first scene and corresponding first camera parameters. The first multi-view image is a first image obtained by capturing the first scene from multiple perspectives using multiple first cameras with different poses. The first camera parameters include the pose information of each first camera. Extract the first feature map corresponding to N stages for each first image. The first feature map includes the first depth feature of each pixel in the corresponding stage; N is a positive integer. Based on the first feature map corresponding to the first image in N stages and the first camera parameters, determine the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage. Based on the regularization network of the Nth stage, the matching cost under each depth value in the Nth stage is regularized to obtain the first depth map of the first source image; wherein, the regularization network includes 2D TransConv and connected ConvLSTM. The first scene is reconstructed into a 3D point cloud based on the first source image and the first depth map; The step of determining the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage based on the first feature map corresponding to the Nth stage of the first image and the first camera parameters includes: Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the depth value of the Nth stage; Based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters, the matching cost for each depth value in the Nth stage is obtained. The step of determining the depth value of the Nth stage based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters includes: Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the matching cost between the first source image and the first reference image at each depth value in the N-1 stages; Based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the predicted depth map of the first source image in the N-1th stage. Based on the predicted depth map of the first source image in the (N-1)th stage, determine the depth value in the Nth stage; The step of obtaining the matching cost for each depth value in the Nth stage based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters includes: For each depth value in the Nth stage, according to the first camera parameters, each pixel in the first source image corresponding to the Nth stage is projected onto the first reference image according to the corresponding depth value to obtain the corresponding projected pixel. The difference in the first depth features between each pixel of the first source image at each depth value in the Nth stage and the corresponding projected pixel is compared to obtain the matching cost at each depth value in the Nth stage. The regularization network based on the Nth stage performs regularization processing on the matching cost at each depth value in the Nth stage to obtain the first depth map of the first source image, including: Based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the probability distribution of the first source image at each depth value in the Nth stage. Based on the probability distribution of the first source image at each depth value in the Nth stage, the predicted depth map of the first source image in the Nth stage is obtained, which is used as the first depth map.

2. The method according to claim 1, characterized in that, The method further includes: The first image in the first multi-view image is traversed, and the first image currently traversed is taken as the first source image. At least one first image in the first multi-view image other than the first source image is taken as the first reference image. Determine the first depth map of the first source image; Determine if there is an untraversed first image in the first multi-view image. If so, continue to traverse multiple first images in the first multi-view image; otherwise, end the traversal and obtain the first depth map corresponding to each first image in the first multi-view image.

3. The method according to claim 1, characterized in that, The method further includes the step of training an N-stage regularization network, including: Acquire a second multi-view image associated with the second scene and corresponding second camera parameters. The second multi-view image is a second image obtained by capturing the second scene from multiple perspectives using multiple second cameras with different poses. The second camera parameters include the pose information of each second camera. Extract the second feature map corresponding to N stages of the second image. The second feature map includes the second depth feature of each pixel in the corresponding stage. Based on the second feature map corresponding to the second image in N stages and the second camera parameters, determine the matching cost between the second source image and the second reference image in the second multi-view image at each depth value in the N stages; Obtain the true depth maps of the second source image at N stages; The N-stage regularization network is trained based on the matching cost at each depth value in the N stages. The training objective is to minimize the loss between the second depth map output by the N-stage regularization network and the corresponding real depth map, and the network parameters of the N-stage regularization network are updated accordingly.

4. The method according to claim 3, characterized in that, The training of the regularized network for N stages based on the matching cost at each depth value in N stages includes: Using the network parameters of the regularized network with N stages as variables, the second depth map of the second source image in N stages is determined based on the matching cost at each depth value in the N stages. For each stage of the regularized network, the first loss function of the regularized network for the corresponding stage is constructed based on the second depth map and the true depth map of the corresponding stage. Based on the weights corresponding to the N stages, the first loss function of each stage is weighted and summed to obtain the second loss function. Find the network parameters of the N-stage regularized network when the value of the second loss function is minimized, and complete the training of the N-stage regularized network.

5. A three-dimensional point cloud reconstruction device, characterized in that, include: The first acquisition module is used to acquire a first multi-view image associated with a first scene and corresponding first camera parameters. The first multi-view image is a first image obtained by capturing the first scene from multiple perspectives using multiple first cameras with different poses. The first camera parameters include the pose information of each first camera. The feature extraction module is used to extract the first feature map corresponding to N stages for each first image. The first feature map includes the first depth feature of each pixel in the corresponding stage; N is a positive integer. The cost determination module is used to determine the matching cost between the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage based on the first feature map corresponding to the Nth stage of the first image and the first camera parameters. The depth estimation module obtains a regularized network based on the Nth stage, and performs regularization processing on the matching cost under each depth value in the Nth stage to obtain the first depth map of the first source image; wherein, the regularized network includes 2D TransConv and connected ConvLSTM; The point cloud reconstruction module is used to reconstruct a three-dimensional point cloud of the first scene based on the first source image and the first depth map. The step of determining the matching cost of the first source image and the first reference image in the first multi-view image at each depth value in the Nth stage based on the first feature map corresponding to the Nth stage of the first image and the first camera parameters includes: Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the depth value of the Nth stage; Based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters, the matching cost for each depth value in the Nth stage is obtained. The step of determining the depth value of the Nth stage based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters includes: Based on the first feature map corresponding to the first N-1 stages of the first image and the first camera parameters, determine the matching cost between the first source image and the first reference image at each depth value in the N-1 stages; Based on the regularization network of the N-1th stage, the matching cost under each depth value in the N-1th stage is regularized to obtain the predicted depth map of the first source image in the N-1th stage. Based on the predicted depth map of the first source image in the (N-1)th stage, determine the depth value in the Nth stage; The step of obtaining the matching cost for each depth value in the Nth stage based on the depth value of the Nth stage, the first feature map of the first image corresponding to the Nth stage, and the first camera parameters includes: For each depth value in the Nth stage, according to the first camera parameters, each pixel in the first source image corresponding to the Nth stage is projected onto the first reference image according to the corresponding depth value to obtain the corresponding projected pixel. The difference in the first depth features between each pixel of the first source image at each depth value in the Nth stage and the corresponding projected pixel is compared to obtain the matching cost at each depth value in the Nth stage. The regularization network based on the Nth stage performs regularization processing on the matching cost at each depth value in the Nth stage to obtain the first depth map of the first source image, including: Based on the regularization network of the Nth stage, the matching cost at each depth value in the Nth stage is regularized to obtain the probability distribution of the first source image at each depth value in the Nth stage. Based on the probability distribution of the first source image at each depth value in the Nth stage, the predicted depth map of the first source image in the Nth stage is obtained, which is used as the first depth map.

6. An electronic device, characterized in that, It includes a processor and a memory, the memory being used to store a computer program, and the processor being used, under the control of the computer program, to execute the method as described in any one of claims 1 to 4.

Images