Depth information processing method based on visual reconstruction

Abstract

The present invention discloses a depth information processing method based on visual reconstruction, which includes the following steps: 1. A robot adjusts the photometric consistency cost value of a pixel point by setting a depth information adjustment amount, and then identifies the type of the texture area where the pixel point is located according to the adjusted photometric consistency cost value; 2. When the robot identifies a pixel point in a first texture area, the robot uses a photometric consistency constraint criterion to process the depth information of the pixel point in the first texture area; when the robot identifies a pixel point in a second texture area, the robot uses photometric consistency cost information and a geometric projection consistency constraint criterion to process the depth information of the pixel point in the second texture area; 3. The robot uses the depth information newly processed in step 2 to perform inter-frame normal vector consistency constraints to obtain the inter-frame normal vector consistency cost value of the corresponding pixel point.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and in particular to a depth information processing method based on visual reconstruction under the constraints of photometric consistency and geometric projection consistency. Background Art

[0002] Visual 3D dense reconstruction is a technique that uses reconstruction algorithms to reconstruct a real scene in 3D by acquiring multiple 2D images from an image acquisition device (such as a camera, scanner, etc.). The photometric consistency-based patch matching algorithm, introduced by Bleyer et al. into multi-view geometry, is the most widely used 3D reconstruction algorithm today. Its core concept is to calculate the depth information of pixels by using the similarity between the photometric information of patches in the current frame and those in neighboring frames. This generally follows four steps: selecting neighboring frames, initializing depth information, determining a depth map, and combining depth maps to obtain a 3D point cloud. The accuracy of the depth map is a key factor affecting the quality of the final output point cloud.

[0003] The 3D reconstruction algorithm based on patch matching can estimate more accurate depth values in areas with rich texture information in the scene (areas with a large number of white pixels). However, in high-resolution images obtained from large-scale scenes, there are often large areas of weak texture in the corresponding high-resolution images, and the information differences between image regions are not obvious. For example, the image information between patches in different frames (such as the luminosity information of the patches) is relatively similar. The algorithm will eventually converge to obtain a limited amount of depth information, making it difficult to extract the globally optimal depth information, causing fuzzy matching problems in some texture areas or destroying the reconstructed details in some texture areas. For obstacles in the room where the image texture is not rich enough (such as flat walls, and texture-rich scenes can be spatial locations with dense pipes), the accuracy and integrity of the subsequently reconstructed 3D point cloud are poor, and the robustness is insufficient, which ultimately affects the reconstruction quality. Summary of the Invention

[0004] To address the above technical deficiencies, the present invention discloses a depth information processing method based on visual reconstruction, which can distinguish between texture-rich areas and texture-weak areas, and on this basis, use the depth information of pixels to perform visual 3D reconstruction. The specific technical solution is as follows:

[0005] A depth information processing method based on visual reconstruction is applied to process images acquired by a robot, wherein a camera installed on the robot acquires each frame of image from different shooting angles. The depth information processing method comprises: step 1, the robot adjusts the photometric consistency cost value of a pixel by setting a depth information adjustment amount, and then identifies the type of texture region in which the pixel is located based on the adjusted photometric consistency cost value; step 2, when the robot identifies that the pixel is within a first texture region, the robot processes the depth information of the pixel within the first texture region using a photometric consistency constraint criterion; when the robot identifies that the pixel is within a second texture region, the robot processes the depth information of the pixel within the second texture region using photometric consistency cost information and a geometric projection consistency constraint criterion; wherein the texture information of the first texture region is greater than that of the second texture region; step 3, the robot uses the most recently processed depth information in step 2 to perform inter-frame normal vector consistency constraints to obtain inter-frame normal vector consistency cost values of corresponding pixels, so that the normal vectors of the pixels in the image are mapped to a three-dimensional point cloud model, wherein the three-dimensional point cloud model is three-dimensional point cloud information of a scene or object to be reconstructed by the robot.

[0006] Furthermore, before executing step 1, it also includes: step 01, the robot obtains a sparse point cloud from each frame image taken from different shooting angles, and then constructs a two-dimensional surface patch from the projected pixel points of the sparse point cloud in the image coordinate system, and then determines the initial depth values of other pixel points in the two-dimensional surface patch except the projected pixel point based on the initial depth value of the projected pixel point and the initial normal vector of the projected pixel point, and then determines the initial depth values of all pixel points in each frame image and the initial normal vector of all pixel points in each frame image; wherein, each projected pixel point of the sparse point cloud in the image coordinate system is configured with an initial depth value and an initial normal vector; the two-dimensional surface patch is configured to be in the corresponding image so as to traverse all pixel points in the image; step 02, based on the similarity between the matching window in the current frame image where the pixel point is located and the matching window in the matching neighboring frame image, the robot uses the initial depth information of the adjacent pixel points of each pixel point in the image to calculate the initial depth of the pixel point. The information is updated to obtain the updated depth information of the pixel point, and then the depth information of the pixel point on the preset neighborhood of the adjacent pixel point is updated to the depth information of the adjacent pixel point to achieve adaptive update of the depth information of the pixel point, until the robot uses the matching window to traverse all the pixels in the current frame image; then, the robot continues to update the depth value of the pixel point by applying a preset depth value increment to obtain the adjusted depth value of the pixel point; the robot continues to update the normal vector of the pixel point by applying a preset normal vector increment to obtain the adjusted normal vector of the pixel point; step 03, repeat step 02 until the number of executions reaches the preset target number of times, and then execute step 1; wherein the preset target number of times is associated with the number of corner points of the image; wherein, for the same frame image, the pixel point traversal starting point required for the robot to execute step 02 twice in a row is symmetrical about the center of the frame image; wherein, the initial depth information includes an initial depth value and an initial normal vector, and the depth information includes a depth value and a normal vector.

[0007] Furthermore, the step 1 specifically includes: after executing the step 03, the depth information of the pixel point newly obtained by the robot includes the adjusted depth value of the pixel point and the adjusted normal vector of the pixel point, so that it becomes the optimized result obtained by repeatedly executing the step 02; wherein, the depth information adjustment amount includes the depth value adjustment amount and the angle adjustment amount; the robot controls the adjusted depth value of each pixel point to increase the depth value adjustment amount, and keeps the adjusted normal vector of each pixel point unchanged, and then calculates the photometric consistency cost value of the pixel point, and records it as the first photometric consistency cost value; at the same time, the robot controls the adjusted depth value of each pixel point to reduce the depth value adjustment amount, and keeps the adjusted normal vector of each pixel point unchanged, and then calculates the photometric consistency cost value of the pixel point, and records it as the second photometric consistency cost value; at the same time, with the line connecting the camera center of the camera and the pixel point as the first rotation axis, the adjusted normal vector of the pixel point is controlled The angle adjustment amount is rotated clockwise along the first axis, and the depth value of the pixel point is kept unchanged, and then the photometric consistency cost value of the pixel point is calculated and recorded as the third photometric consistency cost value; at the same time, with the line connecting the camera center of the camera and the pixel point as the second axis, the adjusted normal vector of the pixel point is controlled to rotate the angle adjustment amount counterclockwise along the second axis, and the depth value of the pixel point is kept unchanged, and then the photometric consistency cost value of the pixel point is calculated and recorded as the fourth photometric consistency cost value; then, the robot determines whether the sum of the first photometric consistency cost value, the second photometric consistency cost value, the third photometric consistency cost value and the fourth photometric consistency cost value is less than the preset cost value threshold. If so, the texture area where the pixel point is located is identified as the second texture area and the pixel point is marked as a pixel point in the second texture area; otherwise, the texture area where the pixel point is located is identified as the first texture area and the pixel point is marked as a pixel point in the first texture area.

[0008] Furthermore, in the step 2, when the robot recognizes that the pixel point is a pixel point in the first texture area, the method for the robot to use the photometric consistency constraint criterion to process the depth information of the pixel point in the first texture area specifically includes: when the robot recognizes that the pixel point is a pixel point in the first texture area through the step 1, the robot sets the adjusted depth value of the pixel point and the adjusted normal vector of the pixel point newly obtained in the step 03 as the reference depth information of the pixel point in the first texture area, and sets the adjusted photometric consistency cost value of the pixel point newly obtained in the step 03 as the reference photometric consistency cost value of the pixel point in the first texture area; for the first texture area of the current frame image, For each pixel point in the processing area, the pixel point is projected onto a matching neighborhood frame image using the internal and external camera parameters corresponding to the camera to form a mapping point in the neighborhood frame image and determine the depth information of the mapping point, and then the depth information of the mapping point is configured as the depth information to be updated of the pixel point, and then a new photometric consistency cost value of the pixel point is calculated based on the depth information to be updated of the pixel point; when the new photometric consistency cost value of the pixel point is less than the baseline photometric consistency cost value of the pixel point, the depth information to be updated of the pixel point is updated to the baseline depth information of the pixel point in the first texture area; wherein the baseline depth information of the pixel point includes the baseline depth value of the pixel point and the baseline normal vector of the pixel point.

[0009] Furthermore, the method in which the robot uses the photometric consistency constraint criterion to process the depth information of the pixel points within the first texture area also includes: the robot controls the updated baseline depth value of the pixel point to increase the baseline depth value increment to obtain the fine-tuned depth value of the pixel point; the robot controls the updated baseline normal vector of the pixel point to increase the baseline normal vector increment to obtain the fine-tuned normal vector of the pixel point; then, based on the fine-tuned depth value of the pixel point and the fine-tuned normal vector of the pixel point, the fine-tuned photometric consistency cost value of the pixel point is calculated; when the fine-tuned photometric consistency cost value of the pixel point is less than the baseline photometric consistency cost value of the pixel point, the fine-tuned depth value of the pixel point is updated to the baseline depth value of the pixel point within the first texture area, and the fine-tuned normal vector of the pixel point is updated to the baseline normal vector of the pixel point within the first texture area.

[0010] Furthermore, in the step 2, when the robot recognizes that the pixel point is a pixel point in the second texture area through the step 1, the method for the robot to process the depth information of the pixel point in the second texture area using the cost information of photometric consistency and the geometric projection consistency constraint criterion includes the following steps: step 21, the robot increases the size of the matching window to obtain a cross-correlation window, and sets the cross-correlation window to be a window that traverses the pixel points from a preset traversal starting point in the second texture area of the image, so that the photometric consistency cost value of the pixel point in the second texture area is represented by the photometric consistency cost value of the corresponding pixel point in the cross-correlation window; then execute step 22; step 22, based on the pixel point obtained in step 03 depth information, calculate the photometric consistency cost value of each pixel in the cross-correlation window; at the same time, calculate the geometric consistency cost value of each pixel in the cross-correlation window based on the depth information of the pixel obtained in step 03; then execute step 23; wherein, the value range of the geometric consistency cost value of the pixel is equal to the value range of the photometric consistency cost value of the pixel; the photometric consistency cost information includes the photometric consistency cost value of the pixel in the second texture area; step 23, the robot sets a first joint weight and applies it to the photometric consistency cost value of each pixel in the cross-correlation window, and sets a second joint weight and applies it to the geometric consistency cost value of each pixel in the cross-correlation window ; Then, the first joint weight and the second joint weight are used to calculate the weighted sum of the photometric consistency cost value and the geometric consistency cost value of each pixel point in the cross-correlation window, and the weighted sum value is set as the joint cost value of the pixel point; then step 24 is executed; step 24, the robot sets the depth information of the adjacent pixel points of each pixel point in the cross-correlation window as the depth information to be updated of the pixel point, and then calculates the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel points of the pixel point based on the depth information to be updated, and then uses the first joint weight and the second joint weight to calculate the weighted sum of the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel points, and sets the weighted sum value as The joint cost value of the adjacent pixel points; then executing step 25; when the joint cost value of the adjacent pixel points described in step 25 and step 24 is less than the joint cost value of the pixel point, the depth information of the adjacent pixel points of the pixel point is set to the updated depth information of the pixel point, so that the depth information to be updated of the pixel point is updated to the updated depth information of the pixel point; and the joint cost value of the adjacent pixel points described in step 24 is updated to the joint cost value of the pixel point; then executing step 26; step 26, repeating steps 24 to 25 until the number of repetitions reaches the first target number, so that the robot uses the cross-correlation window to traverse all pixels in the second texture area; then executing step 27;Before each repetition of step 24, the updated depth information of each pixel is first updated to the depth information of the pixel, and the depth information of the adjacent pixels of each pixel in a predetermined direction is updated to the depth information of the adjacent pixels of the pixel, so as to achieve neighborhood propagation.

[0011] Furthermore, after the robot executes step 26, the following steps are also included: step 27, the robot controls the updated depth information of the pixel point newly obtained in step 26 to increase the preset depth information increment to obtain the incrementally adjusted depth information of the pixel point; then, based on the incrementally adjusted depth information of the pixel point, the photometric consistency cost value and the geometric consistency cost value of the pixel point are calculated, and then the weighted sum of the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel points is calculated using the first joint weight and the second joint weight, and the weighted sum is set as the incrementally adjusted joint cost value of the pixel point. value; then execute step 28; when the incrementally adjusted joint cost value of the pixel point calculated in steps 28 and 27 is less than the joint cost value of the pixel point most recently obtained in step 26, update the incrementally adjusted depth information of the pixel point to the depth information of the pixel point most recently obtained in step 26; then execute step 29; step 29, repeat steps 21 to 28 until the number of repetitions reaches the second target number; then execute step 3; wherein, for the same frame image, each time the robot executes step 21, the preset traversal starting point is set to be located at the corner of the image.

[0012] Furthermore, in step 22, the method for calculating the geometric consistency cost value of each pixel point in the cross-correlation window based on the depth information of the pixel point obtained in step 03 includes: using the internal and external parameters of the camera corresponding to the camera, projecting the pixel point of the cross-correlation window in the second texture area of the current frame image to the matching neighborhood frame image to obtain the mapping point and its depth information; then projecting the mapping point back to the current frame image to obtain the inverse mapping point and its depth information; then calculating the Euclidean distance between the pixel point and the corresponding inverse mapping point, and then normalizing the Euclidean distance according to the size of the current frame image to obtain the geometric consistency cost value of the pixel point; wherein, the cross-correlation window supports the projection of the second texture area of the current frame image to a square area in the matching neighborhood frame image.

[0013] Furthermore, in the step 3, the robot uses the depth information most recently processed in step 2 to perform inter-frame normal vector consistency constraints, and the method for obtaining the inter-frame normal vector consistency cost value of the corresponding pixel points includes: step 31, the robot uses the internal and external parameters of the camera corresponding to the camera to project each pixel point of the current frame image to the matching neighborhood frame image to obtain the mapping point and its normal vector, wherein the normal vector of each pixel point of the current frame image is the normal vector in the depth information most recently processed in step 2; then calculate the inner product of the normal vector of the pixel point and the normal vector of the projected mapping point, and then set the difference between the preset constant and the inner product as the inter-frame normal vector consistency cost value of the pixel point in the current frame image, and determine to form a An inner product calculation method is used to determine the inter-frame normal vector consistency cost value of a pixel point; in step 32, the robot uses the normal vector of the adjacent pixel points of each pixel point in the current frame image as the normal vector of the pixel point to be updated; and then according to the inner product calculation method described in step 31, the depth information to be updated is used to calculate the inter-frame normal vector consistency cost value of the adjacent pixel points of the pixel point; in step 33, when the inter-frame normal vector cost value of the adjacent pixel points described in step 32 is less than the inter-frame normal vector cost value of the pixel point described in step 31, the normal vector of the adjacent pixel points of the pixel point is updated to the normal vector of the pixel point; in step 34, steps 31 to 33 are repeated until the number of repetitions reaches the target propagation number.

[0014] Furthermore, in step 01, for each frame of the image, three projected pixel points are used as vertex pixel points to construct a two-dimensional surface; and based on the conversion relationship between the world coordinate system corresponding to the sparse point cloud and the image coordinate system corresponding to the image, a three-dimensional surface matching the constructed two-dimensional surface is determined; when the semantic annotation information of all vertex pixel points included in the two-dimensional surface to which the other pixel points belong is consistent with the semantic annotation information of the other pixel points, the depth value of the three-dimensional point cloud point corresponding to the vertex pixel point in the three-dimensional surface matched by the two-dimensional surface to which the vertex pixel point belongs is used as the initial depth value of the other pixel point; when the semantic annotation information of some vertex pixel points included in the two-dimensional surface to which the other pixel point belongs is consistent with the semantic annotation information of the other pixel points, the initial depth value of the other pixel point is determined based on the initial depth value of the vertex pixel point; wherein, in the two-dimensional surface to which the other pixel point belongs, the robot sets the initial normal vector of the vertex pixel point to be the initial normal vector of the other pixel point.

[0015] Furthermore, in step 02, based on the similarity between the matching window in the current frame image where the pixel point is located and the matching window in the matched neighborhood frame image, the robot uses the initial depth information of the neighboring pixel points of each pixel point in the image to update the initial depth information of the pixel point, and the method for obtaining the updated depth information of the pixel point specifically includes: for each pixel point in the current frame image, determining the first matching window corresponding to the pixel point, which corresponds to the two-dimensional surface patch; using the internal and external parameters of the camera corresponding to the camera, projecting the pixel point to the matched neighborhood frame image, forming a mapping point in the neighborhood frame image and determining the second matching window corresponding to the mapping point, and determining the depth information of the mapping point; then, based on the depth information of the pixel point, calculating the first matching window corresponding to each pixel point in the current frame image and the second matching window corresponding to the pixel point. The robot calculates a first similarity between the second matching window corresponding to the mapping point obtained by point projection; wherein the size of the first matching window is equal to the size of the matching window; the robot uses the initial depth value of each of the adjacent pixel points of the pixel point as the depth value to be updated of the pixel point, and the robot uses the initial normal vector of each of the adjacent pixel points of the pixel point as the normal vector to be updated of the pixel point; then, based on the depth value to be updated of the pixel point and the normal vector to be updated of the pixel point, calculates the second similarity between the first matching window corresponding to each pixel point in the current frame image and the second matching window corresponding to the mapping point obtained by projecting the pixel point; then, when the first similarity is less than the second similarity, the robot selects the depth information whose second similarity meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel point.

[0016] Furthermore, the method of selecting the depth information whose second similarity meets the first preset condition from the depth information to be updated and setting it as the updated depth information of the pixel point includes: selecting the depth information corresponding to the maximum similarity from at least one depth information to be updated as the updated depth information of the pixel point; wherein the maximum similarity is the second similarity that meets the first preset condition; wherein the depth information to be updated includes the normal vector to be updated and the depth information to be updated; the updated depth information includes the updated normal vector and the updated depth information; wherein the adjacent pixel points of each pixel point include: adjacent pixel points arranged in four mutually perpendicular directions with the pixel point as the center of symmetry, so that there are multiple normal vectors to be updated for each pixel point, and there are also multiple depth values to be updated for each pixel point; wherein the pixel points on the preset neighborhood of adjacent pixel points include: adjacent pixel points arranged in four mutually perpendicular directions with the adjacent pixel point as the center of symmetry.

[0017] Furthermore, in step 27, the method of increasing the preset depth information increment by the updated depth information of the pixel point newly obtained by the robot control step 26 to obtain the incrementally adjusted depth information of the pixel point includes: increasing the preset depth value increment by the updated depth value of the pixel point newly obtained by the robot control step 26 to obtain the incrementally adjusted depth value of the pixel point; increasing the preset normal vector increment by the updated normal vector of the pixel point newly obtained by the robot control step 26 to obtain the incrementally adjusted normal vector of the pixel point; then, based on the incrementally adjusted depth value of the pixel point and the incrementally adjusted normal vector of the pixel point, calculating the third joint cost value between each pixel point in the first correlation window in the current frame image and the mapping point projected by the pixel point in the second correlation window; and then setting the depth information whose third joint cost value meets the second preset condition as the fine-tuned depth information of the pixel point; wherein, the greater the third similarity corresponding to the pixel point, the smaller the adjusted photometric consistency cost value of the pixel point.

[0018] Furthermore, the method of setting the depth information whose third similarity meets the second preset condition as the fine-tuned depth information of the pixel point includes: when the second similarity is greater than the first similarity and the third similarity is greater than the second similarity, the robot sets the depth information corresponding to the third similarity as the fine-tuned depth information of the pixel point.

[0019] Furthermore, in step 25, the depth information of the adjacent pixel points of the pixel point is set as the updated depth information of the pixel point, so that the depth information to be updated of the pixel point is updated to the updated depth information of the pixel point. The method includes: for each pixel point in the second texture area in the current frame image, determining the first correlation window corresponding to the pixel point; using the internal and external camera parameters corresponding to the camera, projecting the pixel point to the matching neighborhood frame image to form a mapping point in the neighborhood frame image and determining the second correlation window corresponding to the mapping point, and determining the depth information of the mapping point; then, based on the depth information of the pixel point, calculating a first joint cost value between each pixel point in the first correlation window in the current frame image and the mapping point projected by the pixel point in the second correlation window; wherein, the first The size of the cross-correlation window is larger than the size of the matching window, and the first joint cost value belongs to the joint cost value; the robot uses the depth value of each of the adjacent pixels of the pixel as the depth value to be updated of the pixel, and the robot uses the normal vector of each of the adjacent pixels of the pixel as the normal vector to be updated of the pixel; then, based on the depth value to be updated of the pixel and the normal vector to be updated of the pixel, calculate the second joint cost value between each pixel in the first cross-correlation window in the current frame image and the mapping point projected by the pixel in the second cross-correlation window; then, when the second joint cost value is less than the first joint cost value, the robot selects the depth information whose second joint cost value meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel.

[0020] Furthermore, the method of selecting the depth information whose second joint cost value meets the first preset condition from the depth information to be updated and setting it as the updated depth information of the pixel point includes: selecting the depth information corresponding to the largest joint cost value from at least one depth information to be updated as the updated depth information of the pixel point; wherein the depth information to be updated includes the normal vector to be updated and the depth information to be updated; the updated depth information includes the updated normal vector and the updated depth information; wherein the adjacent pixel points of each pixel point include: adjacent pixel points arranged in four mutually perpendicular directions with the pixel point as the center of symmetry, so that there are multiple normal vectors to be updated for each pixel point, and there are also multiple depth values to be updated for each pixel point.

[0021] Furthermore, in step 02, the robot continues to update the depth value of the pixel point by applying a preset depth value increment to obtain an adjusted depth value of the pixel point; the robot continues to update the normal vector of the pixel point by applying a preset normal vector increment, and the method for obtaining the adjusted normal vector of the pixel point includes: the robot controls the updated depth value of the pixel point to increase the preset depth value increment to obtain the latest depth value of the pixel point; the robot controls the updated normal vector of the pixel point to increase the preset normal vector increment to obtain the latest normal vector of the pixel point; then, based on the incrementally adjusted depth value of the pixel point and the incrementally adjusted normal vector of the pixel point, a third joint cost value is calculated between each pixel point in the first correlation window in the current frame image and the mapping point projected by the pixel point in the second correlation window; and the depth information whose third joint cost value meets the second preset condition is set as the fine-tuned depth information of the pixel point.

[0022] Furthermore, the method of setting the depth information whose third joint cost value meets the second preset condition as the fine-tuned depth information of the pixel point includes: when the second joint cost value is greater than the first joint cost value and the third joint cost value is less than the second joint cost value, the robot sets the depth information corresponding to the third joint cost value as the fine-tuned depth information of the pixel point.

[0023] Furthermore, the photometric consistency cost value of the pixel point is derived from: the similarity between the first target window corresponding to each pixel point in the current frame image and the second target window corresponding to the mapping point projected by the pixel point; wherein, for the pixel point in the first texture area, the first target window is set to the matching window; for the pixel point in the second texture area, the first target window is set to a window with a size larger than the matching window; the method for determining the similarity between the first target window corresponding to each pixel point in the current frame image and the second target window corresponding to the mapping point projected by the pixel point includes: based on the depth value of each pixel point included in the first target window and the depth value of the corresponding mapping point in the second target window, and based on the normal vector of each pixel point included in the first target window and the normal vector of the corresponding mapping point in the second target window, determining the variance information between the first target window and the second target window; based on the The variance information between the first target window and the second target window is used to determine the similarity between the first target window and the second target window, and then the robot sets the difference between the preset constant and the similarity as the photometric consistency cost value of the pixel point in the current frame image; wherein, the greater the similarity between the first target window and the second target window, the smaller the photometric consistency cost value of the corresponding pixel point; the smaller the similarity between the first target window and the second target window, the larger the photometric consistency cost value of the corresponding pixel point; the robot marks the corresponding photometric consistency cost value as the photometric consistency cost value of the pixel point; wherein, the first target window and the second target window are configured to occupy rectangular areas of the same size, and the second target window is formed by projecting the first target window in the current frame image into the matching neighborhood frame image; the current frame image and the matching neighborhood frame image are two frame images of the same size.

[0024] Furthermore, for each pixel point in the current frame image, based on the transformation relationship between the image coordinate system corresponding to the current frame image, the world coordinate system and the image coordinate system corresponding to the matching neighboring frame image, the mapping point of the pixel point projected to the matching neighboring frame image is determined, and the depth information of the mapping point is determined based on the depth information of the pixel point; wherein, a ray is drawn from the camera from the center of the camera through the pixel point in the current frame image, and the intersection of the ray and the three-dimensional surface corresponding to the ray in space is set as the initial depth value of the pixel point, and the normal vector of the three-dimensional surface is the initial normal vector of the pixel point; wherein, the internal and external parameters of the camera corresponding to the camera are the parameters required for the conversion between coordinate systems, so that after the depth information of the pixel points of each frame image taken at different shooting angles is reconstructed into a local point cloud, each local point cloud is converted and assembled together to form three-dimensional dense point cloud data.

[0025] Compared with the existing technology, the robot implements the photometric consistency constraint criterion and the geometric projection consistency constraint criterion for the strong and weak texture areas in the image respectively. It can not only restore the depth information in the weak texture area to the greatest extent, but also ensure that the original reconstructed details in the rich texture area will not be destroyed. This greatly improves the completeness of the reconstructed point cloud and suppresses the fuzzy matching problem in the weak texture area. The weaker texture is expressed by the weighted sum of the photometric consistency cost value and the geometric consistency cost value of the pixel points in a larger window, which improves the robustness of the algorithm. Ultimately, it can successfully restore relatively accurate depth information in the weak texture area, thereby improving the integrity and quality of the reconstructed point cloud for high-resolution images. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 A flowchart of a depth information processing method based on visual reconstruction is disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the drawings in the embodiments of the present invention. To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention and are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementation methods and the advantages of the present invention. Processing or method depicted as a flowchart. Although the flowchart describes each step as a sequential process, many of the steps can be implemented in parallel, concurrently or simultaneously. In addition, the order of the steps can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0028] It should be noted that the terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or apparatuses.

[0029] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0030] The term "and / or" herein simply describes an association relationship, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, and the existence of B alone. In addition, the term "at least one" herein refers to any combination of at least two of any one or more of a plurality of items. For example, "at least one of A, B, and C" can represent any one or more elements selected from the set consisting of A, B, and C.

[0031] In high-resolution images, there are often large areas of weak texture. In order to perform visual three-dimensional reconstruction of high-resolution images, if a traditional patch matching algorithm is run in the weak texture area, fuzzy matching will occur, making it impossible to recover the depth information of these weak texture areas. In some visual reconstruction scenarios, the reconstruction details in the rich texture area will be destroyed, seriously affecting the integrity and quality of the reconstructed point cloud. An embodiment of the present invention discloses a depth information processing method based on visual reconstruction. The depth information processing method is applied to process images obtained by a robot, wherein a camera installed on the robot obtains each frame of image from different shooting angles, and each frame of image is collected from the working scene where the robot is located. The executor of the depth information processing method provided by the embodiment of the present invention is generally a robot with a certain computing power, and a processor is provided inside the robot. In some possible implementations, the depth information processing method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0032] As an example, Figure 1As shown, the depth information processing method includes the following steps: Step S1, the robot adjusts the photometric consistency cost value of the pixel by setting the depth information adjustment amount, and then identifies the type of texture area where the pixel is located based on the adjusted photometric consistency cost value, and marks the type of the pixel, specifically distinguishing whether the pixel is in a texture-rich area or an area with insufficient texture (weak texture area, insufficient texture information); then the robot executes Step S2. The depth information adjustment amount includes an adjustment amount for the depth value and an adjustment amount for the normal vector, and the adjustment amount for the depth value and the adjustment amount for the normal vector belong to different types of depth information adjustment amounts. In some embodiments, the adjustment amount for the depth value mainly changes the photometric consistency cost value of the pixel by increasing or decreasing the depth value of the pixel, and the adjustment amount for the normal vector mainly changes the same photometric consistency cost value of the pixel by adjusting the posture (angle information) of the normal vector of the pixel (a normal vector belonging to a plane in three-dimensional space, with directionality); then, the different types of depth information adjustment amount adjustments are independent, and the corresponding photometric consistency cost value of the pixel is obtained after adjusting the depth information adjustment amount. Therefore, the robot adjusts the photometric consistency cost value of the pixel point by setting an adjustment amount regarding the depth value, and obtains the photometric consistency cost value under the condition of depth value change (including increase or decrease of the depth value), which can include the photometric consistency cost value after the depth value increases and the photometric consistency cost value after the depth value decreases. The robot adjusts the photometric consistency cost value of the pixel point by setting an adjustment amount regarding the normal vector, and obtains the photometric consistency cost value under the condition of normal vector change (including different clockwise rotations of the normal vector), which can include the photometric consistency cost value after the normal vector rotates clockwise and the photometric consistency cost value after the normal vector rotates counterclockwise. Therefore, step S1 fine-tunes the depth value and normal vector of the pixel point in the image area and calculates the photometric consistency cost value after the disturbance to determine whether the pixel point is located in a rich texture area or a weak texture area.

[0033] In some embodiments, the function performed by step S1 can be modularly processed into a texture detector, that is, the program code corresponding to the function performed by step S1 is stored in a device corresponding to the texture detector, or the function performed by step S1 is hardware-implemented into a texture detector using a logic circuit, so that the texture detector is used to detect the type of texture area in the image, distinguish between weak texture areas and texture-rich areas, and improve the versatility of the function performed by step S1.

[0034] Step S2: When the robot identifies a pixel point as being within the first texture region, the robot uses the photometric consistency constraint criterion to process the depth information of the pixel point within the first texture region; when the robot identifies a pixel point as being within the second texture region, the robot uses the photometric consistency cost information and the geometric projection consistency constraint criterion to process the depth information of the pixel point within the second texture region; therefore, regardless of whether the pixel point is within the first texture region or the second texture region, after the robot traverses each captured frame image or repeatedly traverses each captured frame image, it ultimately determines the updated (after iterative correction) depth information of each pixel point, and then executes step S3. The texture information of the first texture region is greater than that of the second texture region.

[0035] In some embodiments, the texture information of the image area framed by the scanning window can be determined based on the pixel difference between the pixel value of the pixel point and the average pixel value, wherein the larger the sum of the pixel differences within the window, the richer the texture included in the window, and vice versa.

[0036] It should be noted that when a pixel point is in a rich texture area, corresponding to when the pixel point is in the first texture area, the deviation of the projection of the pixel point in the neighboring frame image due to slight perturbations will also cause the photometric information of the matching window of the neighboring frame image (the projection area of the patch in the image coordinate system) to change significantly, causing the photometric consistency cost value of the corresponding pixel point to change significantly. The neighboring frame image here serves as a reference image to provide a representation basis for the depth information of the current frame image. Therefore, in this embodiment, for the rich texture area, the calculation result of the depth information under the original patch (pre-set matching window) size is maintained, and only the photometric consistency constraint is applied to restore the depth information of the rich texture area well, avoiding the problem of detail destruction caused by the introduction of geometric projection consistency. In this embodiment, facing the rich texture area, the robot uses the photometric consistency constraint criterion to process the depth information of the pixel points in the rich texture area, selects a smaller matching window to traverse the pixel points in the first texture area, because the smaller matching window can encompass rich texture information and the algorithm has less computational complexity. Then, only the photometric consistency constraint criterion is used to calculate and update the depth information of the pixel points, aiming to retain the depth information of the details in the reconstructed scene to the greatest extent.

[0037] It should be noted that whenever the depth value and normal vector of a pixel are slightly changed, the position of the pixel projected on the neighboring frame image only deviates a small distance from the original projection position; when the pixel is in a weak texture area, corresponding to when the pixel is in a second texture area, the photometric information in the second texture area is homogenized, that is, the photometric information of the patch of the current frame image and the photometric information of the patch of the neighboring frame image are still relatively similar, so the photometric consistency cost will not change much. Therefore, this embodiment does not use the photometric consistency constraint criterion to process the depth information of the pixel points in the weak texture area; wherein the photometric information of the patch includes statistical information of the pixel values in the matching window (which will be associated with the number of pixels and the average pixel value). After the robot identifies the weak texture and rich texture areas, for the weak texture area, the size of the corresponding matching window is increased to enrich the texture information of the single window in the weak texture area, in order to suppress the fuzzy matching problem caused by the weak texture area. Then, this embodiment introduces the geometric consistency constraint criterion based on the epipolar geometry theory to process the pixels in the second texture area, and finally successfully recovers relatively accurate depth information in the weak texture area.

[0038] In step S3, the robot uses the most recently processed depth information from step S2 to perform an inter-frame normal vector consistency constraint, obtaining an inter-frame normal vector consistency cost value for the corresponding pixel point, so that the normal vector of the pixel point in the image is mapped to a three-dimensional point cloud model, wherein the three-dimensional point cloud model is the three-dimensional point cloud information of the scene or object to be reconstructed by the robot, and is a collection of point clouds corresponding to the depth image converted from each frame image. Regardless of whether the pixel point is within the first texture area or the second texture area, after obtaining the normal vector in the most recently iterated depth information in step S2, the robot calculates the normal vector similarity between the pixel point of the current frame image and the pixel point projected into the neighboring frame image in each pair of matching images, and adds it as a consistency constraint to the cost function. The inter-frame normal vector consistency cost value of the corresponding pixel point is then calculated according to the inter-frame normal vector consistency constraint, with the intention of making the normal vector of each pixel point in the image tend to be consistent, so that more accurate depth information can be integrated into the final point cloud. Preferably, the lower the inter-frame normal vector consistency cost value of the pixel point, the more accurate the normal vector of the pixel point.

[0039] In summary, the robot implements photometric consistency constraints and geometric projection consistency constraints for strong and weak texture areas respectively. It can not only restore the depth information in the weak texture area to the greatest extent, but also ensure that the original reconstructed details in the rich texture area will not be destroyed. This greatly improves the completeness of the reconstructed point cloud, greatly suppresses the occurrence of fuzzy matching problems, improves the robustness of the algorithm, and improves the integrity and quality of the reconstructed point cloud for high-resolution images.

[0040] As an embodiment, before executing step S1, the method further includes:

[0041] Step 01: The robot obtains a sparse point cloud from each frame of images taken from different shooting angles. The robot reads the sparse point cloud and obtains image information and internal and external parameters of the camera, including the intrinsic parameter matrix, the rotation matrix and the translation vector. A two-dimensional surface is then constructed from the projected pixel points of the sparse point cloud in the image coordinate system. Multiple two-dimensional surfaces can be constructed, corresponding to matching windows that traverse different image areas of the same frame image. Based on the initial depth value and the initial normal vector of the projected pixel point, the initial depth value of the other pixel points in the two-dimensional surface except the projected pixel point and the initial normal vector of the other pixel points in the two-dimensional surface except the projected pixel point are determined, and then the initial depth value and initial normal vector of all pixel points in each frame image are determined. Then, the robot executes step 02; wherein, each projected pixel point of the sparse point cloud in the image coordinate system is configured with an initial depth value and an initial normal vector; the two-dimensional surface is configured to be within the corresponding image so that all pixels in the image can be traversed using the matching window.

[0042] In the step 01, a sparse point cloud can be formed by sparse points extracted from a two-dimensional image. Sparse points are point cloud points of pixels with sparse features in three-dimensional space, and are configured with initial depth values and initial normal vectors. There is a coordinate transformation relationship between each point included in the sparse point cloud and each projected pixel point in the two-dimensional patch, so that each projected pixel point is configured with the initial depth value and initial normal vector of the corresponding point in the sparse point cloud. In this embodiment, every three points in the sparse point cloud are triangulated to form a plane, and then the plane is assigned to pixels with non-sparse features. Then, each pixel point is assigned to a corresponding two-dimensional patch, and then each pixel point in the two-dimensional patch is configured with an initial depth value and an initial normal vector to achieve pixel initialization. The two-dimensional patch can be regarded as the plane transformed from the world coordinate system to the image coordinate system. The other pixel points in the two-dimensional patch, except for the projected pixel points, include pixel points with non-sparse features, that is, pixel points that have not been configured with initial depth values and initial normal vectors in advance. Specifically, in step 01, for each frame of the image, three projected pixels are used as vertex pixels to construct a two-dimensional patch; and based on the transformation relationship between the world coordinate system corresponding to the sparse point cloud and the image coordinate system corresponding to the image, a three-dimensional patch matching the constructed two-dimensional patch is determined; when the semantic annotation information of all vertex pixels included in the two-dimensional patch to which the other pixel points belong is consistent with the semantic annotation information of the other pixel points, the depth value of the three-dimensional point cloud point corresponding to the vertex pixel point in the three-dimensional patch to which the vertex pixel points belong is used as the initial depth value of the other pixel points; or, when the semantic annotation information of some vertex pixels included in the two-dimensional patch to which the other pixel points belong is consistent with the semantic annotation information of the other pixel points, the initial depth value of the other pixel points is determined based on the initial depth value of the vertex pixel points; wherein, in the two-dimensional patch to which the other pixel points belong, the robot sets the initial normal vector of the vertex pixel point as the initial normal vector of the other pixel points. It should be noted that the initial depth information includes the initial depth value and the initial normal vector, and the depth information includes the depth value and the normal vector.

[0043] In some embodiments, when multiple two-dimensional patches are constructed, an initial depth map for each two-dimensional patch can be determined, and then the two-dimensional patches included in the current frame image are combined in a specific order to obtain an initial depth map for the current frame image.

[0044] For a two-dimensional face, the depth values of the vertex pixels of the two-dimensional face are known. This embodiment can combine the semantic annotation information of other pixels in the two-dimensional face except the vertex pixels and the initial depth values and semantic annotation information of the vertex pixels to determine the initial depth values of other pixels, and then determine the initial depth map of a two-dimensional face based on the initial depth values of other pixels and the initial depth values of the vertex pixels.

[0045] Considering that for a 2D patch, if its three vertex pixels have the same semantic label (corresponding to semantic annotation information) and normal vector, all pixels in the 2D patch are likely to be located on the same scene plane. Otherwise, the plane may have depth discontinuities. Therefore, the semantic annotation information can be combined here to initialize the depth values of other pixels.

[0046] If it is determined that the semantic annotation information of the vertex pixels included in the two-dimensional patch to which the other pixels belong is consistent with the semantic annotation information of the other pixels, the depth value of the three-dimensional point cloud point corresponding to the other pixels in the three-dimensional patch that matches the two-dimensional patch to which they belong is used as the initial depth value of the other pixels. In a specific application, a ray can be drawn from the camera from the optical center through the other pixels in the image, and the intersection of the ray and the corresponding three-dimensional patch in space is found. In this way, the reprojected depth of the intersection is set as the initial depth value of the other pixels.

[0047] When it is determined that the semantic annotation information of some vertex pixels included in the two-dimensional surface to which other pixels belong is consistent with the semantic annotation information of other pixels, the initial depth values of other pixels are determined based on the initial depth values of some vertex pixels.

[0048] In a specific application, if the semantic labels of other pixels are the same as one or two vertices, the initial depth values of other pixels can be set to the average depth of the same semantic vertices plus a random perturbation of the depth range.

[0049] If it is determined that the semantic annotation information of the vertex pixels included in the projected triangle patch of other pixels is inconsistent with the semantic annotation information of other pixels, the initial depth value of the other pixels is determined based on the depth range of the three-dimensional patch matched with the two-dimensional patch. Here, a depth value can be randomly selected from the depth range as the initial depth value of the other pixels.

[0050] It should be noted that the semantic annotation information can be obtained by manual annotation or generated based on a trained semantic neural network. That is, by inputting the image captured by the camera into the trained semantic neural network, the semantic annotation information of each pixel in the current frame image can be determined.

[0051] Step 02: Based on the similarity between the matching window in the current frame image where the pixel is located and the matching window in the matching neighboring frame image, the robot updates the initial depth information of each pixel in the image using the initial depth information of the neighboring pixels of the pixel to obtain the updated depth information of the pixel. The initial depth information of the neighboring pixels of each pixel is pre-configured by the robot, that is, it is determined by the depth information of the sparse point cloud described in Step 01. The robot calculates the updated photometric consistency cost value of the pixel based on the depth information to be updated. The robot calculates the initial photometric consistency cost value of the pixel based on the initial depth information of the pixel, specifically the photometric consistency cost value between the pixel and the mapping point projected in the matching neighboring frame image. If the photometric consistency cost value to be updated for the pixel is less than the initial photometric consistency cost value for the pixel, the photometric consistency cost value to be updated for the pixel is set to the updated photometric consistency cost value for the pixel, and the depth information to be updated for the pixel is set to the updated depth information for the pixel. Otherwise, the initial photometric consistency cost value for the pixel is set to the updated photometric consistency cost value for the pixel, and the initial depth information for the pixel is set to the updated depth information for the pixel, that is, the photometric consistency cost value for the pixel is maintained unchanged, and the depth information for the pixel is maintained unchanged. Therefore, the initial photometric consistency cost value for the pixel is an initial benchmark, and the depth information of adjacent pixels is subsequently used to calculate a new photometric consistency cost value. If the new photometric consistency cost value is less than the initial photometric consistency cost value, the new depth information is used to replace the initial depth information, so that the updated depth information for the pixel tends to continue to update in the direction of decreasing the corresponding photometric consistency cost value. Then the robot updates the pixel points on the preset neighborhood of the adjacent pixel points to the adjacent pixel points, and continues to adaptively update the depth information of the pixel points based on the similarity between the matching window in the current frame image where the updated adjacent pixel points are located and the matching window in the neighborhood frame image matched by it, and repeats this process to implement neighborhood propagation until the robot uses the matching window to traverse all the pixel points in the current frame image; wherein, the pixel points on the preset neighborhood of the adjacent pixel points include: adjacent pixel points set in four mutually perpendicular directions with the adjacent pixel point as the symmetry center, for example, the four adjacent pixel points in the up, down, left and right directions of the adjacent pixel point, and the traversed pixel points can be excluded; in order to maintain neighborhood propagation to traverse all pixel points in the current frame image, the adjacent pixel points of each pixel point can be updated to the pixel point.

[0052] At the end of the neighborhood propagation, it is also determined that the robot has traversed all the pixels in the current frame image using the matching window and calculated the photometric consistency cost of all the pixels. Specifically, the robot has calculated the photometric consistency cost of all the pixels in the current frame image and their projections to the mapping points in the matching neighborhood frame image. The depth information of each pixel is iteratively processed and optimized in the direction of a smaller corresponding photometric consistency cost. Then, in order to further improve the accuracy of the depth information of each pixel, it is necessary to apply a small increment to fine-tune the corresponding depth information. Based on the similarity between the matching window in the current frame image where the updated adjacent pixel is located and the matching window in the matching neighborhood frame image, the robot continues to update the depth value of the pixel by applying a preset depth value increment to obtain the adjusted depth value of the pixel. The robot continues to update the normal vector of the pixel by applying a preset normal vector increment to obtain the adjusted normal vector of the pixel. In step 02, based on the depth value of each pixel point after adding the preset depth value increment and the normal vector of each pixel point after adding the preset normal vector increment, the adjusted photometric consistency cost value of the pixel point is calculated; if the adjusted photometric consistency cost value of the pixel point is less than the photometric consistency cost value of the pixel point before adjustment (corresponding to the photometric consistency cost value of the pixel point at the end of neighborhood propagation), the depth value of the pixel point after adding the preset depth value increment is updated as the depth value of the pixel point, and the normal vector of the pixel point after adding the preset normal vector increment is updated as the normal vector of the pixel point, and the adjusted photometric consistency cost value of the pixel point is updated as the photometric consistency cost value of the pixel point, so that the fine-tuned depth information is more accurate.

[0053] As an embodiment, in step 02, based on the similarity between the matching window in the current frame image where the pixel point is located and the matching window in the matching neighborhood frame image, the robot uses the initial depth information of the adjacent pixels of each pixel point in the image to update the initial depth information of the pixel point, and the method for obtaining the updated depth information of the pixel point specifically includes: for each pixel point in the current frame image, determining the first matching window corresponding to the pixel point, which corresponds to the two-dimensional surface patch; using the internal and external camera parameters corresponding to the camera, projecting the pixel point to the matching neighborhood frame image, forming a mapping point in the neighborhood frame image and determining the second matching window corresponding to the mapping point, and determining the depth information of the mapping point. In some embodiments, a homography matrix can be calculated based on the initial depth value of the pixel point at the center of the first matching window of the current frame image, the initial normal vector of the pixel point at the center of the first matching window, the intrinsic parameter matrix obtained in step 01, the rotation matrix and the translation vector; according to the triangular geometric relationship formed by the epipolar geometry and the camera projection model disclosed in the prior art, the specific calculation method of the homography matrix includes: recording the ratio of the product of the translation vector and the transposed initial normal vector to the initial depth value as the first matrix; then recording the difference between the rotation matrix and the first matrix as the second matrix; then recording the result of multiplying the intrinsic parameter matrix, the second matrix and the inverse matrix of the intrinsic parameter matrix in sequence as the homography matrix; then the coordinates of the pixel point of the first matching window of the current frame image are converted to the matching neighboring frame image through the homography matrix, and the first matching window is also projected into the second matching window in the matching neighboring frame image through the homography matrix, then the coordinates of the mapping point in the second matching window of the neighboring frame image are obtained, and based on the projection relationship generated by the homography matrix, the initial depth information of the pixel point in the first matching window of the current frame image can be converted into the initial depth information of the mapping point in the second matching window of the neighboring frame image. Then, based on the depth information of the pixel point (including the initial depth information of the pixel point) and the depth information of the mapping point (including the initial depth information of the mapping point), the first similarity between the first matching window corresponding to each pixel point in the current frame image and the second matching window corresponding to the mapping point projected by the pixel point is calculated, wherein the robot records the difference between the value 1 (regarded as a preset constant) and the first similarity as the photometric consistency cost value of the pixel point in the first matching window in the current frame image, and the size of the first matching window is equal to the size of the matching window; the first similarity describes the degree of similarity between the first matching window corresponding to each pixel point in the current frame image and the second matching window corresponding to the mapping point projected by the pixel point, which can correspond to the degree of similarity between the photometric information of the first matching window and the photometric information of the second matching window, which is equivalent to the normalized cross-correlation function, and the corresponding value range is preferably [-1, 1].In particular, in order to maximize the preservation of reconstruction, the side length of the first matching window is 0.0035 times the length of the longest side of the current frame image, wherein the shape of the matching window is a square. Then, on this basis, the robot uses the initial depth value of each of the adjacent pixels of the pixel as the depth value to be updated of the pixel, and the robot uses the initial normal vector of each of the adjacent pixels of the pixel as the normal vector to be updated of the pixel; then, based on the depth value to be updated of the pixel and the normal vector to be updated of the pixel, the robot calculates the second similarity between the first matching window corresponding to each pixel in the current frame image and the second matching window corresponding to the mapping point obtained by projecting the pixel, wherein the greater the second similarity corresponding to the pixel, the smaller the photometric consistency cost value to be updated calculated for the pixel; then, when the first similarity is less than the second similarity, the robot selects the depth information to be updated whose second similarity meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel. In some embodiments, the method of selecting the depth information whose second similarity meets the first preset condition from the depth information to be updated and setting it as the updated depth information of the pixel point includes: selecting the depth value corresponding to the maximum similarity (the depth value and normal vector of the pixel point corresponding to the maximum second similarity) from at least one depth information to be updated as the updated depth information of the pixel point, which belongs to the depth information with the largest second similarity among all the selected depth information to be updated; wherein the maximum similarity is the second similarity that meets the first preset condition, the depth information to be updated includes the normal vector to be updated and the depth information to be updated, and the updated depth information includes the updated normal vector and the updated depth information; in some embodiments, selecting the depth information to be updated from the depth information to be updated The method for setting the depth information whose second similarity meets the first preset condition as the updated depth information of the pixel point includes: selecting the depth information whose second similarity is greater than the preset similarity threshold from at least one depth information to be updated as the updated depth information of the pixel point, which belongs to the depth information to be updated and whose second similarity is greater than the preset similarity threshold; wherein the adjacent pixel points of each pixel point include: adjacent pixel points arranged in four mutually perpendicular directions with the pixel point as the center of symmetry, so that there are multiple normal vectors to be updated for each pixel point, and there are also multiple depth values to be updated for each pixel point. For example, the adjacent pixel points of the pixel point include adjacent pixel points in the upper, lower, left and right directions, a total of 4 adjacent pixel points, which can also be recorded as 4 neighborhood pixel points.

[0054] In some embodiments, for each pixel point in the current frame image, the matching window is configured to be centered on the pixel point and determine a window with a specific square area size corresponding to the pixel point. Preferably, the side length of the matching window is 0.0035 times the length of the longest side of the current frame image; or the matching window is configured to start from a pixel point traversal starting point and traverse all pixel points in the current frame image row by row and column by sliding along a predetermined direction.

[0055] Specifically, in step 02, the robot continues to update the depth value of the pixel by applying a preset depth value increment to obtain the adjusted depth value of the pixel; the robot continues to update the normal vector of the pixel by applying a preset normal vector increment to obtain the adjusted normal vector of the pixel. The method for obtaining the adjusted normal vector of the pixel includes: the robot controls the updated depth value of the pixel to increase the preset depth value increment to obtain the latest depth value of the pixel; the robot controls the updated normal vector of the pixel to increase the preset normal vector increment to obtain the latest normal vector of the pixel; and then, based on the latest depth value of the pixel and the latest normal vector of the pixel, calculate The third similarity between the first matching window corresponding to each pixel in the current frame image and the second matching window corresponding to the mapping point projected by the pixel is calculated; the depth information whose third similarity meets the second preset condition is then set as the fine-tuned depth information of the pixel, including the fine-tuned depth value of the pixel and the fine-tuned normal vector of the pixel; preferably, the value range of the preset depth value increment is: less than or equal to 0.05 times the depth value before fine-tuning, and greater than or equal to -0.05 times the depth value before fine-tuning; the value range of the preset normal vector increment is: less than or equal to 0.02 times the normal vector before fine-tuning, and greater than or equal to -0.02 times the normal vector before fine-tuning. In step 02, the greater the third similarity corresponding to the pixel, the smaller the adjusted photometric consistency cost value of the pixel. Specifically, the robot records the difference between the value 1 (regarded as a preset constant) and the third similarity as the photometric consistency cost value of the pixel point in the first matching window in the current frame image, which can be marked as the third photometric consistency cost value of the pixel point, where the size of the first matching window is equal to the size of the matching window; the third similarity describes the degree of similarity between the first matching window corresponding to each fine-tuned pixel point in the current frame image and the second matching window corresponding to the mapping point obtained by projecting the pixel point, which can correspond to the degree of similarity between the fine-tuned photometric information of the first matching window and the fine-tuned photometric information of the second matching window.

[0056] Furthermore, the method of setting the depth information whose third similarity meets the second preset condition as the fine-tuned depth information of the pixel includes: when the second similarity is greater than the first similarity and the third similarity is greater than the second similarity, the robot sets the depth information corresponding to the third similarity as the fine-tuned depth information of the pixel. Specifically, the robot repeatedly uses the initial depth information of the neighboring pixels of each pixel in the image (or the current depth information of the neighboring pixels) to update the initial depth information of the pixel (or the current depth information of the pixel), and when the calculated second similarity is too large, the robot selects the depth information whose second similarity meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel, until the neighborhood propagation is completed, and the robot continues to compare the newly obtained second similarity with the third similarity. When the third similarity is greater than the second similarity, the robot sets the depth information corresponding to the third similarity as the fine-tuned depth information of the pixel.

[0057] It should be noted that in order to cooperate with the calculation of the similarity between the first matching window corresponding to each pixel point in the current frame image and the second matching window corresponding to the mapping point projected by the pixel point, as well as the photometric consistency cost value of the pixel point, each time the depth information of the pixel point is updated, a new homography matrix is calculated based on the updated depth information of the pixel point and the internal and external parameters of the camera. Then, the pixel point is converted to the matching neighborhood frame image through the homography matrix to obtain a new mapping point, which corresponds to the updated pixel point in the first matching window of the current frame image projected to the mapping point in the second matching window of the matching neighborhood frame image. Based on the projection relationship generated by the homography matrix, the depth information of the new mapping point can be converted. Then, based on the updated depth information of the pixel point and the depth information of the new mapping point, the similarity between the first matching window corresponding to the pixel point after the depth information is updated and the second matching window corresponding to the mapping point projected by the pixel point is calculated, and then the photometric consistency cost value of the pixel point after the depth information is updated is calculated.

[0058] Step 03: Repeat step 02 until the number of executions reaches the preset target number, and determine the neighborhood propagation that has actually been executed for the preset target number of times, where the number of repeated executions is the difference between the preset target number and the value 1. Then execute step 1; wherein the preset target number is associated with the number of corner points of the image; within a limited number of repeated executions, each time the robot executes step 02, the robot will choose to start from a new traversal starting point to process the depth information of the pixel points within the matching window of the current frame image, or control the matching window to slide from a new traversal starting point within the image to cover the same batch of pixel points in the current frame image; wherein, for the same frame image, the pixel point traversal starting point set by the robot for each two consecutive executions of step 02 is symmetrical about the center of the frame image. Preferably, the preset target number is set to 4 times, and the robot repeats step 02 3 times to converge the depth information of each pixel to the optimal value and guide the photometric consistency cost value of the pixel to change in a smaller direction; the shape of the current frame image is a rectangle, and the robot starts traversing from the pixel point in the lower right corner of the current frame image when executing an odd number of times, and starts traversing from the pixel point in the upper left corner of the current frame image when executing an even number of times.

[0059] Combining steps 01 to 03, the process by which the robot determines the depth information of a pixel includes an initialization process for determining the initial depth information from the sparse point cloud data as a benchmark, a comparison of the photometric consistency cost values based on the pixels, a neighborhood propagation process for updating the depth information using adjacent pixels, and a fine-tuning process for updating the depth information by applying an increment of the depth information. The initialization process provides a more accurate basis for updating the depth information, the neighborhood propagation process can update the depth information by combining the photometric consistency cost value verification method of adjacent pixels, and the fine-tuning process optimizes the randomness of the depth information by applying interference, making the final fine-tuned depth information for the pixel more accurate, and further making the three-dimensional point cloud model reconstructed from the image composed of pixels more accurate.

[0060] Based on the above embodiment, step S1 specifically includes:

[0061] After executing step 03, the robot's newly acquired pixel depth information includes the adjusted depth value and the adjusted normal vector of the pixel, making it the optimized result obtained by repeatedly executing step 02. The depth information adjustment amount includes the depth value adjustment amount and the angle adjustment amount.

[0062] The robot controls the adjusted depth value of each pixel point to increase the depth value adjustment amount, and keeps the adjusted normal vector of each pixel point unchanged, and then calculates the photometric consistency cost value of the pixel point, and records it as the first photometric consistency cost value; preferably, the adjusted depth value of the pixel point increases by a depth value adjustment amount to become 1.2 times the adjusted depth value of the pixel point, which is recorded as the depth value of the pixel point after the depth value adjustment amount is increased, and then based on the depth value of the pixel point after the depth value adjustment amount is increased and the adjusted normal vector of the pixel point, the first photometric consistency cost value of the pixel point is calculated.

[0063] At the same time, the robot controls the adjusted depth value of each pixel to reduce the depth value adjustment amount, and keeps the adjusted normal vector of each pixel unchanged, and then calculates the photometric consistency cost value of the pixel, and records it as the second photometric consistency cost value; preferably, the adjusted depth value of the pixel is reduced by a depth value adjustment amount to become 0.8 times the adjusted depth value of the pixel, which is recorded as the depth value of the pixel after the depth value adjustment amount is increased, wherein the depth value adjustment amount is equal to 0.2 times the adjusted depth value of the pixel; then based on the depth value of the pixel after the depth value adjustment amount is reduced and the adjusted normal vector of the pixel, the second photometric consistency cost value of the pixel is calculated.

[0064] At the same time, the robot uses the line connecting the camera center and the pixel point (the pixel point of the current frame image) as the first rotation axis, controls the adjusted normal vector of the pixel point to rotate clockwise along the first rotation axis by the angle adjustment amount, and keeps the depth value of the pixel point unchanged, and then calculates the photometric consistency cost value of the pixel point and records it as the third photometric consistency cost value. Preferably, the angle adjustment amount is 25 degrees.

[0065] At the same time, the robot uses the line connecting the camera center and the pixel point (the pixel point of the current frame image) as the second rotation axis, controls the adjusted normal vector of the pixel point to rotate counterclockwise along the second rotation axis by the angle adjustment amount, and keeps the depth value of the pixel point unchanged, and then calculates the photometric consistency cost value of the pixel point and records it as the fourth photometric consistency cost value. Preferably, the angle adjustment amount is 25 degrees.

[0066] Then, the robot determines whether the sum of the first photometric consistency cost value, the second photometric consistency cost value, the third photometric consistency cost value and the fourth photometric consistency cost value is less than the preset cost value threshold. If so, the robot identifies that the texture area where the pixel point is located is the second texture area and marks the pixel point as a pixel point in the second texture area; otherwise, the robot identifies that the texture area where the pixel point is located is the first texture area and marks the pixel point as a pixel point in the first texture area; wherein, the sum of the first photometric consistency cost value, the second photometric consistency cost value, the third photometric consistency cost value and the fourth photometric consistency cost value can be regarded as the photometric consistency change feature of the pixel point of the current frame image and the pixel point projected to the pixel point of the neighboring frame. In some embodiments, the preset cost value threshold is set to a value of 1, the first texture area is an image area with rich texture, and the second texture area is an image area with weak texture, so as to identify the type of texture area to which the pixel point belongs according to the change characteristics of the corresponding photometric consistency cost value (including the photometric consistency cost value change caused only by the depth value adjustment amount, and the photometric consistency cost value change caused only by the angle adjustment amount); wherein the texture information of the first texture area is more than the texture information of the second texture area; it should be noted that in the process of the robot setting the depth information adjustment amount to adjust the depth information of the pixel point, the photometric consistency cost value of the pixel point is calculated by first calculating the similarity between the matching window corresponding to the pixel point in the current frame image (the pixel point after the depth information is applied with the depth information adjustment amount) and the matching window corresponding to the mapping point projected by the pixel point in the matching neighboring frame image, and then subtracting the similarity using a preset constant to obtain the photometric consistency cost value of the pixel point; wherein the current frame image and the matching neighboring frame image are a pair of matching images obtained by the robot. Therefore, this embodiment is applied to each pair of matching images. By fine-tuning the depth value and normal vector of the pixel points in the area, the fine-tuned photometric consistency cost value is calculated to determine whether the pixel point is located in a rich texture area or a weak texture area, thereby obtaining the photometric consistency change characteristics of the pixel point of the current frame image and the pixel point projected to the pixel point of the neighboring frame, and then based on the photometric consistency change characteristics, accurately identify whether the pixel point is located in a weak texture area.

[0067] As an embodiment, in step S2, when the robot identifies that the pixel is a pixel in the first texture region, the method for the robot to process the depth information of the pixel in the first texture region using the photometric consistency constraint criterion specifically includes: when the robot identifies that the pixel is a pixel in the first texture region through step S1, the robot sets the adjusted depth value and the adjusted normal vector of the pixel most recently obtained in step 03 as the reference depth information of the pixel in the first texture region, and sets the adjusted photometric consistency cost value of the pixel most recently obtained in step 03 as the reference photometric consistency cost value of the pixel in the first texture region. At this time, the robot has calculated the photometric consistency cost value of each pixel in all image regions, which are all photometric consistency cost values between each pixel most recently obtained in step 03 and its projection into the matching neighboring frame image. The reference depth information of the pixel includes the reference depth value and the reference normal vector of the pixel; specifically, the reference depth value of the pixel in the first texture region is the adjusted depth value of the pixel, and the reference normal vector of the pixel in the first texture region is the adjusted normal vector of the pixel.

[0068] Since the acquisition scene where the robot's camera is located is relatively complex, such as occlusion, weak texture, and strong reflection, the depth value and normal vector calculated by the pixel point in the current frame image are not unique, and need to be corrected with the help of the depth information of the pixel point in the neighboring frame image. Therefore, in step S2, when the robot recognizes that the pixel point is a pixel point in the first texture area, for each pixel point in the first texture area of the current frame image, in fact, for each pixel point in the first texture area of the current frame image, the camera internal and external parameters corresponding to the camera are used to project the pixel point to the matching neighboring frame image to form a mapping point in the neighboring frame image, and based on the baseline depth information of the pixel point, the pixel point is projected to the matching neighboring frame image. The robot determines the depth information of the mapping point based on the information, and then configures the newly obtained depth information of the mapping point as the depth information to be updated for the pixel point, which can be understood as the depth information of the pixel point used to temporarily calculate the photometric consistency cost value, that is, as the depth information of the pixel point of the current frame image to participate in the calculation of the photometric consistency cost value, but the depth information to be updated for the pixel point cannot replace the baseline depth information of the pixel point; then, based on the depth information to be updated for the pixel point (serving as the depth information of the pixel point within the matching window of the current frame image) and the depth information of the mapping point (serving as the depth information of the pixel point within the matching window of the neighboring frame image), the new photometric consistency cost value of the pixel point is calculated. It can be understood that the calculation method of the photometric consistency cost value between the pixel point of the matching window in the first texture area of the current frame image and the mapping point of the matching neighboring frame to which it is projected does not change, and what changes is the depth information of the pixel point of the matching window in the first texture area of the current frame image; wherein, the determination of the depth information of the mapping point based on the baseline depth information of the pixel point and the calculation method of the photometric consistency cost value are specifically referred to the related implementation content in step S2, which will not be repeated here.

[0069] In some embodiments, the matching neighborhood frame is a target image among multiple candidate neighborhood frame images corresponding to the current frame image, but the target image may not remain consistent when the robot repeatedly executes step S2, especially the posture transformation relationship between the target image and the current frame image may not remain consistent, so that the depth information of the mapping point projected to the matching neighborhood frame each time step S2 is executed may not be the same.

[0070] Then, when the new photometric consistency cost value of the pixel is less than the baseline photometric consistency cost value of the pixel, the depth information to be updated of the pixel is updated to the baseline depth information of the pixel in the first texture area, which can be marked as the updated baseline depth value of the pixel; and the new photometric consistency cost value of the pixel is updated to the baseline photometric consistency cost value of the pixel in the first texture area, which can be marked as the updated baseline photometric consistency cost value of the pixel. When the new photometric consistency cost value of the pixel is greater than or equal to the baseline photometric consistency cost value of the pixel, the baseline depth information of the pixel in the first texture area remains unchanged, and the baseline photometric consistency cost value of the pixel in the first texture area also remains unchanged. It should be noted that when the robot identifies a pixel as being within the first texture region, it can repeatedly utilize the photometric consistency constraint to process the depth information of the pixel within the first texture region, and use the depth information of the mapping points of the matching neighboring frame to update the depth information of the pixel within the matching window of the current frame image multiple times. In particular, a new update is performed as the matching window slides within the first texture region, until the current frame image is completely traversed (the image traversed by the matching window at a subsequent moment can be updated as the current frame image) or the same frame image is repeatedly traversed multiple times from different traversal starting points (the traversal starting point changes with the number of iterations), and at least all pixels within the first texture region are traversed. Thus, not only is the robot's traversal of the entire frame image more comprehensive, but the baseline depth information of the pixels within the first texture region becomes more accurate after the update, and each pixel of the matching window within the first texture region becomes more accurate after the update. Therefore, the robot maintains the calculation results of the depth information at the original patch (matching window) size, and can well restore the depth information of the rich texture area by only applying the photometric consistency constraint criterion, avoiding the detail destruction problem caused by the introduction of the geometric projection consistency constraint criterion.

[0071] As an embodiment, the method for the robot to determine the depth information of a pixel point in a first texture region using a photometric consistency constraint criterion further includes: when the above embodiment determines that the new photometric consistency cost value of the pixel point is less than the base photometric consistency cost value of the pixel point, the robot controls the updated base depth value of the pixel point (after the depth value of the mapped point is updated) to increase the base depth value increment to obtain the fine-tuned depth value of the pixel point; the robot controls the updated base normal vector of the pixel point (after the normal vector of the mapped point is updated) to increase the base normal vector increment to obtain the fine-tuned normal vector of the pixel point. In some embodiments, when the above embodiment determines that the new photometric consistency cost value of the pixel point is greater than or equal to the base photometric consistency cost value of the pixel point, the robot controls the base depth value of the pixel point (not updated by the depth value of the mapped point) to increase the base depth value increment to obtain the fine-tuned depth value of the pixel point; the robot controls the base normal vector of the pixel point (not updated by the normal vector of the mapped point) to increase the base normal vector increment to obtain the fine-tuned normal vector of the pixel point.

[0072] Then, the fine-tuned photometric consistency cost value of the pixel is calculated based on the fine-tuned depth value of the pixel and the fine-tuned normal vector of the pixel. When the fine-tuned photometric consistency cost value of the pixel is less than the baseline photometric consistency cost value of the pixel, the fine-tuned depth value of the pixel is updated to the baseline depth value of the pixel in the first texture area, the fine-tuned normal vector of the pixel is updated to the baseline normal vector of the pixel in the first texture area, and the fine-tuned photometric consistency cost value of the pixel is updated to the baseline photometric consistency cost value of the pixel in the first texture area. Since the pixel is in a rich texture area, in order to avoid a large amount of disturbance in the depth information causing its projection in the neighboring frame image to deviate too much, this embodiment sets the baseline depth value increment and the baseline normal vector increment to be relatively small to prevent the photometric consistency cost value of the pixel from changing significantly.

[0073] In some embodiments, for each pixel point of the current frame image, the neighborhood frame image that can be transformed to reach more than one frame through the homography matrix corresponding to the depth information of the pixel point is more than one frame, so multiple candidate neighborhood frame images can be obtained from the current frame image, and then, for each candidate neighborhood image of the current frame image, a first matching weight is determined based on the degree of matching between the camera internal and external parameters corresponding to the candidate neighborhood image and the camera internal and external parameters corresponding to the current frame image, and the semantic annotation information of the same target object corresponding to the candidate neighborhood image and the current frame image is determined, and a second matching weight is determined based on the degree of matching between the determined semantic annotation information of the same target object; based on the first matching weight and the second matching weight, a total matching weight corresponding to each candidate neighborhood image is determined; and a candidate neighborhood image with a total matching weight value greater than a preset threshold is selected as the matched neighborhood frame image.

[0074] Here, when determining neighboring frames that match pixels in the current frame, a first matching weight can be determined based on the degree of matching between camera internal and external parameters, and a second matching weight based on the degree of matching between semantic annotation information of the same target object. By considering the degree of matching between semantic annotation information, the selected matching neighboring frames can focus on the same target object related to the scene. This allows the subsequently reconstructed point cloud data to focus more on these target objects, improving the integrity of the 3D reconstruction.

[0075] The matching neighborhood frame image in this embodiment is used as a reference image to provide a basis for depth characterization of the current frame image. The semantic annotation information can provide a strong data basis for depth characterization, which is mainly due to the fact that the depth values corresponding to the pixels of similar or identical semantic labels are usually consistent. Therefore, in addition to determining the first matching weight of the two images (i.e., the current frame image and the candidate neighborhood image) based on the degree of matching between the internal and external parameters of the camera, the second matching weight of the two images can also be determined based on the degree of matching between the semantic annotation information, and then the total matching weight corresponding to each candidate neighborhood image is determined by multiplication or addition of the two matching weights. A candidate neighborhood image with a sufficiently large total matching weight can be selected as the matching neighborhood frame image.

[0076] As an embodiment, in step S2, when the robot recognizes that the pixel point is a pixel point within the second texture region through step S1, the method for the robot to process the depth information of the pixel point within the second texture region using the photometric consistency cost information and the geometric projection consistency constraint criterion includes the following steps:

[0077] Step 21: The robot increases the size of the matching window to obtain a cross-correlation window, and sets the cross-correlation window to be a window that traverses pixels from a preset traversal starting point within the second texture region of the image, so that the photometric consistency cost value of the pixel points within the second texture region is represented by the photometric consistency cost value of the corresponding pixel points within the cross-correlation window, and then executes step 22. In this embodiment, the cross-correlation window covers more texture information within the second texture region than the matching window. Since the texture information of the second texture region is less than that of the first texture region and is a weak texture region, the use of the cross-correlation window in the second texture region of the image does not significantly increase the computational complexity and can also suppress fuzzy matching problems caused by weak texture regions. In this embodiment, the pixels within the second texture region and their mapping points in the matched neighboring frame image are framed by the corresponding cross-correlation window, so that the pixels within the second texture region of the current frame image are derived from the pixels within the cross-correlation window, and the cross-correlation window supports the projection of the second texture region of the current frame image to a square area in the matched neighboring frame image.

[0078] Preferably, when the side length of the matching window is 0.0035 times the length of the longest side of the current frame image, the side length of the cross-correlation window is set to 0.005 times the length of the longest side of the current frame image, thereby ensuring the ability to describe details without increasing the amount of computation. In any frame image, when calculating the photometric consistency cost value for each pixel point in the second texture area, the calculation is performed within the image area size framed by the cross-correlation window. The calculation of the photometric consistency cost value for each pixel point in the second texture area can be recorded as the calculation of the photometric consistency cost value for each pixel point in the cross-correlation window.

[0079] Step 22: Based on the depth information of the pixel points obtained in step 03, calculate the photometric consistency cost value of each pixel point in the cross-correlation window; at the same time, based on the depth information of the pixel points obtained in step 03, calculate the geometric consistency cost value of each pixel point in the cross-correlation window; then execute step 23; wherein, the value range of the geometric consistency cost value of the pixel point is equal to the value range of the photometric consistency cost value of the pixel point; the photometric consistency cost information includes the photometric consistency cost value of the pixel points in the second texture area.

[0080] In step 22, the method for calculating the geometric consistency cost value of each pixel point in the cross-correlation window based on the depth information of the pixel point obtained in step 03 includes: the robot uses the camera internal and external parameters corresponding to the camera to project the pixel point of the cross-correlation window in the second texture area of the current frame image to the matching neighborhood frame image to obtain the mapping point (specifically the two-dimensional coordinate of the mapping point) and its depth information; wherein, referring to step 02 involving the homography matrix calculation method, according to the epipolar geometry and camera projection model, the camera internal and external parameters corresponding to the camera can be used to calculate the homography matrix required for each pixel point of the cross-correlation window in the second texture area of the current frame image, and then the coordinates of the pixel point in the second texture area of the current frame image are projected to the matching neighborhood frame image through the corresponding homography matrix to obtain the coordinates of the mapping point and the depth information of the mapping point; then based on the homography matrix and the depth information of the mapping point, the mapping point is projected Back mapping the current frame image, specifically, projecting the neighboring frame pixel point at the projection position of the pixel point in the matched neighboring frame image back into the second texture area of the current frame image or into the cross-correlation window to obtain the inverse mapping point and its depth information; in some embodiments, without considering the influence of ambient lighting conditions, the depth information of the mapping point is free of noise factors, then in the current frame image, the coordinate position of the inverse mapping point and the coordinate position of the pixel point are coincident, but the depth information converted by the camera projection model in the actual implementation scenario is with noise factors; then it is necessary to calculate the Euclidean distance between the pixel point and the corresponding inverse mapping point, which is equal to the straight-line distance between the two-dimensional coordinates of the pixel point and the two-dimensional coordinates of the corresponding inverse mapping point; in order to unify the value range of the geometric consistency cost value and the value range of the photometric consistency cost value, the robot normalizes the Euclidean distance according to the size of the current frame image to obtain the geometric consistency cost value of the pixel point.

[0081] In some embodiments, the Euclidean distance normalization method includes: the ratio of the Euclidean distance between the pixel point and the corresponding inverse mapping point to 0.5 times the length of the longest side of the current frame image; within the allowable error range, the Euclidean distance between the pixel point and the corresponding inverse mapping point rarely exceeds the length of the longest side of the current frame image; when the geometric consistency cost value is greater than the value 2, the geometric consistency cost value is set to the value 2, so that the value range of the geometric consistency cost value is consistent with the value range of the photometric consistency cost value, that is, unified to a numerical range greater than or equal to 0 and less than or equal to 2.

[0082] In step 22, in order to calculate the photometric consistency cost value of each pixel point in the cross-correlation window (which can also be understood as each pixel point traversed by the cross-correlation window), the robot first calculates the similarity between the first cross-correlation window corresponding to each pixel point in the second texture area of the current frame image and the second cross-correlation window corresponding to the mapping point obtained by projecting the pixel point in the matching neighboring frame image, wherein the first cross-correlation window and the second cross-correlation window both belong to the cross-correlation window; then the robot records the difference between the value 1 (considered as a preset constant) and the similarity (the similarity between the first cross-correlation window and the second cross-correlation window) as the photometric consistency cost value of the pixel point in the first cross-correlation window of the second texture area of the current frame image, and the size of the first cross-correlation window is larger than the size of the matching window. The robot will begin to calculate the photometric consistency cost value of each pixel point in the cross-correlation window in the second texture area based on the initial depth information of the pixel point.

[0083] Step 23. The robot sets a first joint weight and applies it to the photometric consistency cost value of each pixel in the cross-correlation window, and sets a second joint weight and applies it to the geometric consistency cost value of each pixel in the cross-correlation window; then uses the first joint weight and the second joint weight to calculate the weighted sum of the photometric consistency cost value and the geometric consistency cost value of each pixel in the cross-correlation window, and sets the weighted sum as the joint cost value of the pixel; preferably, the sum of the first joint weight and the second joint weight is equal to 1, and the second joint weight is set to 0.25 to meet the matching degree between the photometric consistency cost value and the geometric consistency cost value of the same pixel. In some embodiments, in order to obtain a smaller joint cost value, the first joint weight and the second joint weight can be adaptively adjusted to achieve a specific similarity requirement between a pair of matching images; then execute step 24.

[0084] Step 24: The robot sets the depth information of the adjacent pixel points of each pixel point in the cross-correlation window (which can also be regarded as each pixel point traversed by the cross-correlation window) as the depth information to be updated of the pixel point, and then calculates the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel points of the pixel point based on the depth information to be updated, and then uses the first joint weight and the second joint weight to calculate the weighted sum of the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel point, and sets the weighted sum value as the joint cost value of the adjacent pixel point; then executes step 25.

[0085] Among them, when the photometric information of the pixel points in the cross-correlation window (including pixel value information and the number of pixels) has a greater influence on the expression of image details in the second texture area (the specific spatial position relationship of the point cloud), the first joint weight is increased; when the geometric information of the pixel points in the cross-correlation window (including the Euclidean distance between the pixel points in a pair of matching images) has a greater influence on the expression of image details in the second texture area (the specific spatial position relationship of the point cloud), the second joint weight is increased.

[0086] It should be noted that the adjacent pixel points of each pixel point include: adjacent pixel points arranged in four mutually perpendicular directions with the pixel point as the center of symmetry, so that the number of normal vectors to be updated for each pixel point is 4, and the number of depth values to be updated for each pixel point is also 4. For example, the adjacent pixel points of a pixel point include the adjacent pixel points above, below, left and right, a total of 4 adjacent pixel points, which can also be recorded as 4 neighborhood pixel points. In some embodiments, for each pixel point in the second texture area in the current frame image, the correlation window is configured to determine a window with a specific square area size corresponding to the pixel point with the pixel point in the second texture area as the center, and the correlation window will also cover the image area outside the second texture area in the current frame image during the process of traversing the second texture area, from which the adjacent pixel points of each pixel point in the correlation window can be obtained, regardless of whether the depth information of the pixel point is updated; when the shape of the correlation window and the shape of the matching window are both squares, the side length of the correlation window is greater than the side length of the matching window; the correlation window is configured to start from a pixel point traversal starting point in the second texture area, and traverse all pixel points in the second texture area row by row and column by sliding along a predetermined direction.

[0087] Step 25: When the joint cost value of the adjacent pixel points described in step 24 is less than the joint cost value of the pixel point, the depth information of the adjacent pixel points of the pixel point is set to the updated depth information of the pixel point, so that the depth information to be updated of the pixel point is updated to the updated depth information of the pixel point, and the photometric consistency cost value of the adjacent pixel points described in step 24 is updated to the photometric consistency cost value of the pixel point, and the geometric consistency cost value of the adjacent pixel points described in step 24 is updated to the geometric consistency cost value of the pixel point, and then the joint cost value of the adjacent pixel points described in step 24 is updated to the joint cost value of the pixel point. Step 25 is faced with multiple adjacent pixels. When the depth information of a point is available, according to the updating method of the initial depth information of the pixel point in step 02 of the aforementioned embodiment, based on the similarity between the cross-correlation window in the current frame image where the pixel point is located and the cross-correlation window in the neighboring frame image matched by it, the depth information of the adjacent pixel points of the pixel point is set as the updated depth information of the pixel point. For the specific updating method, please refer to the relevant content of step 02. The difference is that the matching window of step 02 is replaced by a cross-correlation window of a larger size covering the second texture area, so as to realize the use of the cost information of photometric consistency and the geometric projection consistency constraint criterion to process the depth information of the pixel points in the second texture area; then execute step 26.

[0088] In some embodiments, the robot uses the initial depth information of the neighboring pixels of each pixel in the cross-correlation window (which can also be considered as each pixel traversed by the cross-correlation window) as the depth information to be updated for the pixel, and calculates the joint cost value of the neighboring pixels according to the calculation method disclosed in step 24. If the joint cost value of the neighboring pixels is less than the joint cost value of the pixel, the initial depth information of the neighboring pixels of the pixel is updated to the depth information of the pixel as the updated depth information of the pixel, wherein the depth information of the pixel obtained in step 03 includes the initial depth information of the neighboring pixels of each pixel in the cross-correlation window (which can also be considered as each pixel traversed by the cross-correlation window). Then, based on this, the depth value of each pixel in the second texture area of the current frame image is updated. Considering that the neighboring pixels of a pixel often have depth consistency with the pixel in the same type of texture area, more accurate depth information can be determined by spatial propagation (corresponding to the neighborhood propagation mentioned in the previous embodiment). Therefore, it is necessary to repeat steps 24 to 25 to implement neighborhood propagation.

[0089] Specifically, in step 25, the method of setting the depth information of the adjacent pixels of the pixel as the updated depth information of the pixel so that the depth information to be updated of the pixel is updated to the updated depth information of the pixel specifically includes:

[0090] For each pixel point in the second texture area in the current frame image, determine the first correlation window corresponding to the pixel point, which belongs to the correlation window and can be a window centered on the pixel point and larger in size than the matching window; use the internal and external parameters of the camera corresponding to the camera to project the pixel point to the matching neighborhood frame image to form a mapping point in the neighborhood frame image and determine the second correlation window corresponding to the mapping point, and determine the depth information of the mapping point; then, based on the depth information of the pixel point, calculate the first joint cost value between each pixel point in the first correlation window in the current frame image and the mapping point projected by the pixel point in the second correlation window, recorded as the joint cost value of the pixel point in the first correlation window, wherein the size of the first correlation window is larger than the size of the matching window, and the first joint cost value belongs to the joint cost value; the robot calculates the depth value of each of the adjacent pixel points of the pixel point (whose initial value is obtained in step 03) as the depth value to be updated of the pixel point, the robot uses the normal vector of each of the adjacent pixel points of the pixel point as the normal vector to be updated of the pixel point (whose initial value is obtained in step 03); then, based on the depth value to be updated of the pixel point and the normal vector to be updated of the pixel point, calculate the second joint cost value between each pixel point in the first cross-correlation window in the current frame image and the mapping point projected by the pixel point in the second cross-correlation window, wherein the second joint cost value also belongs to the joint cost value, preferably, the depth information corresponding to the second joint cost value is different from the depth information corresponding to the first joint cost value; then, when the second joint cost value is less than the first joint cost value, the robot selects the depth information whose second joint cost value meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel point. Among them, the method of selecting the second joint cost value from the depth information to be updated and setting the depth information that meets the first preset condition as the updated depth information of the pixel point includes: selecting the depth information corresponding to the largest joint cost value from at least one depth information to be updated as the updated depth information of the pixel point; wherein, the largest joint cost value is the second joint cost value that meets the first preset condition; wherein, the adjacent pixel points of each pixel point include: adjacent pixel points set in four mutually perpendicular directions with the pixel point as the center of symmetry, so that there are multiple normal vectors to be updated for each pixel point, and there are also multiple depth values to be updated for each pixel point.

[0091] Step 26, repeat steps 24 to 25 until the number of repetitions reaches the first target number, so that the robot uses the cross-correlation window to traverse all pixels in the second texture area and obtains the updated depth information of each pixel in the second texture area. Then execute step 27. In some embodiments, it can be determined that the robot has updated all pixels in the current frame image, including the adjacent pixels of the pixels in the second texture area. In this embodiment, before the robot repeats step 24 each time, it first updates the updated depth information of each pixel to the depth information of the pixel, and updates the depth information of the adjacent pixels of each pixel in a given direction to the depth information of the adjacent pixels of the pixel, so as to achieve neighborhood propagation. Specifically, before executing the current step 24, the updated depth information of the pixel point in the last executed step 25 can be updated to the depth information of the pixel point (if necessary, one of the adjacent pixels of the pixel point can be updated to the pixel point), and the depth information of the adjacent pixels of each pixel point in the last executed step 24 in a given direction can be updated to the depth information of the adjacent pixels of the pixel point (if necessary, the adjacent pixels of the adjacent pixels of the pixel point in a given direction can be updated to the adjacent pixels of the pixel point). For example, among the upper, lower, left and right adjacent pixels of the pixel point described in step 24, the depth information of a pixel point corresponding to the greatest similarity can be selected in accordance with the depth information of the associated preset conditions involved in step 02.

[0092] It should be noted that when calculating the joint cost value of each pixel point in the cross-correlation window (which can also be understood as each pixel point traversed by the cross-correlation window), the photometric consistency cost value corresponding to the similarity between the two cross-correlation windows (the cross-correlation window of the current frame image and the cross-correlation window projected into the matching neighboring frame image) and the geometric consistency cost value corresponding to the Euclidean distance between a pair of pixels with a projection relationship between the two cross-correlation windows can be used to verify whether the depth information of the selected adjacent pixels can meet the update requirements of the pixels in the second texture area of the current frame image. The two cross-correlation windows here correspond to the first cross-correlation window corresponding to the pixel point in the second texture area of the current frame image and the second cross-correlation window corresponding to the mapping point of the pixel point in the matching neighboring frame image. Since the initial depth values of different adjacent pixels are different, when a pixel point uses a different initial depth value, the mapping point corresponding to the pixel point also changes, so the corresponding second cross-correlation window will also change accordingly. Here, the initial depth value of the appropriate adjacent pixel point can be selected based on the similarity between the first cross-correlation window and the second cross-correlation window to update the current pixel point.

[0093] Step 27, the robot controls the updated depth information of the pixel point newly obtained in step 26 to add the preset depth information increment to obtain the incrementally adjusted depth information of the pixel point; then, the photometric consistency cost value and the geometric consistency cost value of the pixel point are calculated based on the incrementally adjusted depth information of the pixel point, and then the weighted sum of the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel points is calculated using the first joint weight and the second joint weight, and the weighted sum is set as the incrementally adjusted joint cost value of the pixel point; then step 28 is executed; wherein, the specific calculation method of the incrementally adjusted joint cost value of the pixel point can refer to the related content described in steps 22 to 23 of the aforementioned embodiment.

[0094] If the incrementally adjusted joint cost value calculated in steps 28 and 27 for the pixel is less than the joint cost value for the pixel most recently obtained in step 26, the robot updates the incrementally adjusted depth information for the pixel to the depth information most recently obtained in step 26. This can also be considered as updating the incrementally adjusted depth information for the pixel to the depth information for the pixel, making the incrementally adjusted depth information more accurate. Then, step 29 is executed.

[0095] Specifically, after the neighborhood propagation is completed, it is determined that the robot has traversed all pixels in the second texture region using the cross-correlation window. Specifically, the depth information of each pixel in the second texture region is iteratively processed and optimized toward a smaller joint cost. Then, to further improve the accuracy of the depth information of each pixel, a perturbation amount is applied to incrementally adjust the corresponding depth information. Since the second texture region is a weak texture region, the photometric information within the region is homogenized, and the photometric information of the patch of the current frame image is relatively similar to the photometric information of the patch of the neighboring frame image. Therefore, after applying a larger perturbation amount, at least an increment larger than the baseline depth value increment and baseline normal vector increment for the first texture region, the photometric consistency cost and geometric consistency cost of the pixels in the second texture region will not change significantly, and there is no need to use the depth information of the mapped points of the current frame image's pixels projected onto the patch of the neighboring frame image to replace the depth information of the pixels in the current frame image.

[0096] In step 27, the robot controls the updated depth value of the pixel point newly obtained in step 26 to increase the preset depth value increment to obtain the incrementally adjusted depth value of the pixel point; the robot controls the updated normal vector of the pixel point newly obtained in step 26 to increase the preset normal vector increment to obtain the incrementally adjusted normal vector of the pixel point; then, based on the incrementally adjusted depth value of the pixel point and the incrementally adjusted normal vector of the pixel point, a third joint derivative is calculated between each pixel point in the first cross-correlation window in the current frame image and the mapping point projected by the pixel point in the second cross-correlation window. value, as the incrementally adjusted joint cost value of the pixel; then setting the depth information whose third joint cost value meets the second preset condition as the fine-tuned depth information of the pixel, including the fine-tuned depth value of the pixel and the fine-tuned normal vector of the pixel; preferably, the value range of the preset depth value increment is: less than or equal to 0.05 times the depth value before fine-tuning, and greater than or equal to -0.05 times the depth value before fine-tuning; the value range of the preset normal vector increment is: less than or equal to 0.02 times the normal vector before fine-tuning, and greater than or equal to -0.02 times the normal vector before fine-tuning. Specifically, the method of setting the depth information whose third joint cost value meets the second preset condition as the fine-tuned depth information of the pixel includes: when the second joint cost value is greater than the first joint cost value, and the third joint cost value is less than the second joint cost value, the robot sets the depth information corresponding to the third joint cost value as the fine-tuned depth information of the pixel. Specifically, the robot uses the depth information of the adjacent pixels of the pixel in the second texture area (whose initial value is obtained by the step 03) to update the depth information of the pixel. When the calculated second joint cost value is too small, the robot selects the depth information whose second joint cost value meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel. Until the neighborhood propagation involved in step 26 is completed, the robot compares the newly obtained second joint cost value with the third joint cost value. When the third joint cost value is less than the second joint cost value, the robot sets the depth information corresponding to the third joint cost value as the fine-tuned depth information of the pixel.

[0097] Step 29, repeat steps 21 to 28 until the number of repetitions reaches the second target number, so that the robot uses the photometric consistency constraint criterion and the geometric projection consistency constraint criterion to process the depth information of the pixel points in the second texture area, including the update and calculation mentioned in the above steps; then execute the step S3; wherein, for the same frame image, the preset traversal starting point set by the robot each time is located at the corner of the image, then the starting point of the neighborhood propagation involved in steps 24 to 25 can be configured to change with the change of the number of iterations, and the starting points for odd and even times are different, thereby increasing the comprehensiveness of the propagation. Preferably, for the same frame image, the pixel point traversal starting point set by the robot for each two consecutive executions of step 21 is symmetrical about the center of the frame image. Preferably, when the plane geometric shape of a frame image currently being processed is a rectangle, the second target number of times can be set to 3 times, then the robot actually executes steps 21 to 28 4 times, and the number of repeated executions is 3 times, which can converge the depth information of each pixel point in the second texture area to the optimal value, and guide the joint cost value of the pixel point to change in a smaller direction, so that the texture information similarity between the cross-correlation window in the second texture area of the current frame image and the window projected in the matching neighboring frame image is maximized, which is convenient for restoring greater image texture details; when the robot executes an odd number of times, the depth information of each pixel point in the second texture area can be converged to the optimal value, and the joint cost value of the pixel point can be guided ... depth information of the cross-correlation window in the second texture area of the current frame image and the window projected in the matching neighboring frame image can be maximized, which is convenient for restoring greater image texture details; when the robot executes the odd number of times, the depth information of each pixel point in the second texture area can be converged to the optimal value, and the joint cost value of the pixel point can be guided to change in a smaller direction, so as to restore greater image texture details. The robot starts traversing from the pixel point in the lower right corner of the current frame image. When the robot executes an even number of times, it starts traversing from the pixel point in the upper left corner of the current frame image, so as to achieve a more comprehensive traversal of the current frame image and determine more accurate depth information in a more comprehensive spatial propagation method. Especially under the geometric projection consistency constraint criterion, the accuracy of the depth information of the pixel points of the neighboring frame image is required to be high. Therefore, all images or their second texture areas must be repeatedly processed for the second target number of times to correct a relatively accurate depth map. Only then will the geometric consistency cost value obtained by the pixel points in the second texture area have a better effect.

[0098] In summary, in high-resolution images, there are often large areas of weak texture, which makes the traditional patch matching method based on photometric consistency produce fuzzy matching, making it impossible to recover the depth information of these areas, seriously affecting the integrity and quality of the reconstructed point cloud. In the face of this lack of robustness of traditional patch matching algorithms, after identifying the weak texture area (the second texture area), the present invention increases the patch matching window for the weak texture area to enrich the texture information within the window, and increases the amount of computation in exchange for the expression of weak texture, in order to suppress the fuzzy matching problem caused by the weak texture area. Then, a geometric reprojection consistency constraint criterion based on epipolar geometry theory is introduced, which can finally successfully recover relatively accurate depth information in the weak texture area.

[0099] As an embodiment, in the step S3, the robot uses the depth information most recently determined in the step S2 to perform inter-frame normal vector consistency constraints, and the method for obtaining the inter-frame normal vector consistency cost value of the corresponding pixel points includes: step 31, the robot can calculate and construct the homography matrix of the pixel points in the current frame image based on the epipolar geometry and the camera projection model, using the internal and external parameters of the camera corresponding to the camera, and then use the homography matrix to project each pixel point of the current frame image to the matching neighborhood frame image to obtain the mapping point (mainly the two-dimensional coordinates in the image coordinate system) and its normal vector (the normal vector of the three-dimensional facet projected into the three-dimensional space, including the three-dimensional coordinates of the normal vector), wherein the normal vector of each pixel point of the current frame image is the normal vector in the depth information most recently determined in step S2; for the specific calculation method of the homography matrix, please refer to the content of step 02 of the aforementioned embodiment. Then calculate the inner product of the normal vector of the pixel point and the normal vector of the mapping point obtained by its projection, wherein the inner product of the normal vector of the pixel point and the normal vector of the mapping point is the product of the coordinate values of the same dimension of the normal vector of the pixel point and the normal vector of the mapping point, which is used to describe the degree of posture matching of the two normal vectors, and also describes the degree of coplanarity of the corresponding facets; the inner product of the normal vector of the pixel point and the normal vector of the mapping point is specifically: the product of the horizontal coordinate of the normal vector of the pixel point and the horizontal coordinate of the normal vector of the mapping point is recorded as the horizontal coordinate product, the product of the vertical coordinate of the normal vector of the pixel point and the vertical coordinate of the normal vector of the mapping point is recorded as the vertical coordinate product, and the sum of the horizontal coordinate product, the vertical coordinate product and the vertical coordinate product is recorded as the inner product of the normal vector of the pixel point and the normal vector of the mapping point. Then, the difference between the preset constant and the inner product is set as the inter-frame normal vector consistency cost value of the pixel point in the current frame image, and an inner product calculation method for determining the inter-frame normal vector consistency cost value of the pixel point is determined. The preset constant is preferably a value of 1.

[0100] In step 32, the robot uses the normal vectors of the neighboring pixels of each pixel in the current frame image as the normal vector to be updated for the pixel; then, according to the inner product calculation method described in step 31, the depth information to be updated is used to calculate the inter-frame normal vector consistency cost value of the neighboring pixels of the pixel; in some embodiments, the number of neighboring pixels of each pixel is not limited to one, and is located within a matching window in the current frame image. The matching window can be a window centered on the pixel. When the pixel to be traversed by the robot in the current frame image or the next frame image changes, the matching window also changes. The robot can then obtain the normal vectors of multiple neighboring pixels of each pixel, forming multiple normal vectors to be updated and the normal vectors of the corresponding projected mapping points, and then calculate multiple inter-frame normal vector consistency cost values, so as to facilitate the subsequent selection of a reasonable cost value (generally, the one that represents the most similar position of the normal vectors of two pixels with a projected relationship) to update the normal vector of the pixel. The robot then executes step 33.

[0101] When the inter-frame normal vector cost value of the adjacent pixel points described in steps 33 and 32 is less than the inter-frame normal vector cost value of the pixel point described in step 31, the normal vectors of the adjacent pixel points of the pixel point are set to the updated normal vector of the pixel point, so that the normal vectors of the adjacent pixel points of the pixel point are updated to the normal vector of the pixel point, and the inter-frame normal vector cost value of the adjacent pixel points described in step 32 can be updated to the inter-frame normal vector cost value; then step 34 is executed. It should be noted that the robot will record the adjacent pixel points of the pixel point as temporary pixels, which can serve as the original pixel point, and then update the adjacent pixel points of the temporary pixel point to the temporary pixel point, and then update the depth information of the updated temporary pixel point to the depth information of the adjacent pixel points of the pixel point. In some embodiments, the matching window covers the current frame image with the temporary pixel point as the center. Based on this update, steps 32 and 33 are repeatedly executed to update the normal vectors of the pixels within the matching window based on neighborhood propagation.

[0102] It is worth noting that since the robot can obtain multiple normal vectors to be updated and the normal vectors of the corresponding projected mapping points in step 32, the normal vector corresponding to the inter-frame normal vector consistency cost value with the largest cost value or greater than the preset cost value threshold can be selected to update as the normal vector of the pixel point, so that the inter-frame normal vector consistency cost value of each updated pixel point changes in a smaller direction, thereby obtaining a more accurate normal vector.

[0103] Step 34: Repeat steps 31 through 33 until the target number of propagations is reached. This confirms that the robot has calculated the inter-frame normal vector consistency cost for all pixels in the current frame and adaptively updates the normal vectors of the pixels. The robot traverses the same frame twice in succession, setting the pixel traversal starting point symmetrically about the center of the frame. This is effectively the starting point for the matching window traversal of the current frame, achieving a more comprehensive traversal of the frame. The target number of propagations is preferably 1, and the actual number of executions is 2.

[0104] Preferably, if the number of iterations is odd, it starts from the pixel in the lower right corner of the image; if the number of iterations is even, it starts from the pixel in the upper left corner of the image. The number of iterations here is the actual number of executions, which is equal to the sum of the target propagation number and the value 1.

[0105] In summary, this embodiment calculates the normal vector similarity between the current pixel point estimated by the above steps and the pixel point projected to the neighboring frame in each pair of matching images by executing steps 31 to 34, and adds it to the cost function as a consistency constraint, in order to make the normal vector of each pixel point in the matched image tend to be consistent, and then perform inter-frame normal vector consistency constraints on all pixel points, so that the normal vectors of the pixel points in each view tend to be consistent, and the normal vectors of the pixel points in the image are mapped to a three-dimensional point cloud model, so that more accurate depth information can be integrated into the final point cloud, wherein the three-dimensional point cloud model is the three-dimensional point cloud information of the scene or object that the robot needs to reconstruct.

[0106] In the aforementioned embodiment, the photometric consistency cost of a pixel point is derived from the similarity between the first target window corresponding to each pixel point in the current frame image and the second target window corresponding to the mapping point projected by the pixel point. For pixels within the first texture region, the first target window is set to the matching window, including the first matching window; for pixels within the second texture region, the first target window is set to a window larger than the matching window, including the first cross-correlation window. This achieves a balance between the richness of the correlation within different texture regions and the amount of computation required.

[0107] The method for determining the similarity between a first target window corresponding to each pixel point in the current frame image and a second target window corresponding to a mapping point projected from the pixel point includes:

[0108] Based on the depth value of each pixel point included in the first target window and the depth value of the corresponding mapping point in the second target window, and based on the normal vector of each pixel point included in the first target window and the normal vector of the corresponding mapping point in the second target window, variance information between the first target window and the second target window is determined; wherein the depth value and normal vector of the same pixel point can be combined with internal and external parameters of the camera to construct a homography matrix to obtain the depth value and normal vector of the same mapping point.

[0109] A weight for each pixel point included in the first target window is determined based on a pixel difference, a distance difference, a depth value difference, and a normal vector difference between the pixel point and a center pixel point of the first target window. A weight for each mapping point included in the second target window is determined based on a pixel difference, a distance difference, a depth value difference, and a normal vector difference between the mapping point and the center pixel point of the second target window.

[0110] Then, based on the variance information between the first target window and the second target window, the similarity between the first target window and the second target window is determined, and then the robot sets the difference between the preset constant and the similarity as the photometric consistency cost value of the pixel point in the current frame image; wherein, the greater the similarity between the first target window and the second target window, the smaller the photometric consistency cost value of the corresponding pixel point; the smaller the similarity between the first target window and the second target window, the larger the photometric consistency cost value of the corresponding pixel point; the robot marks the corresponding photometric consistency cost value as the photometric consistency cost value of the pixel point; wherein, the first target window and the second target window are configured to occupy rectangular areas of the same size, and the second target window is formed by projecting the first target window in the current frame image into the matching neighborhood frame image; the current frame image and the matching neighborhood frame image are two frame images of the same size.

[0111] Specifically, the pixel difference is used to characterize the color similarity between a pixel and a center pixel, the distance difference is used to characterize the distance between a pixel and a center pixel, the depth difference is used to characterize the spatial position similarity (position in the physical world) between a pixel and a center pixel, and the normal difference is used to characterize the coplanarity of the corresponding surface between a pixel and a center pixel. Therefore, for pixels with smaller pixel difference, smaller distance difference, smaller depth difference, and smaller normal difference, the pixel is more likely to belong to the same plane as the center pixel, and a higher weight can be given in the covariance calculation process. This is mainly due to the fact that such pixels have a higher similarity influence. Similarly, for pixels with larger pixel difference, larger distance difference, larger depth difference, and larger normal difference, the pixel is less likely to belong to the same plane as the center pixel, and a lower weight can be given in the covariance calculation process. Here, the covariance is used to characterize the overall error between the two target windows. The smaller the overall error, the higher the similarity between the two target windows. Conversely, the larger the overall error, the lower the similarity between the two target windows. Therefore, based on the pixel difference, distance difference, depth value difference, and normal vector difference between each pixel in the first target window and the center pixel of the first target window, the weight for the pixel is determined, and the similarity between the first target window and the second target window is obtained by combining the covariance and the determined weight.

[0112] It should be noted that the present invention mentions that the homography matrix represents the mapping relationship between two planes, and the homography matrix is a mapping between points. The homography matrix can provide more constraints on the corresponding points on the two images. After knowing the image point position of a certain point in one image, relative to the basic matrix, the homography matrix can be used to obtain the exact image point position in the other image.

[0113] Based on the above embodiment, for each pixel point in the current frame image, based on the transformation relationship between the image coordinate system corresponding to the current frame image, the world coordinate system, and the image coordinate system corresponding to the matched neighboring frame image, the mapping point of the pixel point projected onto the matched neighboring frame image is determined, and the depth information of the mapping point is determined based on the depth information of the pixel point; wherein, a ray is drawn from the camera from the camera center through the pixel point in the current frame image, and the intersection of the ray and the corresponding three-dimensional surface patch in space is set as the reprojection depth of the intersection as the initial depth value of the pixel point, and the normal vector of the three-dimensional surface patch is the initial normal vector of the pixel point; wherein, the camera intrinsic and extrinsic parameters corresponding to the camera are the parameters required for conversion between coordinate systems, so that after the depth information of the pixel points of each frame image captured at different shooting angles is reconstructed into a local point cloud, each local point cloud (which can be converted from the depth value of each pixel point into a depth map) is converted and aggregated to form a complete three-dimensional dense point cloud data, thereby completing the reconstruction of the three-dimensional point cloud.

[0114] It should be noted that in the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0115] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, the functional units in the various embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0116] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0117] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A depth information processing method based on visual reconstruction, characterized in that: The depth information processing method is applied to process images acquired by a robot, wherein a camera installed on the robot acquires each frame of image from different shooting angles; the depth information processing method comprises: Step 1: The robot adjusts the photometric consistency cost of the pixel by setting the depth information adjustment amount, and then identifies the type of texture area where the pixel is located based on the adjusted photometric consistency cost; Step 2: When the robot recognizes that the pixel point is within the first texture region, the robot uses the photometric consistency constraint criterion to process the depth information of the pixel point within the first texture region; when the robot recognizes that the pixel point is within the second texture region, the robot uses the photometric consistency cost information and the geometric projection consistency constraint criterion to process the depth information of the pixel point within the second texture region; wherein the texture information of the first texture region is greater than that of the second texture region; Step 3: The robot uses the newly processed depth information from step 2 to perform inter-frame normal vector consistency constraints, obtains the inter-frame normal vector consistency cost value of the corresponding pixel points, and maps the normal vector of the pixel points in the image to the three-dimensional point cloud model, where the three-dimensional point cloud model is the three-dimensional point cloud information of the scene or object that the robot needs to reconstruct.

2. The depth information processing method according to claim 1, characterized in that: Before executing step 1, also include: Step 01: The robot obtains a sparse point cloud from each frame of images taken at different shooting angles, and then constructs a two-dimensional surface patch from the projected pixel points of the sparse point cloud in the image coordinate system. Based on the initial depth value and initial normal vector of the projected pixel point, the robot determines the initial depth value of each pixel point in the two-dimensional surface patch except the projected pixel point, and the initial normal vector of each pixel point in the two-dimensional surface patch except the projected pixel point. Then, the robot determines the initial depth value and initial normal vector of all pixels in each frame of the image. Each projected pixel point of the sparse point cloud in the image coordinate system is configured with an initial depth value and an initial normal vector. The two-dimensional surface patch is configured to be within the corresponding image to facilitate traversal of all pixels in the image. Step 02: Based on the similarity between the matching window in the current frame image where the pixel is located and the matching window in the matched neighborhood frame image, the robot uses the initial depth information of the adjacent pixels of each pixel in the image to update the initial depth information of the pixel to obtain the updated depth information of the pixel, and then updates the depth information of the pixel on the preset neighborhood of the adjacent pixel to the depth information of the adjacent pixel to adaptively update the depth information of the pixel until the robot uses the matching window to traverse all the pixels in the current frame image; then, the robot continues to update the depth value of the pixel by applying a preset depth value increment to obtain the adjusted depth value of the pixel; the robot continues to update the normal vector of the pixel by applying a preset normal vector increment to obtain the adjusted normal vector of the pixel; Step 03: Repeat step 02 until the number of executions reaches the preset target number, and then execute step 1; For the same frame image, the starting point of the pixel point traversal required by the robot for two consecutive executions of step 02 is symmetrical about the center of the frame image; The initial depth information includes an initial depth value and an initial normal vector, and the depth information includes a depth value and a normal vector.

3. The depth information processing method according to claim 2, characterized in that: The step 1 specifically includes: After executing step 03, the depth information of the pixel point newly obtained by the robot includes the adjusted depth value of the pixel point and the adjusted normal vector of the pixel point, making it the optimization result obtained by repeatedly executing step 02; wherein the depth information adjustment amount includes the depth value adjustment amount and the angle adjustment amount; The robot controls the adjusted depth value of each pixel to increase the depth value adjustment amount, while keeping the adjusted normal vector of each pixel unchanged, and then calculates the photometric consistency cost value of the pixel point, and records it as the first photometric consistency cost value; At the same time, the robot controls the adjusted depth value of each pixel to reduce the depth value adjustment amount, while keeping the adjusted normal vector of each pixel unchanged, and then calculates the photometric consistency cost value of the pixel point, and records it as the second photometric consistency cost value; At the same time, using the line connecting the camera center and the pixel as the first rotation axis, control the adjusted normal vector of the pixel to rotate clockwise along the first rotation axis by the angle adjustment amount while keeping the depth value of the pixel unchanged. Then, calculate the photometric consistency cost of the pixel and record it as the third photometric consistency cost. At the same time, using the line connecting the camera center and the pixel as the second rotation axis, control the adjusted normal vector of the pixel to rotate counterclockwise along the second rotation axis by the angle adjustment amount while keeping the depth value of the pixel unchanged. Then, calculate the photometric consistency cost of the pixel and record it as the fourth photometric consistency cost. Then, the robot determines whether the sum of the first photometric consistency cost value, the second photometric consistency cost value, the third photometric consistency cost value and the fourth photometric consistency cost value is less than the preset cost value threshold. If so, the robot identifies that the texture area where the pixel point is located is the second texture area and marks the pixel point as a pixel point within the second texture area; otherwise, the robot identifies that the texture area where the pixel point is located is the first texture area and marks the pixel point as a pixel point within the first texture area.

4. The depth information processing method according to claim 2, characterized in that: In step 2, when the robot recognizes that the pixel point is a pixel point within the first texture area, the method for the robot to process the depth information of the pixel point within the first texture area using the photometric consistency constraint criterion specifically includes: When the robot recognizes that the pixel point is a pixel point within the first texture area through step 1, the robot sets the adjusted depth value and the adjusted normal vector of the pixel point most recently obtained in step 03 as the reference depth information of the pixel point within the first texture area, and sets the adjusted photometric consistency cost value of the pixel point most recently obtained in step 03 as the reference photometric consistency cost value of the pixel point within the first texture area; For each pixel point in the first texture area of the current frame image, using the camera internal and external parameters corresponding to the camera, project the pixel point onto the matching neighboring frame image to form a mapping point in the neighboring frame image and determine the depth information of the mapping point, then configure the depth information of the mapping point as the depth information to be updated for the pixel point, and then calculate a new photometric consistency cost value for the pixel point based on the depth information to be updated for the pixel point; When the new photometric consistency cost value of the pixel point is less than the baseline photometric consistency cost value of the pixel point, the depth information to be updated of the pixel point is updated to the baseline depth information of the pixel point in the first texture area; wherein the baseline depth information of the pixel point includes the baseline depth value of the pixel point and the baseline normal vector of the pixel point.

5. The depth information processing method according to claim 4, characterized in that: The method for the robot to process depth information of pixel points in the first texture area using a photometric consistency constraint criterion further includes: The robot controls the updated baseline depth value of the pixel point to increase the baseline depth value increment to obtain the fine-tuned depth value of the pixel point; the robot controls the updated baseline normal vector of the pixel point to increase the baseline normal vector increment to obtain the fine-tuned normal vector of the pixel point; then, based on the fine-tuned depth value of the pixel point and the fine-tuned normal vector of the pixel point, the fine-tuned photometric consistency cost value of the pixel point is calculated; When the fine-tuned photometric consistency cost value of the pixel point is less than the baseline photometric consistency cost value of the pixel point, the fine-tuned depth value of the pixel point is updated to the baseline depth value of the pixel point in the first texture area, and the fine-tuned normal vector of the pixel point is updated to the baseline normal vector of the pixel point in the first texture area.

6. The depth information processing method according to claim 2, characterized in that: In step 2, when the robot recognizes that the pixel point is a pixel point within the second texture region through step 1, the method for the robot to process the depth information of the pixel point within the second texture region using the cost information of photometric consistency and the geometric projection consistency constraint criterion includes the following steps: Step 21: The robot increases the size of the matching window to obtain a cross-correlation window, and sets the cross-correlation window to be a window that traverses pixels from a preset traversal starting point within the second texture region of the image, so that the photometric consistency cost value of the pixel points within the second texture region is represented by the photometric consistency cost value of the corresponding pixel points within the cross-correlation window; then executing step 22; Step 22: Calculate the photometric consistency cost value of each pixel in the cross-correlation window based on the depth information of the pixel obtained in step 03; at the same time, calculate the geometric consistency cost value of each pixel in the cross-correlation window based on the depth information of the pixel obtained in step 03; then execute step 23; wherein the value range of the geometric consistency cost value of the pixel is equal to the value range of the photometric consistency cost value of the pixel; the photometric consistency cost information includes the photometric consistency cost value of the pixel in the second texture area; Step 23: The robot sets a first joint weight and applies it to the photometric consistency cost value of each pixel in the cross-correlation window, and sets a second joint weight and applies it to the geometric consistency cost value of each pixel in the cross-correlation window; then, using the first joint weight and the second joint weight, the robot calculates a weighted sum of the photometric consistency cost value and the geometric consistency cost value of each pixel in the cross-correlation window, and sets the weighted sum as the joint cost value of the pixel; then, the robot executes step 24; Step 24: The robot sets the depth information of the adjacent pixels of each pixel in the cross-correlation window as the depth information to be updated of the pixel, and then calculates the photometric consistency cost value and the geometric consistency cost value of the adjacent pixels of the pixel based on the depth information to be updated. Then, using the first joint weight and the second joint weight, the robot calculates the weighted sum of the photometric consistency cost value and the geometric consistency cost value of the adjacent pixels, and sets the weighted sum as the joint cost value of the adjacent pixels; then, executing step 25; Step 25: When the joint cost value of the adjacent pixels in step 24 is less than the joint cost value of the pixel, the depth information of the adjacent pixels of the pixel is set as the updated depth information of the pixel, so that the depth information to be updated of the pixel is updated to the updated depth information of the pixel; and the joint cost value of the adjacent pixels in step 24 is updated to the joint cost value of the pixel; and then step 26 is executed; Step 26, repeating steps 24 to 25 until the number of repetitions reaches the first target number, so that the robot uses the cross-correlation window to traverse all pixels in the second texture area; then executing step 27; wherein, before each repetition of step 24, the updated depth information of each pixel is first updated to the depth information of the pixel, and the depth information of the adjacent pixels of each pixel in a predetermined direction is updated to the depth information of the adjacent pixels of the pixel, so as to realize neighborhood propagation.

7. The depth information processing method according to claim 6, characterized in that: After the robot executes step 26, the following steps are also included: Step 27: The robot controls the updated depth information of the pixel point obtained in step 26 to increase the preset depth information increment to obtain the incrementally adjusted depth information of the pixel point; then, based on the incrementally adjusted depth information of the pixel point, the photometric consistency cost value and the geometric consistency cost value of the pixel point are calculated; then, using the first joint weight and the second joint weight, a weighted sum of the photometric consistency cost value and the geometric consistency cost value of the adjacent pixel points is calculated, and the weighted sum is set as the incrementally adjusted joint cost value of the pixel point; then, step 28 is executed; In step 28, when the incrementally adjusted joint cost value of the pixel point calculated in step 27 is less than the joint cost value of the pixel point most recently obtained in step 26, the depth information of the pixel point after the incremental adjustment is updated to the depth information of the pixel point most recently obtained in step 26; then step 29 is executed; Step 29, repeat steps 21 to 28 until the number of repetitions reaches a second target number; then execute step 3; wherein, for the same frame image, each time the robot executes step 21, the preset traversal starting point is set to be located at the corner of the image.

8. The depth information processing method according to claim 6, characterized in that: In step 22, the method of calculating the geometric consistency cost value of each pixel in the cross-correlation window based on the depth information of the pixel obtained in step 03 includes: Using the camera internal and external parameters corresponding to the camera, the pixel points of the cross-correlation window in the second texture area of the current frame image are projected into the matching neighboring frame image to obtain the mapping points and their depth information; The mapping point is then projected back to the current frame image to obtain the inverse mapping point and its depth information; the Euclidean distance between the pixel point and the corresponding inverse mapping point is then calculated, and the Euclidean distance is normalized according to the size of the current frame image to obtain the geometric consistency cost value of the pixel point; The cross-correlation window supports projection of the second texture area of the current frame image to a square area in the matching neighboring frame image.

9. The depth information processing method according to claim 2, characterized in that: In step 3, the robot uses the most recently processed depth information from step 2 to perform inter-frame normal vector consistency constraints, and the method for obtaining the inter-frame normal vector consistency cost value of the corresponding pixel point includes: Step 31: The robot uses the camera internal and external parameters corresponding to the camera to project each pixel of the current frame image into the matching neighboring frame image to obtain the mapping point and its normal vector, wherein the normal vector of each pixel of the current frame image is the normal vector in the depth information most recently processed in step 2; then, the inner product of the normal vector of the pixel point and the normal vector of the projected mapping point is calculated, and the difference between the preset constant and the inner product is set as the inter-frame normal vector consistency cost value of the pixel point in the current frame image, and an inner product calculation method for determining the inter-frame normal vector consistency cost value of the pixel point is determined; Step 32: The robot uses the normal vectors of the neighboring pixels of each pixel in the current frame image as the normal vector to be updated for the pixel; and then calculates the inter-frame normal vector consistency cost of the neighboring pixels of the pixel using the depth information to be updated according to the inner product calculation method described in step 31; In step 33, when the inter-frame normal vector cost value of the adjacent pixel points in step 32 is less than the inter-frame normal vector cost value of the pixel point in step 31, the normal vectors of the adjacent pixel points of the pixel point are updated to the normal vector of the pixel point; Step 34: Repeat steps 31 to 33 until the number of repetitions reaches the target number of propagations.

10. The depth information processing method according to claim 2, characterized in that: In step 01, for each frame of the image, a two-dimensional surface is constructed using three projected pixel points as vertex pixel points; and based on the transformation relationship between the world coordinate system corresponding to the sparse point cloud and the image coordinate system corresponding to the image, a three-dimensional surface matching the constructed two-dimensional surface is determined; When the semantic annotation information of all vertex pixels included in the two-dimensional surface patch to which the other pixel points belong is consistent with the semantic annotation information of the other pixel points, the depth value of the three-dimensional point cloud point corresponding to the vertex pixel point in the three-dimensional surface patch matched by the two-dimensional surface patch to which the vertex pixel points belong is used as the initial depth value of the other pixel points; Determining the initial depth value of the other pixel point based on the initial depth value of the vertex pixel point when the semantic annotation information of some vertex pixel points in the two-dimensional surface patch to which the other pixel point belongs is consistent with the semantic annotation information of the other pixel point; Wherein, within the two-dimensional surface to which the other pixel points belong, the robot sets the initial normal vector of the vertex pixel point as the initial normal vector of the other pixel points.

11. The depth information processing method according to claim 2, characterized in that: In step 02, based on the similarity between the matching window in the current frame image where the pixel is located and the matching window in the matching neighboring frame image, the robot updates the initial depth information of each pixel in the image using the initial depth information of the neighboring pixels of the pixel, and the method for obtaining the updated depth information of the pixel specifically includes: For each pixel point in the current frame image, determine a first matching window corresponding to the pixel point, which corresponds to the two-dimensional surface patch; use the camera internal and external parameters corresponding to the camera to project the pixel point onto the matching neighboring frame image to form a mapping point in the neighboring frame image and determine a second matching window corresponding to the mapping point, and determine the depth information of the mapping point; then, based on the depth information of the pixel point, calculate a first similarity between the first matching window corresponding to each pixel point in the current frame image and the second matching window corresponding to the mapping point obtained by projecting the pixel point; wherein the size of the first matching window is equal to the size of the matching window; The robot uses the initial depth value of each of the adjacent pixels of the pixel as the depth value to be updated of the pixel, and the robot uses the initial normal vector of each of the adjacent pixels of the pixel as the normal vector to be updated of the pixel; then, based on the depth value to be updated of the pixel and the normal vector to be updated of the pixel, calculates a second similarity between a first matching window corresponding to each pixel in the current frame image and a second matching window corresponding to a mapping point projected by the pixel; Then, when the first similarity is less than the second similarity, the robot selects the depth information whose second similarity meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel.

12. The depth information processing method according to claim 11, characterized in that: The method of selecting, from the depth information to be updated, the depth information whose second similarity meets the first preset condition and setting it as the updated depth information of the pixel point includes: Selecting depth information corresponding to the maximum similarity from at least one depth information to be updated as updated depth information of the pixel; wherein the maximum similarity is a second similarity that meets the first preset condition; The depth information to be updated includes the normal vector to be updated and the depth information to be updated; the updated depth information includes the updated normal vector and the updated depth information; The adjacent pixels of each pixel include: adjacent pixels arranged in four mutually perpendicular directions with the pixel as the symmetric center, so that there are multiple normal vectors to be updated for each pixel and multiple depth values to be updated for each pixel; The pixel points in the preset neighborhood of the adjacent pixel points include: adjacent pixel points arranged in four mutually perpendicular directions with the adjacent pixel point as the symmetry center.

13. The depth information processing method according to claim 11, characterized in that: In step 27, the robot controls the updated depth information of the pixel point newly obtained in step 26 to increase the preset depth information increment to obtain the depth information of the pixel point after the increment adjustment. The method includes: The robot controls the pixel point's updated depth value newly obtained in step 26 by adding a preset depth value increment to obtain the incrementally adjusted depth value of the pixel point; the robot controls the pixel point's updated normal vector newly obtained in step 26 by adding a preset normal vector increment to obtain the incrementally adjusted normal vector of the pixel point; Then, based on the incrementally adjusted depth value of the pixel point and the incrementally adjusted normal vector of the pixel point, a third joint cost value is calculated between each pixel point in the first correlation window in the current frame image and the mapping point projected by the pixel point in the second correlation window; and the depth information whose third joint cost value meets the second preset condition is set as the fine-tuned depth information of the pixel point; wherein, the greater the third similarity corresponding to the pixel point, the smaller the adjusted photometric consistency cost value of the pixel point.

14. The depth information processing method according to claim 13, characterized in that: The method of setting the depth information having the third similarity meeting the second preset condition as the fine-tuned depth information of the pixel point includes: When the second similarity is greater than the first similarity and the third similarity is greater than the second similarity, the robot sets the depth information corresponding to the third similarity as the fine-tuned depth information of the pixel.

15. The depth information processing method according to claim 7, characterized in that: In step 25, the method of setting the depth information of the adjacent pixels of the pixel as the updated depth information of the pixel so that the depth information to be updated of the pixel is updated to the updated depth information of the pixel includes: For each pixel point in the second texture area of the current frame image, determine a first cross-correlation window corresponding to the pixel point; using the camera internal and external parameters corresponding to the camera, project the pixel point to the matching neighboring frame image to form a mapping point in the neighboring frame image and determine the second cross-correlation window corresponding to the mapping point, and determine the depth information of the mapping point; then, based on the depth information of the pixel point, calculate a first joint cost value between each pixel point in the first cross-correlation window in the current frame image and the mapping point projected by the pixel point in the second cross-correlation window; wherein the size of the first cross-correlation window is larger than the size of the matching window, and the first joint cost value belongs to the joint cost value; The robot uses the depth value of each of the adjacent pixels of the pixel as the depth value to be updated of the pixel, and the robot uses the normal vector of each of the adjacent pixels of the pixel as the normal vector to be updated of the pixel; then, based on the depth value to be updated of the pixel and the normal vector to be updated of the pixel, calculates a second joint cost value between each pixel in the first cross-correlation window in the current frame image and a mapping point projected by the pixel in the second cross-correlation window; Then, when the second joint cost value is less than the first joint cost value, the robot selects the depth information whose second joint cost value meets the first preset condition from the depth information to be updated and sets it as the updated depth information of the pixel point.

16. The depth information processing method according to claim 15, characterized in that: The method of selecting the depth information to be updated, the depth information of which the second joint cost value meets the first preset condition, and setting it as the updated depth information of the pixel point includes: Selecting the depth information corresponding to the largest joint cost value from at least one depth information to be updated as the updated depth information of the pixel; The depth information to be updated includes the normal vector to be updated and the depth information to be updated; the updated depth information includes the updated normal vector and the updated depth information; Among them, the adjacent pixel points of each pixel point include: adjacent pixel points arranged in four mutually perpendicular directions with the pixel point as the symmetry center, so that there are multiple normal vectors to be updated for each pixel point, and there are also multiple depth values to be updated for each pixel point.

17. The depth information processing method according to claim 15, characterized in that: In step 02, the robot continues to update the depth value of the pixel by applying a preset depth value increment to obtain an adjusted depth value of the pixel; The robot continues to update the normal vector of the pixel point by applying a preset normal vector increment. The method for obtaining the adjusted normal vector of the pixel point includes: The robot controls the updated depth value of the pixel point to increase the preset depth value increment to obtain the latest depth value of the pixel point; The robot controls the updated normal vector of the pixel point to increase the preset normal vector increment to obtain the latest normal vector of the pixel point; then, based on the incrementally adjusted depth value of the pixel point and the incrementally adjusted normal vector of the pixel point, calculates a third joint cost value between each pixel point in the first correlation window in the current frame image and the mapping point projected by the pixel point in the second correlation window; and then sets the depth information of the third joint cost value that meets the second preset condition as the fine-tuned depth information of the pixel point.

18. The depth information processing method according to claim 17, characterized in that: The method of setting the depth information of the pixel point whose third joint cost value meets the second preset condition as the fine-tuned depth information includes: When the second joint cost value is greater than the first joint cost value and the third joint cost value is less than the second joint cost value, the robot sets the depth information corresponding to the third joint cost value as the fine-tuned depth information of the pixel point.

19. The depth information processing method according to any one of claims 2 to 18, characterized in that: The photometric consistency cost of the pixel point is derived from the similarity between the first target window corresponding to each pixel point in the current frame image and the second target window corresponding to the mapping point projected by the pixel point; wherein, for the pixel points in the first texture area, the first target window is set as the matching window; for the pixel points in the second texture area, the first target window is set as a window larger than the matching window; The method for determining the similarity between a first target window corresponding to each pixel point in the current frame image and a second target window corresponding to a mapping point projected from the pixel point includes: Determining variance information between the first target window and the second target window based on a depth value of each pixel included in the first target window and a depth value of a corresponding mapping point in the second target window, and based on a normal vector of each pixel included in the first target window and a normal vector of the corresponding mapping point in the second target window; Based on the variance information between the first target window and the second target window, the robot determines the similarity between the first target window and the second target window, and then sets the difference between a preset constant and the similarity as the photometric consistency cost value of the pixel point in the current frame image; The greater the similarity between the first target window and the second target window, the smaller the photometric consistency cost value of the corresponding pixel point; the smaller the similarity between the first target window and the second target window, the larger the photometric consistency cost value of the corresponding pixel point; the robot marks the corresponding photometric consistency cost value as the photometric consistency cost value of the pixel point; The first target window and the second target window are configured to occupy rectangular areas of the same size, and the second target window is formed by projecting the first target window in the current frame image into the matching neighboring frame image; the current frame image and the matching neighboring frame image are two frame images of the same size.

20. The depth information processing method according to claim 10, characterized in that: For each pixel in the current frame image, determine the mapping point where the pixel is projected onto the matching neighboring frame image based on the transformation relationship between the image coordinate system corresponding to the current frame image, the world coordinate system, and the image coordinate system corresponding to the matching neighboring frame image, and determine the depth information of the mapping point based on the depth information of the pixel; A ray is drawn from the camera from the center of the camera through the pixel in the current frame image. The intersection of the ray and the corresponding three-dimensional surface in space is set to the reprojected depth of the intersection as the initial depth value of the pixel, and the normal vector of the three-dimensional surface is the initial normal vector of the pixel; Among them, the internal and external parameters of the camera corresponding to the camera are the parameters required for conversion between coordinate systems, so that after the depth information of the pixel points of each frame image taken at different shooting angles is reconstructed into a local point cloud, each local point cloud is converted and assembled together to form three-dimensional dense point cloud data.

Images