Point cloud reconstruction method, device, electronic device and storage medium

Abstract

The present invention provides a point cloud reconstruction method, device, electronic device and storage medium, wherein the method comprises: based on the posture information of a target camera and a neighborhood camera, determining a candidate image of an image to be reconstructed of a target camera within a candidate matching area of ​​a neighborhood camera; based on the similarity between the candidate image and the image to be reconstructed, determining a paired image of the image to be reconstructed from the candidate image; based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighborhood camera, determining the depth information of the image to be reconstructed, and performing point cloud reconstruction based on the depth information of the image to be reconstructed, thereby overcoming the defect that the point cloud reconstruction effect depends on the accuracy of the camera posture in the traditional solution, and being able to divide the scene at the point cloud level by setting the area range on the basis of a known point cloud, thereby realizing point cloud reconstruction of a specific area, and also providing great convenience for subsequent gridding operations.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a point cloud reconstruction method, device, electronic device and storage medium. Background Art

[0002] At present, 3D cameras can reconstruct the 3D point cloud of the target scene, which has an additional depth dimension information compared to conventional 2D cameras, providing additional data processing freedom for various back-end developments and having important application value. With the vigorous development of 3D (Three Dimension) related industries, 3D cameras and 3D reconstruction technology have gradually become indispensable basic hardware equipment and data processing core technology in computer vision related fields, such as model building, deep learning, 3D animation production, etc.

[0003] Most of the current 3D point cloud reconstruction schemes are based on unknown camera poses, and the camera pose is usually estimated using the feature point matching method. This method has high requirements for the reconstructed scene. When the reconstructed scene does not meet the requirements, for example, scenes with weak textures or scenes with multiple repeated textures, it is often impossible to accurately estimate the camera pose. In addition, since the camera pose is extremely important for the generation of point clouds, the point cloud reconstruction effect is poor. Summary of the invention

[0004] The present invention provides a point cloud reconstruction method, device, electronic device and storage medium, which are used to solve the defect in the prior art that the point cloud reconstruction effect depends on the accuracy of the camera posture.

[0005] The present invention provides a point cloud reconstruction method, comprising:

[0006] Based on the position information of the target camera and the neighboring camera, determine the candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera;

[0007] Determining a pairing image of the image to be reconstructed from the candidate images based on a similarity between the candidate images and the image to be reconstructed;

[0008] Based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighborhood camera, the depth information of the image to be reconstructed is determined, and point cloud reconstruction is performed based on the depth information of the image to be reconstructed.

[0009] According to a point cloud reconstruction method provided by the present invention, the method of determining a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighboring camera based on the pose information of the target camera and the neighboring camera comprises:

[0010] Based on the position information of the target camera and the neighboring camera, determining the change matrix between the target camera and the neighboring camera;

[0011] Based on the change matrix between the target camera and the neighborhood camera, determining the projection of the ray corresponding to the to-be-reconstructed image of the target camera in the coordinate system of the neighborhood camera;

[0012] Based on the projection, a candidate image of the to-be-reconstructed image of the target camera within a candidate matching area of ​​the neighborhood camera is determined.

[0013] According to a point cloud reconstruction method provided by the present invention, determining, based on the projection, a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighborhood camera, comprises:

[0014] Setting a sliding window on the projection, determining a candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera based on a sliding range of the sliding window, and determining a candidate image within the candidate matching area;

[0015] The size of the sliding window is determined based on the size of the to-be-reconstructed image of the target camera.

[0016] According to a point cloud reconstruction method provided by the present invention, the method of determining a paired image of the image to be reconstructed from the candidate image based on the similarity between the candidate image and the image to be reconstructed comprises:

[0017] Determining a structural similarity and / or a position similarity between the candidate image and the image to be reconstructed based on grayscale information of the candidate image and the image to be reconstructed;

[0018] Based on the structural similarity and / or position similarity between the candidate image and the image to be reconstructed, a paired image of the image to be reconstructed is determined from the candidate images.

[0019] According to a point cloud reconstruction method provided by the present invention, the depth information of the image to be reconstructed is determined based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighborhood camera, including:

[0020] Based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighboring camera, respectively determine a first angle and a second angle, wherein the first angle is formed by a ray corresponding to the image to be reconstructed of the target camera and a line connecting the optical centers of the target camera and the neighboring camera, and the second angle is formed by a ray corresponding to the paired image of the neighboring camera and a line connecting the optical centers of the target camera and the neighboring camera;

[0021] Depth information of the image to be reconstructed is determined based on the first angle and the second angle.

[0022] According to a point cloud reconstruction method provided by the present invention, determining the depth information of the image to be reconstructed based on the first angle and the second angle includes:

[0023] Determine candidate depth information of the paired images relative to each neighboring camera based on the first angle and the second angle relative to each neighboring camera and its paired image, and a line connecting the optical centers of the target camera and each neighboring camera;

[0024] Determine the depth information of the image to be reconstructed based on the candidate depth information of the paired images relative to each neighboring camera and the similarity between the paired images of each neighboring camera and the image to be reconstructed.

[0025] According to a point cloud reconstruction method provided by the present invention, the point cloud reconstruction based on the depth information of the image to be reconstructed includes:

[0026] Determining a three-dimensional point cloud of the target camera based on the depth information of the image to be reconstructed;

[0027] Based on the three-dimensional point cloud of the target camera and the change matrix between the target camera and the world coordinate system, determine the three-dimensional point cloud of the target camera in the world coordinate system;

[0028] Based on the point cloud fusion constraint, the 3D point clouds of each target camera in the world coordinate system are fused to obtain the reconstructed point cloud;

[0029] The point cloud fusion constraint includes at least one of a similarity constraint, a visibility constraint, a color correlation constraint, and a specific area constraint.

[0030] The present invention also provides a point cloud reconstruction device, comprising:

[0031] A candidate image determination unit, configured to determine, based on the position information of the target camera and the neighboring camera, a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighboring camera;

[0032] a pairing image determining unit, configured to determine a pairing image of the image to be reconstructed from the candidate images based on a similarity between the candidate images and the image to be reconstructed;

[0033] The point cloud reconstruction unit is used to determine the depth information of the image to be reconstructed based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighborhood camera, and to perform point cloud reconstruction based on the depth information of the image to be reconstructed.

[0034] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the point cloud reconstruction method described in any one of the above are implemented.

[0035] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of any of the above-described point cloud reconstruction methods are implemented.

[0036] The point cloud reconstruction method, device, electronic device and storage medium provided by the present invention determine the candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighborhood camera according to the posture information of the target camera and the neighborhood camera, and determine the paired image of the image to be reconstructed from the candidate image according to the similarity between the candidate image and the image to be reconstructed. The matching method based on similarity is different from the scoring criterion based on normalized cross-correlation in the traditional scheme, and the matching accuracy is significantly improved. The depth information of the image to be reconstructed is determined based on the position information of the image to be reconstructed and its paired image, and the posture information of the target camera and its neighborhood camera, and the point cloud is reconstructed based on the depth information of the image to be reconstructed, which overcomes the defect that the point cloud reconstruction effect depends on the accuracy of the camera posture in the traditional scheme, and can realize point cloud reconstruction of a specific area by setting the area range on the basis of a known point cloud, and also provides great convenience for subsequent gridding operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is a flow chart of the point cloud reconstruction method provided by the present invention;

[0039] Figure 2 The camera array diagram of the target camera and its neighboring cameras provided by the present invention;

[0040] Figure 3 is a flow chart of step 110 in the point cloud reconstruction method provided by the present invention;

[0041] Figure 4 is a schematic diagram of the projection process of the image to be reconstructed provided by the present invention;

[0042] Figure 5 is a schematic diagram of a sliding window provided by the present invention;

[0043] Figure 6 is a flow chart of step 120 in the point cloud reconstruction method provided by the present invention;

[0044] Figure 7 is a schematic diagram of a process for determining depth information provided by the present invention;

[0045] Figure 8 is a schematic diagram of a first angle and a second angle provided by the present invention;

[0046] Fig. 9 is a flow chart of step 420 in the point cloud reconstruction method provided by the present invention;

[0047] Fig.10 It is a flowchart of the point cloud reconstruction process provided by the present invention;

[0048] Fig.11 It is an overall flow chart of the point cloud reconstruction method provided by the present invention;

[0049] Fig.12 It is a structural schematic diagram of a point cloud reconstruction device provided by the present invention;

[0050] Fig.13 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Most of the current point cloud reconstruction solutions first obtain the camera pose and sparse point cloud through the camera pose estimation method, and then obtain the dense point cloud of the scene based on the sparse point cloud. There are three common dense point cloud reconstruction methods, namely, patch-based diffuse point cloud reconstruction method, voxel-based point cloud reconstruction method and depth map fusion-based point cloud reconstruction method.

[0053] Among them, the patch-based diffuse point cloud reconstruction method is a method that uses the diffusion of sparse point clouds to obtain dense point clouds, that is, based on the estimated camera pose, the sparse point cloud is obtained by using SFM (Structure From Motion), and then the dense point cloud of the scene is obtained after diffusion.

[0054] The voxel-based point cloud reconstruction method is a traditional method for generating regular point clouds. It can generate regular point clouds and is convenient for extracting the plane of objects. However, its accuracy is affected by the resolution of spatial division, and it is difficult to handle scenes with high precision and large scale.

[0055] The point cloud reconstruction method based on depth map fusion is to obtain the camera pose through SFM, select sparse point clouds according to certain rules, perform depth estimation based on the selected sparse point clouds, and perform nonlinear optimization on each seed point. It has a wide range of application scenarios and is suitable for dense point cloud reconstruction of complex objects.

[0056] However, the point cloud reconstruction processes of the above three point cloud reconstruction schemes all rely on the camera pose and sparse point cloud obtained by SFM, that is, the point cloud reconstruction effect depends on the camera pose obtained by SFM and the accuracy of the sparse point cloud. The estimation of the camera pose usually adopts the feature point matching method, which has high requirements for the reconstructed scene. When the reconstructed scene cannot meet the requirements, for example, scenes with weak textures, scenes with multiple repeated textures, etc., it is often impossible to accurately estimate the accurate camera pose, which leads to poor point cloud reconstruction effect. Moreover, most of the current point cloud reconstruction methods are to perform mask operations on the input image, and this operation will inevitably lose the information of the input image.

[0057] In view of the above situation, the present invention provides a point cloud reconstruction method, which aims to divide the scene at the point cloud level and realize point cloud reconstruction of a specific area. Figure 1 It is a flow chart of the point cloud reconstruction method provided by the present invention, such as Figure 1 As shown, the method includes:

[0058] Step 110 : determining, based on the position information of the target camera and the neighboring camera, candidate images of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera.

[0059] Specifically, the three-dimensional point cloud reconstruction method provided in the embodiment of the present invention first executes step 110, calibrates the camera array through the calibration plate, determines the target camera and its posture information from the camera array, and uses the posture information and geometric information constraints of the target camera to quickly determine the search area of ​​the target camera, thereby determining the camera in the search area and the posture information of the camera, which is the neighborhood camera of the target camera; it should be noted that the posture information here includes not only the position information of the corresponding camera, but also the posture information, that is, the pitch posture, shooting angle, etc. of the corresponding camera.

[0060] In addition, the number of neighborhood cameras here can be one or more. Figure 2 The camera array diagram of the target camera and its neighboring cameras provided by the present invention is as follows: Figure 2As shown, in order to reconstruct the point cloud more accurately, in the embodiment of the present invention, the number of neighborhood cameras is determined to be 4. Wherein, A represents the target camera, and B, C, D and E are the neighborhood cameras of the target camera A respectively.

[0061] Subsequently, the image to be reconstructed of the target camera in the candidate matching area of ​​its neighborhood camera and the candidate images within the candidate matching area can be determined based on the pose information of the target camera and its neighborhood cameras. The specific process may be that the projection matrix between the target camera and the neighborhood camera, i.e., the change matrix, is firstly marked with reference to the pose information of the target camera and the neighborhood camera. Then, based on the change matrix, the image to be reconstructed of the target camera is projected to the coordinate system of its neighborhood camera, and the projection area, i.e., the candidate matching area, and the candidate images within the candidate matching area are determined.

[0062] Step 120: Determine a paired image of the image to be reconstructed from the candidate images based on the similarity between the candidate images and the image to be reconstructed.

[0063] Specifically, after determining the candidate images of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera in step 110, step 120 is performed to determine the paired image of the image to be reconstructed according to the image matching strategy, that is, according to the similarity between the candidate image and the image to be reconstructed, determine the paired image of the image to be reconstructed from the candidate images. This process specifically includes the following steps:

[0064] First, determine the similarity between each candidate image and the image to be reconstructed in the candidate matching area. The similarity here may be position similarity and / or structural similarity, and may be determined based on the grayscale information of the candidate image and the image to be reconstructed;

[0065] Then, according to the similarity between the candidate image and the image to be reconstructed, the matching score between each candidate image and the image to be reconstructed in the candidate matching area can be determined. The matching score is used to indicate the degree of similarity between the corresponding candidate image and the image to be reconstructed. The higher the matching score, the higher the similarity between the corresponding candidate image and the image to be reconstructed, and the greater the possibility that the candidate image is the paired image of the image to be reconstructed of the target camera in its neighboring camera; conversely, the lower the matching score, the lower the similarity between the corresponding candidate image and the image to be reconstructed, and the less likely the candidate image is the paired image of the image to be reconstructed of the target camera in its neighboring camera.

[0066] Thereafter, based on the matching scores of the candidate images, the image closest to the image to be reconstructed of the target camera can be determined from the candidate images in the candidate matching area, that is, the paired image of the image to be reconstructed of the target camera in its neighborhood camera. Specifically, the matching scores of the candidate images can be sorted, and the candidate image with the highest matching score and the candidate image corresponding to the matching score can be determined. The candidate image is the paired image of the image to be reconstructed.

[0067] Step 130 , based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighborhood camera, the depth information of the image to be reconstructed is determined, and point cloud reconstruction is performed based on the depth information of the image to be reconstructed.

[0068] Specifically, after determining the paired image of the image to be reconstructed in step 120, step 130 can be executed to determine the depth information of the image to be reconstructed according to the position information of the image to be reconstructed and its paired image, and the pose information of the target camera and its neighborhood cameras, and perform point cloud reconstruction according to the depth information of the image to be reconstructed. The specific process includes the following steps:

[0069] First, after determining the paired image of the image to be reconstructed in step 120, the position information of the paired image in the neighborhood camera can be determined. In addition, the position information of the image to be reconstructed in the target camera needs to be determined.

[0070] Subsequently, according to the position information of the image to be reconstructed and its paired image, as well as the pose information of the target camera and its neighboring cameras, the candidate depth information of the paired image of the image to be reconstructed is determined, and based on the candidate depth information of the paired image, the depth information of the image to be reconstructed of the target camera is determined;

[0071] After that, the depth information of the images to be reconstructed of each target camera can be fused, and point cloud reconstructed according to the fused depth information, thereby realizing point cloud reconstruction with camera pose as input, through image matching strategy and depth information of the image to be reconstructed. That is, based on the known point cloud, the scene can be divided at the point cloud level by setting the area range, thereby realizing point cloud reconstruction of a specific area.

[0072] The point cloud reconstruction method provided by the present invention determines the candidate images of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera according to the posture information of the target camera and the neighboring camera, and determines the paired image of the image to be reconstructed from the candidate images according to the similarity between the candidate image and the image to be reconstructed. The matching method based on similarity is different from the scoring criterion based on normalized cross-correlation in the traditional scheme, and significantly improves the matching accuracy. Based on the position information of the image to be reconstructed and its paired image, and the posture information of the target camera and its neighboring camera, the depth information of the image to be reconstructed is determined, and point cloud reconstruction is performed based on the depth information of the image to be reconstructed, which overcomes the defect that the point cloud reconstruction effect depends on the accuracy of the camera posture in the traditional scheme, and can realize point cloud reconstruction of a specific area by setting the area range on the basis of a known point cloud, and also provides great convenience for subsequent gridding operations.

[0073] Based on the above embodiments, Figure 3 is a flow chart of step 110 in the point cloud reconstruction method provided by the present invention, such as Figure 3 As shown, step 110 includes:

[0074] Step 111, determining a change matrix between the target camera and the neighboring camera based on the position information of the target camera and the neighboring camera;

[0075] Step 112, based on the change matrix between the target camera and the neighboring camera, determine the projection of the ray corresponding to the image to be reconstructed of the target camera in the coordinate system of the neighboring camera;

[0076] Step 113: Based on the projection, determine the candidate images of the to-be-reconstructed image of the target camera within the candidate matching area of ​​the neighboring camera.

[0077] Specifically, in step 110, the process of determining the candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera according to the pose information of the target camera and the neighboring camera can be divided into the following three steps:

[0078] Step 111, first, determine the pose information of the target camera and its neighboring cameras. This process has been described in detail above and will not be repeated here. Then, based on the pose information of the target camera and its neighboring cameras, mark the change matrix between the target camera and the neighboring cameras. The change matrix here can also be understood as the projection matrix of the image to be reconstructed of the target camera projected onto the coordinate system of the neighboring camera.

[0079] Step 112, according to the change matrix between the target camera and the neighboring camera determined in step 111, the image to be reconstructed of the target camera is projected into the coordinate system of the neighboring camera, thereby obtaining the projection of the ray corresponding to the image to be reconstructed of the target camera in the coordinate system of the neighboring camera.

[0080] The following takes the target camera A and a single neighborhood camera B as an example to illustrate the projection process of the image to be reconstructed:

[0081] Figure 4 is a schematic diagram of the projection process of the image to be reconstructed provided by the present invention, such as Figure 4 As shown, according to the change matrix between the target camera A and the neighboring camera B, the image P to be reconstructed of the target camera A is projected to the coordinate system of the neighboring camera B, and the optical center O of the target camera A is obtained. A The projections P1 and P2 of the ray passing through the image to be reconstructed P in the coordinate system of the neighborhood camera B are the endpoints.

[0082] Step 113, after determining the projection of the corresponding ray of the image to be reconstructed of the target camera in the coordinate system of the neighboring camera in step 112, the area where the projection is located can be determined, and the area can be used as the candidate matching area of ​​the image to be reconstructed of the target camera in the neighboring camera. Thereafter, the candidate image in the candidate matching area can be determined.

[0083] The method provided by the embodiment of the present invention projects the image to be reconstructed of the target camera and determines the candidate matching area according to the projection. The projection-based matching and positioning method greatly improves the matching rate and avoids a large amount of matching screening work. It not only promotes the progress of the point cloud reconstruction task, but also provides key benefits for the point cloud reconstruction task.

[0084] Based on the above embodiment, the projection of the ray corresponding to the image to be reconstructed of the target camera in the coordinate system of the neighboring camera can be calculated by the following formula:

[0085]

[0086] in, represents the change matrix between the target camera A and the neighboring camera B, K B represents the intrinsic parameter matrix of the neighborhood camera B, (x A ,y A , z A ) represents the optical center O of the target camera A A is the endpoint and the point on the ray passing through the image to be reconstructed P, (u B , v B ) means (x A ,y A , z A ) is projected in the coordinate system of the neighboring camera B, f x 、f y 、c x 、c y , as well as They are all parameters in the pose information of the target camera and its neighborhood cameras.

[0087] Based on the above embodiment, step 113 includes:

[0088] A sliding window is set on the projection, and based on the sliding range of the sliding window, a candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera is determined, and a candidate image within the candidate matching area is determined;

[0089] The size of the sliding window is determined based on the size of the image to be reconstructed from the target camera.

[0090] Specifically, in step 113, when determining the candidate images of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighborhood camera, a sliding window can be set on the projection of the corresponding ray of the image to be reconstructed of the target camera obtained in step 112 in the coordinate system of the neighborhood camera, and the sliding window is slid along the projection to determine the coverage range of the sliding window when sliding. This coverage range is the sliding range of the sliding window. The candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera can be determined through this sliding range. To be specific, this sliding range can be directly used as the candidate matching area, or further screening can be performed on the basis of this sliding range, and the screened sliding range can be used as the candidate matching area; thereafter, the candidate images within the candidate matching area can be determined.

[0091] It should be noted that the size of the sliding window determines the size of the candidate matching area, and further determines the accuracy of the candidate matching area; if the size is too large, more candidate graphics will be obtained, which will make the subsequent image matching process take a lot of time and effort; correspondingly, if the size is too small, some candidate images will be missed, and it is easy to fail to include the image closest to the image to be reconstructed into the candidate image, which will lead to deviations in subsequent image matching. The matched paired image is not the candidate image closest to the image to be reconstructed. Based on this, if it is necessary to determine a suitable candidate matching area, the size of the sliding window needs to be determined based on the size of the image to be reconstructed, which can be twice, three times, four times, etc., the size of the image to be reconstructed. Figure 5 is a schematic diagram of a sliding window provided by the present invention, such as Figure 5 As shown, in the embodiment of the present invention, the size of the sliding window (15×15 pixels) is determined to be three times the size of the image to be reconstructed (5×5 pixels), and F1 in the figure represents the candidate image.

[0092] The method provided by the embodiment of the present invention establishes a sliding window on the projection, and determines the candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera through the sliding range of the sliding window, so that the candidate image within the candidate matching area can be determined, which not only improves the image matching rate, but also can better ensure the accuracy of image matching, thereby achieving a balance between speed and accuracy.

[0093] Based on the above embodiments, Figure 6 is a flow chart of step 620 in the point cloud reconstruction method provided by the present invention, such as Figure 6 As shown, step 120 includes:

[0094] Step 121, determining the structural similarity and / or position similarity between the candidate image and the image to be reconstructed based on the grayscale information of the candidate image and the image to be reconstructed;

[0095] Step 122: determining a paired image of the image to be reconstructed from the candidate images based on the structural similarity and / or position similarity between the candidate images and the image to be reconstructed.

[0096] Specifically, in step 120, the process of determining the paired image of the image to be reconstructed from the candidate images according to the similarity between the candidate images and the image to be reconstructed can be divided into the following two steps:

[0097] Step 121, firstly, according to the grayscale information of the image to be reconstructed of the target camera and the grayscale information of the candidate images of the neighborhood camera, the structural similarity and / or positional similarity between each candidate image in the candidate matching area and the image to be reconstructed is calculated. The structural similarity can characterize the degree of similarity between the corresponding candidate image and the image to be reconstructed. For example, when both are triangles or squares, it can be preliminarily determined that the structural similarity between the two is high. The positional similarity is used to characterize the degree of similarity between the position of the corresponding candidate image in the neighborhood camera and the position of the image to be reconstructed in the target camera. For example, when both are located in the upper left corner or the lower right corner of their respective cameras, it can be preliminarily determined that the positional similarity between the two is high.

[0098] Step 122, according to the structural similarity or position similarity obtained in step 121, the matching score of each candidate image in the candidate matching area is determined. The matching score is used to indicate the degree of similarity between the corresponding candidate image and the image to be reconstructed. The higher the matching score, the higher the similarity between the corresponding candidate image and the image to be reconstructed, and the greater the possibility that the candidate image is the paired image of the image to be reconstructed of the target camera in its neighborhood camera; conversely, the lower the matching score, the lower the similarity between the corresponding candidate image and the image to be reconstructed, and the less likely the candidate image is the paired image of the image to be reconstructed of the target camera in its neighborhood camera.

[0099] Furthermore, according to the matching scores of the candidate images, the paired images of the image to be reconstructed of the target camera in the camera in its neighborhood can be determined from the candidate images in the candidate matching area, that is, the candidate images are sorted in order from high to low or from low to high according to the matching scores, and the candidate image with the highest matching score and the corresponding matching score is determined, and the candidate image is the paired image of the image to be reconstructed.

[0100] In addition, in step 122, the position similarity and structural similarity can be combined to jointly determine the matching score between the candidate image and the image to be reconstructed. The matching score determined based on the similarity at multiple levels can more accurately characterize the degree of similarity between the corresponding candidate image and the image to be reconstructed, thereby improving the accuracy of image matching and providing a strong guarantee for the determination of paired images.

[0101] The method provided by the embodiment of the present invention determines the process of pairing images from candidate images through the position similarity and / or structural similarity between the candidate image and the image to be reconstructed. Different from the matching method based on the NCC (Normalized Cross Correlation) class in the traditional scheme, the introduction of position similarity and / or structural similarity can not only more accurately analyze the reliability of the pairing of the two, but also ensure the accuracy of the pairing of the two.

[0102] Based on the above embodiment, the position similarity, structural similarity, and matching score between the candidate image and the image to be reconstructed can be calculated by the following formula:

[0103] The calculation formula of the position similarity between the candidate image and the image to be reconstructed is as follows:

[0104]

[0105] Among them, P represents the image to be reconstructed, F1 represents the candidate image in the candidate matching area, and P i Represents the grayscale information of the i-th pixel in the image to be reconstructed, represents the grayscale information of the i-th pixel in the candidate image, dist(P, F1) represents the position similarity between the candidate image and the image to be reconstructed, and 25 refers to the pixel (5×5).

[0106] The calculation formula of the structural similarity between the candidate image and the image to be reconstructed can be expressed as:

[0107]

[0108] Among them, sim(P, F1) represents the structural similarity between the candidate image and the image to be reconstructed.

[0109] The calculation formula of the matching score between the candidate image and the image to be reconstructed is as follows:

[0110]

[0111] Wherein, scale(P, F1) represents the matching score between the candidate image and the image to be reconstructed.

[0112] Based on the above embodiments, Figure 7 is a schematic diagram of the process of determining the depth information provided by the present invention, such as Figure 7 As shown, in step 130, based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighboring camera, the depth information of the image to be reconstructed is determined, including:

[0113] Step 710, based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighboring camera, respectively determine a first angle and a second angle, wherein the first angle is formed by a ray corresponding to the image to be reconstructed of the target camera and a line connecting the optical centers of the target camera and the neighboring camera, and the second angle is formed by a ray corresponding to the paired image of the neighboring camera and a line connecting the optical centers of the target camera and the neighboring camera;

[0114] Step 720: Determine depth information of the image to be reconstructed based on the first angle and the second angle.

[0115] Specifically, in step 130, the process of determining the depth information of the image to be reconstructed according to the position information of the image to be reconstructed and its paired image, and the pose information of the target camera and its neighboring cameras, specifically includes the following steps:

[0116] First, execute step 710 to determine the position information of the image to be reconstructed in the target camera and the position information of the paired image of the image to be reconstructed in the neighboring camera, and based on the position information of the two, as well as the pose information of the target camera and its neighboring camera, determine a first angle between a ray corresponding to the image to be reconstructed of the target camera and a line connecting the optical centers of the target camera and the neighboring camera, and a second angle between a ray corresponding to the paired image of the neighboring camera and a line connecting the optical centers of the target camera and the neighboring camera;

[0117] The following takes the target camera A and the single neighborhood camera B as an example to illustrate the process of determining the first angle and the second angle:

[0118] Figure 8 is a schematic diagram of the first angle and the second angle provided by the present invention, such as Figure 8 As shown, the image to be reconstructed P and the paired image F B The position information of the target camera A and the neighboring camera B is used to determine the image P to be reconstructed of the target camera A corresponding to the ray O. A P, the line O connecting the optical center of the target camera A and the neighboring camera B A O B The first angle α between AB , and the paired image F of the neighboring camera B B Corresponding ray O B F B, connecting the optical center of the target camera A and the neighboring camera B A O B The second angle β between AB .

[0119] Then, step 720 may be executed to calculate candidate depth information of the paired image of the image to be reconstructed according to the first angle and the second angle, that is, Figure 8 O A d B , and the depth information of the image to be reconstructed is determined based on the candidate depth information. It should be noted that in this process, the similarity between the paired image and the image to be reconstructed, as well as the depth information of the paired image, can also be combined to jointly determine the depth information of the image to be reconstructed, so as to achieve point cloud reconstruction based on this depth information.

[0120] The method provided by the embodiment of the present invention can quickly obtain the first angle and the second angle through the position information of the image to be reconstructed and the paired image on the basis of the known camera posture. The candidate depth information of the paired image and the depth information of the image to be reconstructed can be efficiently obtained through the fast triangulation method. Compared with the complex triangulation algorithm in the traditional scheme, the fast triangulation method provided by the embodiment of the present invention can improve the measurement rate while ensuring the accuracy of the measurement results.

[0121] Based on the above embodiments, Fig. 9 is a flow chart of step 420 in the point cloud reconstruction method provided by the present invention, such as Fig. 9 As shown, step 720 includes:

[0122] Step 721, determining candidate depth information relative to the paired images of each neighboring camera based on the first angle and the second angle relative to each neighboring camera and its paired image, and the optical center connection line between the target camera and each neighboring camera;

[0123] Step 722 : Determine the depth information of the image to be reconstructed based on the candidate depth information of the paired images of each neighboring camera and the similarity between the paired images of each neighboring camera and the image to be reconstructed.

[0124] Specifically, in step 720, the process of determining the depth information of the image to be reconstructed according to the first angle and the second angle can be divided into the following two steps:

[0125] Step 721, after the first angle and the second angle of a single neighborhood camera and its paired image are determined through the aforementioned steps, the aforementioned process may be repeated to determine the first angle and the second angle of each neighborhood camera and its paired image relative to the target camera; subsequently, candidate depth information relative to the paired image of each neighborhood camera may be calculated based on the first angle and the second angle relative to each neighborhood camera and its paired image, and the optical center connection line between the target camera and each neighborhood camera.

[0126] Step 722, after the similarity between the paired image of a single neighborhood camera and the image to be reconstructed is determined through the aforementioned steps, the similarity determination process can be repeated to determine the similarity between the paired images of each neighborhood camera and the image to be reconstructed; thereafter, the depth information of the image to be reconstructed can be weighted based on the similarity between the paired images of each neighborhood camera and the image to be reconstructed, and the candidate depth information relative to the paired images of each neighborhood camera obtained in step 721.

[0127] The method provided by the embodiment of the present invention can quickly determine the candidate depth information of the paired images relative to each neighboring camera after calculating the first angle and the second angle relative to each neighboring camera and its paired image by a fast triangulation method, and weight the candidate depth information of the paired images relative to each neighboring camera and the similarity between the paired images of each neighboring camera and the image to be reconstructed, so as to efficiently calculate the depth information of the image to be reconstructed. The depth information of the image to be reconstructed determined by combining various aspects of the candidate depth information has a wider information inclusion degree and a stronger generalization, thereby providing strong support for the point cloud reconstruction process based on this depth information.

[0128] Based on the above embodiment, the candidate depth information of the paired images of each neighboring camera and the calculation formula of the image to be reconstructed are shown in the following formula:

[0129] Among them, the paired image F of the neighborhood camera B B The candidate depth information can be calculated by the following formula:

[0130]

[0131] Among them, depth B and O A d B represents the paired image F of the neighboring camera B B The candidate depth information, O A O B represents the optical center line between the target camera A and the neighboring camera B, α AB Represents the image P to be reconstructed of the target camera A corresponding to the ray O A P and O A O B The first angle betweenAB represents the paired image F of the neighboring camera B B Corresponding ray O B F B With O A O B The second angle between.

[0132] Paired image F of neighborhood camera C C The candidate depth information can be calculated by the following formula:

[0133]

[0134] Among them, depth C and O A d C represents the paired image F of the neighborhood camera C C The candidate depth information, O A O C represents the optical center line between the target camera A and the neighboring camera C, α AC Represents the image P to be reconstructed of the target camera A corresponding to the ray O A P and O A O C The first angle between AC represents the paired image F of the neighborhood camera C C Corresponding ray O C F C With O A O C The second angle between.

[0135] Paired image F of neighborhood camera D D The candidate depth information can be calculated by the following formula:

[0136]

[0137] Among them, depth D and O A d D Denotes the paired image F of the neighborhood camera D D The candidate depth information, O A O D represents the optical center line between the target camera A and the neighboring camera D, α AD Represents the image P to be reconstructed of the target camera A corresponding to the ray O A P and O A O D The first angle between AD Denotes the paired image F of the neighborhood camera D D Corresponding ray O D F D With O A OD The second angle between.

[0138] Paired image F of neighboring camera E E The candidate depth information can be calculated by the following formula:

[0139]

[0140] Among them, depth E and O A d E Denotes the paired image F of the neighborhood camera E E The candidate depth information, O A O E represents the optical center line between the target camera A and the neighboring camera E, α AE Represents the image P to be reconstructed of the target camera A corresponding to the ray O A P and O A O E The first angle between AE Denotes the paired image F of the neighborhood camera E E Corresponding ray O E F E With O A O E The second angle between.

[0141] The depth information of the image P to be reconstructed can be calculated by the following formula:

[0142]

[0143] Among them, depth represents the depth information of the image P to be reconstructed, scale B represents the paired image F of the neighboring camera B B The pairing score between the image to be reconstructed P, scale C represents the paired image F of the neighborhood camera C C The pairing score between the image P to be reconstructed, scale D Denotes the paired image F of the neighborhood camera D D The pairing score between the image P to be reconstructed, scale E Denotes the paired image F of the neighborhood camera E E The pairing score between it and the image P to be reconstructed.

[0144] Based on the above embodiments, Fig.10 It is a flowchart of the point cloud reconstruction process provided by the present invention, such as Fig.10 As shown, in step 130, point cloud reconstruction is performed based on the depth information of the image to be reconstructed, including:

[0145] Step 1010, determining a three-dimensional point cloud of a target camera based on depth information of the image to be reconstructed;

[0146] Step 1020, based on the three-dimensional point cloud of the target camera and the change matrix between the target camera and the world coordinate system, determine the three-dimensional point cloud of the target camera in the world coordinate system;

[0147] Step 1030, based on the point cloud fusion constraint, the three-dimensional point clouds of each target camera in the world coordinate system are fused to obtain a reconstructed point cloud; the point cloud fusion constraint includes at least one of a similarity constraint, a visibility constraint, a color correlation constraint, and a specific area constraint.

[0148] Specifically, in step 130, the process of performing point cloud reconstruction according to the depth information of the image to be reconstructed is essentially a process of fusing the depth information of the image to be reconstructed of each target camera and performing point cloud reconstruction according to the fused depth information, which specifically includes the following steps:

[0149] First, execute step 1010 to determine the three-dimensional point cloud of the target camera based on the depth information of the image to be reconstructed;

[0150] Then, step 1020 is executed to determine the transformation matrix from the target camera to the world coordinate system according to the position and posture information of the target camera, and then the three-dimensional point cloud of the target camera is converted into a three-dimensional point cloud in the world coordinate system according to the transformation matrix. After obtaining the three-dimensional point cloud of a single target camera in the world coordinate system, the above process can be repeated to determine the three-dimensional point cloud of each target camera in the world coordinate system.

[0151] After that, step 1030 can be executed to fuse the three-dimensional point clouds of each target camera in the world coordinate system. The fusion can obtain the reconstructed point cloud after point cloud reconstruction. It should be noted that the three-dimensional point cloud fusion of each target camera must also comply with the point cloud fusion constraint, that is, the three-dimensional point cloud of each target camera in the world coordinate system must be fused based on the point cloud fusion constraint. The point cloud fusion constraint here includes one or more of similarity constraint, visibility constraint, color correlation constraint and specific area constraint.

[0152] In point cloud fusion constraints, similarity constraints:

[0153] Since the calculated depth information of the image to be reconstructed of the target camera will inevitably have certain errors, when fusing the three-dimensional point clouds of each target camera in the world coordinate system, the similarity between the image to be reconstructed and the paired images of each neighboring camera can be used as a reference. That is, in the three-dimensional space, if there are other points within the agreed threshold range of a certain point, the point corresponding to the maximum matching score is selected from the matching scores of these points, and the depth information of the point is used as the depth information of the image to be reconstructed of the target camera.

[0154] Visibility constraints:

[0155] If a point is outside the agreed threshold range, but it passes through other points when projected to any neighboring camera, the point is considered invisible and deleted.

[0156] Color correlation constraints:

[0157] When the point is projected into the visible camera, quadratic linear interpolation is used in each visible camera to calculate the RGB value of the point in each visible camera, and the correlation CNCC (Color Normalized Cross Correlation) between each RGB value is calculated, and point cloud fusion is performed based on the color correlation.

[0158] Specific area constraints:

[0159] After fusing the 3D point clouds of each target camera in the world coordinate system, it is necessary to estimate the dense point cloud of a specific area (specified area) through the region selection module to reduce the impact of the background on the mesh reconstruction, thereby providing benefits for improving the efficiency of subsequent mesh reconstruction tasks.

[0160] The method provided in the embodiment of the present invention fuses the three-dimensional point clouds of each target camera in the world coordinate system based on the point cloud fusion constraints, introduces CNCC in the deep fusion process, uses RGB information to constrain the deep fusion, and improves the efficiency of subsequent mesh reconstruction through specific area constraints.

[0161] Based on the above embodiment, the three-dimensional point cloud of the target camera in the world coordinate system and the calculation formula of CNCC are shown as follows:

[0162] Among them, the three-dimensional point cloud of the target camera in the world coordinate system can be calculated by the following formula:

[0163]

[0164] in, Represents the change matrix between the target camera A and the world coordinate system, represents the 3D point cloud of the target camera A, X w Represents the three-dimensional point cloud of target camera A in the world coordinate system.

[0165] The calculation formula of CNCC is as follows:

[0166]

[0167] Among them, P irepresents the i-th visible camera among the visible cameras, P j represents the jth visible camera among the visible cameras, It represents the number of combinations of two visible cameras in the visible camera, NNC represents the normalized cross correlation, and CNNC represents the correlation between each RGB value.

[0168] Based on the above embodiment, in step 130, the depth information of the image to be reconstructed is determined based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighboring camera, and then further includes:

[0169] Determine the three-dimensional coordinates of the central pixel point of the image to be reconstructed;

[0170] Based on the three-dimensional coordinates of the central pixel point, determine the three-dimensional coordinates of each pixel point in the image to be reconstructed;

[0171] Based on the three-dimensional coordinates of each pixel point, determine the projection point of each pixel point in the coordinate system of the neighborhood camera;

[0172] Based on the three-dimensional coordinates of each projection point and the color scale of each viewing angle, a depth optimization strategy is constructed, and based on the depth optimization strategy, the depth information of the reconstructed image is calibrated.

[0173] Specifically, in step 130, after determining the depth information of the image to be reconstructed based on the position information of the image to be reconstructed and the paired image, as well as the posture information of the target camera and the neighborhood camera, before performing point cloud reconstruction based on the depth information of the image to be reconstructed, a depth optimization strategy can also be constructed to calibrate the depth information of the image to be reconstructed, so that the subsequent point cloud reconstruction task based on the depth information of the image to be reconstructed is more accurate and the point cloud reconstruction effect is better.

[0174] The depth information calibration process specifically includes the following steps:

[0175] First, determine the three-dimensional coordinates of the central pixel of the image to be reconstructed;

[0176] Then, the three-dimensional coordinates of each pixel point in the image to be reconstructed are calculated according to the three-dimensional coordinates of the central pixel point of the image to be reconstructed;

[0177] Then, according to the change matrix between the target camera and the neighboring camera, each pixel point in the image to be reconstructed is projected into the coordinate system of the neighboring camera, and the projection point of each pixel point in the coordinate system of the neighboring camera is determined;

[0178] Afterwards, a depth optimization strategy can be constructed according to the three-dimensional coordinates of each projection and the color scale of the viewing angle of each neighborhood camera, and the depth information of the reconstructed image can be optimized and calibrated based on the depth optimization strategy.

[0179] In an embodiment of the present invention, a pixel-by-pixel depth calculation mode is proposed, which uses the center pixel to calibrate the depth information to achieve fast fine-tuning of the depth information of the reconstructed image, and the fine-tuning range is small and the accuracy is high. In addition, the depth estimation is upgraded to a pixel-by-pixel estimation, which provides an excellent depth value for the subsequent depth information fusion process.

[0180] Based on the above embodiment, the calculation formula of the three-dimensional coordinates of the central pixel point of the image to be reconstructed, the three-dimensional coordinates of each pixel point in the image to be reconstructed, and the depth optimization strategy (depth optimization function) is as follows:

[0181] Among them, the three-dimensional coordinates of the central pixel point of the image to be reconstructed can be calculated by the following formula:

[0182]

[0183] Among them, O A The optical center of the target camera A, depth represents the depth information of the image to be reconstructed of the target camera A, represents the direction vector from the imaging optical center of the target camera A to the image to be reconstructed, and X represents the three-dimensional coordinates of the central pixel point of the image to be reconstructed.

[0184] The calculation formula for the three-dimensional coordinates of each pixel in the image to be reconstructed is as follows:

[0185]

[0186] Among them, h s and h t They respectively represent the two components of the normal vector of the image to be reconstructed in the imaging plane, and X(i, j) represents the three-dimensional coordinates of each pixel point in the image to be reconstructed.

[0187] The depth optimization strategy (depth optimization function) can be expressed by the following formula:

[0188] I A (i, j) = c k *I k (P k (X(i,j)))

[0189]

[0190] Among them, I A (i, j) represents the RGB value of the three-dimensional coordinates of the projection point of each pixel in the image to be reconstructed in the coordinate system of each neighboring camera, c k represents the color scale of the neighboring camera’s view angle k, I k (i, j) represents the RGB value of the three-dimensional coordinates of each projection point under viewing angle k, Pk represents the change matrix between the image to be reconstructed and each neighboring camera under the viewing angle k, and E represents the depth optimization function.

[0191] Fig.11 is the overall flow chart of the point cloud reconstruction method provided by the present invention, such as Fig.11 As shown, the method includes:

[0192] Step 1111, determining a change matrix between the target camera and the neighboring camera based on the position information of the target camera and the neighboring camera;

[0193] Step 1112, based on the change matrix between the target camera and the neighboring camera, determine the projection of the ray corresponding to the image to be reconstructed of the target camera in the coordinate system of the neighboring camera;

[0194] Step 1113, setting a sliding window on the projection, determining a candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera based on the sliding range of the sliding window, and determining a candidate image within the candidate matching area; the size of the sliding window is determined based on the size of the image to be reconstructed of the target camera;

[0195] Step 1121, determining the structural similarity and / or position similarity between the candidate image and the image to be reconstructed based on the grayscale information of the candidate image and the image to be reconstructed;

[0196] Step 1122, determining a paired image of the image to be reconstructed from the candidate images based on the structural similarity and / or position similarity between the candidate images and the image to be reconstructed;

[0197] Step 1131, based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighboring camera, respectively determine a first angle and a second angle, wherein the first angle is formed by a ray corresponding to the image to be reconstructed of the target camera and a line connecting the optical centers of the target camera and the neighboring camera, and the second angle is formed by a ray corresponding to the paired image of the neighboring camera and a line connecting the optical centers of the target camera and the neighboring camera;

[0198] Step 1132, determining candidate depth information relative to the paired images of each neighboring camera based on the first angle and the second angle relative to each neighboring camera and its paired image, and a line connecting the optical centers of the target camera and each neighboring camera;

[0199] Step 1133, determining the depth information of the image to be reconstructed based on the candidate depth information of the paired images relative to each neighboring camera and the similarity between the paired images of each neighboring camera and the image to be reconstructed;

[0200] Step 1134, determining a three-dimensional point cloud of the target camera based on the depth information of the image to be reconstructed;

[0201] Step 1135, determining the three-dimensional point cloud of the target camera in the world coordinate system based on the three-dimensional point cloud of the target camera and the change matrix between the target camera and the world coordinate system;

[0202] Step 1136, based on the point cloud fusion constraint, the three-dimensional point clouds of each target camera in the world coordinate system are fused to obtain a reconstructed point cloud, and the point cloud fusion constraint includes at least one of a similarity constraint, a visibility constraint, a color correlation constraint, and a specific area constraint.

[0203] The point cloud reconstruction method provided by the present invention only needs to take the camera posture as input, and through the image matching strategy, the depth information of the image to be reconstructed of the target camera can be determined, and on this basis, the depth information of the image to be reconstructed of each target camera is fused, and the point cloud is reconstructed based on the fused depth information. That is, on the basis of the known point cloud, by setting the area range, the scene can be divided at the point cloud level to realize point cloud reconstruction of a specific area.

[0204] The point cloud reconstruction device provided by the present invention is described below. The point cloud reconstruction device described below and the point cloud reconstruction method described above can be referenced to each other.

[0205] Fig.12 is a schematic diagram of the structure of the point cloud reconstruction device provided by the present invention, such as Fig.12 As shown, the device comprises:

[0206] A candidate image determination unit 1210 is used to determine a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighboring camera based on the position information of the target camera and the neighboring camera;

[0207] a pairing image determining unit 1220, configured to determine a pairing image of the image to be reconstructed from the candidate images based on a similarity between the candidate images and the image to be reconstructed;

[0208] The point cloud reconstruction unit 1230 is used to determine the depth information of the image to be reconstructed based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighborhood camera, and perform point cloud reconstruction based on the depth information of the image to be reconstructed.

[0209] The point cloud reconstruction device provided by the present invention determines the candidate images of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera according to the posture information of the target camera and the neighboring camera, and determines the paired image of the image to be reconstructed from the candidate images according to the similarity between the candidate image and the image to be reconstructed. The matching method based on similarity is different from the scoring criterion based on normalized cross-correlation in the traditional scheme, and significantly improves the matching accuracy. Based on the position information of the image to be reconstructed and its paired image, and the posture information of the target camera and its neighboring camera, the depth information of the image to be reconstructed is determined, and point cloud reconstruction is performed based on the depth information of the image to be reconstructed, which overcomes the defect that the point cloud reconstruction effect depends on the accuracy of the camera posture in the traditional scheme, and can realize point cloud reconstruction of a specific area by setting the area range on the basis of a known point cloud, and also provides great convenience for subsequent gridding operations.

[0210] Based on the above embodiment, the candidate image determination unit 1210 is used to:

[0211] Based on the position information of the target camera and the neighboring camera, determining the change matrix between the target camera and the neighboring camera;

[0212] Based on the change matrix between the target camera and the neighborhood camera, determining the projection of the ray corresponding to the to-be-reconstructed image of the target camera in the coordinate system of the neighborhood camera;

[0213] Based on the projection, a candidate image of the to-be-reconstructed image of the target camera within a candidate matching area of ​​the neighborhood camera is determined.

[0214] Based on the above embodiment, the candidate image determination unit 1210 is used to:

[0215] Setting a sliding window on the projection, determining a candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera based on a sliding range of the sliding window, and determining a candidate image within the candidate matching area;

[0216] The size of the sliding window is determined based on the size of the to-be-reconstructed image of the target camera.

[0217] Based on the above embodiment, the paired image determining unit 1220 is used to:

[0218] Determining a structural similarity and / or a position similarity between the candidate image and the image to be reconstructed based on grayscale information of the candidate image and the image to be reconstructed;

[0219] Based on the structural similarity and / or position similarity between the candidate image and the image to be reconstructed, a paired image of the image to be reconstructed is determined from the candidate images.

[0220] Based on the above embodiment, the device further includes a depth information determining unit, configured to:

[0221] Based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighboring camera, respectively determine a first angle and a second angle, wherein the first angle is formed by a ray corresponding to the image to be reconstructed of the target camera and a line connecting the optical centers of the target camera and the neighboring camera, and the second angle is formed by a ray corresponding to the paired image of the neighboring camera and a line connecting the optical centers of the target camera and the neighboring camera;

[0222] Depth information of the image to be reconstructed is determined based on the first angle and the second angle.

[0223] Based on the above embodiment, the depth information determining unit is used to:

[0224] Determine candidate depth information of the paired images relative to each neighboring camera based on the first angle and the second angle relative to each neighboring camera and its paired image, and a line connecting the optical centers of the target camera and each neighboring camera;

[0225] Determine the depth information of the image to be reconstructed based on the candidate depth information of the paired images relative to each neighboring camera and the similarity between the paired images of each neighboring camera and the image to be reconstructed.

[0226] Based on the above embodiment, the point cloud reconstruction unit 1230 is used to:

[0227] Determining a three-dimensional point cloud of the target camera based on the depth information of the image to be reconstructed;

[0228] Based on the three-dimensional point cloud of the target camera and the change matrix between the target camera and the world coordinate system, determine the three-dimensional point cloud of the target camera in the world coordinate system;

[0229] Based on the point cloud fusion constraint, the 3D point clouds of each target camera in the world coordinate system are fused to obtain the reconstructed point cloud;

[0230] The point cloud fusion constraint includes at least one of a similarity constraint, a visibility constraint, a color correlation constraint, and a specific area constraint.

[0231] Based on the above embodiment, the device further includes a depth information calibration unit, which is used to:

[0232] Determine the three-dimensional coordinates of the central pixel point of the image to be reconstructed;

[0233] Based on the three-dimensional coordinates of the central pixel point, determining the three-dimensional coordinates of each pixel point in the image to be reconstructed;

[0234] Based on the three-dimensional coordinates of each pixel point, determining the projection point of each pixel point in the coordinate system of the neighborhood camera;

[0235] A depth optimization strategy is constructed based on the three-dimensional coordinates of each projection point and the color scale of each viewing angle, and the depth information of the image to be reconstructed is calibrated based on the depth optimization strategy.

[0236] Fig.13 An example of a physical structure diagram of an electronic device is shown in FIG. Fig.13 As shown, the electronic device may include: a processor 1310, a communication interface 1320, a memory 1330 and a communication bus 1340, wherein the processor 1310, the communication interface 1320 and the memory 1330 communicate with each other through the communication bus 1340. The processor 1310 may call the logic instructions in the memory 1330 to execute the point cloud reconstruction method, which includes: based on the position information of the target camera and the neighboring camera, determining the candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera; based on the similarity between the candidate image and the image to be reconstructed, determining the paired image of the image to be reconstructed from the candidate image; based on the position information of the image to be reconstructed and the paired image, and the position information of the target camera and the neighboring camera, determining the depth information of the image to be reconstructed, and performing point cloud reconstruction based on the depth information of the image to be reconstructed.

[0237] In addition, the logic instructions in the above-mentioned memory 1330 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0238] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the point cloud reconstruction method provided by the above methods, and the method includes: based on the posture information of the target camera and the neighborhood camera, determining the candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighborhood camera; based on the similarity between the candidate image and the image to be reconstructed, determining the pairing image of the image to be reconstructed from the candidate image; based on the position information of the image to be reconstructed and the pairing image, and the posture information of the target camera and the neighborhood camera, determining the depth information of the image to be reconstructed, and performing point cloud reconstruction based on the depth information of the image to be reconstructed.

[0239] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the point cloud reconstruction method provided by the above-mentioned methods, the method comprising: determining, based on the posture information of a target camera and a neighborhood camera, a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighborhood camera; determining, from the candidate images, a pairing image of the image to be reconstructed based on the similarity between the candidate image and the image to be reconstructed; determining, based on the position information of the image to be reconstructed and the pairing image, and the posture information of the target camera and the neighborhood camera, the depth information of the image to be reconstructed, and performing point cloud reconstruction based on the depth information of the image to be reconstructed.

[0240] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0241] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0242] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A point cloud reconstruction method, characterized in that: include: Based on the position information of the target camera and the neighboring camera, determine the candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera; Determining a pairing image of the image to be reconstructed from the candidate images based on a similarity between the candidate images and the image to be reconstructed; Determine the depth information of the image to be reconstructed based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighborhood camera, and perform point cloud reconstruction based on the depth information of the image to be reconstructed; The determining, based on the similarity between the candidate image and the image to be reconstructed, a pairing image of the image to be reconstructed from the candidate image comprises: Determining a structural similarity and / or a position similarity between the candidate image and the image to be reconstructed based on grayscale information of the candidate image and the image to be reconstructed; Based on the structural similarity and / or position similarity between the candidate image and the image to be reconstructed, a paired image of the image to be reconstructed is determined from the candidate images.

2. The point cloud reconstruction method according to claim 1, characterized in that: The step of determining, based on the position information of the target camera and the neighboring camera, a candidate image of the image to be reconstructed of the target camera within the candidate matching area of ​​the neighboring camera comprises: Based on the position information of the target camera and the neighboring camera, determining the change matrix between the target camera and the neighboring camera; Based on the change matrix between the target camera and the neighborhood camera, determining the projection of the ray corresponding to the to-be-reconstructed image of the target camera in the coordinate system of the neighborhood camera; Based on the projection, a candidate image of the to-be-reconstructed image of the target camera within a candidate matching area of ​​the neighborhood camera is determined.

3. The point cloud reconstruction method according to claim 2, characterized in that: The step of determining, based on the projection, a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighborhood camera comprises: Setting a sliding window on the projection, determining a candidate matching area of ​​the image to be reconstructed of the target camera within the neighborhood camera based on a sliding range of the sliding window, and determining a candidate image within the candidate matching area; The size of the sliding window is determined based on the size of the to-be-reconstructed image of the target camera.

4. The point cloud reconstruction method according to any one of claims 1 to 3, characterized in that: The determining the depth information of the image to be reconstructed based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighboring camera, comprises: Based on the position information of the image to be reconstructed and the paired image, and the posture information of the target camera and the neighboring camera, respectively determine a first angle and a second angle, wherein the first angle is formed by a ray corresponding to the image to be reconstructed of the target camera and a line connecting the optical centers of the target camera and the neighboring camera, and the second angle is formed by a ray corresponding to the paired image of the neighboring camera and a line connecting the optical centers of the target camera and the neighboring camera; Depth information of the image to be reconstructed is determined based on the first angle and the second angle.

5. The point cloud reconstruction method according to claim 4, characterized in that: The determining, based on the first angle and the second angle, depth information of the image to be reconstructed includes: Determine candidate depth information of the paired images relative to each neighboring camera based on the first angle and the second angle relative to each neighboring camera and its paired image, and a line connecting the optical centers of the target camera and each neighboring camera; Determine the depth information of the image to be reconstructed based on the candidate depth information of the paired images relative to each neighboring camera and the similarity between the paired images of each neighboring camera and the image to be reconstructed.

6. The point cloud reconstruction method according to any one of claims 1 to 3, characterized in that: The point cloud reconstruction based on the depth information of the image to be reconstructed includes: Determining a three-dimensional point cloud of the target camera based on the depth information of the image to be reconstructed; Based on the three-dimensional point cloud of the target camera and the change matrix between the target camera and the world coordinate system, determine the three-dimensional point cloud of the target camera in the world coordinate system; Based on the point cloud fusion constraint, the 3D point clouds of each target camera in the world coordinate system are fused to obtain the reconstructed point cloud; The point cloud fusion constraint includes at least one of a similarity constraint, a visibility constraint, a color correlation constraint, and a specific area constraint.

7. A point cloud reconstruction device, characterized in that: include: A candidate image determination unit, configured to determine, based on the position information of the target camera and the neighboring camera, a candidate image of the image to be reconstructed of the target camera within a candidate matching area of ​​the neighboring camera; a pairing image determining unit, configured to determine a pairing image of the image to be reconstructed from the candidate images based on a similarity between the candidate images and the image to be reconstructed; A point cloud reconstruction unit, configured to determine the depth information of the image to be reconstructed based on the position information of the image to be reconstructed and the paired image, and the pose information of the target camera and the neighborhood camera, and to perform point cloud reconstruction based on the depth information of the image to be reconstructed; The paired image determination unit is used for: Determining a structural similarity and / or a position similarity between the candidate image and the image to be reconstructed based on grayscale information of the candidate image and the image to be reconstructed; Based on the structural similarity and / or position similarity between the candidate image and the image to be reconstructed, a paired image of the image to be reconstructed is determined from the candidate images.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the point cloud reconstruction method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the point cloud reconstruction method according to any one of claims 1 to 6 are implemented.

Images