A method and system for estimating the confidence of three-dimensional reconstruction target position

Abstract

The present invention relates to a method and system for estimating the confidence of a three-dimensional reconstructed target position, the method comprising: using a deep learning model to estimate the depth value of each pixel of the target in the current frame, and calculating the coordinates of the target in the camera coordinate system according to the depth value of each pixel; clustering the contour point cloud of each target based on the coordinates of the target in the camera coordinate system using a clustering algorithm to obtain a target point set; performing plane fitting on the target point set to obtain an inner point set and an outer point set of a plane model; and calculating the confidence of target reconstruction according to the inner point set and the outer point set of the plane model. The present invention improves the robustness of the model and the accuracy of target reconstruction by combining the deep learning model to estimate the depth value of pixels in multiple frames of images with the clustering method, and provides an accuracy evaluation standard.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and three-dimensional reconstruction, and in particular relates to a method and system for estimating the confidence of a three-dimensional reconstruction target position. Background Art

[0002] Monocular 3D refers to the process of reconstructing the 3D information of the environment space within the field of view through image frame information using a monocular vision sensor. Monocular 3D reconstruction mainly uses two methods: multi-view geometry and deep learning. Compared with the traditional multi-view geometry intersection method, the deep learning strategy has the following two significant advantages: (1) The reconstruction result does not depend on the camera pose, thereby improving the flexibility and reliability of reconstruction. In traditional multi-view geometry mapping, the camera pose is generally obtained through IMU (Inertial Measurement Unit), and its measurement results are affected by environmental noise and accumulated errors; (2) It can obtain pixel-by-pixel 3D spatial information, effectively improving the completeness of reconstruction. The multi-view intersection method is based on the depth solution of the target feature points, which makes it difficult to obtain dense 3D information in the image; while deep learning predicts the depth of the image pixel by pixel, which can not only restore the 3D information of all targets in the field of view, but also effectively reconstruct the spatial contours of the target of interest, greatly improving the completeness of the 3D information of the spatial target.

[0003] The camera will continuously observe the target and predict its spatial orientation information during the movement. In order to improve the accuracy of target depth prediction, it is necessary to fuse multiple consecutive frames of observation. Environmental factors, such as illumination changes between consecutive frames, dynamic targets, and astigmatism on the surface of objects, will bring additional noise and errors to the neural network, thereby affecting the accuracy of single-frame reconstruction. Summary of the invention

[0004] In order to solve the problem of target position uncertainty analysis in deep learning 3D reconstruction and improve the accuracy and reliability of target 3D reconstruction, a first aspect of the present invention provides a method for estimating the confidence of the target position of 3D reconstruction, including: using a deep learning model to estimate the depth value of each pixel of the target in the current frame, and calculating the coordinates of the target in the camera coordinate system according to the depth value of each pixel; based on the coordinates of the target in the camera coordinate system, using a clustering algorithm to cluster the contour point cloud of each target to obtain a target point set; performing plane fitting on the target point set to obtain an inner point set and an outer point set of a plane model; and calculating the confidence of the target reconstruction according to the inner point set and the outer point set of the plane model.

[0005] In some embodiments of the present invention, the calculating of the coordinates of the target in the camera coordinate system according to the depth value of each pixel is achieved by the following steps:

[0006]

[0007] Among them, (X c i , Y c i ) represents the coordinates of the i-th pixel in the camera coordinate system; Z i represents the depth value of the i-th pixel; (u i ,v i ) represents the coordinates of the i-th pixel in the current frame image; (f x , f y , C x , C y ) is the camera calibration internal parameter.

[0008] In some embodiments of the present invention, the method of clustering the contour point cloud of each target based on the coordinates of the target in the camera coordinate system using a clustering algorithm to obtain a target point set includes the following steps: calculating the Euclidean distance of each point in the target contour point cloud in the camera coordinate system; dividing the target contour point cloud into a noise point set and a target point set based on the Euclidean distance and the K-Means clustering algorithm.

[0009] In some embodiments of the present invention, the calculation of the confidence of target reconstruction based on the inner point set and the outer point set of the plane model is implemented by the following steps:

[0010]

[0011] Among them, bf represents the confidence, p is the ratio of the total number of internal points to the total number of all points, and KL is the weighted sum of the divergence coefficients of the internal points in three dimensions.

[0012] Furthermore, the confidence level is normalized to the interval [0, 1].

[0013] In the above embodiment, the method of estimating the depth value of each pixel of the target in the current frame by using the deep learning model includes: estimating the depth value of each pixel of the target in the current frame by using the deep learning model The depth value of each pixel is calculated based on the scale factor. Perform scale recovery.

[0014] According to a second aspect of the present invention, a system for estimating the confidence of a three-dimensional reconstructed target position is provided, comprising: an estimation module for estimating the depth value of each pixel of the target in the current frame using a deep learning model, and calculating the coordinates of the target in the camera coordinate system based on the depth value of each pixel; a clustering module for clustering the contour point cloud of each target based on the coordinates of the target in the camera coordinate system using a clustering algorithm to obtain a target point set; a fitting module for performing plane fitting on the target point set to obtain an inner point set and an outer point set of a plane model; and a calculation module for calculating the confidence of the target reconstruction based on the inner point set and the outer point set of the plane model.

[0015] Furthermore, the clustering module includes a calculation unit and a division unit. The calculation unit is used to calculate the Euclidean distance of each point in the target contour point cloud in the camera coordinate system; the division unit is used to divide the target contour point cloud into a noise point set and a target point set based on the Euclidean distance and the K-Means clustering algorithm.

[0016] The third aspect of the present invention provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for estimating the confidence of the three-dimensional reconstructed target position provided in the first aspect of the present invention.

[0017] A fourth aspect of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method for estimating the confidence of the three-dimensional reconstructed target position provided in the first aspect of the present invention.

[0018] The beneficial effects of the present invention are:

[0019] 1. The present invention proposes a method for estimating the confidence of target position in 3D reconstruction based on deep learning, which can effectively evaluate the error size of the 3D reconstruction result of the current frame;

[0020] 2. The present invention provides an accuracy evaluation standard for target 3D reconstruction, guiding the improvement of target reconstruction accuracy;

[0021] 3. The present invention provides a frame-by-frame multi-target reconstruction confidence index, based on which the error distribution law of targets of different types and in different scenarios is statistically analyzed, and then samples are added to the training set in a targeted manner to improve the network robustness;

[0022] 4. The present invention provides information weight values ​​for multi-frame image fusion, thereby improving data fusion accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1A schematic diagram of a basic flow chart of a method for estimating the confidence level of a three-dimensional reconstruction target position in some embodiments of the present invention;

[0024] Figure 2 A schematic diagram of a specific flow chart of a method for estimating the confidence level of a three-dimensional reconstruction target position in some embodiments of the present invention;

[0025] Figure 3 A schematic diagram of a specific structure of a system for estimating the confidence level of a three-dimensional reconstruction target position in some embodiments of the present invention;

[0026] Figure 4 It is a schematic diagram of the structure of an electronic device in some embodiments of the present invention. DETAILED DESCRIPTION

[0027] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0028] refer to Figure 1 In a first aspect of the present invention, a method for estimating the confidence of a three-dimensional reconstructed target position is provided, comprising: S100. estimating the depth value of each pixel of the target in the current frame by using a deep learning model, and calculating the coordinates of the target in the camera coordinate system according to the depth value of each pixel; S200. clustering the contour point cloud of each target based on the coordinates of the target in the camera coordinate system by using a clustering algorithm to obtain a target point set; S300. performing plane fitting on the target point set to obtain an inner point set and an outer point set of a plane model; S400. calculating the confidence of the target reconstruction according to the inner point set and the outer point set of the plane model.

[0029] It can be understood that the above-mentioned target corresponds to a three-dimensional reconstruction target, especially a monocular three-dimensional reconstruction target. The present invention does not limit the number of targets, which means that the method for estimating the confidence of the three-dimensional reconstruction target position provided by the present invention is not only applicable to a single three-dimensional reconstruction target, but also to multiple three-dimensional reconstruction targets. The above-mentioned deep learning model includes at least AlexNet, VGG16, VGG19, MARANet, Resnet18, Resnet50, Inception series, FCN or CNN, etc., a multi-layer neural network for target detection or visual recognition.

[0030] In step S100 of some embodiments of the present invention, the calculation of the coordinates of the target in the camera coordinate system according to the depth value of each pixel is implemented by the following steps:

[0031]

[0032] Among them, (X c i , Yc i ) represents the coordinates of the i-th pixel in the camera coordinate system; Z i represents the depth value of the i-th pixel; (u i ,v i ) represents the coordinates of the i-th pixel in the current frame image; (f x , f y , C x , C y ) is the camera calibration internal parameter.

[0033] refer to Figure 2 In step S200 of some embodiments of the present invention, the clustering of the contour point cloud of each target based on the coordinates of the target in the camera coordinate system using a clustering algorithm to obtain a target point set includes the following steps: S201. Calculating the Euclidean distance of each point in the target contour point cloud in the camera coordinate system; S203. Based on the Euclidean distance and K-Means clustering algorithm, dividing the target contour point cloud into a noise point set and a target point set.

[0034] Specifically, based on step S100 and the target contour point cloud Ω(X i , Y i , Z i ), perform K-Means clustering algorithm on it, set the number of categories (take the number of categories as 2) and the Euclidean distance threshold, and divide the target contour point cloud into two categories according to the Euclidean distance threshold: the one outside the threshold (greater than the threshold) is the noise point set Ω N , the other category within the threshold (less than or equal to the threshold) is the reconstructed target point Ω1.

[0035] In step S300 of some embodiments of the present invention, performing plane fitting on the target point set to obtain the inner point set and the outer point set of the plane model includes: performing plane fitting on the target point set Ω1 based on the RANSAC algorithm, and distinguishing the inner point Ω on the plane model generated thereby. in (Interior point set) and exterior point Ωo ut (external point set), calculate Ω in The depth value of the 0.85-0.95 quantile is preferred. 0.95 ), and use it as the depth value of the target camera coordinate system in the current frame. The depth value obtained at this time is highly consistent with the actual value.

[0036] It can be understood that the selection of 0.85-0.95 quantiles complies with the 2sigma principle of the normal distribution and is in good agreement with the actual data, and 0.95 as the best embodiment further improves the reconstruction accuracy of the target.

[0037] In step S400 of some embodiments of the present invention, the calculation of the confidence of the target reconstruction based on the inner point set and the outer point set of the plane model is implemented by the following steps:

[0038]

[0039] Among them, bf represents the confidence, p is the ratio (ratio) of the total number of internal points to the total number of all points, and KL is the weighted sum of the divergence coefficients of the internal points in three dimensions (the three dimensions of the target contour point cloud).

[0040] Furthermore, the confidence level is normalized to the interval [0, 1]. Specifically, the calculation is as follows:

[0041]

[0042] Among them, O max , O min are the maximum and minimum values ​​of bf respectively, and belief is the final reconstruction confidence.

[0043] In step S100 of the above embodiment, the step of estimating the depth value of each pixel of the target in the current frame by using the deep learning model includes: estimating the depth value of each pixel of the target in the current frame by using the deep learning model The depth value of each pixel is calculated based on the scale factor. Perform scale restoration. Specifically, based on the pre-trained network weight information, calculate the depth value of each pixel in the current frame Since the monocular predicted depth is a relative value, it needs to be scaled and calculated as follows: in is the scale factor, obtained through measurement.

[0044] Example 2

[0045] refer to Figure 3 According to a second aspect of the present invention, a system 1 for estimating the confidence of a three-dimensional reconstructed target position is provided, comprising: an estimation module 11, for estimating the depth value of each pixel of the target in the current frame by using a deep learning model, and calculating the coordinates of the target in the camera coordinate system according to the depth value of each pixel; a clustering module 12, for clustering the contour point cloud of each target by using a clustering algorithm based on the coordinates of the target in the camera coordinate system to obtain a target point set; a fitting module 13, for performing plane fitting on the target point set to obtain an inner point set and an outer point set of a plane model; and a calculation module 14, for calculating the confidence of target reconstruction according to the inner point set and the outer point set of the plane model.

[0046] Furthermore, the clustering module 12 includes a calculation unit and a division unit. The calculation unit is used to calculate the Euclidean distance of each point in the target contour point cloud in the camera coordinate system; the division unit is used to divide the target contour point cloud into a noise point set and a target point set based on the Euclidean distance and the K-Means clustering algorithm.

[0047] Example 3

[0048] refer to Figure 4 According to a third aspect of the present invention, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method of the first aspect of the present invention.

[0049] The electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the electronic device 500 are also stored. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0050] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 4 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 4 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0051] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above functions defined in the method of the embodiment of the present disclosure are executed. It should be noted that the computer-readable medium described in the embodiment of the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more conductors, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wire, optical cable, RF (radio frequency), etc., or any suitable combination of the foregoing.

[0052] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device. The computer-readable medium carries one or more computer programs. When the one or more programs are executed by the electronic device, the electronic device:

[0053] Computer program code for performing the operations of embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, Python, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on a user's computer, partially on a user's computer, as a separate software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0054] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0055] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for estimating the confidence of a three-dimensional reconstruction target position, characterized in that: include: The deep learning model is used to estimate the depth value of each pixel of the target in the current frame, and the coordinates of the target in the camera coordinate system are calculated according to the depth value of each pixel; Based on the coordinates of the target in the camera coordinate system, the contour point cloud of each target is clustered using a clustering algorithm to obtain a target point set; Performing plane fitting on the target point set to obtain an inner point set and an outer point set of the plane model; According to the inner point set and outer point set of the plane model, the confidence of the target reconstruction is calculated: in, bf Indicates confidence, p is the ratio of the total number of internal points to the total number of all points, KL It is the weighted sum of the divergence coefficients of the interior points in three dimensions.

2. The method for estimating the confidence of a three-dimensional reconstruction target position according to claim 1, characterized in that: The calculation of the coordinates of the target in the camera coordinate system according to the depth value of each pixel is achieved by the following steps: ， ； in,( , ) indicates the i The coordinates of the pixel in the camera coordinate system; Z i Indicates i The depth value of pixels; ( u i , v i ) indicates the i The coordinates of pixels in the current frame image; Calibrate the intrinsic parameters for the camera.

3. The method for estimating the confidence level of a three-dimensional reconstruction target position according to claim 1, characterized in that: The method of clustering the contour point cloud of each target using a clustering algorithm based on the coordinates of the target in the camera coordinate system to obtain a target point set includes the following steps: Calculate the Euclidean distance of each point in the target contour point cloud in the camera coordinate system; Based on the Euclidean distance and K-Means clustering algorithm, the target contour point cloud is divided into a noise point set and a target point set.

4. The method for estimating the confidence of a three-dimensional reconstruction target position according to claim 1, characterized in that: It also includes normalizing the confidence scores to the interval [0,1].

5. The method for estimating the confidence level of a three-dimensional reconstruction target position according to any one of claims 1 to 4, characterized in that: The method of estimating the depth value of each pixel of the target in the current frame by using the deep learning model includes: Use the deep learning model to estimate the depth value of each pixel of the target in the current frame ; The depth value of each pixel is calculated based on the scale factor. Perform scale recovery.

6. A system for estimating the confidence of a three-dimensional reconstructed target position, characterized in that: include: An estimation module, used to estimate the depth value of each pixel of the target in the current frame by using a deep learning model, and calculate the coordinates of the target in the camera coordinate system according to the depth value of each pixel; A clustering module is used to cluster the contour point cloud of each target based on the coordinates of the target in the camera coordinate system using a clustering algorithm to obtain a target point set; A fitting module, used for performing plane fitting on the target point set to obtain an inner point set and an outer point set of the plane model; A calculation module is used to calculate the confidence of target reconstruction based on the inner point set and the outer point set of the plane model: ; in, bf Indicates confidence, p is the ratio of the total number of internal points to the total number of all points, KL It is the weighted sum of the divergence coefficients of the interior points in three dimensions.

7. The three-dimensional reconstruction target position confidence estimation system according to claim 6, characterized in that: The clustering module includes a calculation unit and a division unit. The calculation unit is used to calculate the Euclidean distance of each point in the target contour point cloud in the camera coordinate system; The division unit is used to divide the target contour point cloud into a noise point set and a target point set based on the Euclidean distance and K-Means clustering algorithm.

8. An electronic device comprising: one or more processors; A storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement the method for estimating the confidence of the three-dimensional reconstruction target position as described in any one of claims 1 to 5.

9. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method for estimating the confidence level of a three-dimensional reconstruction target position as claimed in any one of claims 1 to 5 is implemented.

Images