Three-dimensional reconstruction and positioning method for weak texture object based on binocular vision

Abstract

The invention provides a three-dimensional reconstruction and positioning method for a weak texture object based on binocular vision, and the method comprises the steps: processing an obtained image of an object according to a camera distortion parameter in combination with an image distortion removing method, so as to form a distortion-removed image; processing the distorted image through binocular stereo correction to determine a corrected image; performing stereo matching on the left camera correction image and the right camera correction image according to a CRESTereo deep learning network to determine a disparity map; semantic segmentation is carried out on the left camera correction image or the right camera correction image based on a Unet network to obtain semantic information, a depth map is determined according to the optical center distance of the binocular camera, the focal length of the binocular camera and the disparity map, and the semantic information and the depth map are combined to determine a dense point cloud model of the object; and performing point cloud matching on the dense point cloud model and a standard point cloud model of the object to obtain depth information of a complete object point cloud so as to determine three-dimensional reconstruction and positioning of the object.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a three-dimensional reconstruction and positioning method of a weak-texture object based on binocular vision, a three-dimensional reconstruction and positioning system of a weak-texture object based on binocular vision, and a computer-readable storage medium. Background Art

[0002] In the field of computer vision, precise object positioning can provide accurate position information for defect detection, target object grasping, polishing and other downstream tasks.

[0003] Existing object positioning technologies can accurately locate objects with rich textures and regular shapes, but traditional solutions rely on multiple sets of cameras to ensure that the field of view covers the entire object, or rely on technical means such as attaching multiple labels on the surface of the object to achieve accurate positioning of the object. Therefore, the existing technology lacks technical means to accurately locate objects with weak textures and irregular shapes through a small number of viewing angle images.

[0004] In order to overcome the above-mentioned defects of the prior art, the art is in urgent need of a three-dimensional reconstruction and positioning technology for weak-texture objects based on binocular vision, which can realize three-dimensional reconstruction and precise positioning of weak-texture and irregular objects under a small number of camera perspectives. Summary of the invention

[0005] A brief summary of one or more aspects is given below to provide a basic understanding of these aspects. This summary is not an exhaustive overview of all conceived aspects, and is neither intended to identify the key or critical elements of all aspects nor to define the scope of any or all aspects. Its only purpose is to give some concepts of one or more aspects in a simplified form as a prelude to a more detailed description that will be given later.

[0006] In order to overcome the above-mentioned defects of the prior art, the present invention provides a three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision, a three-dimensional reconstruction and positioning system of weak-texture objects based on binocular vision, and a computer-readable storage medium, which can realize three-dimensional reconstruction and precise positioning of weak-texture and irregular objects under a small number of camera viewing angles.

[0007] Specifically, the above-mentioned three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision provided in the first aspect of the present invention includes the steps of: S1: processing the acquired image of the object according to the camera distortion parameters in combination with the image dedistortion method to form a dedistorted image; S2: processing the dedistorted image through binocular stereo correction to determine the corrected image, and the corrected image includes a left camera corrected image and a right camera corrected image; S3: stereo matching the left camera corrected image and the right camera corrected image according to the CRESTereo deep learning network to determine the disparity map; S4: semantically segmenting the left camera corrected image or the right camera corrected image based on the Unet network to obtain semantic information, and determining the depth map according to the optical center distance of the binocular cameras, the focal length of the binocular cameras and the disparity map, combining the semantic information with the depth map to determine the dense point cloud model of the object; and S5: performing point cloud matching on the dense point cloud model with the standard point cloud model of the object to obtain the depth information of the complete object point cloud to determine the three-dimensional reconstruction and positioning of the object.

[0008] Preferably, in one embodiment of the present invention, the image of the object is captured by the binocular camera, and camera-related data of the binocular camera is obtained by calibrating internal and external parameters of the camera, and the image includes a left camera image captured by a left camera of the binocular camera and a right camera image captured by a right camera of the binocular camera.

[0009] Preferably, in an embodiment of the present invention, the camera distortion parameters of the binocular camera include a radial distortion coefficient and a tangential distortion coefficient.

[0010] Preferably, in an embodiment of the present invention, the step S1 comprises: determining the pixel value of the dedistorted image by using a mapping relationship between the image before being processed by the image dedistortion method and the image after being processed by the image dedistortion method, and utilizing distortion transformation and interpolation method.

[0011] Preferably, in one embodiment of the present invention, the step S2 includes: jointly transforming the camera coordinate systems of the left camera and the right camera so that the imaging of the left camera calibrated image and the right camera calibrated image are on the same plane, the viewing angles of the left camera calibrated image and the right camera calibrated image are parallel, and the longitudinal coordinates of all points on the same epipolar line of the left camera calibrated image and the right camera calibrated image are consistent.

[0012] Preferably, in one embodiment of the present invention, the step S5 includes: obtaining the local feature descriptor of the dense point cloud model and the local feature descriptor of the standard point cloud model; using KDTree and a hash table to process the local feature descriptor of the dense point cloud model and the local feature descriptor of the standard point cloud model to determine an initial change matrix; and according to the initial transformation matrix, using the standard point cloud model as the source point cloud and the dense point cloud model as the target point cloud, performing secondary matching through the Fast-GICP algorithm to obtain the depth information of the complete object point cloud.

[0013] Preferably, in one embodiment of the present invention, the step of obtaining the local feature descriptor of the dense point cloud model and the local feature descriptor of the standard point cloud model includes: constructing spherical coordinates for each point of the dense point cloud model and the standard point cloud model in turn and dividing them into different sectors; and counting the neighbor point information in the different sectors to determine the local feature descriptor of the dense point cloud model and the local feature descriptor of the standard point cloud model.

[0014] Preferably, in one embodiment of the present invention, the step of using KDTree and a hash table to process the local feature descriptors of the dense point cloud model and the local feature descriptors of the standard point cloud model to determine the initial change matrix includes: using KDTree and a hash table to process the local feature descriptors of the dense point cloud model and the local feature descriptors of the standard point cloud model to determine matching point cloud pairs; using DBSCAN to perform clustering to determine a plurality of point clusters based on the composition conditions of neighbor points and point clusters within a preset distance threshold range; and determining the initial change matrix based on the offset of the centers of the outer bounding boxes of the two point clusters.

[0015] In addition, the three-dimensional reconstruction and positioning system of weak-texture objects based on binocular vision provided according to the second aspect of the present invention includes a memory and a processor. The memory stores computer instructions. The processor is connected to the memory and is configured to execute the computer instructions stored in the memory to implement the three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision provided in any one of the above embodiments.

[0016] In addition, the computer readable storage medium provided according to the third aspect of the present invention stores computer instructions. When the computer instructions are executed by the processor, the three-dimensional reconstruction and positioning method of weak texture objects based on binocular vision provided by any of the above embodiments is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above features and advantages of the present invention can be better understood after reading the detailed description of the embodiments of the present disclosure in conjunction with the following drawings. In the drawings, the components are not necessarily drawn to scale, and components with similar related properties or features may have the same or similar reference numerals.

[0018] Figure 1 A flowchart of a method for 3D reconstruction and positioning of a weak-texture object based on binocular vision according to some embodiments of the present invention is shown;

[0019] Figure 2 A schematic diagram of a three-dimensional reconstruction and positioning system for weak-texture objects based on binocular vision according to some embodiments of the present invention is shown;

[0020] Figure 3 A diagram showing the calibration result of camera internal and external parameters provided according to an embodiment of the present invention is shown;

[0021] Figure 4 A flowchart of constructing a dense point cloud model according to some embodiments of the present invention is shown;

[0022] Figure 5 A schematic diagram of an image processed by image dedistortion and binocular stereo correction according to an embodiment of the present invention is shown;

[0023] Figure 6 shows a disparity map provided according to an embodiment of the present invention;

[0024] Figure 7 A flowchart of object positioning based on point cloud matching provided according to some embodiments of the present invention is shown; and

[0025] Figure 8 A diagram showing a three-dimensional reconstruction result of an object provided according to some embodiments of the present invention is shown.

[0026] Reference numerals:

[0027] 100: 3D reconstruction and positioning methods;

[0028] S1~S6: steps;

[0029] 200: 3D reconstruction and positioning system;

[0030] 210: Memory;

[0031] 211: Computer readable storage medium;

[0032] 220: Processor;

[0033] 230: Camera; and

[0034] 500: Multiple lines. DETAILED DESCRIPTION

[0035] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the aspects described below in conjunction with the accompanying drawings and specific embodiments are only exemplary and should not be construed as limiting the scope of protection of the present invention in any way.

[0036] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0037] In addition, the terms "upper", "lower", "left", "right", "top", "bottom", "horizontal" and "vertical" used in the following description should be understood as the directions shown in the paragraph and the related drawings. Such relative terms are only used for the convenience of description and do not mean that the device described therein must be manufactured or operated in a specific direction, and therefore should not be understood as limiting the present invention.

[0038] It is understood that although the terms "first", "second", "third", etc. may be used herein to describe various components, regions, layers and / or parts, these components, regions, layers and / or parts should not be limited by these terms, and these terms are only used to distinguish different components, regions, layers and / or parts. Therefore, the first component, region, layer and / or part discussed below may be referred to as a second component, region, layer and / or part without departing from some embodiments of the present invention.

[0039] As mentioned above, existing object positioning technologies can accurately locate objects with rich textures and regular shapes, but traditional solutions rely on multiple sets of cameras to ensure that the field of view covers the entire object, or rely on technical means such as attaching multiple labels on the surface of the object to achieve accurate positioning of the object. Therefore, the existing technology lacks technical means to achieve accurate positioning of objects with weak textures and irregular shapes through a small number of perspective images.

[0040] In order to overcome the above-mentioned defects of the prior art, the present invention provides a three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision, a three-dimensional reconstruction and positioning system of weak-texture objects based on binocular vision, and a computer-readable storage medium, which can realize three-dimensional reconstruction and precise positioning of weak-texture and irregular objects under a small number of camera viewing angles.

[0041] In some non-limiting embodiments, the three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision provided by the first aspect of the present invention can be implemented via the three-dimensional reconstruction and positioning system of weak-texture objects based on binocular vision provided by the second aspect of the present invention. Specifically, the three-dimensional reconstruction and positioning system of weak-texture objects based on binocular vision can be configured with a memory and a processor. The memory includes but is not limited to the computer-readable storage medium provided by the third aspect of the present invention, on which computer instructions are stored. The processor is connected to the memory and is configured to execute the computer instructions stored on the memory to implement the three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision provided by the first aspect of the present invention.

[0042] The following will first describe the working principle of the above-mentioned three-dimensional reconstruction and positioning system for weak-texture objects based on binocular vision in conjunction with some embodiments of the three-dimensional reconstruction and positioning method for weak-texture objects based on binocular vision. Those skilled in the art will understand that these embodiments of the three-dimensional reconstruction and positioning method for weak-texture objects based on binocular vision are only some non-limiting implementation methods provided by the present invention, which are intended to clearly demonstrate the main concept of the present invention and provide some specific solutions that are convenient for the public to implement, rather than to limit all functions or all working modes of the three-dimensional reconstruction and positioning system for weak-texture objects based on binocular vision. Similarly, the three-dimensional reconstruction and positioning system for weak-texture objects based on binocular vision is also only a non-limiting implementation method provided by the present invention, and does not constitute a limitation on the execution subject and execution order of each step in these three-dimensional reconstruction and positioning methods for weak-texture objects based on binocular vision.

[0043] Please refer to Figure 1 and Figure 2 , Figure 1 A flowchart of a method for 3D reconstruction and positioning of a weak-texture object based on binocular vision according to some embodiments of the present invention is shown. Figure 2 A schematic diagram of a three-dimensional reconstruction and positioning system for weak-texture objects based on binocular vision provided according to some embodiments of the present invention is shown.

[0044] exist Figure 2In the illustrated embodiment, the three-dimensional reconstruction and positioning system 200 for weak-texture objects based on binocular vision may include a memory 210 and a processor 220, the memory 210 is configured with a computer-readable storage medium 211, and the computer-readable storage medium 211 may store computer instructions. The processor 220 may be connected to the memory 210 and configured to execute the computer instructions stored on the memory 210 to implement the three-dimensional reconstruction and positioning method 100 for weak-texture objects based on binocular vision. The three-dimensional reconstruction and positioning system 200 may also be connected to a plurality of cameras 230, preferably, the camera 230 may be a binocular camera, and the binocular camera may include a left camera and a right camera.

[0045] like Figure 1 As shown, the 3D reconstruction and positioning method 100 of a weak-texture object based on binocular vision includes step S1: processing the acquired image of the object according to camera distortion parameters in combination with an image dedistortion method to form a dedistorted image.

[0046] The three-dimensional reconstruction and positioning system 200 can collect images of the detected object through a binocular camera. The collected images may include a left camera image collected by a left camera of the binocular camera and a right camera image collected by a right camera of the binocular camera.

[0047] In a preferred embodiment, the binocular camera sensor connected to the 3D reconstruction and positioning system 200 may be RealSense D435i (manufactured by Intel Corporation), and the image size collected by the binocular camera may be a height of h=1080 and a width of w=1920.

[0048] In addition, the 3D reconstruction and positioning system 200 can obtain camera-related data of the binocular camera through camera internal and external parameter calibration, thereby determining the relationship between the cameras and the coordinate relationship between the camera and the detected object. Here, the 3D reconstruction and positioning system 200 can perform camera internal and external parameter calibration through the calibration tool of Matlab (a mathematical software) and the Zhang Zhengyou calibration method.

[0049] Please refer to Figure 3 , Figure 3 The result of calibrating the external parameters of each camera installed in the experimental site using the calibration tool based on Matlab is shown in FIG. Figure 3 shown.

[0050] Specifically, in the process of camera intrinsic calibration in a preferred embodiment, the 3D reconstruction and positioning system 200 can collect 30 images of a chessboard with a size of 18×18×12 mm, and ensure that the chessboard in each image occupies the entire field of view of the camera. In the process of camera extrinsic calibration, a chessboard is simulated by attaching markers every 1m on the ground and an image of the chessboard formed on the ground is collected. Afterwards, the camera coordinate system and the image coordinate system can be transformed through the camera intrinsic parameter matrix K. The specific change process is as follows:

[0051]

[0052] in, represents the coordinates in the image coordinate system, Represents the coordinates in the camera coordinate system, z is the depth value, is the camera intrinsic parameter matrix K. Through perspective projection, the 3D reconstruction and positioning system 200 can realize the conversion from 2D (two-dimensional, 2-Dimensional) to 3D (three-dimensional, 3-Dimensional) coordinates.

[0053] The camera extrinsic matrix [R|t] can be used to realize the transformation between the world coordinate system and the camera coordinate system. The transformation relationship is as follows:

[0054]

[0055] Among them, t is the translation vector, R is the rotation matrix, Represents the coordinates in the world coordinate system.

[0056] Here, the image coordinate system takes the center of the image plane as the coordinate origin, and the X-axis and Y-axis are parallel to the two vertical sides of the image plane respectively; the camera coordinate system is a three-dimensional coordinate system centered on the camera, with the optical center of the camera as the coordinate origin, the X-axis and Y-axis are parallel to the X-axis and Y-axis of the image coordinate system respectively, and the Z-axis is the optical axis. Those skilled in the art can determine the world coordinate system by determining a reference coordinate system, and then determine the actual three-dimensional coordinates of a point in the camera coordinate system based on the rotation and translation transformation between the camera coordinate system and the world coordinate system.

[0057] In addition, through the calibration of camera internal and external parameters, the camera-related data acquired by the 3D reconstruction and positioning system 200 may also include camera distortion parameters. Here, the camera distortion parameters may include:

[0058] (k1,k2,p1,p2,k3),

[0059] Among them, k1, k2, k3 are radial distortion coefficients, and p1 and p2 are tangential distortion coefficients.

[0060] Please refer to Figure 4 , Figure 4 A flowchart for constructing a dense point cloud model provided according to some embodiments of the present invention is shown.

[0061] Please combine Figure 1 and Figure 4 , the 3D reconstruction and positioning system 200 can execute step S1, and process the acquired image according to the determined camera distortion coefficient in combination with the image dedistortion method to further acquire a dedistorted image after dedistortion. In this process, the 3D reconstruction and positioning system 200 can determine the pixel value of the dedistorted image by using the mapping relationship between the image before and after the image dedistortion method, and by using the distortion transformation and interpolation method. Figure 4 As shown, the image processed by the image dedistortion method includes an image captured by a left camera and an image captured by a right camera.

[0062] Afterwards, the 3D reconstruction and positioning method 100 may include step S2: processing the dedistorted image by binocular stereo correction to determine a corrected image, where the corrected image includes a left camera corrected image and a right camera corrected image.

[0063] The 3D reconstruction and positioning system 200 can transform the camera coordinate systems of the left camera and the right camera together so that the images of the left camera correction image and the right camera correction image are on the same plane, the viewing angles of the left camera correction image and the right camera correction image are parallel, and the longitudinal coordinates of all points on the same epipolar line of the left camera correction image and the right camera correction image are consistent. In a preferred embodiment, the 3D reconstruction and positioning system 200 can use the function of OpenCV (Open Source Computer Vision Library, a cross-platform computer vision library) to perform binocular stereo correction on the dedistorted image.

[0064] Specifically, the camera-related data used for binocular stereo calibration may include left camera internal parameters, right camera internal parameters, the size of the calibrated image, the rotation matrix of the camera coordinate system of the left camera to the camera coordinate system of the right camera, and the translation matrix. The 3D reconstruction and positioning system 200 may first use the rotation matrix of the camera coordinate system of the right camera to the camera coordinate system of the left camera. Transform the perspective of the right camera so that it is parallel to the perspective of the left camera. Then, use the correction transformation to make the images of the left and right cameras coplanar. Here, the correction transformation of the camera coordinate system of the left and right cameras is as follows:

[0065]

[0066] Among them, R L Represents the rotation matrix of aligning the image captured by the left camera to the same imaging plane, R RRepresents the rotation matrix of the image captured by the right camera aligned to the same imaging plane. The image captured by the left camera and the image captured by the right camera are the original images directly captured by the binocular camera. rect is the correction matrix, denoted by Specifically, it can be expressed as:

[0067]

[0068] Among them, r1 is the x coordinate system of the left camera after rotation and alignment to the same imaging plane. ′ The direction cosines of the coordinate axis relative to the three axes of the original coordinate system, x ′ The direction of the coordinate axis is consistent with the baseline of the binocular camera, that is, the line connecting the optical centers of the left and right cameras. r2 represents the y coordinate system of the left camera after rotation and alignment to the same imaging plane. ′ The direction cosines of the coordinate axis relative to the three axes of the original coordinate system. r3 represents the z axis of the left camera coordinate system after rotation and alignment to the same imaging plane. ′ The direction cosines of the coordinate axes relative to the three axes of the original coordinate system, r2 and r3 satisfy the right-hand criterion.

[0069] Please refer to Figure 5 , Figure 5 A schematic diagram of an image processed by image dedistortion and binocular stereo correction according to an embodiment of the present invention is shown.

[0070] like Figure 5 As shown in FIG. 1 , the images captured by the left camera and the right camera are transformed into the left camera correction image and the right camera correction image after image dedistortion and binocular stereo correction. To reflect the left and right camera alignment effect, multiple lines 500 are added to the left camera correction image and the right camera correction image. After alignment, the three-dimensional point reprojection of the same object ( Figure 5 The multiple lines 500 (the vertices of the cube are used for demonstration) have the same Y coordinates in the left and right cameras, that is, they are on the same horizontal line. This means that the viewing angles of the left and right cameras are aligned and the imaging is aligned.

[0071] Please continue to refer to Figure 1 and Figure 4 The three-dimensional reconstruction and positioning method 100 may include step S3: stereo matching the left camera rectified image and the right camera rectified image according to the CRESTereo deep learning network (Cascaded Recurrent Stereo matching network) to determine a disparity map.

[0072] Please refer to Figure 6 , Figure 6 A disparity map provided according to an embodiment of the present invention is shown.

[0073] In a preferred embodiment, the disparity map of the 3D reconstruction and positioning system 200 can be determined according to the disparity map determined by the CRESTereo deep learning network as follows: Figure 6 As shown. The three-dimensional reconstruction and positioning system 200 uses the adaptive group correlation layer structure of the CRESTereo deep learning network to find the similar parts of the left camera corrected image and the right camera corrected image, that is, to find the pixel points of the left camera corrected image and the right camera corrected image corresponding to the same point in the real world. After that, the difference between the two corresponding pixel points in the two images is calculated to obtain the depth information of the pixel point, and the disparity map is determined accordingly. Specifically, the position (u, v) of the left camera corrected image and the pixel position (u ′ ,v ′ ) can be related as follows:

[0074]

[0075] Where Corr(u,v,d) is the correlation of the dth matching pair in the left camera rectified image, and the range of d is only on the epipolar line. Here, the plane formed by the real-world points and the optical centers of the left and right cameras can be the epipolar line. The intersection line between the plane and the imaging planes of the left and right cameras can be the epipolar line. The position (u′,v′) of the right camera rectified image and the position (u,v) of the left camera rectified image constitute a matching pair, C is the number of feature channels of the CRESTereo deep learning network, and P l It represents the feature map of the left camera rectified image after being extracted by the convolutional neural network (CNN) of the CRESTereo deep learning network, P r It represents the feature map of the right camera rectified image after being extracted by the convolutional neural network of the CRESTereo deep learning network.

[0076] In this way, by stereo matching the left camera rectified image and the right camera rectified image based on the CRESTereo deep learning network, the 3D reconstruction and positioning system 200 can improve the feature extraction of weak texture objects, thereby obtaining accurate pixel-by-pixel disparity.

[0077] Please continue to refer to Figure 1 The 3D reconstruction and positioning system 200 can execute step S4: perform semantic segmentation on the left camera corrected image or the right camera corrected image based on the Unet network (a semantic segmentation algorithm) to obtain semantic information, and determine the depth map according to the optical center distance of the binocular cameras, the focal length of the binocular cameras and the disparity map, and combine the semantic information with the depth map to determine a dense point cloud model of the object.

[0078] In a preferred embodiment, the 3D reconstruction and positioning system 200 performs semantic segmentation on the left camera rectified image based on the Unet network to obtain semantic information, thereby distinguishing the pixels of the detected object from the background pixel values, and the foreground and background of the detected object are separated. Specifically, the backbone network used by the Unet network is the VGG network (Visual Geometry Group Network), and the 3D reconstruction and positioning system 200 sets the label of the object to be detected to obj and the background label to background. Afterwards, the 3D reconstruction and positioning system 200 traverses each pixel of the segmented image, sets the BGR (a picture format) value of the pixel belonging to the object (obj) to (0,0,1), and sets the BGR value of other pixels to (0,0,0). Here, in the preferred embodiment, the segmented image is the left camera rectified image.

[0079] Afterwards, the distance between the optical centers of the binocular cameras, the focal length of the binocular cameras and the disparity map determined in step S4 are combined to determine the depth map. In this preferred embodiment, the 3D reconstruction and positioning system 200 can continue to traverse the pixels of the left camera calibrated image after semantic segmentation, and convert the disparity of the pixels with BGR values ​​of (0,0,1) into depth values. The conversion process is as follows:

[0080]

[0081] Among them, b is the distance between the optical centers of the left and right cameras, that is, the baseline length of the binocular camera, f is the focal length of the binocular camera, and d is the parallax value of the position (x, y) in the corrected image of the left camera.

[0082] Afterwards, if Figure 4 As shown, the 3D reconstruction and positioning system 200 can combine semantic information with the depth map, and use the semantic information obtained by semantic segmentation to obtain only the depth information of the object, thereby obtaining a dense point cloud model of the object to complete the dense reconstruction of the object. Here, the dense point cloud model can be determined based on the conversion between the image pixels and the world coordinate system, and the conversion relationship is as follows:

[0083]

[0084] Among them, (u, v) is the pixel coordinate, that is, the position (u, v) in the corrected image of the left camera, (X w ,Y w ,Z) is the corresponding coordinate of the pixel coordinate in the world coordinate system, (c x ,c y ) is the coordinate of the optical center in the image coordinate system.

[0085] Please continue to refer to Figure 1The 3D reconstruction and positioning system 200 may execute step S5: performing point cloud matching between the dense point cloud model and the standard point cloud model of the object to obtain depth information of the complete object point cloud to determine the 3D reconstruction and positioning of the object.

[0086] Since the dense point cloud model obtained using only a single set of binocular cameras cannot reconstruct the complete shape of the object, and the standard point cloud model of the object prepared in advance does not have depth information, it is necessary to match the two point cloud models to combine the depth information and shape information of the two point cloud models.

[0087] In a preferred embodiment, the 3D reconstruction and positioning system 200 may first obtain a local feature descriptor of a dense point cloud model and a local feature descriptor of a standard point cloud model.

[0088] Please refer to Figure 7 , Figure 7 A flowchart of object positioning based on point cloud matching provided according to some embodiments of the present invention is shown.

[0089] like Figure 7 As shown, the 3D reconstruction and positioning system 200 performs point cloud matching on a dense point cloud model with depth information and a standard point cloud model of an object with a complete shape. In order to better achieve point cloud matching, the 3D reconstruction and positioning system 200 can construct local feature descriptors to perform point cloud matching. Specifically, the 3D reconstruction and positioning system 200 can sequentially construct spherical coordinates for each point of the dense point cloud model and the standard point cloud model and divide them into different sectors, and then count the neighbor point information in different sectors (including the number of neighbor points of the point in different sectors, the relative position of the point and the adjacent points, the relative azimuth angle, and the relative polar angle information). Based on the neighbor point information, the 3D reconstruction and positioning system 200 can construct a two-dimensional normalized histogram and count the distribution characteristics of the point cloud data at different angles to represent the local feature descriptors of the dense point cloud model and the local feature descriptors of the standard point cloud model. In one embodiment, the 3D reconstruction and positioning system 200 can sequentially use each point of the dense point cloud model and the standard point cloud model as the sphere center, establish a sphere with a radius of 5, divide it into 6 sectors in equal proportion, and establish 4 concentric spheres inside the sectors. The radii of the concentric spheres are as follows:

[0090]

[0091] Among them, R j represents the radius of the concentric sphere, r min represents the minimum radius threshold, r max represents the maximum radius threshold. In this embodiment, r min =1,j∈[1,5],J=5,r max =5.

[0092] Thus, the 3D reconstruction and positioning system 200 can construct a dense point cloud model of the object to be positioned and a local feature descriptor of the standard point cloud model.

[0093] Afterwards, the 3D reconstruction and positioning system 200 can locate the object according to the determined local feature descriptor. First, the 3D reconstruction and positioning system 200 can use KDTree (K-Dimensional Tree) and hash tables to process the local feature descriptors of the dense point cloud model and the local feature descriptors of the standard point cloud model for initial transformation. The 3D reconstruction and positioning system 200 can use KDTree and hash tables to find point cloud pairs that match the dense point cloud model and the standard point cloud model, and calculate the displacement relationship between similar point cloud pairs as the initial change matrix.

[0094] exist Figure 7 In the illustrated embodiment, after the 3D reconstruction and positioning system 200 finds matching point cloud pairs based on local feature descriptors, it uses DBSCAN (Density-Based Spatial Clustering of Applications with Noise, a clustering algorithm) for clustering. The 3D reconstruction and positioning system 200 can determine multiple point clusters based on the neighboring points and the formation conditions of the point clusters within a preset distance threshold range. The initial change matrix is ​​then determined by the offset of the centers of the outer bounding boxes of the two point clusters. For example, the maximum distance between two points that are considered as neighboring points (i.e., the preset distance threshold range of neighboring points) can be set to 0.2, and the minimum number of points that can constitute a single independent cluster (i.e., the formation condition of the point cluster) can be set to 8 to determine multiple point clusters. The initial change matrix is ​​then determined by the offset of the centers of the outer bounding boxes of the two point clusters, thereby obtaining the overall offset of the two point clusters:

[0095] T1=P target_center -P source_center ,

[0096] Among them, T1 represents the initial change matrix, P target_center and P source_center They are respectively the center of the outer bounding box of part of the dense point cloud model and the center of the outer bounding box of the standard point cloud model.

[0097] like Figure 7As shown, according to the initial transformation matrix, the 3D reconstruction and positioning system 200 uses the standard point cloud model as the source point cloud and the dense point cloud model as the target point cloud, and performs secondary matching through the Fast-GICP (Fast-Generalized iterative closest point) algorithm to obtain the depth information of the complete object point cloud. Specifically, the 3D reconstruction and positioning system 200 can set the relevant parameters in the Fast-GICP algorithm: the farthest correlation distance is 5, and the thread is 4. Thus, the 3D reconstruction and positioning system 200 achieves the acquisition of the depth information of the complete object point cloud through fine matching to complete the object positioning based on point cloud matching. The object positioning based on point cloud matching can avoid the problem of large positioning error caused by incomplete point cloud.

[0098] In this way, the three-dimensional reconstruction and positioning system 200 constructs a local feature descriptor for a dense point cloud model that contains depth information but incomplete shape and a standard point cloud model that does not contain depth information but has a complete shape, and uses KDTree to match them. On this basis, it performs initial transformation and secondary fine matching based on Fast-GICP, thereby obtaining accurate object positioning.

[0099] Please refer to Figure 8 , Figure 8 A diagram showing a three-dimensional reconstruction result of an object provided according to some embodiments of the present invention is shown.

[0100] like Figure 8 As shown, the three-dimensional reconstruction and positioning system 200 can complete the three-dimensional reconstruction and positioning of the object through the acquired depth information of the complete object point cloud.

[0101] In summary, the three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision provided by the present invention can realize accurate positioning of objects by relying on only a small number of cameras. By applying stereo matching based on the CREStereo deep learning neural network model to the left and right camera images after dedistortion and stereo correction, the feature extraction of weak-texture objects is improved. Combined with the foreground and background segmentation of the object, the depth information of the object to be positioned is quickly and accurately obtained, and then a dense point cloud model is obtained; the dense point cloud model containing depth information but incomplete shape is matched with the standard point cloud model that does not contain depth information but has a complete shape, so as to obtain accurate object positioning. In this way, three-dimensional reconstruction and accurate positioning of weak-texture and irregular objects are realized under the perspective of a small number of cameras.

[0102] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0103] Those skilled in the art will appreciate that information, signals, and data may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips cited throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0104] Those skilled in the art will further appreciate that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps are generally described above in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. The technician may implement the described functionality in different ways for each specific application, but such implementation decisions should not be interpreted as resulting in a departure from the scope of the present invention.

[0105] The various illustrative logic modules and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in cooperation with a DSP core, or any other such configuration.

[0106] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, a storage medium may be integrated into a processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor and the storage medium may reside in a user terminal as discrete components.

[0107] In one or more exemplary embodiments, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented as a computer program product in software, each function may be stored on or transmitted by a computer-readable medium as one or more instructions or codes. Computer-readable media include both computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one place to another. Storage media may be any available medium that can be accessed by a computer. As an example and not limitation, such a computer-readable medium may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer. Any connection is also properly referred to as a computer-readable medium. For example, if the software is transmitted from a website, a server, or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. Disk and disc as used herein include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, wherein disk often reproduces data magnetically, while disc reproduces data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0108] The previous description of the disclosure is provided to enable any person skilled in the art to make or use the disclosure. Various modifications to the disclosure will be apparent to those skilled in the art, and the general principles defined herein may be applied to other variations without departing from the spirit or scope of the disclosure. Thus, the disclosure is not intended to be limited to the examples and designs described herein, but should be granted the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A three-dimensional reconstruction and positioning method for weak-texture objects based on binocular vision, characterized in that: Includes steps: S1: processing the acquired image of the object according to the camera distortion parameter in combination with the image dedistortion method to form a dedistorted image; S2: Processing the dedistorted image by binocular stereo correction to determine a corrected image, wherein the corrected image includes a left camera corrected image and a right camera corrected image; S3: According to the CRESTereo deep learning network, stereo matching is performed on the left camera rectified image and the right camera rectified image to determine a disparity map; S4: performing semantic segmentation on the left camera rectified image or the right camera rectified image based on a Unet network to obtain semantic information, and determining a depth map according to the optical center distance of the binocular cameras, the focal length of the binocular cameras and the disparity map, and combining the semantic information with the depth map to determine a dense point cloud model of the object; as well as S5: performing point cloud matching on the dense point cloud model and the standard point cloud model of the object to obtain depth information of the complete object point cloud to determine the three-dimensional reconstruction and positioning of the object.

2. The three-dimensional reconstruction and positioning method according to claim 1, characterized in that: The image of the object is collected by the binocular camera, and the camera-related data of the binocular camera is obtained by calibrating the internal and external parameters of the camera. The image includes a left camera image collected by the left camera of the binocular camera and a right camera image collected by the right camera of the binocular camera.

3. The three-dimensional reconstruction and positioning method according to claim 1, characterized in that: The camera distortion parameters of the binocular camera include radial distortion coefficient and tangential distortion coefficient.

4. The three-dimensional reconstruction and positioning method according to claim 1, characterized in that: The step S1 comprises: The pixel values ​​of the dedistorted image are determined by using the mapping relationship between the image before and after being processed by the dedistortion method, and by using distortion transformation and interpolation methods.

5. The three-dimensional reconstruction and positioning method according to claim 1, characterized in that: The step S2 comprises: The camera coordinate systems of the left camera and the right camera are transformed together so that the images of the left camera correction image and the right camera correction image are on the same plane, the viewing angles of the left camera correction image and the right camera correction image are parallel, and the longitudinal coordinates of all points on the same epipolar line of the left camera correction image and the right camera correction image are consistent.

6. The three-dimensional reconstruction and positioning method according to claim 1, characterized in that: The step S5 comprises: Acquire a local feature descriptor of the dense point cloud model and a local feature descriptor of the standard point cloud model; Processing the local feature descriptors of the dense point cloud model and the local feature descriptors of the standard point cloud model using KDTree and a hash table to determine an initial change matrix; and According to the initial transformation matrix, with the standard point cloud model as the source point cloud and the dense point cloud model as the target point cloud, a secondary matching is performed through the Fast-GICP algorithm to obtain the depth information of the complete object point cloud.

7. The three-dimensional reconstruction and positioning method according to claim 6, characterized in that: The step of obtaining the local feature descriptor of the dense point cloud model and the local feature descriptor of the standard point cloud model comprises: sequentially constructing spherical coordinates for each point of the dense point cloud model and the standard point cloud model and dividing the points into different sectors; and Neighbor point information in the different sectors is counted to determine a local feature descriptor of the dense point cloud model and a local feature descriptor of the standard point cloud model.

8. The three-dimensional reconstruction and positioning method according to claim 6, characterized in that: The step of using KDTree and a hash table to process the local feature descriptor of the dense point cloud model and the local feature descriptor of the standard point cloud model to determine an initial change matrix includes: Using KDTree and a hash table to process the local feature descriptors of the dense point cloud model and the local feature descriptors of the standard point cloud model to determine a matching point cloud pair; Using DBSCAN to perform clustering, and determining a plurality of point clusters according to the conditions for constituting neighbor points and point clusters within a preset distance threshold range; and An initial change matrix is ​​determined according to the offset of the centers of the outer bounding boxes of the two point clusters.

9. A three-dimensional reconstruction and positioning system for weak-texture objects based on binocular vision, characterized in that: include: a memory having computer instructions stored thereon; as well as A processor is connected to the memory and is configured to execute computer instructions stored in the memory to implement the three-dimensional reconstruction and positioning method of weak-texture objects based on binocular vision as described in any one of claims 1 to 8.

10. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by the processor, the three-dimensional reconstruction and positioning method of a weak-texture object based on binocular vision as claimed in any one of claims 1 to 8 is implemented.

Images