Open type three-dimensional reconstruction method, automatic depth positioning method, equipment and robot

Abstract

The invention provides an open type three-dimensional reconstruction method. The three-dimensional reconstruction method comprises the following steps: collecting a multi-view image of a target by using a visual module; obtaining a dense point cloud based on a three-dimensional reconstruction algorithm; and post-processing the dense point cloud to obtain a dense target point cloud. According to the method, dense target point clouds are screened through original open type three-dimensional reconstruction and AI post-processing, so that the data volume of subsequent processing and analysis is reduced, the data processing efficiency is effectively improved, and the three-dimensional reconstruction speed on a conventional computing power platform is greatly improved. In addition, the invention also provides an automatic depth positioning method, equipment and a robot.

Description

Technical Field

[0001] The present application relates to the field of three-dimensional reconstruction technology, and in particular, to an open three-dimensional reconstruction method, an automatic depth positioning method, a device and a robot. Background Art

[0002] Three-dimensional reconstruction technology generates three-dimensional maps by acquiring three-dimensional data of the scene, and has shown broad application prospects in many fields such as autonomous driving, obstacle avoidance navigation, virtual reality, augmented reality, architecture, and medical care.

[0003] The development of artificial intelligence and machine learning models has made it possible for robots to enter home, commercial and industrial scenarios. In the actual task execution process, robots need to accurately perceive the surrounding environment so that they can understand the complex environment and predict the future state. The application of 3D reconstruction technology can help robots understand the spatial structure and layout of the surrounding environment, provide strong support for the robot's motion positioning, path planning, obstacle avoidance navigation, etc.; it can also obtain the 3D shape and structure information of environmental objects, so as to more accurately identify the object, effectively improving the accuracy of the robot when performing tasks such as grasping, operation, and classification.

[0004] In addition to identifying individual objects, robots also need to understand the semantic relationship of the entire scene to determine their own position, posture, and motion prediction. 3D reconstruction technology can help robots analyze the distribution, relative position, and semantic information of objects in a scene, thereby achieving an understanding of the entire scene, which helps robots perform tasks and make decisions in complex environments.

[0005] However, the current 3D reconstruction technology has many drawbacks and can no longer meet the needs of the development of artificial intelligence. 3D reconstruction technology is mainly divided into two categories. One is 3D reconstruction SfM (Structure from Motion) based on traditional geometric methods; the other is 3D reconstruction based on neural radiation fields, such as NERF (Neural Radiance Fields) and its extension 3DGS (3DGaussian Splatting). Although SfM has low requirements for computing power and relatively fast reconstruction speed, it has problems such as poor accuracy and adaptability; although algorithms such as NERF have high accuracy, they have high requirements for computing power.

[0006] For example, even if the NERF-based 3D reconstruction algorithm is run on a high-performance computing platform (such as NVIDIA RTX3090), it will take dozens of minutes to reconstruct the scene. On a conventional computing platform, such as the RK3588 development platform with a computing power of 6TOPS (computing power is a few tenths of RTX 3090), the reconstruction time will be unacceptable for the above scenario.

[0007] Therefore, how to achieve both a significant increase in the speed of 3D reconstruction and a higher accuracy of 3D reconstruction on a conventional computing platform to meet the application requirements of actual scenarios (including kitchen robot scenarios) is an urgent problem to be solved. Summary of the invention

[0008] On the one hand, the present application provides an open three-dimensional reconstruction method for automatic depth targets, characterized in that the three-dimensional reconstruction method includes: using a visual module to collect multi-perspective images of the target; obtaining a dense point cloud based on a three-dimensional reconstruction algorithm; and post-processing the dense point cloud to obtain a dense target point cloud.

[0009] In one embodiment, the method for obtaining the dense point cloud includes: obtaining a reconstructed sparse point cloud through the three-dimensional reconstruction algorithm; providing scale information to the three-dimensional reconstruction algorithm to correct the scale of the sparse point cloud; and obtaining a dense point cloud based on the corrected sparse point cloud.

[0010] In one embodiment, the method for obtaining the sparse point cloud includes: extracting feature points of the multi-view image through a three-dimensional reconstruction algorithm; matching feature points of different images in the multi-view image; calculating the disparity of the matched feature points between the different images; estimating the three-dimensional coordinates of the matched feature points in the three-dimensional space using the disparity and the pose information of the visual module; and combining all the estimated three-dimensional coordinates to reconstruct the sparse point cloud.

[0011] In one embodiment, the method for obtaining the dense point cloud includes: processing the pose information of the visual module, and processing at least one of the corrected sparse point cloud and the multi-view image to obtain the dense point cloud.

[0012] In one embodiment, the post-processing includes: processing the multi-view image using a pre-trained neural network model to determine the position of the target in the multi-view image; calculating the position range of the target in three-dimensional space by combining the pose information of the visual module and the position information of the target in the multi-view image; and screening out point clouds within the position range in the dense point cloud.

[0013] In one embodiment, the position range of the target in the three-dimensional space is calculated to satisfy: K ​​= T cw T obj , where K is the target pose homogeneous matrix in the world coordinate system, T obj is the target pose homogeneous matrix in the visual module coordinate system, T cw is the homogeneous matrix of the visual module pose in the world coordinate system, T obj satisfy R 3×3is the rotation matrix of the target in three-dimensional space, t 3×1 is the centroid coordinate of the dense target point cloud, t 3×1 satisfy [x i ,y i ,z i ] is the coordinate of the i-th point in the three-dimensional space, and the method for obtaining the centroid coordinates satisfies: [x 3d ,y 3d ,z 3d ] T =H[x 2d ,y 2d ,1] T , where [x 3d ,y 3d ,z 3d ] represents the three-dimensional coordinates of the target point in three-dimensional space, x 2d and 2d They respectively represent the coordinates of the target point in the two-dimensional image, and H represents the transformation matrix from the two-dimensional image to the three-dimensional point cloud calculated by the three-dimensional reconstruction algorithm.

[0014] In one embodiment, the method for acquiring the multi-view images includes: using a robotic arm to drive the visual module to move and acquire images from multiple viewpoints, wherein the end effector of the robotic arm is coupled to the visual module, and the coordinate system of the end effector is aligned with the coordinate system of the visual module.

[0015] In one embodiment, the method for providing the scale information includes: calibrating the robotic arm to determine the position and posture of the end effector in the coordinate system of the robotic arm base; moving the robotic arm from an initial position to a reference position, wherein, in the process of the robotic arm moving to the reference position, the vision module captures the multi-view image, and the control system of the robotic arm records the position and posture of the end effector; measuring the actual distance from the initial position to the reference position, and establishing a scale reference based on the recorded position and posture of the end effector; comparing the distance with the distance estimated from the sparse point cloud by the three-dimensional reconstruction algorithm to obtain a size factor, wherein the scale information includes the size reference and the size factor.

[0016] In one embodiment, the distance estimated from the sparse point cloud by the three-dimensional reconstruction algorithm includes at least one of: the estimated distance between the matched feature points, the estimated distance from the visual module to the feature points of the multi-view image, and the estimated distance from the feature points in the multi-view image to the reference position.

[0017] In one embodiment, the pose estimation of the end effector satisfies: in, is the estimated pose of the end effector given all measurements at time step k, K k is the Kalman gain at time step k, z k is the measurement vector at time step k, H k is the observation matrix at time step k.

[0018] In one embodiment, the sparse point cloud includes a sparse target point cloud and a sparse environment point cloud, and the dense point cloud includes a dense target point cloud and a dense environment point cloud.

[0019] On the other hand, the present application provides an automatic deep vision positioning method based on open three-dimensional reconstruction, the positioning method comprising: the three-dimensional reconstruction method as described in any of the above embodiments, and aligning the dense target point cloud with the posture information of the visual module to achieve the positioning of the target.

[0020] In one embodiment, the registration includes: mapping each point in the dense target point cloud to a corresponding two-dimensional position in the multi-view image to obtain two-dimensional information; based on the posture information of the visual module, fusing the two-dimensional information with the three-dimensional information in the dense target point cloud to give each point in the dense target point cloud depth information; calculating the position and / or posture of the target in three-dimensional space through the posture information of the visual module and the depth information.

[0021] On the other hand, the present application provides an automatic depth vision positioning device, including: a processing unit and a storage unit connected thereto; the storage unit is used to store a computer program; the processing unit is used to call the computer program stored in the storage unit to execute the three-dimensional reconstruction method described in any one of the above embodiments, or execute the positioning method described in any one of the above embodiments.

[0022] On the other hand, the present application provides an automatic deep vision positioning robot, including: a vision module for acquiring multi-view images; an algorithm module for obtaining a dense point cloud; and a post-processing module for post-processing the dense point cloud to obtain a dense target point cloud.

[0023] In one embodiment, the robot further includes: a visual positioning module, which is used to align the dense target point cloud with the position and posture information of the visual module to achieve positioning of the target.

[0024] In one embodiment, the algorithm module is also used to reconstruct a sparse point cloud, and the robot also includes: a scale module, which provides scale information to the algorithm module to correct the scale of the sparse point cloud, wherein the algorithm module processes the corrected sparse point cloud to obtain the dense point cloud.

[0025] In one embodiment, the algorithm module includes a target screening algorithm, and the target screening algorithm satisfies: K = T cw T obj , where K is the target pose homogeneous matrix in the world coordinate system, T obj is the target pose homogeneous matrix in the visual module coordinate system, T cw is the homogeneous matrix of the visual module pose in the world coordinate system, T obj satisfy R 3×3 is the rotation matrix of the target in three-dimensional space, t 3×1 is the centroid coordinate of the dense target point cloud, t 3×1 satisfy [x i ,y i ,z i ] is the coordinate of the i-th point in three-dimensional space, and the centroid coordinates are selected to satisfy: [x 3d ,y 3d ,z 3d ] T =H[x 2d ,y 2d ,1] T , where [x 3d ,y 3d ,z 3d ] represents the three-dimensional coordinates of the target point in three-dimensional space, x 2d and 2d They respectively represent the coordinates of the target point in the two-dimensional image, and H represents the calculated transformation matrix from the two-dimensional image to the three-dimensional point cloud, wherein, during the post-processing process, the dense target point cloud is obtained by the target screening algorithm.

[0026] In one embodiment, the scale module includes a robotic arm, which is capable of driving the vision module to move, and the vision module acquires images from multiple perspectives, wherein the robotic arm includes an end effector, the end effector is coupled to the vision module, and the coordinate system of the end effector is aligned with the coordinate system of the vision module.

[0027] The open 3D reconstruction method of automatic depth target provided by the present application may have at least one of the following beneficial effects:

[0028] According to the three-dimensional reconstruction method in some embodiments of the present application, a dense target point cloud is obtained through post-processing operations, and the screening category can be customized, and the point cloud of the target object can be directly screened out from the dense point cloud, thereby improving the pertinence and efficiency of the processing;

[0029] According to the three-dimensional reconstruction method in some embodiments of the present application, post-processing can specifically correct errors in the reconstruction process, improve the accuracy of point cloud data, and make the reconstructed three-dimensional model closer to the real object.

[0030] According to the three-dimensional reconstruction method in some embodiments of the present application, combined with the visual module and the position information in the image, the position range of the target in the three-dimensional space can be calculated more accurately, ensuring that the correspondence between the dense target point cloud and the actual target object is more accurate.

[0031] According to the three-dimensional reconstruction method in some embodiments of the present application, by screening out point clouds related to the target, it is helpful to reduce the amount of data for subsequent processing and analysis, which can effectively improve data processing efficiency and achieve a significant increase in the three-dimensional reconstruction speed.

[0032] The automatic depth visual positioning method based on open 3D reconstruction provided in the present application may have at least one of the following beneficial effects:

[0033] According to the positioning method in some embodiments of the present application, by aligning the dense target point cloud and the posture information of the visual module, the position of the target in the three-dimensional space can be accurately determined, which is conducive to improving the positioning accuracy.

[0034] According to the positioning method in some embodiments of the present application, the system's robustness to environmental changes, such as illumination changes, occlusion, etc., can be improved through registration. It also helps to identify and locate multiple targets in complex scenes, and ensures the accuracy of identification and positioning even when the target is partially occluded or there is environmental interference.

[0035] The features and advantages of the embodiments of the present invention will be further described in the following description, and in part will become apparent from the description, or understood by practicing the present invention. The purposes and other advantages of the embodiments of the present invention can be achieved and obtained by the structures and / or functions specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Other features, purposes and advantages of the present application will become more apparent through the following detailed description of non-limiting embodiments in conjunction with the accompanying drawings. In the accompanying drawings:

[0037] Figure 1 is a flowchart of an open 3D reconstruction method of an automatic depth target according to an exemplary embodiment of the present application.

[0038] Figure 2 is a schematic diagram of a system architecture according to an exemplary embodiment of the present application.

[0039] Figure 3 is another schematic diagram of a system architecture according to an exemplary embodiment of the present application.

[0040] Figure 4 is a schematic diagram of a sparse point cloud in a three-dimensional reconstruction method according to an exemplary embodiment of the present application.

[0041] Figure 5 It is a schematic diagram of the structure of a robotic arm according to an exemplary embodiment of the present application.

[0042] Figure 6 is a schematic diagram of a dense point cloud in a three-dimensional reconstruction method according to an exemplary embodiment of the present application.

[0043] Figure 7 is a schematic diagram of a dense target point cloud in a three-dimensional reconstruction method according to an exemplary embodiment of the present application.

[0044] Figure 8 is a flow chart of an automatic depth vision positioning method according to an exemplary embodiment of the present application.

[0045] Fig. 9 It is a schematic diagram of the structure of an automatic depth vision positioning device according to an exemplary embodiment of the present application.

[0046] Fig.10 is a block diagram of a non-volatile computer-readable storage medium according to an exemplary embodiment of the present application.

[0047] Fig.11 It is a schematic diagram of the structure of an automatic deep vision positioning robot according to an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to better understand the present application, a more detailed description will be made of various aspects of the present application with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application, and are not intended to limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0049] It should be noted that in this specification, expressions such as "first", "second", etc. are only used to distinguish one feature from another feature, and do not represent any limitation on the features, especially do not represent any order of precedence. In the specification, references to "one embodiment", "embodiment", "example embodiment", "some embodiments", etc. indicate that the described embodiment may include a specific feature, structure or characteristic, but each embodiment may not necessarily include the specific feature, structure or characteristic. In addition, these phrases do not necessarily refer to the same embodiment.

[0050] It should be understood that expressions such as “include,” “including,” “have,” “contain” and / or “comprise” in this specification are open rather than closed expressions, which indicate the existence of stated features, elements and / or parts, but do not exclude the existence of one or more other features, elements, parts and / or combinations thereof.

[0051] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments in this application can be combined with each other. In addition, unless explicitly limited or contradictory to the context, the specific steps included in the method recorded in this application are not necessarily limited to the recorded order, but can be performed in any order or in parallel.

[0052] The features, principles and other aspects of the present application will be described in detail below in conjunction with the accompanying drawings.

[0053] In the related art, an image containing depth information is obtained through a camera such as RGB-D, and a 3D point cloud model is directly generated based on this depth image combined with the camera's intrinsic parameters. The multi-view depth map collected is fused after pose alignment, and the fused point cloud is denoised and optimized to form a dense point cloud. The inventors of the present application found that the field of view of depth sensors such as RGB-D is usually small, requiring multiple scans to cover a larger area, and are sensitive to lighting conditions. They are prone to failure in strong light or reflective surfaces, and are difficult to handle occlusion and dynamic scenes. In addition, the accuracy of the depth sensor is limited, which will affect the reconstruction quality of the 3D point cloud model.

[0054] In the related technology, 3D models are reconstructed through three-dimensional reconstruction technology, but the current three-dimensional reconstruction technology has many drawbacks. For example, algorithms based on neural radiation fields (such as NERF and 3DGS) can achieve high-quality three-dimensional reconstruction by simulating the continuous volume density and color of the scene through neural networks. As an extension of NERF, 3DGS uses Gaussian ellipsoids to model scenes and achieves fast rendering by rasterizing the Gaussian ellipsoid into images, while maintaining the advantages of explicit representation, facilitating downstream tasks such as dynamic reconstruction, geometry editing, and physical simulation. Although it can provide higher accuracy and better detail capture, it has very high requirements on computing power, is not suitable for low-computing power platforms, is difficult to apply on a large scale, and the reconstruction speed takes tens of minutes, which needs to be further improved.

[0055] For another example, SfM restores the three-dimensional structure of a scene by analyzing images from multiple perspectives. Although it has low computing power requirements and a relatively fast reconstruction speed, its accuracy and adaptability are relatively poor, making it difficult to meet the needs of high-precision reconstruction. In addition, the inventors of this application further discovered that in the field of three-dimensional reconstruction technology, SfM describes the geometric transformation between images based on geometric methods through homography matrices, which only provides relative scale information and cannot reflect the actual physical size of the object.

[0056] The present application provides an open three-dimensional reconstruction method for automatic depth targets, which can at least improve or solve the above-mentioned problems.

[0057] Figure 1 is a flow chart of an open 3D reconstruction method of an automatic depth target according to an exemplary embodiment of the present application. Figure 1 As shown, the present application provides a three-dimensional reconstruction method 1000 including:

[0058] Step S1100, using a visual module to collect multi-view images of the target;

[0059] Step S1200, obtaining a dense point cloud based on a three-dimensional reconstruction algorithm; and

[0060] Step S1300 , post-processing the dense point cloud to obtain a dense target point cloud.

[0061] It should be understood that the steps shown in the 3D reconstruction method 1000 are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously or in different steps. Figure 1 The following is executed in the order shown. Figures 1 to 6 The above steps S1100 to S1300 are further described.

[0062] Step S1100: Use a visual module to capture multi-view images of the target.

[0063] Figure 2 is a schematic diagram of a system architecture according to an exemplary embodiment of the present application. Figure 1 to Figure 2 As shown, a multi-view image of a target can be collected by the visual module 110, and the multi-view image includes both target information and environmental information around the target. The number of multi-view images can be multiple, and the visual module 110 can capture images from different positions with the same or different postures (angles).

[0064] In one possible implementation, the visual module 110 can be used to shoot the target at different positions at multiple angles to collect a series of images. In another possible implementation, the position and / or posture of the visual module 110 can be fixed, and the position and / or posture of the target can be changed to collect a series of images.

[0065] Exemplarily, the vision module 110 includes a sensor that provides two-dimensional image information, such as a combination of one or more of an RGB camera, a SLR camera, a mirrorless camera, an industrial camera, a camera, and a mobile phone lens.

[0066] It should be noted that compared with sensors that provide depth information (such as RGB-D depth cameras, etc.), the visual module 110 is not easily affected by distance, lighting conditions and surface characteristics of objects, and can ensure accuracy in measuring objects at long distances or on smooth or transparent surfaces; the depth camera will introduce depth information, greatly increase the amount of system data processing, and will affect the speed of three-dimensional reconstruction by multiples; in addition, the accuracy and reliability of the depth camera is difficult to guarantee, and the cost of a depth camera that can meet the accuracy and reliability requirements will be dozens of times the cost of the visual module 110.

[0067] Step S1200, obtaining a dense point cloud based on a three-dimensional reconstruction algorithm.

[0068] Figure 3 FIG. 1 is another schematic diagram of a system architecture according to an exemplary embodiment of the present application. Figures 1 to 3 As shown, the reconstructed sparse point cloud can be first obtained by the 3D reconstruction algorithm in the algorithm board. In a possible implementation, the step of obtaining the sparse point cloud includes extracting feature points of the multi-view image by the 3D reconstruction algorithm; then matching the feature points of different images in the multi-view image; then calculating the disparity of the matched feature points between different images; and estimating the 3D coordinates of the matched feature points in the 3D space using the disparity and the pose (including position and posture) information of the visual module 110.

[0069] Figure 4 Schematic diagram of a sparse point cloud in a three-dimensional reconstruction method according to an exemplary embodiment of the present application. Figure 4As shown, after obtaining the three-dimensional coordinates of the feature points, the estimated three-dimensional coordinates of all feature points can be combined to reconstruct a sparse point cloud.

[0070] Exemplarily, the three-dimensional reconstruction algorithm includes a sparse algorithm, and the sparse algorithm can be based on OpenMVG (Open Multiple View Geometry), and a sparse point cloud can be obtained through OpenMVG and subsequent operations.

[0071] Exemplarily, feature points with distinguishing properties can be extracted from multi-view images, and these feature points will be used in subsequent matching and three-dimensional reconstruction. Feature points with scale invariance and / or rotation invariance are extracted from the multi-view images collected by the visual module 110 by at least one of the feature detection algorithms SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features), and ORB (Oriented FAST and Rotated BRIEF) in OpenMVG. Each feature point will have a descriptor that describes the local features of the point, and the descriptor includes a vector form for the subsequent feature point matching process.

[0072] Exemplarily, by matching feature points of different images in a multi-view image (including multiple images), a correspondence between multi-view images is established. The most likely matching pair is found from a set of feature points using at least one of the algorithms, such as FLANN (Fast Library for Approximate Nearest Neighbors) and RANSAC (Random Sample Consensus). Matching will be based on matching criteria such as the nearest neighbor distance ratio (NNDR) and / or symmetric matching. FLANN can be used to match feature points between different images; NNDR can be used to filter out incorrect matching pairs and retain high-quality matching results.

[0073] For example, the disparity between the matched feature points in different images is calculated to provide a basis for the subsequent three-dimensional coordinate estimation, and then the disparity value is calculated based on the matched feature point pairs. The geometric relationship between the image plane and the three-dimensional space is established using the intrinsic parameter matrix and the extrinsic parameter matrix (pose information) of the visual module 110.

[0074] It should be noted that the technical term "disparity" described in the context of this application refers to the difference in pixel positions of the same feature point in different images. For each pair of matched feature points, the difference in pixel positions in different images, i.e., the disparity value, is calculated.

[0075] Exemplarily, the three-dimensional coordinates of the matched feature points in the three-dimensional space are estimated by using the disparity and the position information of the visual module 110. This design is conducive to converting the matched feature points from the two-dimensional image coordinates to the three-dimensional space coordinates. Furthermore, for each pair of matched feature points, the three-dimensional coordinates can be calculated based on the triangulation method using the disparity and the position information of the visual module 110, and the estimation result of the three-dimensional coordinates is optimized by minimizing the reprojection error to ensure the accuracy of the estimation.

[0076] For example, all estimated 3D coordinates are combined into a sparse point cloud to provide a basis for subsequent 3D reconstruction and application. All estimated 3D coordinate points are combined into a point cloud data set. Deduplication and filtering algorithms can also be applied to remove duplicate points and noise points to improve the quality of the point cloud.

[0077] The inventors of the present application have found that since the reconstruction results of the 3D reconstruction technology (such as SfM) only contain relative scale information but not absolute scale information, the 2D images collected by the vision module 110 cannot provide the absolute scale information. Therefore, the solution of combining the 3D reconstruction algorithm with the vision module 110 cannot reflect the real size and / or actual position of the target object in the 3D space.

[0078] It should be noted that the technical term "scale information" described in the context of this application refers to data that can provide actual physical dimensions or sizes. "Absolute scale information" refers to scale information that directly corresponds to the actual dimensions in the real world. For example, the length of an object in three-dimensional space is m meters, and the length m is the absolute scale information. "Relative scale information" refers to the proportional relationship between the various parts of an object, but does not involve the actual physical dimensions.

[0079] Based on the above problems, the solution of the present application provides additional open scale information for the 3D reconstruction algorithm to correct the scale of the sparse point cloud.

[0080] In open 3D reconstruction, the accuracy of scale information is the key to ensuring the accuracy of the 3D model. The accuracy and resolution of the vision module 110, the occlusion and surface reflection of the target object, the information inconsistency when the data provided by multiple sensors are fused, the accuracy of the calibration and calibration of the measuring equipment (such as Inertial Measurement Unit, IMU, or other equipment), or the calibration parameters change over time and are not updated in time, will all cause errors in the scale information, affect the accuracy of the scale information, and further cause the reconstructed 3D model to be distorted.

[0081] The present application introduces a robotic arm 120 as a measuring device, and couples the end effector 121 of the robotic arm 120 with the visual module 110 to drive the visual module 110 and the robotic arm 120 to move together. In order to achieve accurate spatial positioning, image data (two-dimensional) and physical position (three-dimensional) data fusion of the robotic arm 120, the present application sets the coordinate system of the end effector 121 to be aligned with the coordinate system of the visual module 110.

[0082] Calibration plates, feature point matching or other techniques can be used to determine the spatial relationship between the two coordinate systems and make necessary adjustments. Alternatively, the visual module 110 can be rotated and / or translated to ensure that the coordinate system of the visual module 110 completely matches the coordinate system of the end effector 121. The alignment of the above two coordinate systems can make the positions and directions of the end effector 121 and the visual module 110 in space correlated and consistent with each other. The origin of the coordinate system of the end effector 121 is consistent with the position of the origin of the coordinate system of the visual module 110 in space, that is, they share the same spatial position. In addition, the axes of the above two coordinate systems (including the x, y, and z axes) are in the same direction in space, so that the x-axis of the coordinate system of the end effector 121 is aligned with the x-axis of the coordinate system of the visual module 110, pointing to the same spatial direction.

[0083] In some exemplary embodiments, the coupling method of the end effector 121 and the vision module 110 includes a fixed connection, such as a threaded screw connection, welding or bonding, or a combination thereof. In other exemplary embodiments, the coupling method includes a non-fixed connection, such as a snap connection, a clamping connection, a magnetic connection, or a combination thereof, so that the end effector 121 and the vision module 110 are easily decoupled after the mating is completed.

[0084] Exemplarily, the method for acquiring multi-view images may also use the robotic arm 120 to drive the visual module 110 to move, and capture a series of images from multiple viewpoints during the movement.

[0085] Figure 5 is a schematic diagram of the structure of a robotic arm according to an exemplary embodiment of the present application. Figure 5 As shown, the robot arm 120 includes an end effector 121 , a base 122 , and a control system 123 .

[0086] The method for providing scale information includes: first calibrating the robot 120 to determine the posture (including position and posture) of the end effector 121 of the robot 120 in the coordinate system of the base 122 of the robot 120; then controlling the robot 120 to move from the initial position to the reference position, which is a predetermined position of known coordinates. In the process of the robot 120 moving to the reference position, the visual module 110 collects multi-view images, and the control system 123 of the robot 120 records the posture of the end effector 121 during image acquisition; then measuring the actual distance S from the initial position to the reference position of the robot 120, and establishing a scale reference η based on the recorded posture of the end effector 121; comparing the distance S with the distance L estimated from the sparse point cloud by the three-dimensional reconstruction algorithm to obtain the size factor The dimensional information provided in this application includes the dimension reference η and the dimension factor

[0087] Exemplarily, the robot 120 is calibrated to determine the pose (including position and posture) of the end effector 121 in the coordinate system of the base 122. Calibration is performed using a calibration plate (such as a chessboard) on which feature points of known size are located. The calibration plate can be placed within the working range of the robot 120, and a multi-view image of the calibration plate can be captured using the vision module 110. The internal and external parameters of the vision module 110 are calculated using a calibration algorithm. The pose of the end effector 121 in the coordinate system of the base 122 is calculated using the known position of the calibration plate and the captured multi-view images.

[0088] Exemplarily, the method for calibrating the robot 120 includes setting a marking point, and operating the robot 120 to drive the end effector 121 to move to a series of preset positions; at each preset position in the series of positions, the marking point is photographed using the visual module 110 to obtain a series of images, and the position of the end effector 121 relative to the marking point is recorded. Thereafter, the precise position information of the marking point is extracted from the obtained series of images, and the various joint angles and / or encoder data of the robot 120 when it moves to each of the above-mentioned preset positions are recorded. A mathematical model is established based on the precise position information of the extracted marking point, the various joint angles and / or encoder data of the robot 120 to estimate the position and posture of the end effector 121.

[0089] The robot arm 120 may include multiple joints. The more joints there are, the more degrees of freedom the robot arm 120 has, and the more postures the end effector 121 can achieve. As an example, the number of joints of the robot arm 120 includes 3, 4, 5, 6 or more.

[0090] Exemplarily, the robot arm 120 is moved from an initial position to a known reference position by controlling the movement of the robot arm 120. During the movement of the robot arm 120, the vision module 110 collects multi-view images to provide data support for the establishment of the scale reference η.

[0091] The actual distance S between the initial position and the reference position of the robot arm 120 can be measured by a measuring tool (such as a ruler, a laser rangefinder, etc.), and the precise distance moved by the robot arm 120 can be obtained to provide real data for the establishment of the scale reference η. The geometric relationship between the initial position and the reference position is established using the geometric relationship and the intrinsic and extrinsic information of the camera. Based on the recorded posture of the end effector 121 and the measured actual distance S, the scale reference η is established to provide a reference for subsequent scale correction. The scale reference η plays a role of reference and calibration in the measurement process, which is conducive to ensuring the accuracy and reliability of the measurement results.

[0092] Exemplarily, the actual distance S measured is compared with the distance L estimated by the 3D reconstruction algorithm to obtain a size factor to correct the scale of the 3D model. The distance L estimated from the sparse point cloud includes, for example, at least one of the estimated distance between matched feature points, the estimated distance from the vision module 110 to the feature point of the multi-view image, and the estimated distance from the feature point in the multi-view image to the reference position.

[0093] Compare the actual distance S measured with the estimated distance L and calculate the size factor Right now:

[0094] It is easy to understand that through the above steps, a sparse point cloud can be generated from the multi-view image, and by calibrating the robot arm 120, collecting multi-view images, measuring the actual distance S and calculating the size factor Steps such as establishing the scale benchmark η and size factor The scale factor η provides a reference point or object of known size that can be used to determine the actual size of other measurements. In 3D reconstruction and image analysis, the scale factor η also helps eliminate scale ambiguity and ensure that the reconstructed model and analysis results are accurate in size. It is used to adjust and convert the relative sizes between different systems and data sets. In the construction of three-dimensional models, the scale factor Adjust the dimensions of the model to match the actual observed data or expected results.

[0095] Scaling datum η and scale factor They play a synergistic role in ensuring the accuracy, consistency and comparability of the estimated data of the real data. The two together realize the precise scale correction of the three-dimensional model, which not only ensures the accuracy of the three-dimensional model, but also provides reliable scale information for robot operation, providing important support in the three-dimensional reconstruction and robot movement process.

[0096] It should be noted that due to the scale information (including the size reference η and the size factor ) has a crucial impact on the accuracy of model reconstruction. Imperfect system models or errors caused by external disturbances will cause scale information to drift over time, ultimately distorting the reconstructed model. The present application further designs the pose estimation of the end effector 121 in a dynamic system to provide accurate and real-time pose data and optimize scale information.

[0097] Exemplarily, the pose estimation of the end effector 121 satisfies:

[0098]

[0099] In the formula, is the estimated pose of the end effector given all measurements at time step k, K k is the Kalman gain at time step k, z k is the measurement vector at time step k, H k is the observation matrix at time step k. Based on the Kalman filter, the predicted state estimate and the new measurement are used to update the state estimate, and the Kalman gain K k The measurement weights are adjusted, taking into account the measurement uncertainty and prediction uncertainty, and the update step minimizes the error of the state estimate, which is conducive to providing the optimal estimate.

[0100] The present application improves the robustness and real-time performance of the estimation, optimizes the scale correction process, and enhances the system's adaptability to complex environments and tasks, thereby effectively ensuring scale accuracy in high-precision three-dimensional reconstruction by providing the real-time posture information of the end effector 121 to the scale correction acquisition process.

[0101] As a possible implementation, the calculated scale factor Applied to the entire sparse point cloud, the coordinates of each point are scaled to eliminate scale ambiguity. In some implementations, multiple rounds of optimization and iteration are required, and the scale correctness of the point cloud is verified to obtain a corrected sparse point cloud, and then a dense point cloud can be obtained based on the corrected sparse point cloud.

[0102] Figure 6 Schematic diagram of a dense point cloud in a three-dimensional reconstruction method according to an exemplary embodiment of the present application. Figure 6As shown, a 3D reconstruction algorithm may be used to process at least one of the corrected sparse point cloud, the multi-view image, and the pose information of the visual module 110 to obtain a dense point cloud.

[0103] Combining sparse point clouds and pose information can improve the accuracy of depth estimation. Pose information can provide accurate shooting position and direction, which helps to calculate depth more accurately. Combining sparse point clouds and multi-view images can generate a more complete point cloud model, especially when there are occlusions in the scene or sparse areas of sparse point clouds. Multi-view images provide scene information observed from different angles, which helps to supplement information from other perspectives when there is insufficient data in certain areas due to occlusion or missing texture. For complex scenes, such as kitchen environments and lighting environments, the fusion of the above three types of information can better handle the complexity of the scene and improve the accuracy, robustness and efficiency of dense point cloud reconstruction.

[0104] Continue to refer Figure 3 In addition to the sparse algorithm, the three-dimensional reconstruction algorithm also includes a dense algorithm. The dense algorithm can be based on OpenMVS (Open Multi-View Stereo). Through the corrected sparse point cloud and the pose information of the visual module 110, the position and direction of each visual module 110 relative to the global coordinate system can be determined. Feature points are extracted in each image, and descriptors are generated for these feature points. Feature points are matched between multiple views to establish a correspondence between images. For each pixel in the scene, the matched feature points and pose information are used to estimate its depth by, for example, triangulation or other methods, and the MVS (Multi-View Stereo) algorithm, such as PMVS (Patch-Based Multi-View Stereo), COLMAP, etc., is applied to calculate the depth information of each pixel. The MVS algorithm can consider the information of multiple views and determine the optimal depth value of each pixel through voting or optimization strategies. Multiple depth maps can then be fused into a consistent global depth map. According to the fused depth map and the internal parameters (focal length, principal point, etc.) of the visual module 110, the depth information is converted into points in three-dimensional space to construct a dense point cloud.

[0105] For example, the generated dense point cloud may be further optimized, including denoising, clustering, smoothing or downsampling, to improve the quality of the point cloud and reduce noise.

[0106] The applicant of the present invention also found that when the visual module 110 collects multi-view images of the target, it will capture the target and its surrounding environment into the picture. Therefore, the sparse point cloud obtained based on the multi-view image includes a sparse target point cloud corresponding to the target and a sparse environment point cloud corresponding to the environment; the dense point cloud obtained based on the scale-corrected sparse point cloud includes a dense target point cloud and a dense environment point cloud. At this time, the dense point cloud contains a large number of points, each of which contains position information, and even color and normal information. Processing this huge amount of data information requires a lot of computing resources.

[0107] Step S1300 , post-processing the dense point cloud to obtain a dense target point cloud.

[0108] Since processing the above-mentioned huge amount of data on a conventional computing platform requires a lot of computing resources, resulting in a long time-consuming 3D reconstruction speed, this application also introduces AI (Artificial Intelligence) post-processing steps to address this problem. Through AI screening in the post-processing process, the dense target point cloud can be effectively identified and segmented, and redundant data can be greatly filtered out, thereby saving computing resources and achieving a reconstruction speed increase of dozens to hundreds of times.

[0109] Figure 7 Schematic diagram of a dense target point cloud in a three-dimensional reconstruction method according to an exemplary embodiment of the present application. Figure 7 As shown, the post-processing process includes using a pre-trained neural network model to process the multi-view image to determine the position of the target in the multi-view image; combining the posture information of the visual module 110 and the position information of the target in the multi-view image to calculate the position range of the target in the three-dimensional space; and filtering out the point cloud within the position range in the dense point cloud to obtain a dense target point cloud.

[0110] Exemplarily, the neural network model is trained before post-processing, and a number of pictures (e.g., no less than 100 pictures) are taken of the target object to be screened, and image annotation and neural network model training are performed to form a pre-trained neural network model. The pre-trained neural network model includes, for example, a combination of one or more of YOLO, Faster R-CNN, and Mask R-CNN.

[0111] Process the multi-view images and perform target detection on the images of each view to ensure that the target position of each view is accurately identified. Apply a trained model such as YOLO to process the image, identify the target object, and determine its position in the two-dimensional image (for example, the coordinates or confidence of the bounding box), and then record the target position detected in each view. Combine the pose information of the visual module 110 and the position information of the target in the multi-view image to calculate the position range of the target in three-dimensional space. The calculation method, for example, includes calculating the bounding box (BoundingBox) or the minimum circumscribed sphere (Minimum Bounding Sphere) of the target point cloud.

[0112] Exemplarily, the 3D coordinates of all points are extracted from the dense point cloud generated by 3D reconstruction, the point cloud within the target position range is screened out, and the screened point cloud is denoised and filtered to remove noise points and abnormal points, and the processed point cloud is combined into a dense target point cloud corresponding to the target in the scene (such as Figure 7 box).

[0113] For example, the position information of the visual module 110 during image acquisition recorded by the control system 123 of the robot 120 can be used. The geometric relationship between the image plane and the three-dimensional space is established using the internal and external reference information of the visual module 110, and the two-dimensional coordinates of the target in the image are converted into three-dimensional rays in the coordinate system of the visual module 110. The three-dimensional rays of multiple perspectives are combined to calculate the position range of the target in the three-dimensional space using a triangulation method.

[0114] This application innovatively proposes a method for calculating the position range of a target in three-dimensional space, which satisfies:

[0115] K=T cw T obj ,as well as

[0116] [x 3d ,y 3d ,z 3d ] T =H[x 2d ,y 2d ,1] T ,

[0117] Where K is the homogeneous matrix of the target pose in the world coordinate system, T obj is the target pose homogeneous matrix in the visual module coordinate system, T cw is the homogeneous matrix of the visual module pose in the world coordinate system, [x 3d ,y 3d ,z 3d ] represents the three-dimensional coordinates of the target point in three-dimensional space, x 2d and2d They respectively represent the coordinates of the target point in the two-dimensional image, and H represents the calculated transformation matrix from the two-dimensional image to the three-dimensional point cloud.

[0118] Then, the centroid coordinates of the dense target point cloud can be obtained. The centroid coordinates t 3×1 satisfy:

[0119]

[0120] [x i ,y i ,z i ] is the coordinate of the i-th point in three-dimensional space, and the coordinate mean is calculated by the eight vertices of the bounding box of the three-dimensional target point cloud Then calculate the covariance matrix Then perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalues ​​λ1≥λ2≥λ3 and the corresponding eigenvectors Get the target rotation matrix R in three-dimensional space 3×3 Thus, the pose homogeneous matrix T of the target in the visual module coordinate system can be obtained: obj , 0 1×3 is a zero vector, and the homogeneous matrix T of the visual module pose in the world coordinate system is known cw , and then get the target pose homogeneous matrix K = T in world coordinates cw T obj .

[0121] It should be noted that the technical term "world coordinate system" described in the context of this application refers to a predefined reference coordinate system that is used to provide a unified reference frame for the entire workspace or environment. This coordinate system is fixed, and all other coordinate systems (including the coordinate system of the robot 120) are defined relative to this coordinate system. As an example, the position of the robot 120, the visual module 110, etc. in the world coordinate system can be determined by coordinate transformation.

[0122] It is easy to understand that the above centroid coordinates and covariance matrix are helpful to accurately determine the position of the target in three-dimensional space; the rotation matrix obtained by eigenvalue decomposition can effectively estimate the attitude of the target, that is, the position and orientation of the target; by simplifying the complex point cloud data into the centroid and rotation matrix, it is easier to process and analyze the data. Since the covariance matrix and eigenvalue decomposition can reduce the influence of a single point, the above method has good robustness to noise and outliers. In addition, by considering all points in the point cloud, the above method provides a comprehensive description of the overall shape and position of the target, so that the calculation can be performed quickly and the calculation efficiency is improved. These calculations are compatible with a variety of sensors and data sources, and can be generalized. The calculated position range and attitude information can also be used for visualization to help understand and analyze the three-dimensional structure of the target.

[0123] The reconstruction accuracy error of the open 3D reconstruction method / solution based on this application is less than 1% (image acquisition is performed at a distance of several meters from the target). When the distance is closer, the reconstruction accuracy is further improved. The reconstruction speed of the 3D model on a conventional computing power platform is completed within 2 minutes. Compared with traditional 3D reconstruction technology, the speed of this application scheme has been improved by dozens to hundreds of times, and it has excellent reconstruction accuracy.

[0124] On the other hand, the present application provides an automatic depth vision positioning method based on open three-dimensional reconstruction. Figure 8 is a flow chart of an automatic depth vision positioning method according to an exemplary embodiment of the present application. Figure 8 As shown, the positioning method 2000 provided by the present application includes:

[0125] Step S1100 to step S1300; and

[0126] Step S2100, aligning the dense target point cloud with the position and posture information of the visual module to achieve the positioning of the target.

[0127] It should be understood that the steps shown in the positioning method 2000 are not exclusive, and other steps may be performed before, after, or between any of the steps shown. In addition, some of the steps shown may be performed simultaneously or in different steps. Figure 7 The following is executed in the order shown. Figure 8 The above steps S1100 to S2100 are further described.

[0128] In a possible implementation, the dense target points selected by the three-dimensional reconstruction can be registered with the visual module 110 to realize the positioning of the target. Exemplarily, the registration includes mapping each point in the dense target point cloud to the corresponding two-dimensional position in the multi-view image to obtain two-dimensional information; based on the posture information of the visual module 110, the two-dimensional information obtained by projection is fused with the three-dimensional information in the dense target point cloud, so as to give each point in the dense target point cloud depth information, and the depth information includes the actual distance from the visual module 110 to each point in the point cloud, indicating the distance of the point from the camera; through the posture information of the visual module 110 and the above-mentioned depth information, the position and / or posture of the target in the three-dimensional space can be calculated. The calculation result is an accurate three-dimensional coordinate and posture, which reflects the exact position and direction of the target in space.

[0129] Exemplarily, the intrinsic parameter matrix and extrinsic parameter matrix of the visual module 110 can be used, and the image of each perspective can be projected and transformed respectively, and the two-dimensional position of each point in each perspective image can be recorded to ensure that the position of each point in multiple perspectives is accurately mapped, so as to realize the projection of each point in the dense target point cloud from three-dimensional space to the two-dimensional image plane.

[0130] For example, the geometric relationship between the image plane and the three-dimensional space can be established by combining the position information of the visual module 110 during image acquisition recorded by the control system 123 with the internal and external reference information. By fusing the two-dimensional information and the three-dimensional information, the two-dimensional position is back-projected back to the three-dimensional space, and the depth information of each point in the dense target point cloud is calculated.

[0131] Through the above registration process, the precise positioning and orientation of the target in three-dimensional space can be obtained, providing high-quality data support for the machine's recognition of targets in space.

[0132] On the other hand, the present application provides an automatic depth vision positioning device. Fig. 9 Schematic diagram of the structure of the automatic depth vision positioning device according to the exemplary embodiment of the present application. Fig. 9 As shown, the automatic depth vision positioning device 200 includes a processing unit 210 and a storage unit 220 connected thereto. The storage unit 220 is used to store computer programs. In one possible implementation, the processing unit 210 is used to call the computer program stored in the storage unit 220 to execute the three-dimensional reconstruction method in any of the above implementations. In another possible implementation, the processing unit 210 is used to call the computer program stored in the storage unit 220 to execute the automatic depth vision positioning method in any of the above implementations.

[0133] The processing unit 210 serves as the control center of the automatic depth vision positioning device 200. It can be a general-purpose central processing unit (CPU) or other processors, such as a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.; or any conventional general-purpose processor, such as a microprocessor, etc.

[0134] As an example, the processing unit 210 may include one or more CPUs, such as Figure 5 As shown in the CPU0 and CPU1, the storage unit 220 can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0135] In a possible implementation, the storage unit 220 may exist independently of the processing unit 210. The storage unit 220 may be connected to the processing unit 210 via, for example, a bus 230, and is used to store data, instructions, or program codes. When the processing unit 210 calls and executes the instructions or program codes stored in the storage unit 220, the method provided in the embodiment of the present application can be implemented.

[0136] In another possible implementation, the storage unit 220 may also be integrated with the processing unit 210 .

[0137] Exemplarily, the automatic depth vision positioning device 200 can also include a communication interface 240 for connecting with other devices through a communication network, and the communication network can be Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc. The communication interface 240 can include a receiving unit for receiving data and a sending unit for sending data.

[0138] The bus 230 may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For convenience of representation, Fig. 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0139] Need to explain, Fig. 9 The structure shown in the figure does not constitute a limitation on the automatic depth vision positioning device 200, except Fig. 9 In addition to the components shown, the automatic depth vision positioning device 200 may include Fig. 9 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0140] Optionally, Fig. 9 The automatic depth vision positioning device 200 shown in FIG. 2 is a chip.

[0141] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the above-mentioned processing unit 210 or the function of the processing unit 210. When a memory is integrated in the digital processing chip, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the actions performed by the model training device or the natural language processing device in the above-mentioned embodiment according to the program code stored in the external memory.

[0142] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0143] In addition, in the drawings of the device embodiments provided in the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.

[0144] In addition, the present invention also proposes the following embodiments:

[0145] A non-volatile computer-readable storage medium. Fig.10 is a schematic diagram of the structure of a non-volatile computer-readable storage medium according to an exemplary embodiment of the present application. Fig.10 As shown, the readable storage medium 300 is composed of a semiconductor device 310 and a peripheral circuit 320. The readable storage medium 300 stores a computer program, and when the computer program is executed, it can execute the three-dimensional reconstruction method in any of the above embodiments, and / or execute the automatic depth vision positioning method in any of the above embodiments.

[0146] The computer-readable storage medium 300 includes a device capable of temporarily or permanently storing instructions and data for the automatic deep vision positioning device 200 to operate in a specific manner. As used herein, the term "computer-readable storage medium" does not itself encompass transient electrical or electromagnetic signals (e.g., on a carrier wave propagating through the medium). Therefore, the computer-readable storage medium 300 can be considered to be tangible and non-transitory.

[0147] Illustratively, non-limiting examples of non-transitory tangible computer-readable storage media may include, but are not limited to, non-volatile memory, such as flash memory or read-only memory (ROM), volatile memory, such as static random access memory (RAM) or dynamic access memory, buffer memory, cache memory, optical storage media, magnetic storage media and devices, network accessible or cloud storage, and other types of storage and / or any suitable combination thereof.

[0148] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0149] A person of ordinary skill in the art may understand that all or part of the steps in the method of the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium 300. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium 300. As a possible example, the computer-readable storage medium 300 may also be a disk, an optical disk, or a solid-state hard disk, etc.

[0151] On the other hand, the present application provides an automatic deep vision positioning robot. Fig.11 Schematic diagram of the structure of an automatic depth vision positioning robot according to an exemplary embodiment of the present application. Fig.11 As shown, the robot 100 includes a vision module 110 , a robotic arm 120 , an algorithm module 130 and a post-processing module 140 .

[0152] Need to explain, Fig.11 The structure shown in the figure does not constitute a limitation on the robot 100, except Fig.11 In addition to the components shown, the robot 100 may include Fig.11 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.

[0153] In one possible implementation, the robot 100 has learning and adaptive capabilities, and can continuously optimize control strategies based on historical operating data and environmental changes to improve operating efficiency and accuracy.

[0154] As an example, the vision module 110 can be used to collect multi-view images, the algorithm module 120 can receive data transmitted by the vision module 110 and the robot arm 120, and apply multiple types of algorithms to output a point cloud model. For example, the algorithm module 120 is an algorithm board, through which a dense point cloud can be obtained. The post-processing module 140 can be used to post-process the dense point cloud to obtain a dense target point cloud. The vision module 110, the algorithm module 120 and the post-processing module 140 together constitute a complete three-dimensional reconstruction and target recognition system. The collaboration of different modules in the system ensures the realization of the entire process from image acquisition to target positioning.

[0155] Exemplarily, the robotic arm 120 includes an end effector 121. The robotic arm 120 is coupled to the vision module 110 via the end effector 121 to achieve the common movement of the vision module 110 and the robotic arm 120. Exemplarily, the coordinate system of the end effector 121 is aligned with the coordinate system of the vision module 110. The alignment of the above two coordinate systems can make the positions and directions of the end effector 121 and the vision module 110 in space correlated and consistent with each other. The origin of the coordinate system of the end effector 121 and the origin of the coordinate system of the vision module 110 are consistent in position in space, that is, they share the same spatial position.

[0156] When the visual module 110 acquires images from multiple perspectives, the posture of the visual module 110 can be determined by the posture of the end effector 121. For example, the end effector 121 is a manipulator, and the visual module 110 is an RGB camera. The manipulator 120 can move according to the instructions issued by the robot 100, and the manipulator 120 drives the manipulator to grab the RGB camera; or, the RGB camera is pre-installed on the manipulator so that the RGB camera can move with the manipulator.

[0157] In a possible implementation, the robot 100 further includes a scale module 140, which can provide scale information to the algorithm module 130 to correct the scale of the sparse point cloud. The scale information includes a scale reference η and a scale factor. Scaling datum η and scale factor They have a synergistic effect in ensuring the accuracy, consistency and comparability of the estimated data of the real data. The two together realize the precise scale correction of the three-dimensional model, which not only ensures the accuracy of the three-dimensional model, but also provides reliable scale information for the operation of the robot 100, providing important support for the movement of the robot 100.

[0158] As an example, the position information of the visual module 110 during image acquisition recorded by the control system 123 of the robot 120 can be used. The geometric relationship between the image plane and the three-dimensional space is established using the internal and external reference information of the visual module 110, and the two-dimensional coordinates of the target in the image are converted into three-dimensional rays in the coordinate system of the visual module 110. The three-dimensional rays of multiple perspectives are combined to calculate the position range of the target in the three-dimensional space using a triangulation method.

[0159] Exemplarily, the algorithm module 130 includes a sparse algorithm, a dense algorithm, and a target screening algorithm. Through the sparse algorithm, a sparse point cloud can be obtained; through the dense algorithm, the algorithm module 130 can process the corrected sparse point cloud to obtain a dense point cloud; combined with the target screening algorithm, the post-processing module 140 can calculate and obtain a dense target point cloud in the process of AI post-processing the dense point cloud. Further, the target screening algorithm satisfies:

[0160] K=T cw T obj ,as well as

[0161] [x 3d ,y 3d ,z 3d ] T =H[x 2d ,y 2d ,1] T ,

[0162] In the above formula, K is the homogeneous matrix of the target pose in the world coordinate system, T obj is the target pose homogeneous matrix in the visual module coordinate system, T cw is the homogeneous matrix of the visual module pose in the world coordinate system, [x 3d ,y 3d ,z 3d ] represents the three-dimensional coordinates of the target point in three-dimensional space, x 2d and 2d They respectively represent the coordinates of the target point in the two-dimensional image, and H represents the calculated transformation matrix from the two-dimensional image to the three-dimensional point cloud.

[0163] Dense target point cloud centroid coordinates t 3×1 satisfy:

[0164]

[0165] [x i ,y i ,z i ] is the coordinate of the i-th point in three-dimensional space, and the coordinate mean is calculated by the eight vertices of the bounding box of the three-dimensional target point cloud Then calculate the covariance matrix Then perform eigenvalue decomposition on the covariance matrix C to obtain the eigenvalues ​​λ1≥λ2≥λ3 and the corresponding eigenvectors The pose homogeneous matrix under And the homogeneous matrix T of the visual module pose in the world coordinate system is known cw , and then get the target pose homogeneous matrix K = T in world coordinates cw T obj .

[0166] The target screening algorithm can calculate the position range of the target in three-dimensional space, thereby screening out dense target point clouds.

[0167] Exemplarily, the post-processing module 140 includes a pre-trained neural network model. Before post-processing, the neural network model needs to be trained, and several pictures of the target object to be screened are taken, and image annotation and deep learning training of the neural network model are performed to form a pre-trained neural network model. The pre-trained neural network model includes, for example, a combination of one or more of YOLO, Faster R-CNN, and Mask R-CNN.

[0168] In a possible implementation, the robot 100 further includes a positioning module 150, which is used to align the dense target point cloud with the position information of the vision module 110 to achieve the positioning of the target. As an example, the positioning result includes accurate three-dimensional coordinates and posture, reflecting the exact position and direction of the target in space. The positioning result can be used in a variety of applications, such as robot navigation, automated detection, augmented reality, etc.

[0169] For robot scenarios in the field of artificial intelligence, after providing scale information, the end effector 121 (such as a manipulator) of the robot arm 120 can accurately grasp the target according to the positioning result.

[0170] The "target" described in the context of this application may refer to a target object. As an example, the target includes a handle of a kitchen utensil, such as a frying basket handle (corresponding to Figure 7 The dense point cloud shown in the middle box), the handle of a water cup, the handle of a frying pan, etc. As another example, the target includes food or non-food items, such as apples, cucumbers, milk in a box, French fries, plates, seasoning boxes or bottles, etc.

[0171] Exemplarily, the robot 100 can be used in many scenarios such as medical care, home, retail, security, education, etc. For example, the robot 100 is applied to cooking or automated food preparation processes in kitchen scenarios to achieve accurate and rapid identification and classification of ingredients or kitchen utensils, and perform grasping, cooking and / or other operation tasks.

[0172] The above description is only an implementation method of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of protection involved in the present application is not limited to the technical solution formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the technical concept. For example, the above features are replaced with the technical features with similar functions disclosed in this application (but not limited to) by each other to form a technical solution.

Claims

1. An automatic deep open 3D reconstruction method, characterized in that: The three-dimensional reconstruction method comprises: Use the visual module to collect multi-view images of the target; Obtaining dense point cloud based on 3D reconstruction algorithm; and The dense point cloud is post-processed to obtain a dense target point cloud.

2. The three-dimensional reconstruction method according to claim 1, characterized in that: The method for obtaining the dense point cloud includes: Obtaining a reconstructed sparse point cloud by using the three-dimensional reconstruction algorithm; Providing scale information to the 3D reconstruction algorithm to correct the scale of the sparse point cloud; A dense point cloud is obtained based on the corrected sparse point cloud.

3. The three-dimensional reconstruction method according to claim 2, characterized in that: The method for obtaining the sparse point cloud includes: Extracting feature points of the multi-view images by a three-dimensional reconstruction algorithm; Matching feature points of different images in the multi-view images; Calculate the disparity of the matched feature points between the different images; estimating the three-dimensional coordinates of the matched feature points in the three-dimensional space by using the disparity and the pose information of the visual module; All the estimated three-dimensional coordinates are combined to reconstruct the sparse point cloud.

4. The three-dimensional reconstruction method according to claim 3, characterized in that: The method for obtaining the dense point cloud includes: The pose information of the visual module is processed, and at least one of the corrected sparse point cloud and the multi-view image is processed to obtain the dense point cloud.

5. The three-dimensional reconstruction method according to claim 2, characterized in that: The post-processing includes: Processing the multi-view image using a pre-trained neural network model to determine the position of the target in the multi-view image; Calculating the position range of the target in three-dimensional space by combining the posture information of the visual module and the position information of the target in the multi-view image; and Point clouds located within the position range are screened out from the dense point cloud.

6. The positioning method according to claim 5, characterized in that: Calculate the position range of the target in three-dimensional space to satisfy: K=T cw T obj , Among them, K is the target pose homogeneous matrix in the world coordinate system, T obj is the target pose homogeneous matrix in the visual module coordinate system, T cw is the homogeneous matrix of the visual module pose in the world coordinate system, T obj satisfy R 3×3 is the rotation matrix of the target in three-dimensional space, t 3×1 is the centroid coordinate of the dense target point cloud, t 3×1 satisfy [x i ,y i ,z i ] is the coordinate of the i-th point in three-dimensional space, The method for obtaining the centroid coordinates satisfies: [x 3d ,y 3d ,z 3d ] T =H[x 2d ,y 2d ,1] T , Among them, [x 3d ,y 3d ,z 3d ] represents the three-dimensional coordinates of the target point in three-dimensional space, x 2d and 2d They respectively represent the coordinates of the target point in the two-dimensional image, and H represents the transformation matrix from the two-dimensional image to the three-dimensional point cloud calculated by the three-dimensional reconstruction algorithm.

7. The three-dimensional reconstruction method according to claim 4, characterized in that: The method for acquiring the multi-view images includes: A robotic arm is used to drive the visual module to move and to collect images from multiple perspectives, wherein an end effector of the robotic arm is coupled to the visual module, and a coordinate system of the end effector is aligned with a coordinate system of the visual module.

8. The three-dimensional reconstruction method according to claim 7, characterized in that: The method of providing the scale information includes: Calibrate the robotic arm to determine the position and posture of the end effector in the coordinate system of the robotic arm base; The robotic arm is moved from an initial position to a reference position, wherein, in the process of the robotic arm moving to the reference position, the visual module collects the multi-view images, and the control system of the robotic arm records the position and posture of the end effector; Measuring the actual distance from the initial position to the reference position, and establishing a scale reference based on the recorded position and posture of the end effector; The spacing is compared with a distance estimated from the sparse point cloud by the three-dimensional reconstruction algorithm to obtain a size factor, wherein the scale information includes the size reference and the size factor.

9. The three-dimensional reconstruction method according to claim 8, characterized in that: The distance estimated from the sparse point cloud by the 3D reconstruction algorithm includes: At least one of the estimated distances between the matched feature points, the estimated distances from the visual module to the feature points of the multi-view image, and the estimated distances from the feature points in the multi-view image to the reference position.

10. The three-dimensional reconstruction method according to claim 8, characterized in that: The pose estimation of the end effector satisfies: in, is the estimated pose of the end effector given all measurements at time step k, K k is the Kalman gain at time step k, z k is the measurement vector at time step k, H k is the observation matrix at time step k.

11. The three-dimensional reconstruction method according to claim 3, characterized in that: The sparse point cloud includes a sparse target point cloud and a sparse environment point cloud, and the dense point cloud includes a dense target point cloud and a dense environment point cloud.

12. An automatic depth vision positioning method based on open three-dimensional reconstruction, characterized in that: The positioning method comprises: The three-dimensional reconstruction method according to any one of claims 1 to 11, and The dense target point cloud is aligned with the position and posture information of the visual module to achieve positioning of the target.

13. The positioning method according to claim 12, characterized in that: The registration includes: Mapping each point in the dense target point cloud to a corresponding two-dimensional position in the multi-view image to obtain two-dimensional information; Based on the pose information of the visual module, the two-dimensional information is fused with the three-dimensional information in the dense target point cloud to give depth information to each point in the dense target point cloud; The position and / or posture of the target in three-dimensional space is calculated through the posture information of the visual module and the depth information.

14. An automatic depth vision positioning device, characterized in that: The device comprises: a processing unit and its connected storage unit; The storage unit is used to store computer programs; The processing unit is used to call the computer program stored in the storage unit to execute the three-dimensional reconstruction method described in any one of claims 1-11, or to execute the positioning method described in any one of claims 12-13.

15. An automatic deep vision positioning robot, characterized in that: The robot comprises: Vision module, used to collect multi-view images; Algorithm module, used to obtain dense point cloud; The post-processing module is used to post-process the dense point cloud to obtain a dense target point cloud.

16. The robot according to claim 15, characterized in that: Also includes: The positioning module is used to align the dense target point cloud with the position and posture information of the visual module to achieve the positioning of the target.

17. The robot according to claim 15, characterized in that: The algorithm module is also used to reconstruct a sparse point cloud, and the robot further includes: A scale module provides scale information to the algorithm module to correct the scale of the sparse point cloud, wherein the algorithm module processes the corrected sparse point cloud to obtain the dense point cloud.

18. The robot according to claim 16, characterized in that: The algorithm module includes a target screening algorithm, and the target screening algorithm satisfies: K=T cw T obj , Among them, K is the target pose homogeneous matrix in the world coordinate system, T obj is the target pose homogeneous matrix in the visual module coordinate system, T cw is the homogeneous matrix of the visual module pose in the world coordinate system, T obj satisfy R 3×3 is the rotation matrix of the target in three-dimensional space, t 3×1 is the centroid coordinate of the dense target point cloud, t 3×1 satisfy [x i ,y i ,z i ] is the coordinate of the i-th point in three-dimensional space, The centroid coordinates are selected to satisfy: [x 3d ,y 3d ,z 3d ] T =H[x 2d ,y 2d ,1] T , Among them, [x 3d ,y 3d ,z 3d ] represents the three-dimensional coordinates of the target point in three-dimensional space, x 2d and 2d They represent the coordinates of the target point in the two-dimensional image, H represents the calculated transformation matrix from the two-dimensional image to the three-dimensional point cloud, Wherein, during the post-processing process, the dense target point cloud is obtained by the target screening algorithm.

19. The robot according to claim 17, characterized in that: The scale module includes a mechanical arm, which can drive the visual module to move, and the visual module collects images from multiple perspectives. Wherein, the robotic arm includes an end effector, the end effector is coupled to the vision module, and the coordinate system of the end effector is aligned with the coordinate system of the vision module.

Images