Multi-view fast three-dimensional reconstruction method and system

Abstract

The application discloses a kind of multi-viewpoint fast three-dimensional reconstruction method and system, the method includes based on the effective physical observation range of current job station environment setting 3D camera to target object;The first frame image of target object obtained is converted into first point cloud data;Second frame image of the next best view angle obtained based on the space grid model constructed from first point cloud data is converted into second point cloud data;Based on the spatial real pose transformation matrix of default initial view angle and next best view angle, rough registration is completed;The three-dimensional reconstruction model of target object is generated with the spatial real pose transformation matrix as initial value;The multi-viewpoint fast three-dimensional reconstruction method and system, adaptive adjustment, active viewpoint planning, and introduce robot kinematics prior data to obtain the spatial pose transformation matrix of camera in real time, and the matrix is directly used as the good initial pose of two frame point cloud registration, solve the problem of lack of initial pose of existing algorithm.

Description

Technical Field

[0001] This invention belongs to the field of 3D reconstruction technology, and in particular to a method and system for rapid 3D reconstruction from multiple viewpoints. Background Technology

[0002] With the development of industrial automation, mobile AGVs and robotic arms are increasingly used in complex material handling and assembly scenarios. In order to achieve precise grasping and obstacle avoidance, it is usually necessary to reconstruct the target object and the working environment in three dimensions.

[0003] Existing 3D reconstruction technologies mainly face the following problems:

[0004] 1. Due to occlusion, single-viewpoint 3D imaging can often only acquire partial point cloud information of the object's surface, failing to form a complete 3D model, which leads to the failure of subsequent robotic arm grasping pose planning or collisions.

[0005] 2. Traditional multi-view stereo vision (MVS) or multi-angle point cloud stitching methods usually employ blind omnidirectional scanning or random viewpoint acquisition, resulting in a large amount of redundant data.

[0006] 3. Existing point cloud registration algorithms (such as the standard ICP algorithm) suffer from slow iteration speed and are prone to getting trapped in local optima when a good initial pose is lacking, failing to meet the stringent requirements of "speed" and "real-time performance" in industrial settings. Current methods do not effectively incorporate motion information from hardware actuators to provide reliable initial pose priors for the registration algorithm. Summary of the Invention

[0007] Purpose of the invention: To provide a multi-viewpoint fast 3D reconstruction method and system to solve the problems of slow 3D reconstruction speed, low point cloud stitching efficiency, and high data redundancy in the prior art, especially the slow iteration speed and easy getting trapped in local optima due to the lack of a good initial pose during point cloud registration.

[0008] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0009] Firstly, a multi-viewpoint rapid 3D reconstruction method is provided, the method comprising:

[0010] After the 3D camera adjusts the exposure parameters based on the current work station environment, the effective physical observation range of the 3D camera for the target object is set according to the three-dimensional physical boundary parameters of the current work station.

[0011] Within the effective physical observation range, the first frame image of the target object obtained by the 3D camera at the default initial viewpoint is converted into first point cloud data in the first local coordinate system;

[0012] Based on the spatial grid model constructed from the first point cloud data, several candidate viewpoints are obtained in the missing viewpoint region, and the next best viewpoint is obtained by calculating the several candidate viewpoints.

[0013] The second frame image of the target object acquired from the next optimal viewpoint is converted into second point cloud data in the second local coordinate system;

[0014] Based on the spatial true pose transformation matrix of the default initial view and the next optimal view, coarse registration of the first point cloud data and the second point cloud data is completed.

[0015] Using the spatial true pose transformation matrix as the initial value, the first point cloud data and the second point cloud data are iteratively aligned until data fusion is completed. The fused data is then filtered to generate a three-dimensional reconstruction model of the target object.

[0016] In some possible embodiments, the 3D camera adjusts its exposure parameters based on the current workstation environment. After establishing a connection with the robot actuator, the 3D camera automatically executes an exposure adjustment procedure based on histogram analysis according to the lighting conditions of the current workstation environment. The exposure adjustment procedure includes:

[0017] Acquire a two-dimensional grayscale image of the current workstation environment under the current lighting conditions and calculate its grayscale histogram distribution to obtain the energy center peak of the grayscale histogram;

[0018] Based on the set target gray-level mean range, the exposure time of the 3D camera is adaptively adjusted using an iterative step method until the energy center peak of the gray-level histogram converges to the target gray-level mean range.

[0019] In a further embodiment, setting the effective physical observation range of the 3D camera for the target object based on the three-dimensional physical boundary parameters of the current workstation includes:

[0020] A workstation parameter database is obtained by establishing the three-dimensional physical boundary parameters of each different workstation. Based on the corresponding ID generated for each workstation, the corresponding three-dimensional physical boundary parameters are called and mapped to the effective physical observation range of the target object by the 3D camera.

[0021] The effective physical observation range is the region of interest and the effective depth truncation range after removing redundant background data that exceeds the physical boundary of the target object.

[0022] In a further embodiment, converting the first frame image of the target object obtained by the 3D camera at a default initial viewpoint into first point cloud data in a first local coordinate system includes:

[0023] Based on the intrinsic parameter matrix of the 3D camera, the first frame image is converted into first point cloud data in the first local coordinate system;

[0024] The first frame image includes a first frame depth map and a first frame color map acquired synchronously.

[0025] In a further embodiment, the spatial grid model constructed based on the first point cloud data obtains several candidate viewpoints in the missing viewpoint region, including:

[0026] Based on the first point cloud data, an octree or voxel grid spatial grid model is constructed. The spatial grid model is divided into occupied state region, idle state region and unknown state region according to the spatial state.

[0027] The boundary voxel between the extracted occupied state region and the unknown state region is used as the missing boundary to form the missing viewpoint region;

[0028] The candidate pose set generated in the missing viewpoint region is input into the kinematic constraint model and the collision detection model. After eliminating invalid poses, the remaining valid pose set is used as the candidate viewpoints.

[0029] In a further embodiment, the calculation of the next optimal viewpoint from the plurality of candidate viewpoints includes:

[0030] Calculate the initial information entropy of each voxel in the spatial grid model. , means as follows:

[0031] ;

[0032] in, For voxels, Let be the initial occupancy probability of each voxel, when When the voxel is an unknown voxel in the unknown state region;

[0033] One of the candidate viewpoints is selected as the current candidate viewpoint. A ray casting algorithm is then used to calculate the voxel sequence that each simulated ray passes through as it traverses the spatial mesh model, and a ray visibility attenuation factor is introduced. The occlusion effect of the preceding voxel on the subsequent unknown voxel is simulated as follows:

[0034] ;

[0035] in, Indicates the first The simulated ray traversing the first Individual factors, For the simulated ray, the first preposition The known occupancy probability of an individual element, and the expected observation information entropy gain of the current candidate viewpoint. It is expressed as follows:

[0036] ;

[0037] in, As the current candidate viewpoint, For the first The total number of primes that a simulated ray passes through before hitting a boundary or being completely blocked. The total number of simulated rays emitted by the 3D camera is discretized using its inherent field of view and resolution parameters;

[0038] Based on the robot motion cost penalty term, a comprehensive benefit evaluation index function is constructed for the current candidate viewpoint. , means as follows:

[0039] ;

[0040] in, For the robot actuator from the current joint posture Move to the current candidate viewpoint The total displacement of the robotic arm joint angles or the motion time required for the posture; and These are preset weighting coefficients;

[0041] The candidate viewpoint that yields the maximum value of the comprehensive benefit evaluation index function after calculation among several candidate viewpoints is selected as the next optimal viewpoint.

[0042] In a further embodiment, the coarse registration of the first point cloud data and the second point cloud data based on the spatial true pose transformation matrix of the default initial view and the next optimal view includes:

[0043] The movement data during the movement from the default initial viewpoint to the next optimal viewpoint is obtained, and the spatial relative pose transformation matrix is ​​obtained through the forward kinematic equations. , means as follows:

[0044] ;

[0045] in, This represents the actual pose matrix of the robot actuators under the default initial view. This is the actual pose matrix of the robot actuator from the next optimal perspective;

[0046] By combining the spatial relative pose transformation matrix and the hand-eye calibration matrix, the true spatial pose transformation matrix of the 3D camera between the default initial viewpoint and the next optimal viewpoint is obtained. , means as follows:

[0047] ;

[0048] in, , ; For hand-eye calibration matrix, This represents the global coordinate matrix of the 3D camera under the default initial viewpoint. This is the global coordinate matrix of the 3D camera at the next optimal viewpoint.

[0049] The spatial true pose transformation matrix is ​​applied to the second point cloud data to align the coordinate system of the second point cloud data to the neighborhood of the coordinate system of the first point cloud data, thus completing the coarse registration, as shown below:

[0050] ;

[0051] in, For the second point cloud data The non-homogeneous coordinates of any point in the second local coordinate system in three-dimensional space. To add 1 to the second point cloud data Expanded into a four-dimensional homogeneous coordinate vector, For the second point cloud data The coarse-aligned second-order coordinates of any point in the first local coordinate system.

[0052] In a further embodiment, the iterative alignment of the first point cloud data and the second point cloud data until data fusion is completed includes:

[0053] A point-to-plane iterative nearest point algorithm is used for fine registration until the number of iterations converges to single digits to obtain finely registered multi-frame point cloud data. The finely registered multi-frame point cloud data is then transformed into a unified global coordinate system for stitching and fusion.

[0054] In a further embodiment, generating a 3D reconstruction model of the target object after filtering the fused data includes:

[0055] In the multi-view overlapping area, the fused finely registered multi-frame point cloud data is filtered by statistical outlier removal, and redundant points and noise are removed to establish a fused point cloud set;

[0056] Find the nearest spatial sampling point in Euclidean space from any spatial sampling point in the fused point cloud set. After finding the nearest neighbor, the spatial sampling point and its corresponding neighbor are calculated sequentially. The Euclidean geometric spatial distance between the nearest points is used to obtain the sampling point in this space. Corresponding average nearest neighbor distance , means as follows:

[0057] ;

[0058] in, For this spatial sampling point to the th Euclidean geometric spatial distance between neighboring points;

[0059] right The average distance is calculated from each spatial sampling point. and the global standard deviation characterizing the degree of distance dispersion. , means as follows:

[0060] ;

[0061] ;

[0062] The average nearest neighbor distance satisfies the following relationship: The spatial sampling point set is a filtered dense and clean point cloud. This is a preset standard deviation multiple threshold factor;

[0063] Normals are estimated from the filtered dense clean point cloud to output a complete 3D reconstruction model with surface topology.

[0064] Secondly, a multi-viewpoint rapid 3D reconstruction system is provided, including:

[0065] The initial setup module is configured to set the effective physical observation range of the 3D camera for the target object according to the three-dimensional physical boundary parameters of the current work station after the 3D camera adjusts the exposure parameters based on the current work station environment.

[0066] The first point cloud acquisition module is configured to convert the first frame image of the target object obtained by the 3D camera at the default initial viewpoint into first point cloud data in the first local coordinate system within the effective physical observation range.

[0067] The optimal viewpoint acquisition module is configured to obtain the missing viewpoint region based on the spatial grid model constructed from the first point cloud data, and to obtain the next optimal viewpoint after calculating several candidate viewpoints in the missing viewpoint region.

[0068] The second point cloud acquisition module is configured to convert the second frame image of the target object acquired from the next best viewpoint into second point cloud data in a second local coordinate system.

[0069] The coarse registration module is configured to complete the coarse registration of the first point cloud data and the second point cloud data based on the spatial true pose transformation matrix of the default initial view and the next best view.

[0070] The model building module is configured to use the spatial real pose transformation matrix as the initial value, iteratively align the first point cloud data and the second point cloud data until data fusion is completed, and generate a three-dimensional reconstruction model of the target object after filtering the fused data.

[0071] The beneficial effects of this invention are as follows: by sequentially acquiring images of the target object from multiple viewpoints and generating corresponding point cloud data, it is possible to acquire more comprehensive information about the target object and obtain the next best viewpoint based on the previous point cloud data, accurately determining the best acquisition position. This not only avoids subsequent failures in robotic arm grasping pose planning or collisions, but also avoids the generation of a large amount of redundant data caused by blind acquisition, thus improving reconstruction efficiency.

[0072] By adaptively adjusting the parameters of the 3D camera, actively planning the viewpoint, and introducing prior kinematic data of the robot to obtain the real spatial pose transformation matrix of the 3D camera in real time, and using this real spatial pose transformation matrix as a good initial pose for the registration of two point clouds, a fast coarse registration is performed. This solves the problem of existing algorithms lacking a good initial pose, significantly reduces the number of iterations of algorithms such as ICP, and achieves efficient and high-quality 3D reconstruction of the target object. Attached Figure Description

[0073] Figure 1 This is a flowchart of a multi-viewpoint rapid 3D reconstruction method in one embodiment of the present invention. Detailed Implementation

[0074] In the following description, numerous specific details are set forth in order to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described in order to avoid obscuring the invention.

[0075] The present invention will be further described in detail below with reference to the accompanying drawings.

[0076] Example 1: Refer to Figure 1 This is a flowchart of a multi-viewpoint rapid 3D reconstruction method disclosed in this embodiment. The method includes:

[0077] S1: After the robot actuator establishes a connection with the 3D camera, and the 3D camera adjusts its exposure parameters based on the current workstation environment, the effective physical observation range of the 3D camera for the target object is set according to the three-dimensional physical boundary parameters of the current workstation. The robot actuator includes an end effector and a base.

[0078] The 3D camera adjusts its exposure parameters based on the current workstation environment as follows:

[0079] S1-1: After initializing the parameters, the 3D camera automatically executes an exposure adjustment program based on histogram analysis according to the lighting conditions of the current workstation environment;

[0080] The exposure adjustment procedure specifically involves: acquiring a two-dimensional grayscale image in the current workstation environment in real time and calculating its grayscale histogram distribution; statistically analyzing the pixel ratio of overexposed highlight areas (grayscale value close to 255) and defective dark areas (grayscale value close to 0); and adaptively adjusting the exposure time of the 3D camera using an iterative stepping method by setting a target grayscale mean range until the energy center peak of the grayscale histogram converges to the target grayscale mean range. This ensures that the structured light stripes emitted by the 3D camera have the best contrast on the surface of the target object, avoiding phase saturation point cloud loss due to high reflectivity or low signal-to-noise ratio point cloud holes due to insufficient illumination.

[0081] The method of setting the effective physical observation range of the 3D camera for the target object based on the three-dimensional physical boundary parameters of the current work station includes:

[0082] S1-2: Obtain the work station parameter database established by the three-dimensional physical boundary parameters of each different work station, call the corresponding three-dimensional physical boundary parameters based on the corresponding ID generated for each work station, and map them as the effective physical observation range of the target object by the 3D camera.

[0083] Specifically, the three-dimensional physical boundary parameters are the coordinates of the three-dimensional spatial clipping frame preset in the robot's reference coordinate system. The mapping process is as follows: Real-time acquisition of joint encoder data representing the current physical posture of the robot actuator; solving for the current robot end-effector pose matrix using forward kinematics equations; and performing a cascaded transformation using pre-stored hand-eye calibration matrices to construct a spatial transformation matrix from the robot's reference coordinate system to the current camera coordinate system. Using this spatial transformation matrix, the coordinates of the 3D spatial clipping box are transformed using linear coordinate projection to calculate the boundary parameters and effective depth range of the 3D Region of Interest (ROI) in the current camera coordinate system. Finally, by calling the 3D camera's SDK low-level configuration interface, the boundary parameters and depth range in the camera coordinate system are sent as configuration instructions to the camera's hardware registers, thus mapping them to the effective physical observation range of the 3D camera for the target object. This allows the 3D camera to directly truncate and remove redundant background point cloud data exceeding this effective physical observation range at the low-level hardware level during the original point cloud stream generation stage.

[0084] The effective observation physical range is the region of interest and the effective depth truncation range after removing redundant background data that exceeds the physical boundary of the target object. Specifically, a workstation parameter database is established in advance through offline teaching or CAD model import. The workstation parameter database records the unique identifier (ID) of each workstation and its three-dimensional physical bounding box coordinates in the robot reference coordinate system. When determining the ROI, the coordinates of each 3D vertex of the invoked 3D physical bounding box are extracted. Combined with the current robot end-effector pose matrix and the camera's intrinsic parameter matrix, a 3D-to-2D perspective projection calculation is performed to obtain the 2D pixel bounding rectangle of this boundary on the camera's photosensitive plane, thus defining the hardware-level ROI. Simultaneously, a distance range along the camera's optical axis is set as the effective depth cutoff range (e.g., for a material bin station, only depth data within 0.5m-1.5m of the camera along the optical axis is retained). The ROI and depth cutoff range are directly sent to the camera's hardware registers, filtering out redundant data from the station background and ground at the initial stage of point cloud generation. Through early ROI definition, automatic exposure optimization, and subsequent active viewpoint planning for blind spots, meaningless repeated shooting is avoided, allowing for complete reconstruction with only a minimal number of viewpoints. Combining hardware-level automatic exposure control and depth range truncation, high-quality basic point clouds can still be acquired even in highly reflective or complex lighting factory environments.

[0085] S2: Within the effective physical observation range, the first frame image of the target object obtained by the 3D camera at the default initial viewpoint is converted into the first point cloud data in the first local coordinate system; wherein, the default initial viewpoint is a pre-set fixed observation pose designed to maximize the global field of view to cover the physical boundary of the work station; the first local coordinate system is a camera coordinate system with the camera optical center as the origin and the optical axis as the Z-axis.

[0086] The process of converting the first frame image of the target object obtained by the industrial 3D camera at a default initial viewpoint into first point cloud data in a first local coordinate system includes:

[0087] S2-1: Based on the intrinsic parameter matrix of the 3D camera, the first frame image is converted into first point cloud data in a first local coordinate system; specifically, the first frame image includes a first frame depth map and a first frame color map acquired synchronously. The specific mathematical implementation process of the conversion is as follows: the main control program locally analyzes and traverses each effective pixel point in the first frame depth map that is located within the effective physical observation range, and extracts its two-dimensional pixel coordinates. and the corresponding depth value Subsequently, the internal parameters of the 3D camera are invoked, including the horizontal pixel focal length. Vertical pixel focal length Horizontal pixel coordinates of optical center and the vertical pixel coordinates of the optical center Using the pinhole camera's inverse projection geometric model, each effective pixel is substituted into the 3D space reconstruction formula for conversion;

[0088] The abscissa in the three-dimensional space under the first local coordinate system is represented as:

[0089] ;

[0090] The ordinate in three-dimensional space is represented as:

[0091] ;

[0092] The three-dimensional spatial depth coordinates are represented as:

[0093] ;

[0094] Thus, the two-dimensional coordinates and depth information of each pixel are transformed into three-dimensional spatial coordinates in the camera's first local coordinate system. The data is then mapped to the corresponding color features of the color map, thereby generating the first point cloud data. .

[0095] S2-2: Subsequently, a voxel grid filter is used to downsample the first point cloud data to balance the amount of data and the preservation of geometric features. Specifically, the downsampling process is as follows: the voxel size parameter of the three-dimensional voxel grid is preset according to the structural refinement of the target object (for example, set to a 5mm×5mm×5mm cubic grid); the original dense first point cloud data is spatially segmented using the voxel grid, and the centroid of the spatial coordinates of all three-dimensional point clouds falling into each tiny cubic grid is calculated; finally, the centroid is used as the only retained sampling point in the grid to replace the original multiple discrete points, thereby eliminating microscopic noise at the bottom layer and significantly compressing the amount of point cloud data, achieving the goal of balancing the computational load of subsequent algorithms and the preservation of key geometric topological features of the target object.

[0096] S3: Based on the spatial grid model constructed from the first point cloud data, obtain several candidate viewpoints in the missing viewpoint region, and calculate the next best viewpoint from the several candidate viewpoints.

[0097] The spatial grid model constructed based on the first point cloud data obtains several candidate viewpoints in the missing viewpoint region, including:

[0098] S3-1: Construct a spatial grid model of an octree or voxel grid based on the first point cloud data. The spatial grid model is divided into occupied state region, idle state region and unknown state region according to the spatial state.

[0099] S3-2: The boundary voxel between the extracted occupied state region and the unknown state region is used as the missing boundary to form the missing viewpoint region;

[0100] S3-3: Using the geometric center or normal direction of the missing viewpoint region (boundary voxel) as a reference, and combined with the effective observation distance of the 3D camera, construct a virtual sampling sphere or sampling hemisphere around the missing viewpoint region; perform discretization sampling on the sampling sphere or sampling hemisphere to generate a set of candidate poses facing the missing viewpoint region; further, input the set of candidate poses into the robot's kinematic constraint model and collision detection model, eliminate invalid poses that exceed the robot's kinematic reach or pose a risk of environmental collision, and retain the set of valid poses as the candidate viewpoints;

[0101] The process of calculating several candidate viewpoints in the missing viewpoint region to obtain the next optimal viewpoint includes:

[0102] S3-4: The information entropy gain of the candidate viewpoint to the unknown state region is simulated and calculated using a ray casting algorithm, and the candidate viewpoint with the largest gain is selected as the next best viewpoint.

[0103] Specifically, the calculation and selection process for the next optimal viewpoint is as follows:

[0104] First, the "unknown voxels" constituting the unknown state region are defined. These unknown voxels refer to three-dimensional mesh units located within the effective physical observation range and not penetrated or illuminated by the measurement light from the 3D camera during historical scanning. Their initial occupancy probability is preset to [value missing]. The main control program calculates each voxel in the space based on Shannon's information entropy theory. Initial information entropy The calculation formula is shown below:

[0105] ;

[0106] As can be seen from the above formula, when the voxel is an unknown voxel, its information entropy is... When a voxel has been identified as having a defined occupied state ( ) or idle state ( When ), its information entropy Subsequently, the main control program iterates through each candidate viewpoint in the set of valid poses; for the current candidate viewpoint... Based on the inherent field of view (FOV) and resolution parameters of the 3D camera, the emitted signal is discretized and simulated. A spatial measurement simulation ray is used; the sequence of voxels along the path of each simulated ray as it passes through the spatial grid model is calculated using a ray-casting algorithm, and a ray visibility attenuation factor is introduced. This is used to simulate the occlusion effect of the preceding voxel on the subsequent unknown voxel, and its calculation formula is shown in the following formula:

[0107] ;

[0108] in, Indicates the first The simulated ray traversing the first Individual factors, For the preceding first ray The known occupancy probability of the individual element. Based on this, the candidate viewpoint... Expected observation information entropy gain The calculation formula is shown below:

[0109] ;

[0110] in, For the first The total prime number of all rays that pass through before hitting a boundary or being completely blocked. To balance motion efficiency and collision avoidance safety in industrial settings, this invention further introduces a robot motion cost penalty term, constructing a comprehensive benefit evaluation index function for candidate viewpoints. As shown in the following formula:

[0111] ;

[0112] in, For the robot actuator from the current joint posture Move to target candidate viewpoint The total displacement of the robotic arm joint angles or the motion time required for the posture; and These are preset weighting coefficients. The main control program calculates and adjusts the function accordingly. The candidate viewpoint that yields the maximum value is determined as the next optimal viewpoint and converted into robot joint movement commands for issuance.

[0113] S4: Control the robot actuator to smoothly move the 3D camera to the next optimal viewing angle, and acquire the second frame image of the target object from the next optimal viewing angle. Use the intrinsic parameter matrix of the 3D camera to convert it into second point cloud data in the local coordinate system. The second frame image includes a second frame depth map and a second frame color map acquired synchronously.

[0114] In this embodiment, since the 3D camera undergoes spatial displacement along with the robot, the second point cloud data obtained here is... The second local coordinate system in which the camera is located is the "second camera local coordinate system" established when the camera is in the next optimal viewing angle; its origin position and axis in three-dimensional space are the same as those obtained from the first point cloud data. The first local coordinate system of the "first camera local coordinate system" established when the camera is in the default initial view is completely different from the first local coordinate system of the camera. The two belong to independent local spatial coordinate systems.

[0115] S5: Obtain the robot arm joint encoder data during the movement from the default initial viewpoint to the next optimal viewpoint, calculate the relative pose using the forward kinematic equations, and after cascading the hand-eye calibration matrix, accurately obtain the spatial true pose transformation matrix of the 3D camera between the default initial viewpoint (first camera local coordinate system) and the next optimal viewpoint (second camera local coordinate system). Using the aforementioned spatial true pose transformation matrix With the second point cloud data The second point cloud data is obtained by homogeneous matrix multiplication of the three-dimensional spatial coordinates in the local coordinate system of the second camera. The data is uniformly transformed to the spatial neighborhood of the local coordinate system of the first camera, thereby eliminating most of the pose deviations caused by hardware motion and efficiently completing the first point cloud data. And the second point cloud data Rapid coarse registration.

[0116] Specifically, the coarse registration of the first point cloud data and the second point cloud data based on the spatial true pose transformation matrix of the default initial view and the next optimal view includes:

[0117] S5-1: Obtain the movement data during the movement from the default initial viewpoint to the next optimal viewpoint, and obtain the spatial relative pose transformation matrix through the forward kinematic equations. Specifically, this scheme constructs a rigorous robot kinematic chain coordinate system to completely eliminate multi-matrix confusion: the robot reference coordinate system is defined as... The default initial viewpoint coordinate system for the robot's end effector is... The robot's end effector coordinate system after moving to the next optimal viewpoint is: During movement, the system reads the encoder data of the servo motors of each joint of the robot actuator in real time, calculates the rotation angle of each joint, and substitutes it into the standard DH (Denavit-Hartenberg) kinematic matrix for cascaded multiplication to obtain the actual pose matrix of the robot's end effector relative to the robot's reference coordinate system under the default initial view and the next optimal view. and By performing matrix inversion and left multiplication, the spatial relative pose transformation matrix of the robot's end effector between two positions is calculated. Its mathematical formula is shown below:

[0118] ;

[0119] S5-2: Combining the spatial relative pose transformation matrix with the hand-eye calibration matrix of the robot end effector, obtain the spatial true pose transformation matrix of the 3D camera between the default initial viewpoint and the next optimal viewpoint. The hand-eye calibration matrix is ​​a pre-calibrated homogeneous spatial rigid body transformation matrix from the robot end effector coordinate system to the 3D camera coordinate system, denoted as... Since the global coordinates of a 3D camera at the default initial viewpoint and the next optimal viewpoint can be represented as follows: and Therefore, in order to eliminate the influence of the robotic arm link offset on the camera imaging space, the spatial relative pose transformation matrix is... With hand-eye calibration matrix By performing cascaded coordinate transformations, the true spatial pose transformation matrix of the 3D camera itself can be calculated. The standard calculation formula is shown below:

[0120] ;

[0121] As can be seen from the above equation, the spatial true pose transformation matrix It is a 3×3 rotation matrix and 3×1 translation vector The 4×4 homogeneous transformation matrix is ​​clearly distinguished from the relative pose matrix of the end effector itself. The spatial true pose transformation matrix is ​​then... The system is applied to the second point cloud data to align the coordinate system of the second point cloud data to the neighborhood of the coordinate system of the first point cloud data, thus completing the coarse registration.

[0122] Specifically, the coordinate multiplication is implemented as follows:

[0123] Extract the second point cloud data The non-homogeneous three-dimensional coordinates of any point in the second camera's local coordinate system It is expanded into a four-dimensional homogeneous coordinate vector by padding with 1. Subsequently, the spatial true pose transformation matrix is... By performing a homogeneous matrix left multiplication with the four-dimensional homogeneous coordinate vector, the coarsely aligned homogeneous coordinates of the point in the local coordinate system of the first camera are calculated. The implementation method is shown in the following formula:

[0124] ;

[0125] in, Represents a 3×3 rotation matrix. Representing a 3×1 translation vector, the main control program traverses and transforms the second point cloud data. All points in the first point cloud data are used to instantly bring the entire frame of second point cloud data closer together in geometric space and align it to the nearest neighbor of the first point cloud data coordinate system.

[0126] S6: Using the spatial true pose transformation matrix as the initial value, iteratively align the first point cloud data and the second point cloud data until data fusion is completed, and then filter the fused data to generate the three-dimensional reconstruction model of the target object.

[0127] The initial value mentioned above has a clear mathematical representation in the algorithm, that is, it is directly obtained by solving the homogeneous transformation matrix. Assigned as the initial transformation estimation matrix for the subsequent fine registration iterative optimization solver. (i.e., order) ).

[0128] The iterative alignment of the first point cloud data and the second point cloud data until data fusion is completed includes:

[0129] S6-1: Use the point-to-plane iterative nearest point algorithm for fine registration until the number of iterations converges to a single digit to obtain finely registered multi-frame point cloud data. Then, convert the finely registered multi-frame point cloud data to a unified global coordinate system for stitching and fusion.

[0130] Furthermore, in this scheme, the unified global coordinate system is physically equivalent to the first camera local coordinate system under the default initial viewpoint.

[0131] Specifically, the splicing and fusion process is as follows:

[0132] The fine-fit optimizer is in the initial value The normal vector of the first point cloud plane is calculated within the nearest neighbor region. By minimizing the least squares error function of the distance from the source point to the target plane, 3-5 local optimum iterations are completed within milliseconds to solve for the final fine registration homogeneous matrix. ;use The second point cloud data is fully projected and transformed into the local coordinate system of the first camera, and then cascaded and accumulated with the first point cloud data to achieve seamless stitching of multi-viewpoint point clouds in a common three-dimensional spatial network data structure.

[0133] The process of generating a 3D reconstruction model of the target object after filtering the fused data includes:

[0134] S6-2: In the multi-viewpoint overlapping area, the fused finely registered multi-frame point cloud data is subjected to statistical outlier removal filtering to remove redundant points and noise;

[0135] S6-3: Calculate the point cloud data for each frame and its... The distance distribution characteristics of neighboring points are used to remove outliers whose distance is greater than a set standard deviation, resulting in a filtered dense and clean point cloud.

[0136] Specifically, firstly, a KD-Tree spatial index structure is used to efficiently construct the nearest neighbor topology of the fused point cloud set; for any spatial sampling point in the point cloud... Find the nearest neighbor in Euclidean geometric space using a KD-Tree fast search. Proximity points (of which) The terms are preset positive integers, preferably between 20 and 50, and are calculated sequentially. Arrive here Euclidean spatial distance between neighboring points Based on this, the average nearest neighbor distance corresponding to this point is calculated, as follows:

[0137] ;

[0138] After traversing the entire fused point cloud set, all After sampling each spatial point, the main control program globally calculates the average distance of all points. and the global standard deviation characterizing the degree of distance dispersion. The mathematical definition of its standard deviation is as follows:

[0139] ;

[0140] ;

[0141] The system presets a standard deviation multiple threshold factor. (Typically set between 1.0 and 2.0). The main control program determines point by point: if the average nearest neighbor distance of a spatial sampling point satisfies the following relationship... If the value is less than or equal to the actual geometric surface of the object, the system defines it as a noise outlier and removes it directly from memory; otherwise, it satisfies the relational expression. All valid spatial sampling point sets are retained as high-precision, noise-free dense clean point clouds, completely eliminating the logical contradiction of traditional parallel production of two sets of point clouds: "dense" and "discrete".

[0142] S6-4: Normal estimation is performed on the filtered dense clean point cloud, and the Poisson surface reconstruction algorithm is used to fit the dense clean point cloud into a continuous triangular mesh model, outputting a complete target 3D reconstruction model with surface topology, providing high-precision data support for the robot arm's subsequent safe and collision-free grasping and path planning.

[0143] Example 2: Based on Example 1, this example discloses a multi-viewpoint rapid 3D reconstruction system, which includes at least a robot actuator containing a joint encoder for changing the observation viewpoint and outputting posture data. Optionally, the robot actuator is an AGV robot with a base and a multi-axis robotic arm, and its robotic arm end effector or gimbal is equipped with an industrial-grade 3D camera such as Mech-Eye.

[0144] The system includes:

[0145] The initial setup module is configured to set the effective physical observation range of the 3D camera for the target object according to the three-dimensional physical boundary parameters of the current work station after the 3D camera adjusts the exposure parameters based on the current work station environment.

[0146] The first point cloud acquisition module is configured to convert the first frame image of the target object obtained by the 3D camera at the default initial viewpoint into first point cloud data in the first local coordinate system within the effective physical observation range.

[0147] The optimal viewpoint acquisition module is configured to obtain the missing viewpoint region based on the spatial grid model constructed from the first point cloud data, and to obtain the next optimal viewpoint after calculating several candidate viewpoints in the missing viewpoint region.

[0148] The second point cloud acquisition module is configured to convert the second frame image of the target object acquired from the next best viewpoint into second point cloud data in a second local coordinate system.

[0149] The coarse registration module is configured to complete the coarse registration of the first point cloud data and the second point cloud data based on the spatial true pose transformation matrix of the default initial view and the next best view.

[0150] The model building module is configured to use the spatial real pose transformation matrix as the initial value, iteratively align the first point cloud data and the second point cloud data until data fusion is completed, and generate a three-dimensional reconstruction model of the target object after filtering the fused data.

[0151] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0152] The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present invention, various equivalent transformations can be made to the technical solutions of the present invention, and all such equivalent transformations fall within the protection scope of the present invention.

Claims

1. A multi-viewpoint fast 3D reconstruction method, characterized in that: The method includes: After the 3D camera adjusts the exposure parameters based on the current work station environment, the effective physical observation range of the 3D camera for the target object is set according to the three-dimensional physical boundary parameters of the current work station. Within the effective physical observation range, the first frame image of the target object obtained by the 3D camera at the default initial viewpoint is converted into first point cloud data in the first local coordinate system; Based on the spatial grid model constructed from the first point cloud data, several candidate viewpoints are obtained in the missing viewpoint region, and the next best viewpoint is obtained by calculating the several candidate viewpoints. The spatial grid model constructed based on the first point cloud data obtains several candidate viewpoints in the missing viewpoint region, including: Based on the first point cloud data, an octree or voxel grid spatial grid model is constructed. The spatial grid model is divided into occupied state region, idle state region and unknown state region according to the spatial state. The boundary voxel between the extracted occupied state region and the unknown state region is used as the missing boundary to form the missing viewpoint region; The candidate pose set generated in the missing viewpoint region is input into the kinematic constraint model and the collision detection model. After removing invalid poses, the remaining valid pose set is used as the candidate viewpoints. The calculation of the next optimal viewpoint from the candidate viewpoints includes: Calculate the initial information entropy of each voxel in the spatial grid model. , means as follows: ； in, For voxels, Let be the initial occupancy probability of each voxel, when When the voxel is an unknown voxel in the unknown state region; One of the candidate viewpoints is selected as the current candidate viewpoint. A ray casting algorithm is then used to calculate the voxel sequence that each simulated ray passes through as it traverses the spatial mesh model, and a ray visibility attenuation factor is introduced. The occlusion effect of the preceding voxel on the subsequent unknown voxel is simulated as follows: ； in, Indicates the first The simulated ray traversing the first Individual factors, For the simulated ray, the first preposition The known occupancy probability of an individual element. To simulate ray indexing, , For the first Voxel indexes on simulated rays, the expected observation information entropy gain of the current candidate viewpoint It is expressed as follows: ； in, As the current candidate viewpoint, For candidate viewpoint index, For the first The total number of primes that a simulated ray passes through before hitting a boundary or being completely blocked. The total number of simulated rays emitted by the 3D camera is discretized using its inherent field of view and resolution parameters; Based on the robot motion cost penalty term, a comprehensive benefit evaluation index function is constructed for the current candidate viewpoint. , means as follows: ； in, For the robot actuator from the current joint posture Move to the current candidate viewpoint The total displacement of the robotic arm joint angles or the motion time required for the posture; and These are preset weighting coefficients; The candidate viewpoint that yields the maximum value of the comprehensive benefit evaluation index function after solving among a number of candidate viewpoints is selected as the next best viewpoint; The second frame image of the target object acquired from the next optimal viewpoint is converted into second point cloud data in the second local coordinate system; Based on the spatial true pose transformation matrix of the default initial view and the next optimal view, coarse registration of the first point cloud data and the second point cloud data is completed. Using the spatial true pose transformation matrix as the initial value, the first point cloud data and the second point cloud data are iteratively aligned until data fusion is completed. The fused data is then filtered to generate a three-dimensional reconstruction model of the target object.

2. The multi-viewpoint rapid 3D reconstruction method according to claim 1, characterized in that: The 3D camera adjusts its exposure parameters based on the current workstation environment. After establishing a connection with the robot actuator, the 3D camera automatically executes an exposure adjustment program based on histogram analysis according to the lighting conditions of the current workstation environment. The exposure adjustment program includes: Acquire a two-dimensional grayscale image of the current workstation environment under the current lighting conditions and calculate its grayscale histogram distribution to obtain the energy center peak of the grayscale histogram; Based on the set target gray-level mean range, the exposure time of the 3D camera is adaptively adjusted using an iterative step method until the energy center peak of the gray-level histogram converges to the target gray-level mean range.

3. The multi-viewpoint rapid 3D reconstruction method according to claim 1, characterized in that: The process of setting the effective physical observation range of the 3D camera for the target object based on the three-dimensional physical boundary parameters of the current work station includes: A workstation parameter database is obtained by establishing the three-dimensional physical boundary parameters of each different workstation. Based on the corresponding ID generated for each workstation, the corresponding three-dimensional physical boundary parameters are called and mapped to the effective physical observation range of the target object by the 3D camera. The effective physical observation range is the region of interest and the effective depth truncation range after removing redundant background data that exceeds the physical boundary of the target object.

4. The multi-viewpoint rapid 3D reconstruction method according to claim 1, characterized in that: The process of converting the first frame image of the target object obtained by the 3D camera at a default initial viewpoint into first point cloud data in a first local coordinate system includes: Based on the intrinsic parameter matrix of the 3D camera, the first frame image is converted into first point cloud data in the first local coordinate system; The first frame image includes a first frame depth map and a first frame color map acquired synchronously.

5. The multi-viewpoint rapid 3D reconstruction method according to claim 1, characterized in that: The coarse registration of the first point cloud data and the second point cloud data based on the spatial true pose transformation matrix of the default initial view and the next optimal view includes: The movement data during the movement from the default initial viewpoint to the next optimal viewpoint is obtained, and the spatial relative pose transformation matrix is ​​obtained through the forward kinematic equations. , means as follows: ； in, This represents the actual pose matrix of the robot actuators under the default initial view. This is the actual pose matrix of the robot actuator from the next optimal perspective; By combining the spatial relative pose transformation matrix and the hand-eye calibration matrix, the true spatial pose transformation matrix of the 3D camera between the default initial viewpoint and the next optimal viewpoint is obtained. , means as follows: ； in, , ; For hand-eye calibration matrix, This represents the global coordinate matrix of the 3D camera under the default initial viewpoint. This is the global coordinate matrix of the 3D camera at the next optimal viewpoint. The spatial true pose transformation matrix is ​​applied to the second point cloud data to align the coordinate system of the second point cloud data to the neighborhood of the coordinate system of the first point cloud data, thus completing the coarse registration, as shown below: ； in, For the second point cloud data The non-homogeneous three-dimensional coordinates of any point in the second local coordinate system. To add 1 to the second point cloud data Expanded into a four-dimensional homogeneous coordinate vector, For the second point cloud data The coarse-aligned second-order coordinates of any point in the first local coordinate system.

6. The multi-viewpoint rapid 3D reconstruction method according to claim 1, characterized in that: The iterative alignment of the first point cloud data and the second point cloud data until data fusion is completed includes: A point-to-plane iterative nearest point algorithm is used for fine registration until the number of iterations converges to single digits to obtain finely registered multi-frame point cloud data. The finely registered multi-frame point cloud data is then transformed into a unified global coordinate system for stitching and fusion.

7. The multi-viewpoint rapid 3D reconstruction method according to claim 6, characterized in that: The process of generating a 3D reconstruction model of the target object after filtering the fused data includes: In the multi-view overlapping area, the fused finely registered multi-frame point cloud data is filtered by statistical outlier removal, and redundant points and noise are removed to establish a fused point cloud set; Find the nearest spatial sampling point in Euclidean space from any spatial sampling point in the fused point cloud set. After finding the nearest neighbor, the spatial sampling point and its corresponding neighbor are calculated sequentially. The Euclidean geometric spatial distance between the nearest points is used to obtain the sampling point in this space. Corresponding average nearest neighbor distance , means as follows: ； in, For this spatial sampling point to the th Euclidean geometric distance between neighboring points For nearest neighbor index, This represents the total number of neighboring points; right The average distance is calculated from each spatial sampling point. and the global standard deviation characterizing the degree of distance dispersion. , means as follows: ； ； in, For spatial sampling point index, The total number of spatial sampling points, and the average nearest neighbor distance satisfy the following relationship: The spatial sampling point set is a filtered dense and clean point cloud. This is a preset standard deviation multiple threshold factor; Normals are estimated from the filtered dense clean point cloud to output a complete 3D reconstruction model with surface topology.

8. A multi-viewpoint rapid 3D reconstruction system, based on the multi-viewpoint rapid 3D reconstruction method according to any one of claims 1-7, characterized in that: include: The initial setup module is configured to set the effective physical observation range of the 3D camera for the target object according to the three-dimensional physical boundary parameters of the current work station after the 3D camera adjusts the exposure parameters based on the current work station environment. The first point cloud acquisition module is configured to convert the first frame image of the target object obtained by the 3D camera at the default initial viewpoint into first point cloud data in the first local coordinate system within the effective physical observation range. The optimal viewpoint acquisition module is configured to obtain the missing viewpoint region based on the spatial grid model constructed from the first point cloud data, and to obtain the next optimal viewpoint after calculating several candidate viewpoints in the missing viewpoint region. The second point cloud acquisition module is configured to convert the second frame image of the target object acquired from the next best viewpoint into second point cloud data in a second local coordinate system. The coarse registration module is configured to complete the coarse registration of the first point cloud data and the second point cloud data based on the spatial true pose transformation matrix of the default initial view and the next best view. The model building module is configured to use the spatial real pose transformation matrix as the initial value, iteratively align the first point cloud data and the second point cloud data until data fusion is completed, and generate a three-dimensional reconstruction model of the target object after filtering the fused data.

Images