Methods, terminal devices, and storage media for rapid fusion of point cloud data from multiple 3D cameras

Abstract

This invention relates to the field of image processing and discloses a method, terminal device, and storage medium for rapid fusion of point cloud data from multiple 3D cameras. The method includes: acquiring calibration data from multiple 3D cameras, including RGB images, depth maps, and intrinsic parameter matrices, wherein the calibration board must simultaneously appear within the field of view of multiple 3D cameras; generating a three-dimensional point cloud coordinate set based on the calibration data, and calculating a transformation matrix between a reference camera coordinate system and a non-reference camera coordinate system based on the three-dimensional point cloud coordinate set; registering the three-dimensional point cloud coordinate set to a preset reference camera coordinate system based on the transformation matrix to obtain merged target point cloud data, which includes RGB color values. This invention achieves multi-view complementarity, can cover the physical blind spots of a single camera, and solves the problems of insufficient field of view and point cloud loss caused by occlusion.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to a multi-3D camera point cloud data rapid fusion method, a terminal device and a storage medium. BACKGROUND

[0002] With the development of computer vision and three-dimensional reconstruction technology, point cloud data, as an important expression form of three-dimensional space information, has been widely applied in the fields of automatic driving, robot navigation, industrial detection and virtual reality.

[0003] The method of acquiring point cloud data by a 3D camera includes a structured light method, a time-of-flight method, binocular stereo vision and a laser radar, etc. However, due to the field of view limitation and the occlusion problem, a traditional single camera often cannot completely cover the target scene, resulting in incomplete point cloud data. A new technical means is needed to solve the above technical problems. SUMMARY

[0004] In view of this, the embodiments of the present application provide a multi-3D camera point cloud data rapid fusion method, a terminal device and a storage medium, which can solve the problem of incomplete point cloud data in the related art.

[0005] The first aspect of the present application provides a multi-3D camera point cloud data rapid fusion method, comprising:

[0006] Collecting calibration data of multiple 3D cameras, the calibration data including an RGB image, a depth image and an intrinsic matrix, wherein a calibration board needs to appear in the field of view of the multiple 3D cameras at the same time;

[0007] According to the calibration data, a three-dimensional point cloud coordinate point set is generated, and a transformation matrix between a reference camera coordinate system and a non-reference camera coordinate system is calculated according to the three-dimensional point cloud coordinate point set;

[0008] According to the transformation matrix, the three-dimensional point cloud coordinate point set is registered into a preset reference camera coordinate system to obtain merged target point cloud data, and the target point cloud data contains RGB color values.

[0009] Further, the step of calculating the transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system according to the three-dimensional point cloud coordinate point set comprises:

[0010] In the three-dimensional point cloud coordinate point set, a first point set under the reference camera coordinate system and a second point set under the non-reference camera coordinate system are obtained;

[0011] The first center of the first point set and the second center of the second point set are calculated;

[0012] According to the first center and the second center, respectively, a decentralized processing is performed to obtain a decentralized point set;

[0013] According to the decentralized point set, a covariance matrix is calculated;

[0014] The covariance matrix is singular value decomposed to obtain a rotation matrix;

[0015] According to the first center, the second center and the rotation matrix, a translation vector is calculated;

[0016] According to the rotation matrix and the translation vector, the transformation matrix is constructed.

[0017] Further, before the step of obtaining the first point set in the reference camera coordinate system and the second point set in the non-reference camera coordinate system, the method further comprises:

[0018] In the plurality of 3D cameras, a target camera is calibrated as a reference camera to determine the reference camera coordinate system and the non-reference camera coordinate system.

[0019] Further, the step of registering the environment point cloud obtained by each camera into the reference camera coordinate system according to the transformation matrix to obtain the merged target point cloud data comprises:

[0020] According to the transformation matrix, the three-dimensional point cloud coordinate point set is converted into a homogeneous coordinate;

[0021] According to the transformation matrix, an application transformation is performed on the homogeneous coordinate to obtain a point cloud set registered into the reference camera coordinate system;

[0022] According to all point cloud sets, merging is performed to obtain the merged target point cloud data.

[0023] Further, the step of generating a three-dimensional point cloud coordinate point set according to the calibration data comprises:

[0024] According to the RGB image, a corner point set in the calibration board is determined;

[0025] According to the depth image and the intrinsic matrix, the corner point set is restored from the pixel coordinate system to the reference camera coordinate system to obtain the three-dimensional point cloud coordinate point set.

[0026] Further, the step of restoring the corner point set from the pixel coordinate system to the reference camera coordinate system to obtain the three-dimensional point cloud coordinate point set according to the depth image and the intrinsic matrix comprises:

[0027] According to the RGB image, a corner point set in the calibration board is determined;

[0028] Based on the depth map, the depth value corresponding to each corner point in the corner point set is obtained, and based on the focal length parameter in the intrinsic parameter matrix, the pixel coordinate system and the reference camera coordinate system are transformed to convert the corner point set into an initial three-dimensional point cloud coordinate point set.

[0029] Based on the depth value, invalid values ​​are removed from the initial three-dimensional point cloud coordinate point set to obtain the three-dimensional point cloud coordinate point set.

[0030] Furthermore, the multiple 3D cameras are two or more 3D cameras, and the multiple 3D cameras are deployed on the same horizontal plane.

[0031] Furthermore, after the step of registering the three-dimensional point cloud coordinate set to the reference camera coordinate system according to the transformation matrix to obtain the merged target point cloud data, the method further includes:

[0032] Based on the target point cloud data, perform a material grabbing operation.

[0033] Secondly, embodiments of the present invention provide a terminal device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method for rapid fusion of point cloud data from multiple 3D cameras.

[0034] Thirdly, embodiments of the present invention provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described method for rapid fusion of point cloud data from multiple 3D cameras.

[0035] Fourthly, embodiments of the present invention provide a computer program product that, when run on a terminal device, causes the terminal device to execute the aforementioned method for rapid fusion of point cloud data from multiple 3D cameras.

[0036] The beneficial effects of this invention compared to existing technologies are as follows: It simultaneously acquires calibration data from multiple 3D cameras, and the calibration board must cover the field of view of all cameras. This achieves multi-view complementarity, covering the physical blind spots of a single camera and solving the problems of insufficient field of view and point cloud loss due to occlusion. Furthermore, a 3D point cloud coordinate set is generated based on the calibration data, and a rigid transformation matrix from the non-reference camera to the reference camera is calculated based on this. This simplifies the traditional, cumbersome multi-step calibration into a single calculation, significantly improving calibration efficiency while ensuring spatial consistency of point cloud fusion. The target point cloud data includes RGB color values, which also addresses the deficiency of lacking color information in point clouds, further improving the integrity of the point cloud data. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a schematic diagram of an embodiment of the method for rapid fusion of point cloud data from multiple 3D cameras according to the present invention;

[0039] Figure 2 This is a schematic diagram of a specific embodiment of step S102 of the method for rapid fusion of point cloud data from multiple 3D cameras in this invention.

[0040] Figure 3 This is a schematic diagram of a specific embodiment of step S103 of the method for rapid fusion of point cloud data from multiple 3D cameras in this invention.

[0041] Figure 4 This is a schematic diagram of another specific embodiment of step S102 of the method for rapid fusion of point cloud data from multiple 3D cameras in this invention;

[0042] Figure 5 This is a schematic diagram of one embodiment of the terminal device in this invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are protected by this invention.

[0044] It should be noted that the terms "comprising," "including," and "having," and any variations thereof, in the specification, claims, and accompanying drawings of this invention, are intended to cover non-exclusive inclusion. For example, a process, method, terminal, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. In the claims, specification, and accompanying drawings of this invention, relational terms such as "first" and "second" are used merely to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such immediate relationship or order between these entities / operations / objects.

[0045] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0046] With the development of computer vision and 3D reconstruction technology, point cloud data, as an important form of expression of 3D spatial information, has been widely used in fields such as autonomous driving, robot navigation, industrial inspection and virtual reality.

[0047] Methods for acquiring point cloud data using 3D cameras include structured light, time-of-flight, binocular stereo vision, and LiDAR. However, traditional single cameras often fail to fully cover the target scene due to field-of-view limitations and occlusion issues, resulting in incomplete point cloud data. A new technological approach is needed to address these problems.

[0048] In view of this, embodiments of the present invention provide a method, terminal device, and storage medium for rapid fusion of point cloud data from multiple 3D cameras. This method simultaneously acquires calibration data from multiple 3D cameras, and the calibration board must cover the field of view of all cameras. It achieves multi-view complementarity, covering the physical blind spots of a single camera and solving the problems of insufficient field of view and point cloud loss due to occlusion. Furthermore, a three-dimensional point cloud coordinate point set is generated based on the calibration data, and a rigid transformation matrix from the non-reference camera to the reference camera is calculated based on this. This simplifies the traditionally cumbersome multi-step calibration into a single calculation, significantly improving calibration efficiency while ensuring spatial consistency of point cloud fusion. The target point cloud data includes RGB color values, which also addresses the deficiency of lacking color information in point clouds, further improving the integrity of the point cloud data.

[0049] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0050] Figure 1 This illustration shows a flowchart of a method for rapid fusion of point cloud data from multiple 3D cameras, provided by an embodiment of the present invention. This method can be applied to terminal devices. Terminal devices can be mobile phones, tablets, laptops, ultra-mobile personal computers (UMPCs), netbooks, etc.

[0051] Specifically, the above-mentioned method for rapid fusion of point cloud data from multiple 3D cameras may include the following steps S101 to S103.

[0052] Step S101: Collect calibration data from multiple 3D cameras. The calibration data includes RGB images, depth maps, and intrinsic parameter matrices. The calibration board must appear in the field of view of the multiple 3D cameras simultaneously.

[0053] In an embodiment of the present invention, when the triggering condition is met, multiple 3D cameras (e.g., four) are triggered to simultaneously acquire calibration data. The calibration data includes RGB images, depth maps, and intrinsic parameter matrices.

[0054] Specifically, the RGB image is used to identify the location of the inner corner points of the calibration board; the depth image records the depth value of each pixel; and the intrinsic parameter matrix contains camera-specific parameters such as focal length.

[0055] It is important to note that the calibration board appears simultaneously in the depth and RGB fields of view of all cameras, so that all cameras can capture the complete data from the same calibration board.

[0056] In one example, the point cloud field of view achievable by each camera and the working area covered within that field of view are determined. Four cameras are triggered to acquire four sets of RGB images, depth maps, and intrinsic parameters of the cameras at the corresponding resolutions. The RGB images and depth maps must be aligned, and the four sets of calibration data must include the same calibration board.

[0057] Based on the four sets of depth maps and intrinsic parameter matrices collected, the four sets of calibration board interior corner point sets are transformed from the pixel coordinate system to the camera coordinate system to obtain the 3D point cloud coordinate set of the interior corner points, as shown in the following formula: ;

[0058] in, For pixel coordinates, For column indexes, This is a row index. This is the depth value, representing the distance of a pixel in the camera coordinate system. This is the intrinsic parameter matrix. , These are the focal length and the scaling factors for the horizontal and vertical directions, respectively. , It is the coordinate of the camera's optical axis on the image plane. and These are the horizontal and vertical coordinates in the camera coordinate system, based on the pixel positions of the depth image. and depth value Perform scaling and translation. This represents the depth value.

[0059] When the depth value of a certain pixel in the depth map When the condition is met, it indicates that the point has no depth information, and the corresponding world coordinates are set to undefined (represented by NaN), as shown in the following formula:

[0060]

[0061] Optionally, the multiple 3D cameras may consist of two or more 3D cameras, and these cameras are deployed on the same horizontal plane. Specifically, two or more 3D cameras are fixed on the same bracket, keeping all cameras on the same horizontal plane. This deployment method ensures that the field of view of each camera is uniformly covered in the horizontal direction, effectively avoiding overlapping or missing views due to height differences. During calibration, the calibration board is placed in the overlapping field of view area of ​​the four cameras, so that the calibration board appears simultaneously in the depth and RGB field of view of all cameras. The horizontal layout maximizes the utilization of the field of view coverage within a limited height.

[0062] Step S102: Generate a three-dimensional point cloud coordinate point set based on the calibration data, and calculate the transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system based on the three-dimensional point cloud coordinate point set.

[0063] In an embodiment of the present invention, the RGB image of each camera is processed into grayscale to identify the pixel coordinates of the corner points within the calibration board; combined with the depth value of the depth map and the intrinsic parameter matrix, the corner points are transformed from the pixel coordinate system to the three-dimensional coordinates in the camera coordinate system. If the depth value is 0, it is judged as an invalid point and marked as undefined (NaN), and finally the three-dimensional point cloud coordinate set in the coordinate system of each camera is obtained.

[0064] In a specific example, if the number of 3D cameras is set to 4, after obtaining the 3D point cloud coordinates of the interior corner points of the same calibration board in the coordinate systems of the four cameras, the transformation matrices between the remaining cameras and the reference camera are calculated. Here, the camera in the upper left corner is set as the reference camera, and the coordinate system of the upper left camera is set as the reference coordinate system. That is, the transformation matrix from the coordinate system of the lower left camera to the coordinate system of the upper left camera, and the transformation matrix from the coordinate system of the upper right camera to the coordinate system of the lower left camera are calculated respectively. Transformation matrix of the top-left camera coordinate system The transformation matrix from the bottom right camera coordinate system to the top left camera coordinate system. For example, for each non-reference camera, obtain its point set P (three-dimensional coordinates from the corner points of the calibration plate) in its coordinate system and the corresponding point set Q (three-dimensional coordinates of the same calibration plate corner point in the reference coordinate system) in the reference camera coordinate system; calculate the rotation matrix R and translation vector t between each non-reference camera and the reference camera respectively; construct the transformation matrix in the corresponding direction from the rotation matrix R and translation vector t.

[0065] Specifically, let the set of three-dimensional points in the reference camera coordinate system be... The remaining 3D point sets in the camera coordinate system are Point set and The formulas for performing centralized computation and decentralization are as follows:

[0066] ;

[0067] ;

[0068] ;

[0069] .

[0070] in, The geometric center (centroid) of the non-reference camera point set P; The geometric center (centroid) of the reference camera point set Q; The point set after decentering the non-reference camera point set P; The point set is the decentralized point set of the baseline camera point set Q.

[0071] Calculate the covariance matrix of the decentralized point set. Then, singular value decomposition is performed on it to obtain the rotation matrix and translation vector. Finally, the rotation matrix R and translation vector t are used to construct the transformation matrix. The formula is as follows:

[0072] ;

[0073] ;

[0074] ;

[0075] .

[0076] in, It is a homogeneous transformation matrix constructed from the rotation matrix R and the translation vector t, and it is the general form of the transformation matrix from all non-reference cameras to the reference camera. For four 3D cameras, one of which is the reference camera, the transformation matrices H corresponding to the other three non-reference cameras (such as the transformation matrices from the lower left, upper right, and lower right cameras to the reference camera) are all constructed according to... The specific instance generated by the structure and construction method (obtaining R through singular value decomposition and obtaining t by combining the center of the point set) is used. This is because it can simultaneously integrate rotation and translation transformations, achieving rigid registration of non-reference camera point clouds to the reference coordinate system through a single matrix operation. This ensures spatial consistency during multi-view point cloud fusion, solving the problems of insufficient field of view and missing point clouds from a single camera. U and V are orthogonal matrices. It is a singular value matrix.

[0077] Step S103: According to the transformation matrix, the three-dimensional point cloud coordinate point set is registered to the preset reference camera coordinate system to obtain the merged target point cloud data, which includes RGB color values.

[0078] In an embodiment of the present invention, the point cloud in the non-reference camera coordinate system is converted into homogeneous coordinates; a transformation matrix is ​​applied to perform coordinate transformation to register the point cloud to the reference coordinate system. All registered point clouds (including the reference camera point cloud) are merged into a single target point cloud; during merging, the color information of the original RGB image is preserved simultaneously, so that the target point cloud contains both geometric coordinates and color attributes.

[0079] In a specific example, following the previous example, the number of 3D cameras is 4. For each non-reference camera, the second point set P in its coordinate system and the first point set Q in the reference camera coordinate system are obtained from the 3D point cloud coordinate set. The geometric centers of P and Q are calculated respectively. After centering the two point sets, the covariance matrix is ​​obtained. The rotation matrix R is obtained by singular value decomposition of this matrix. Then, the translation vector t is calculated by combining the centers of the two point sets and R. Finally, the transformation matrix from the non-reference camera coordinate system to the reference camera coordinate system is constructed from R and t, that is, the transformation matrix from the lower left camera coordinate system to the upper left camera coordinate system is obtained. Transformation matrix from the upper right camera coordinate system to the upper left camera coordinate system The transformation matrix from the bottom right camera coordinate system to the top left camera coordinate system. .

[0080] Obtain the transformation matrix from the lower left camera coordinate system to the upper left camera coordinate system. Transformation matrix from the upper right camera coordinate system to the upper left camera coordinate system The transformation matrix from the bottom right camera coordinate system to the top left camera coordinate system. Then, the transformation matrix of each camera coordinate system is calculated to obtain three sets of point clouds transformed from the camera coordinate system to the reference camera coordinate system. Specifically, for the three non-reference cameras at the lower left, upper right, and lower right corners, their corresponding transformation matrices are called to perform coordinate transformation. For example, the 3D points in the point cloud of each non-reference camera are converted to homogeneous coordinates, and the coordinates in the reference coordinate system are calculated through matrix multiplication, thereby generating three sets of point clouds registered to the reference coordinate system.

[0081] Compare the three sets of point clouds with the point clouds in the original reference camera coordinate system. The points are then merged to obtain the stitched point cloud. The specific process is as follows:

[0082] Each point in the environmental point cloud acquired by the camera in the lower left corner Convert it to homogeneous coordinates and apply the transformation matrix Perform a transformation to obtain a point cloud set transformed to the upper left coordinate system. The formula is as follows:

[0083] ;

[0084] Use the same method to obtain the point cloud set transformed from the upper right corner to the upper left corner. The point cloud that transforms from the bottom right corner to the top left corner Four groups , , , The points are then merged to obtain the merged target point cloud data. The merging process involves preprocessing the four sets of point clouds to remove NaN points with invalid depth values ​​and isolated noise points. Subsequently, a point set overlay method is used to merge all valid points into a single set, while retaining the RGB color attributes of each point. For duplicate points in overlapping areas, points from the reference camera's point cloud are prioritized, while points from non-reference camera points are selected based on their higher color consistency, ultimately forming target point cloud data containing complete scene information.

[0085] Optionally, a material grasping operation is performed based on the target point cloud data. If connected, after point cloud registration and merging, a complete target point cloud containing RGB color values ​​is generated, covering the entire working area. The target point cloud data is input to a downstream execution system (such as a robotic arm control module), which identifies the material's position and orientation based on the geometric and color attributes of the point cloud, triggering the grasping action.

[0086] The beneficial effects of this invention compared to existing technologies are as follows: It simultaneously acquires calibration data from multiple 3D cameras, and the calibration board must cover the field of view of all cameras. This achieves multi-view complementarity, covering the physical blind spots of a single camera and solving the problems of insufficient field of view and point cloud loss due to occlusion. Furthermore, a 3D point cloud coordinate set is generated based on the calibration data, and a rigid transformation matrix from the non-reference camera to the reference camera is calculated based on this. This simplifies the traditional, cumbersome multi-step calibration into a single calculation, significantly improving calibration efficiency while ensuring spatial consistency of point cloud fusion. The target point cloud data includes RGB color values, which also addresses the deficiency of lacking color information in point clouds, further improving the integrity of the point cloud data.

[0087] Traditional point cloud stitching methods rely on multiple manual adjustments to camera pose or calibration object position, which is time-consuming and prone to introducing errors. Therefore, this invention proposes an optional embodiment:

[0088] Reference Figure 2 , Figure 2 This is a schematic diagram of a specific embodiment of step S102 of the method for rapid fusion of point cloud data from multiple 3D cameras in this invention. Step S102 also includes the following specific implementation methods.

[0089] Step S1021: In the three-dimensional point cloud coordinate point set, obtain the first point set in the reference camera coordinate system and the second point set in the non-reference camera coordinate system.

[0090] In an embodiment of the present invention, from the already generated set of 3D point cloud coordinates, a 3D point set in the reference camera coordinate system is selected as the first point set, and a 3D point set in a non-reference camera coordinate system is selected as the second point set. These point sets all originate from the interior corner points of the same calibration plate.

[0091] Step S1022: Calculate the first center of the first point set and the second center of the second point set.

[0092] In an embodiment of the present invention, the geometric center (first center) of the first point set and the geometric center (second center) of the second point set P are calculated respectively. The center is obtained by taking the arithmetic mean of all three-dimensional coordinate components in the point set.

[0093] Step S1023: Decentralization is performed on the first center and the second center respectively to obtain a decentralized point set.

[0094] In an embodiment of the present invention, each point in the first point set Q is subtracted from its first center coordinate to obtain the decentralized point set Qcentered; similarly, each point in the second point set P is subtracted from its second center coordinate to obtain the decentralized point set Pcentered. This step eliminates the translation differences between point sets.

[0095] Step S1024: Calculate the covariance matrix based on the decentralized point set.

[0096] In an embodiment of the present invention, the covariance matrix H between the decentralized point sets Pcentered and Qcentered is calculated. This matrix is ​​used to characterize the spatial distribution correlation between the two point sets.

[0097] Step S1025: Perform singular value decomposition on the covariance matrix to obtain the rotation matrix.

[0098] In an embodiment of the present invention, singular value decomposition is performed on the covariance matrix H to obtain three decomposition matrices. The rotation matrix R is then calculated using the decomposition results.

[0099] Step S1026: Calculate the translation vector based on the first center, the second center, and the rotation matrix.

[0100] In an embodiment of the present invention, the translation vector T is calculated using the rotation matrix R and the coordinates of the first center and the second center.

[0101] Step S1027: Construct the transformation matrix based on the rotation matrix and the translation vector.

[0102] In an embodiment of the present invention, the rotation matrix R and the translation vector T are combined into a homogeneous transformation matrix.

[0103] Optionally, before step S1021, the target camera is calibrated as the reference camera among the multiple 3D cameras to determine the reference camera coordinate system and the non-reference camera coordinate systems. Specifically, one target camera is selected as the reference camera among the multiple 3D cameras. In specific implementations, a camera with a fixed position is selected as the reference. The local coordinate system of the reference camera is determined as the reference coordinate system. This coordinate system will serve as the unified spatial reference for the point cloud registration of all non-reference cameras. The remaining cameras are defined as non-reference cameras, and their coordinate systems are non-reference coordinate systems. Subsequent steps will calculate the transformation relationships from these non-reference coordinate systems to the reference coordinate system.

[0104] In this embodiment of the invention, the rotation and translation relationships between point sets can be accurately calculated using decentralized and singular value decomposition mathematical methods, thus constructing the optimal rigid transformation matrix. Compared to traditional methods that rely on manual calibration or iterative optimization, this significantly improves calibration efficiency and accuracy.

[0105] Traditional point cloud stitching methods have limitations; most algorithms only fuse geometric coordinates, ignoring RGB color information, thus diminishing the value of point clouds in object recognition tasks. Based on this, this invention proposes an optional embodiment:

[0106] Reference Figure 3 , Figure 3 This is a schematic diagram of a specific embodiment of step S103 of the method for rapid fusion of point cloud data from multiple 3D cameras in this invention. Step S103 also includes the following specific implementation methods.

[0107] Step S1031: Convert the three-dimensional point cloud coordinate set into homogeneous coordinates according to the transformation matrix.

[0108] In an embodiment of the present invention, the coordinate set of a three-dimensional point cloud in a non-reference camera coordinate system is converted into a homogeneous coordinate form. Specifically, a dimension is added to each three-dimensional point (x, y, z) to form a homogeneous coordinate representation of (x, y, z, 1), thereby satisfying the dimensionality requirements of matrix operations.

[0109] Step S1032: Apply the transformation to the homogeneous coordinates according to the transformation matrix to obtain a point cloud set registered to the reference camera coordinate system.

[0110] In an embodiment of the present invention, a matrix multiplication operation is performed on the transformed homogeneous coordinates using a calculated transformation matrix. This step maps the point cloud in the non-reference coordinate system to the reference camera coordinate system, generating a registered point cloud set.

[0111] Step S1033: Merge all point cloud sets to obtain the merged target point cloud data.

[0112] In an embodiment of the present invention, the point cloud set of the reference camera itself is merged with all registered point cloud sets. The merging process preserves the original RGB color information, ultimately generating target point cloud data containing complete geometric structure and color attributes.

[0113] In this embodiment of the invention, efficient registration and merging of point clouds from multiple cameras are achieved through homogeneous coordinate transformation and batch operation of rigid transformation matrices, significantly improving the integrity of 3D reconstruction.

[0114] Traditional point cloud generation methods require processing depth and color maps separately, followed by manual alignment of feature points, which is cumbersome. Therefore, this invention proposes an alternative embodiment:

[0115] Reference Figure 4 , Figure 4 This is a schematic diagram of another specific embodiment of step S102 of the method for rapid fusion of point cloud data from multiple 3D cameras in this invention. Step S102 also includes the following specific implementation methods.

[0116] Step S1028: Determine the set of corner points in the calibration board based on the RGB diagram.

[0117] In an embodiment of the present invention, the RGB images acquired by each 3D camera are converted to grayscale, and the two-dimensional pixel coordinates of the corner points inside the calibration board are identified by an image processing algorithm. The corner points constitute the feature point set of the calibration board in the pixel coordinate system.

[0118] Step S1029: Based on the depth map and intrinsic parameter matrix, the corner point set is restored from the pixel coordinate system to the reference camera coordinate system to obtain the three-dimensional point cloud coordinate point set.

[0119] In an embodiment of the present invention, the corner point set is mapped from the pixel coordinate system to the reference camera coordinate system by combining the depth values ​​of corresponding corner points in the depth map and the camera's intrinsic parameter matrix. This process directly calculates the three-dimensional spatial coordinates of each corner point.

[0120] After completing the coordinate system transformation of all corner points, the 3D point cloud coordinate set of the calibration board in the reference camera coordinate system is output. This point set contains the geometric information of the complete surface structure of the calibration board.

[0121] In this embodiment of the invention, the three-dimensional reconstruction of the calibration board corner points is achieved through the collaborative processing of RGB images and depth maps, which can significantly improve calibration efficiency and data reliability. For example, the RGB image provides high-resolution feature recognition, effectively avoiding positioning errors caused by noise in the depth map.

[0122] Traditional point cloud generation methods do not filter invalid points in the depth map, which can easily lead to coordinate offsets in subsequent stitching. Based on this, the present invention proposes an optional embodiment:

[0123] Step S102 also includes the following specific implementation methods.

[0124] Step S102010: Determine the set of corner points in the calibration board based on the RGB diagram.

[0125] Step S102011: Based on the depth map, obtain the depth value corresponding to each corner point in the corner point set, and perform the conversion between the pixel coordinate system and the reference camera coordinate system according to the focal length parameter in the intrinsic parameter matrix, so as to convert the corner point set into an initial three-dimensional point cloud coordinate point set.

[0126] In this embodiment of the invention, the depth value corresponding to each inner corner point of the calibration board is extracted from the depth map, and the focal length parameter is read from the camera intrinsic parameter matrix. This step ensures the completeness of the original data required for subsequent coordinate transformation. Using the focal length parameter and the depth value, the pixel coordinates (u, v) of the corner point are converted into initial three-dimensional coordinates (Xc, Yc, Zc) in the reference camera coordinate system. The transformation process is directly implemented through scaling and translation calculations, without the need for iterative optimization, generating an initial three-dimensional point cloud coordinate set.

[0127] Step S102012: Based on the depth value, invalid values ​​are removed from the initial three-dimensional point cloud coordinate point set to obtain the three-dimensional point cloud coordinate point set.

[0128] In this embodiment of the invention, points with a depth value of 0 (indicating no valid depth information) are marked as invalid points (NaN), while the calculated three-dimensional coordinates of other points with depth values ​​greater than 0 are retained. This step filters out erroneous data caused by occlusion or measurement failure, ensuring the purity of the point set.

[0129] The final output contains only the coordinates of the three-dimensional point cloud of valid points, which will be directly used for subsequent transformation matrix calculations.

[0130] In this embodiment of the invention, the reliability and computational efficiency of point cloud data are significantly improved by verifying the validity of depth values ​​and transforming closed-loop coordinates. Invalid depth points can be automatically removed, effectively preventing erroneous data from contaminating subsequent transformation matrix calculations.

[0131] like Figure 5The diagram illustrates a terminal device according to an embodiment of the present invention. The terminal device 500 may include a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501, such as a multi-3D camera point cloud data fast fusion program. When the processor 501 executes the computer program 503, it implements the steps described in the various multi-3D camera point cloud data fast fusion embodiments.

[0132] A computer program can be divided into one or more modules / units. One or more modules / units are stored in memory 502 and executed by processor 501 to complete the present invention. One or more modules / units can be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a terminal device.

[0133] The terminal device may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of a terminal device and does not constitute a limitation on the terminal device. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, a terminal device may also include input / output devices, network access devices, buses, etc.

[0134] The processor 501 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0135] The memory 502 can be an internal storage unit of the terminal device, such as the hard drive or RAM of the terminal device. The memory 502 can also be an external storage device of the terminal device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 502 can include both internal and external storage units of the terminal device. The memory 502 is used to store computer programs and other programs and data required by the terminal device. The memory 502 can also be used to temporarily store data that has been output or will be output.

[0136] It should be noted that, for the sake of convenience and brevity, the structure of the terminal device described above can also be referred to the specific description of the structure in the method embodiment, which will not be repeated here.

[0137] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement the steps in the above-described method for rapid fusion of point cloud data from multiple 3D cameras.

[0138] This invention provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps in the above-described method for rapid fusion of point cloud data from multiple 3D cameras.

[0139] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for various specific applications, but such implementations should not be considered beyond the scope of this invention.

[0141] In the embodiments provided by this invention, it should be understood that the disclosed terminal devices and methods can be implemented in other ways. For example, the terminal device embodiments described above are merely illustrative. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0145] The embodiments described above are merely illustrative of the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for fast fusion of multi-3D camera point cloud data, characterized in that, The method comprises the following steps: Collecting calibration data of multiple 3D cameras, the calibration data comprising RGB images, depth images and intrinsic matrices, wherein the calibration board needs to appear in the field of view of the multiple 3D cameras at the same time; According to the calibration data, a three-dimensional point cloud coordinate point set is generated, and a transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system is calculated according to the three-dimensional point cloud coordinate point set; According to the transformation matrix, the three-dimensional point cloud coordinate point set is registered into a preset reference camera coordinate system to obtain merged target point cloud data, and the target point cloud data contains RGB color values; The step of registering the environment point cloud obtained by each camera into the reference camera coordinate system to obtain the merged target point cloud data according to the transformation matrix comprises: According to the transformation matrix, the three-dimensional point cloud coordinate point set is converted into homogeneous coordinates; According to the transformation matrix, the homogeneous coordinates are applied to obtain a point cloud set registered into the reference camera coordinate system; According to all point cloud sets, merging is performed to obtain the merged target point cloud data; the step of generating a three-dimensional point cloud coordinate point set according to the calibration data comprises: According to the RGB image, a corner point set in the calibration board is determined; According to the depth image and the intrinsic matrix, the corner point set is restored from the pixel coordinate system to the reference camera coordinate system to obtain the three-dimensional point cloud coordinate point set; the step of generating a three-dimensional point cloud coordinate point set according to the calibration data comprises: According to the RGB image, a corner point set in the calibration board is determined; According to the depth image, the depth value corresponding to each corner point in the corner point set is obtained, and the conversion between the pixel coordinate system and the reference camera coordinate system is performed according to the focal length parameter in the intrinsic matrix to convert the corner point set into an initial three-dimensional point cloud coordinate point set; According to the depth value, the invalid value of the initial three-dimensional point cloud coordinate point set is removed to obtain the three-dimensional point cloud coordinate point set.

2. The method of claim 1, wherein, The step of calculating the transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system according to the three-dimensional point cloud coordinate point set comprises: In the three-dimensional point cloud coordinate point set, a first point set under the reference camera coordinate system and a second point set under the non-reference camera coordinate system are obtained; The first center of the first point set and the second center of the second point set are calculated; According to the first center and the second center, respectively, a decentering process is performed to obtain a decentered point set; According to the decentered point set, a covariance matrix is calculated; The covariance matrix is singular value decomposed to obtain a rotation matrix; According to the first center, the second center and the rotation matrix, a translation vector is calculated; According to the rotation matrix and the translation vector, the transformation matrix is constructed.

3. The method of claim 2, wherein, Before the step of obtaining the first point set under the reference camera coordinate system and the second point set under the non-reference camera coordinate system, the method further comprises: In the multiple 3D cameras, a target camera is calibrated as a reference camera to determine the reference camera coordinate system and the non-reference camera coordinate system.

4. The method of claim 1, wherein, The multiple 3D cameras are more than two 3D cameras, and the multiple 3D cameras are deployed on the same horizontal plane.

5. The method of claim 1, wherein, After the step of registering the three-dimensional point cloud coordinate point set into the reference camera coordinate system according to the transformation matrix to obtain the merged target point cloud data, the method further comprises: According to the target point cloud data, a material grabbing operation is performed.

6. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the multi-3D camera point cloud data fast fusion method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 6. The computer program is executed by the processor to realize the steps of the multi-3D camera point cloud data fast fusion method according to any one of claims 1 to 5.

Images