Fast fusion method for point cloud data of multiple 3D cameras, terminal equipment and storage medium

Abstract

The invention is suitable for the field of image processing, and discloses a fast fusion method for point cloud data of multiple 3D cameras, terminal equipment and a storage medium. The multi-3D-camera point cloud data fast fusion method comprises the steps that calibration data of a plurality of 3D cameras are collected, the calibration data comprise an RGB image, a depth image and an internal reference matrix, and a calibration plate needs to appear in the view fields of the 3D cameras at the same time; generating a three-dimensional point cloud coordinate point set according to the calibration data, and calculating a transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system according to the three-dimensional point cloud coordinate point set; and according to the transformation matrix, registering the three-dimensional point cloud coordinate point set to a preset reference camera coordinate system to obtain merged target point cloud data, the target point cloud data including RGB color values. According to the invention, multi-view complementation is realized, a physical blind area of a single camera can be covered, and the problem of point cloud missing caused by insufficient view field and shielding is solved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of image processing, and particularly relates to a multi-3D camera point cloud data fast fusion method, a terminal device and a storage medium. BACKGROUND

[0002] With the development of computer vision and three-dimensional reconstruction technology, point cloud data, as an important expression form of three-dimensional space information, has been widely applied in the fields of automatic driving, robot navigation, industrial detection and virtual reality.

[0003] The method of acquiring point cloud data by a 3D camera includes a structured light method, a time-of-flight method, binocular stereo vision and a laser radar, etc. However, due to the field of view limitation and the occlusion problem, the traditional single camera often cannot completely cover the target scene, resulting in incomplete point cloud data. A new technical means is needed to solve the above technical problems. SUMMARY

[0004] In view of this, the embodiments of the present application provide a multi-3D camera point cloud data fast fusion method, a terminal device and a storage medium, which can solve the problem of incomplete point cloud data in the related art.

[0005] The first aspect of the present application provides a multi-3D camera point cloud data fast fusion method, comprising: Collecting calibration data of multiple 3D cameras, the calibration data including an RGB image, a depth image and an intrinsic matrix, wherein a calibration board needs to appear in the field of view of the multiple 3D cameras at the same time; According to the calibration data, generating a three-dimensional point cloud coordinate point set, and according to the three-dimensional point cloud coordinate point set, calculating a transformation matrix between a reference camera coordinate system and a non-reference camera coordinate system; According to the transformation matrix, registering the three-dimensional point cloud coordinate point set into a preset reference camera coordinate system to obtain merged target point cloud data, the target point cloud data containing RGB color values.

[0006] Further, the step of calculating the transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system according to the three-dimensional point cloud coordinate point set comprises: In the three-dimensional point cloud coordinate point set, obtaining a first point set under the reference camera coordinate system and a second point set under the non-reference camera coordinate system; Calculating a first center of the first point set and a second center of the second point set; According to the first center and the second center, respectively performing decentralization processing to obtain a decentralized point set; According to the decentralized point set, calculating a covariance matrix; Performing singular value decomposition on the covariance matrix to obtain a rotation matrix; calculating a translation vector according to the first center, the second center and the rotation matrix; constructing the transformation matrix according to the rotation matrix and the translation vector.

[0007] Further, before the step of obtaining the first point set in the reference camera coordinate system and the second point set in the non-reference camera coordinate system, the method further comprises: designating a target camera as a reference camera in the plurality of 3D cameras to determine the reference camera coordinate system and the non-reference camera coordinate system.

[0008] Further, the step of registering the environment point cloud obtained by each camera into the reference camera coordinate system according to the transformation matrix to obtain the merged target point cloud data comprises: converting the three-dimensional point cloud coordinate point set into homogeneous coordinates according to the transformation matrix; applying a transformation to the homogeneous coordinates according to the transformation matrix to obtain a point cloud set registered into the reference camera coordinate system; merging according to all point cloud sets to obtain the merged target point cloud data.

[0009] Further, the step of generating a three-dimensional point cloud coordinate point set according to the calibration data comprises: determining a corner point set in the calibration board according to the RGB image; restoring the corner point set from the pixel coordinate system to the reference camera coordinate system according to the depth image and the intrinsic matrix to obtain the three-dimensional point cloud coordinate point set.

[0010] Further, the step of restoring the corner point set from the pixel coordinate system to the reference camera coordinate system according to the depth image and the intrinsic matrix to obtain the three-dimensional point cloud coordinate point set comprises: determining a corner point set in the calibration board according to the RGB image; obtaining a depth value corresponding to each corner point in the corner point set according to the depth image, and performing conversion between the pixel coordinate system and the reference camera coordinate system according to the focal length parameter in the intrinsic matrix to convert the corner point set into an initial three-dimensional point cloud coordinate point set; performing invalid value elimination on the initial three-dimensional point cloud coordinate point set according to the depth value to obtain the three-dimensional point cloud coordinate point set.

[0011] Further, the plurality of 3D cameras is two or more 3D cameras, and the plurality of 3D cameras are deployed on the same horizontal plane.

[0012] Further, after the step of registering the three-dimensional point cloud coordinate point set into the reference camera coordinate system according to the transformation matrix to obtain the merged target point cloud data, the method further comprises: According to the target point cloud data, a material grabbing operation is performed.

[0013] In a second aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the multi-3D camera point cloud data fusion method when executing the computer program.

[0014] In a third aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of the multi-3D camera point cloud data fusion method.

[0015] In a fourth aspect, an embodiment of the present application provides a computer program product, which, when executed on a terminal device, causes the terminal device to perform the multi-3D camera point cloud data fusion method.

[0016] Compared with the prior art, the embodiment of the present application has the beneficial effects that: the calibration data of multiple 3D cameras is collected at the same time, and the calibration board needs to cover the field of view of all cameras. Multi-view complementation is achieved, the physical blind area of a single camera can be covered, and the problem of point cloud missing caused by insufficient field of view and occlusion is solved. Further, three-dimensional point cloud coordinate point sets are generated according to the calibration data, and a rigid transformation matrix from a non-reference camera to a reference camera is calculated based on the three-dimensional point cloud coordinate point sets. The traditional multi-step calibration can be simplified to single calculation, the calibration efficiency is significantly improved, and the spatial consistency of point cloud fusion is ensured. The target point cloud data contains RGB color values, and the defect of lack of color information of point cloud is also solved, and the integrity of point cloud data is further improved. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0018] Figure 1 An embodiment of the multi-3D camera point cloud data fusion method in the embodiment of the present application is shown in the figure; Figure 2 An embodiment of the step S102 of the multi-3D camera point cloud data fusion method in the embodiment of the present application is shown in the figure; Figure 3A specific embodiment diagram of step S103 of the multi-3D camera point cloud data fast fusion method in the embodiment of the present application is shown in the figure; Figure 4 Another specific embodiment diagram of step S102 of the multi-3D camera point cloud data fast fusion method in the embodiment of the present application is shown in the figure; Figure 5 An embodiment diagram of the terminal device in the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0019] In order to make the objects, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection of the present application.

[0020] It should be noted that the terms "include", "contain" and "have" in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusion. For example, a process, method, terminal, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device. In the claims, specification and drawings of the present application, the relationship terms such as "first" and "second" are only used to distinguish one entity / operation / object from another entity / operation / object, and do not necessarily require or imply any such real-time relationship or sequence between the entities / operations / objects.

[0021] In this document, the term "embodiment" means that the specific features, structures or characteristics described in connection with the embodiment can be included in at least one embodiment of the present application. The phrase appears in various places in the specification does not necessarily refer to the same embodiment, nor is it mutually exclusive or alternative to other embodiments. Those skilled in the art explicitly and implicitly understand that the embodiments described herein can be combined with other embodiments.

[0022] With the development of computer vision and three-dimensional reconstruction technology, point cloud data as an important expression form of three-dimensional space information has been widely used in automatic driving, robot navigation, industrial detection and virtual reality, etc.

[0023] The method for acquiring point cloud data by a 3D camera includes a structured light method, a time-of-flight method, binocular stereo vision, and a laser radar, etc. However, due to the field-of-view limitation and the occlusion problem, a traditional single camera often cannot completely cover a target scene, resulting in incomplete point cloud data. A new technical means is needed to solve the above technical problems.

[0024] In view of this, the embodiment of the present application provides a multi-3D camera point cloud data rapid fusion method, a terminal device and a storage medium, simultaneously collecting calibration data of multiple 3D cameras, and the calibration board needs to cover the field of view of all cameras. The multi-angle complementation is realized, the physical blind area of a single camera can be covered, and the problem of point cloud missing caused by insufficient field of view and occlusion is solved. Further, a three-dimensional point cloud coordinate point set is generated according to the calibration data, and a rigid transformation matrix of a non-reference camera to a reference camera is calculated based on the three-dimensional point cloud coordinate point set. The traditional cumbersome multi-step calibration can be simplified to single calculation, the calibration efficiency is significantly improved, and the spatial consistency of point cloud fusion is ensured. The target point cloud data contains RGB color values, and the defect of point cloud lacking color information can also be solved, further improving the integrity of point cloud data.

[0025] In order to illustrate the technical scheme of the present application, specific embodiments will be described below.

[0026] Figure 1 A multi-3D camera point cloud data rapid fusion method implementation flowchart provided by the embodiment of the present application is shown, and the method can be applied to a terminal device. The terminal device can be a mobile phone, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a netbook, etc.

[0027] Specifically, the above-mentioned multi-3D camera point cloud data rapid fusion method can include the following steps S101 to S103.

[0028] Step S101, collecting calibration data of multiple 3D cameras, the calibration data including RGB images, depth images and intrinsic matrices, wherein the calibration board needs to appear in the field of view of the multiple 3D cameras at the same time.

[0029] In the embodiment of the present application, when the trigger condition is reached, the multiple 3D cameras (for example, four) are triggered to synchronously collect calibration data. The calibration data includes RGB images, depth images, and intrinsic matrices.

[0030] Specifically, the RGB image is used to identify the inner corner point position of the calibration board; the depth image records the depth value of each pixel; and the intrinsic matrix contains the focal length and other camera intrinsic parameters.

[0031] It should be noted that the calibration board appears in the depth field of view and the RGB field of view of all cameras at the same time, so that all cameras can capture complete data of the same calibration board.

[0032] In one example, the point cloud field of view that each camera can reach and the working area that can be covered under the field of view. Four cameras are triggered to collect four sets of RGB images, depth images and camera intrinsic parameters under the corresponding resolution, wherein the RGB images and the depth images are aligned, and the four sets of calibration data contain the same block of calibration board.

[0033] According to the four sets of depth images and the intrinsic parameter matrix collected, the inner corner point set of the calibration board is converted from the pixel coordinate system to the camera coordinate system to obtain the three-dimensional point cloud coordinate point set of the inner corner point, and the formula is as follows: is the pixel coordinate, is the column index, is the row index. is the depth value, which represents the distance of the pixel in the camera coordinate system. is the intrinsic parameter matrix, , is the focal length, and f_x and f_y are the horizontal and vertical scaling factors, respectively; , is the coordinate of the camera optical axis on the image plane. and are the horizontal and vertical coordinates in the camera coordinate system, which are scaled and translated based on the pixel position and the depth value of the depth image. is the depth value.

[0034] When the depth value of a certain pixel in the depth image is , it means that the point has no depth information, and the corresponding world coordinate is set to be undefined (represented by NaN), and the formula is as follows:

[0035] Optionally, the plurality of 3D cameras is two or more 3D cameras, and the plurality of 3D cameras are deployed on the same horizontal plane. Specifically, two or more 3D cameras are fixed on the same bracket to keep all cameras on the same horizontal plane. This deployment method can make the field of view of each camera uniformly cover the horizontal direction, effectively avoiding the missing of the overlapping angle caused by the height difference. When calibrating, the calibration board is placed in the overlapping field of view of the four cameras, so that the calibration board appears in the depth and RGB field of view of all cameras at the same time. The horizontal layout can maximize the use of the field of view coverage range under the limited height.

[0036] In step S102, according to the calibration data, a three-dimensional point cloud coordinate point set is generated, and a transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system is calculated according to the three-dimensional point cloud coordinate point set.

[0037] ​​In the embodiment of the present application, the RGB image of each camera is processed into grayscale, and the pixel coordinates of the inner corner points of the calibration board are identified; the inner corner points are converted from the pixel coordinate system to the three-dimensional coordinates in the camera coordinate system by combining the depth value of the depth map and the intrinsic matrix. If the depth value is 0, it is judged as an invalid point, and marked as not a number (NaN). Finally, the three-dimensional point cloud coordinate point set in the coordinate system of each camera is obtained.

[0038] In a specific example, the number of 3D cameras is set to 4. After obtaining the three-dimensional point cloud coordinate point set of the inner corner points of the same calibration board in the coordinate system of the four cameras, the transformation matrix between the remaining cameras and the reference camera is calculated. Here, the camera at the top left corner is set as the reference camera, and the coordinate system of the camera at the top left corner is set as the reference coordinate system, that is, the transformation matrix from the coordinate system of the camera at the bottom left corner to the coordinate system of the camera at the top left corner, the transformation matrix from the coordinate system of the camera at the top right corner to the coordinate system of the camera at the top left corner, and the transformation matrix from the coordinate system of the camera at the bottom right corner to the coordinate system of the camera at the top left corner are calculated. The transformation matrix from the coordinate system of the camera at the top left corner to the coordinate system of the camera at the bottom left corner The transformation matrix from the coordinate system of the camera at the top left corner to the coordinate system of the camera at the bottom right corner For example, for each non-reference camera, the point set P (three-dimensional coordinates of the corner points of the calibration board) in its coordinate system and the corresponding point set Q (three-dimensional coordinates of the same corner points of the calibration board in the reference coordinate system) in the reference camera coordinate system are obtained; the rotation matrix R and the translation vector t between each non-reference camera and the reference camera are calculated; and the transformation matrix in the corresponding direction is constructed from the rotation matrix R and the translation vector t.

[0039] Specifically, let the three-dimensional point set in the reference camera coordinate system be and the three-dimensional point set in the coordinate system of the remaining cameras be The center calculation and decentralization of the point sets and are performed, and the formulas are as follows: ; ; ; .

[0040] wherein is the geometric center (centroid) of the non-reference camera point set P; is the geometric center (centroid) of the reference camera point set Q; is the point set after decentralization of the non-reference camera point set P; is the point set after decentralization of the reference camera point set Q.

[0041] The covariance matrix of the point set after decentralization is calculated and singular value decomposition is performed thereon to solve the rotation matrix and the translation vector, and finally the rotation matrix R and the translation vector t are constructed into the transformation matrix , the formula is as follows: ; ; ; .

[0042] wherein, is a homogeneous transformation matrix constructed by a rotation matrix R and a translation vector t, and is a general form of all non-reference camera to reference camera transformation matrices. For 4 3D cameras, wherein 1 is a reference camera, and the rest 3 are non-reference cameras, the transformation matrix H (such as the transformation matrix of the lower left corner, upper right corner, and lower right corner cameras to the reference camera) corresponding to each non-reference camera is generated according to the structure and construction method of , and R is obtained by singular value decomposition, and t is obtained by combining the center of the point set. is used because it can integrate rotation and translation transformation at the same time, realize rigid registration of non-reference camera point cloud to the reference coordinate system through single matrix operation, ensure spatial consistency during multi-view point cloud fusion, and solve the problem of insufficient single camera field of view and point cloud loss. U and V are orthogonal matrices, and S is a singular value matrix.

[0043] In step S103, the three-dimensional point cloud coordinate point set is registered into the preset reference camera coordinate system according to the transformation matrix, and the merged target point cloud data is obtained, wherein the target point cloud data contains RGB color values.

[0044] In the embodiments of the present application, the point cloud under the non-reference camera coordinate system is converted into a homogeneous coordinate; the transformation matrix is applied for coordinate transformation, so as to register the point cloud to the reference coordinate system. All the registered point clouds (including the reference camera point cloud) are merged into a single target point cloud; the color information of the original RGB image is simultaneously retained during the merging, so that the target point cloud contains both geometric coordinates and color attributes.

[0045] In one specific example, immediately after the above example, the number of 3D cameras is 4, and for each non-reference camera, the second point set P under its coordinate system and the first point set Q under the reference camera coordinate system are obtained from the three-dimensional point cloud coordinate point set; the geometric centers of P and Q are calculated respectively, and the covariance matrix is obtained after the two point sets are processed by decentralization, and the rotation matrix R is obtained by singular value decomposition of the matrix, and then the translation vector t is calculated by combining the centers of the two point sets and R, and finally the transformation matrix of each non-reference camera to the reference camera coordinate system is constructed by R and t, that is, the transformation matrix of the lower left corner camera coordinate system to the upper left corner camera coordinate system , the transformation matrix of the upper right corner camera coordinate system to the upper left corner camera coordinate system a transformation matrix from the lower right camera coordinate system to the upper left camera coordinate system .

[0046] obtain a transformation matrix from the lower left camera coordinate system to the upper left camera coordinate system a transformation matrix from the upper right camera coordinate system to the upper left camera coordinate system a transformation matrix from the lower right camera coordinate system to the upper left camera coordinate system Afterwards, each camera coordinate system is calculated with the corresponding transformation matrix to obtain three groups of point clouds under the reference camera coordinate system, specifically, for the lower left corner, the upper right corner, and the lower right corner of the non-reference camera, the corresponding transformation matrix is called to perform coordinate conversion. For example, the three-dimensional points in each non-reference camera point cloud are converted into homogeneous coordinates, and the coordinates under the reference coordinate system are calculated by matrix multiplication, thereby generating three groups of point clouds registered to the reference coordinate system.

[0047] The three groups of point clouds are merged with the point clouds under the original reference camera coordinate system to obtain the spliced point cloud, the specific process is as follows: Each point in the environment point cloud obtained by the lower left corner camera is converted into homogeneous coordinates and transformed by applying the transformation matrix to obtain the point cloud set transformed to the upper left coordinate system , the formula is as follows: ; The upper right corner is transformed to the left upper corner by the same method , the lower right corner is transformed to the left upper corner , four groups of , , , are merged to obtain the merged target point cloud data. Among them, for the merging operation, the four groups of point clouds are preprocessed to remove NaN points with invalid depth values and isolated noise points; subsequently, the point set stacking method is used to merge, all valid points are integrated into a single set, while the RGB color attribute of each point is retained; for the repeated points in the overlapping area, the points of the reference camera point cloud are preferentially retained, and the points of the non-reference camera are selected with higher color consistency, and finally the target point cloud data containing complete scene information is formed.

[0048] Optionally, according to the target point cloud data, a material grabbing operation is performed. Access, after completing point cloud registration and merging, a complete target point cloud containing RGB color values is generated, covering the entire working area. The target point cloud data is input to a downstream execution system (such as a robot control module), the material position and posture are identified based on the geometry and color attributes of the point cloud, and the grabbing action is triggered.

[0049] The beneficial effects of the embodiments of the present application compared with the prior art are: the calibration data of multiple 3D cameras is collected at the same time, and the calibration board needs to cover the field of view of all cameras. Multi-view complementation is achieved, the physical blind area of a single camera can be covered, and the problem of missing point cloud caused by insufficient field of view and occlusion is solved. Further, a three-dimensional point cloud coordinate point set is generated according to the calibration data, and a rigid transformation matrix of the non-reference camera to the reference camera is calculated based on this. The traditional cumbersome multi-step calibration can be simplified to a single calculation, which significantly improves the calibration efficiency while ensuring the spatial consistency of point cloud fusion. The target point cloud data contains RGB color values, which can also solve the defect of lack of color information of point cloud, and further improve the integrity of point cloud data.

[0050] The traditional point cloud splicing method relies on multiple manual interventions to adjust the camera pose or the position of the calibration object, which is time-consuming and prone to errors. Based on this, an optional embodiment of the present application is proposed: Reference Figure 2 , Figure 2 This is a specific embodiment diagram of step S102 of the multi-3D camera point cloud data fast fusion method in the embodiments of the present application. Step S102 further includes the following specific embodiments.

[0051] Step S1021, in the three-dimensional point cloud coordinate point set, a first point set in the reference camera coordinate system and a second point set in the non-reference camera coordinate system are obtained.

[0052] In the embodiments of the present application, in the generated three-dimensional point cloud coordinate point set, the three-dimensional point set in the reference camera coordinate system is selected as the first point set, and the three-dimensional point set in the non-reference camera coordinate system is selected as the second point set. These point sets are all from the internal corner points of the same calibration board.

[0053] Step S1022, the first center of the first point set and the second center of the second point set are calculated.

[0054] In the embodiments of the present application, the geometric center (first center) of the first point set and the geometric center (second center) of the second point set P are calculated respectively. The center is obtained by calculating the arithmetic mean of all three-dimensional coordinate components in the point set.

[0055] Step S1023, according to the first center and the second center, respectively, the decentralization processing is performed to obtain a decentralized point set.

[0056] In an embodiment of the present application, each point of the first point set Q is subtracted by its first center coordinate to obtain a decentered point set Qcentered; similarly, each point of the second point set P is subtracted by its second center coordinate to obtain a decentered point set Pcentered. This step eliminates the translation difference between the point sets.

[0057] In step S1024, a covariance matrix is calculated according to the decentered point sets.

[0058] In an embodiment of the present application, a covariance matrix H between the decentered point sets Pcentered and Qcentered is calculated. The matrix is used to represent the spatial distribution correlation between the two point sets.

[0059] In step S1025, a singular value decomposition is performed on the covariance matrix to obtain a rotation matrix.

[0060] In an embodiment of the present application, a singular value decomposition is performed on the covariance matrix H to obtain three decomposition matrices. The rotation matrix R is calculated by the decomposition result.

[0061] In step S1026, a translation vector is calculated according to the first center, the second center and the rotation matrix.

[0062] In an embodiment of the present application, the translation vector T is calculated by the rotation matrix R, the first center and the second center coordinates.

[0063] In step S1027, the transformation matrix is constructed according to the rotation matrix and the translation vector.

[0064] In an embodiment of the present application, the rotation matrix R and the translation vector T are combined into a homogeneous transformation matrix.

[0065] Optionally, before step S1021, a target camera is designated as a reference camera in the plurality of 3D cameras to determine the reference camera coordinate system and the non-reference camera coordinate system. Specifically, in the plurality of 3D cameras, a target camera is selected as a reference camera. In practice, a camera with a fixed position is selected as the reference. The local coordinate system of the reference camera is determined as the reference coordinate system. The coordinate system will serve as a unified spatial reference for the registration of point clouds of all non-reference cameras. The remaining cameras are defined as non-reference cameras, and their coordinate systems are non-reference coordinate systems. Subsequent steps will calculate the transformation relationship between these non-reference coordinate systems and the reference coordinate system.

[0066] In an embodiment of the present application, the rotation and translation relationship between point sets can be accurately calculated by the mathematical methods of decentering and singular value decomposition, and the optimal rigid transformation matrix can be constructed. Compared with the traditional methods relying on manual calibration or iterative optimization, the calibration efficiency and accuracy are significantly improved.

[0067] The traditional point cloud splicing method has a bottleneck, and most algorithms only fuse geometric coordinates, ignoring the RGB color information, which weakens the value of point cloud in object recognition tasks. Based on this, an optional embodiment is proposed in the present application: Referring to Figure 3 , Figure 3 is a specific embodiment diagram of step S103 of the multi-3D camera point cloud data fast fusion method in the embodiment of the present application. Step S103 further includes the following specific embodiments.

[0068] Step S1031, converting the three-dimensional point cloud coordinate point set into homogeneous coordinates according to the transformation matrix.

[0069] In the embodiment of the present application, the three-dimensional point cloud coordinate point set in the non-reference camera coordinate system is converted into homogeneous coordinate form. The specific operation is to add a dimension to each three-dimensional point (x, y, z) to form a homogeneous coordinate representation of (x, y, z, 1) to meet the dimension requirement of matrix operation.

[0070] Step S1032, performing an application transformation on the homogeneous coordinates according to the transformation matrix to obtain a point cloud set registered to the reference camera coordinate system.

[0071] In the embodiment of the present application, the transformed homogeneous coordinates are subjected to matrix multiplication operation using the calculated transformation matrix. This step maps the point cloud in the non-reference camera coordinate system into the reference camera coordinate system, generating a registered point cloud set.

[0072] Step S1033, merging all point cloud sets to obtain the target point cloud data after merging.

[0073] In the embodiment of the present application, the point cloud set of the reference camera itself is merged with all the registered point cloud sets. The merging process retains the original RGB color information, and finally generates the target point cloud data containing complete geometric structure and color attributes.

[0074] In the embodiment of the present application, through the batch operation of homogeneous coordinate conversion and rigid transformation matrix, efficient registration and merging of multi-camera point cloud are realized, which significantly improves the integrity of three-dimensional reconstruction.

[0075] The traditional point cloud generation method needs to process depth map and color map respectively, and then manually aligns the feature points, which is tedious. Based on this, an optional embodiment is proposed in the present application: Referring to Figure 4 , Figure 4 is another specific embodiment diagram of step S102 of the multi-3D camera point cloud data fast fusion method in the embodiment of the present application. Step S102 further includes the following specific embodiments.

[0076] Step S1028, according to the RGB image, determine the corner point set in the calibration board.

[0077] In an embodiment of the application, the RGB image collected by each 3D camera is grayed, and the two-dimensional pixel coordinate position of the internal corner point of the calibration board is identified by an image processing algorithm. The corner points constitute a feature point set of the calibration board in the pixel coordinate system.

[0078] Step S1029, according to the depth map and the intrinsic matrix, restore the corner point set from the pixel coordinate system to the reference camera coordinate system, and obtain the three-dimensional point cloud coordinate point set.

[0079] In an embodiment of the application, the depth value of the corresponding corner point in the depth map and the intrinsic matrix of the camera are combined to map the corner point set from the pixel coordinate system to the reference camera coordinate system. This process directly calculates the three-dimensional space coordinates of each corner point.

[0080] After completing the coordinate system conversion of all corner points, the three-dimensional point cloud coordinate point set of the calibration board in the reference camera coordinate system is output. The point set contains the geometric information of the complete surface structure of the calibration board.

[0081] In an embodiment of the application, through the cooperative processing of the RGB image and the depth map, the three-dimensional restoration of the corner points of the calibration board is realized, which can significantly improve the calibration efficiency and data reliability. For example, the RGB image provides high-resolution feature recognition, which effectively avoids the positioning deviation caused by depth map noise.

[0082] The traditional point cloud generation method does not filter the invalid points in the depth map, which can easily cause coordinate offset in subsequent splicing. Based on this, an optional embodiment of the application is proposed: Step S102 also includes the following specific embodiments.

[0083] Step S102010, according to the RGB image, determine the corner point set in the calibration board.

[0084] Step S102011, according to the depth map, obtain the depth value corresponding to each corner point in the corner point set, and according to the focal length parameter in the intrinsic matrix, perform the conversion between the pixel coordinate system and the reference camera coordinate system to convert the corner point set into an initial three-dimensional point cloud coordinate point set.

[0085] In this embodiment of the present invention, the depth value corresponding to each interior corner point of the calibration plate is extracted from the depth map, and the focal length parameter is read from the camera intrinsic parameter matrix. This step ensures the completeness of the raw data required for subsequent coordinate transformation. Using the focal length parameter and depth value, the pixel coordinates (u, v) of the corner point are converted to the initial 3D coordinates (Xc, Yc, Zc) in the reference camera coordinate system. The conversion process is directly implemented through scaling and translation calculations, without the need for iterative optimization, to generate the initial 3D point cloud coordinate point set.

[0086] Step S102012: Eliminate invalid values ​​from the initial three-dimensional point cloud coordinate point set according to the depth value to obtain the three-dimensional point cloud coordinate point set.

[0087] In this embodiment of the present invention, points with a depth value of 0 (indicating no valid depth information) are marked as invalid points (NaN). The calculated 3D coordinates of all other points with depth values ​​greater than 0 are retained. This step filters out erroneous data caused by occlusion or measurement failure, ensuring the purity of the point set.

[0088] The final output is a three-dimensional point cloud coordinate point set containing only valid points, which will be directly used for subsequent transformation matrix calculations.

[0089] In the embodiment of the present invention, the reliability and computational efficiency of point cloud data are significantly improved through depth value validity verification and closed solution coordinate conversion. Invalid depth points can be automatically eliminated, effectively avoiding erroneous data from contaminating subsequent transformation matrix calculations.

[0090] like Figure 5 FIG2 is a schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device 500 may include a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable by the processor 501, such as a multi-3D camera point cloud data fast fusion program. When the processor 501 executes the computer program 503, the steps described in the aforementioned multi-3D camera point cloud data fast fusion embodiments are implemented.

[0091] The computer program can be divided into one or more modules / units, which are stored in the memory 502 and executed by the processor 501 to implement the present invention. One or more modules / units can be a series of computer program instruction segments that can perform specific functions. These instruction segments are used to describe the execution process of the computer program in the terminal device.

[0092] The terminal device may include, but is not limited to, a processor 501 and a memory 502. Those skilled in the art will appreciate that Figure 5The terminal device is only an example and does not constitute a limitation on the terminal device, and can include more or fewer components, or combine certain components, or different components, for example, the terminal device can also include an input / output device, a network access device, a bus, and the like.

[0093] The processor 501 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0094] The memory 502 can be an internal storage unit of the terminal device, for example, a hard disk or a memory of the terminal device. The memory 502 can also be an external storage device of the terminal device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the memory 502 can include both the internal storage unit and the external storage device of the terminal device. The memory 502 is used to store computer programs and other programs and data required by the terminal device. The memory 502 can also be used to temporarily store data that has been output or will be output.

[0095] It should be noted that, for the convenience and brevity of description, the structure of the terminal device described above can also refer to the specific description of the structure in the method embodiments, which will not be described here.

[0096] The embodiment of the present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, and the computer program is executed by a processor to realize the steps in the above multi-3D camera point cloud data fusion method.

[0097] The embodiment of the present application provides a computer program product, when the computer program product is run on a mobile terminal, so that the mobile terminal executes the steps in the above multi-3D camera point cloud data fusion method.

[0098] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0099] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0100] In the embodiments provided by the present application, it should be understood that the disclosed terminal device and method can be implemented in other ways. For example, the terminal device embodiments described above are only schematic. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0101] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0102] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0103] The integrated module / unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer readable storage medium. The computer program can implement the steps of each method embodiment when executed by a processor. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms. The computer readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the contents included in the computer readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.

[0104] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method for rapid fusion of multi-3D camera point cloud data, characterized in that: include: Collect calibration data from multiple 3D cameras, including RGB images, depth maps, and intrinsic parameter matrices. The calibration plate must appear within the field of view of the multiple 3D cameras simultaneously. Generate a three-dimensional point cloud coordinate point set according to the calibration data, and calculate a transformation matrix between a reference camera coordinate system and a non-reference camera coordinate system according to the three-dimensional point cloud coordinate point set; According to the transformation matrix, the three-dimensional point cloud coordinate point set is aligned to a preset reference camera coordinate system to obtain merged target point cloud data, wherein the target point cloud data includes RGB color values.

2. The method for rapid fusion of multi-3D camera point cloud data according to claim 1, wherein: The step of calculating the transformation matrix between the reference camera coordinate system and the non-reference camera coordinate system according to the three-dimensional point cloud coordinate point set includes: Acquire, from the three-dimensional point cloud coordinate point set, a first point set in the reference camera coordinate system and a second point set in the non-reference camera coordinate system; Calculating a first center of the first point set and a second center of the second point set; Decentralizing the first center and the second center to obtain a decentralized point set; Calculating a covariance matrix based on the decentralized point set; Performing singular value decomposition on the covariance matrix to obtain a rotation matrix; Calculate a translation vector according to the first center, the second center, and the rotation matrix; The transformation matrix is ​​constructed according to the rotation matrix and the translation vector.

3. The method for rapid fusion of multi-3D camera point cloud data according to claim 2, wherein: Before the step of obtaining the first point set in the reference camera coordinate system and the second point set in the non-reference camera coordinate system, the method further includes: A target camera is calibrated as a reference camera among the multiple 3D cameras to determine the reference camera coordinate system and the non-reference camera coordinate system.

4. The method for rapid fusion of multi-3D camera point cloud data according to claim 1, wherein: The step of registering the environment point cloud acquired by each camera to the reference camera coordinate system according to the transformation matrix to obtain the merged target point cloud data includes: Converting the three-dimensional point cloud coordinate point set into homogeneous coordinates according to the transformation matrix; Performing an applied transformation on the homogeneous coordinates according to the transformation matrix to obtain a point cloud set registered to the reference camera coordinate system; All point cloud sets are merged to obtain the merged target point cloud data.

5. The method for rapid fusion of multi-3D camera point cloud data according to claim 1, wherein: The step of generating a three-dimensional point cloud coordinate point set according to the calibration data includes: Determine a set of corner points in the calibration plate according to the RGB image; According to the depth map and the intrinsic parameter matrix, the corner point set is restored from the pixel coordinate system to the reference camera coordinate system to obtain the three-dimensional point cloud coordinate point set.

6. The method for rapid fusion of multi-3D camera point cloud data according to claim 1, wherein: The step of generating a three-dimensional point cloud coordinate point set according to the calibration data includes: Determine a set of corner points in the calibration plate according to the RGB image; According to the depth map, obtain the depth value corresponding to each corner point in the corner point set, and perform a conversion between a pixel coordinate system and a reference camera coordinate system according to the focal length parameter in the intrinsic parameter matrix to convert the corner point set into an initial three-dimensional point cloud coordinate point set; Invalid values ​​are eliminated from the initial three-dimensional point cloud coordinate point set according to the depth value to obtain the three-dimensional point cloud coordinate point set.

7. The method for rapid fusion of multi-3D camera point cloud data according to claim 6, characterized in that: The multiple 3D cameras are more than two 3D cameras, and the multiple 3D cameras are deployed on the same horizontal plane.

8. The method for rapid fusion of multi-3D camera point cloud data according to claim 6, wherein: After the step of registering the three-dimensional point cloud coordinate point set to the reference camera coordinate system according to the transformation matrix to obtain merged target point cloud data, the method further includes: Execute a material grabbing operation according to the target point cloud data.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for rapid fusion of multi-3D camera point cloud data are implemented as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for rapid fusion of multi-3D camera point cloud data as claimed in any one of claims 1 to 8 are implemented.

Images