Multi-view point cloud registration method, device and equipment and computer readable storage medium

Abstract

The application provides a multi-view point cloud registration method, device and equipment and a computer readable storage medium. The method comprises the following steps: collecting a multi-view depth video of an object to be detected, cutting the multi-view depth video into depth image frames, and converting the depth image frames into point cloud data; extracting the point cloud data to obtain multi-view point clouds of the object to be detected, wherein the multi-view point clouds comprise overhead point clouds and side-view point clouds; performing coarse registration on the overhead point clouds and the completed side-view point clouds, and performing slicing on the point clouds after coarse registration; and performing fine registration on the overhead point clouds and the completed side-view point clouds according to the centroids of the overlapping areas of each slice. The application improves the accuracy of point cloud registration, provides technical support for subsequent three-dimensional point cloud reconstruction, is conducive to realizing non-contact measurement of livestock body size and weight and body condition evaluation, and reduces the occurrence of livestock stress reaction.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a multi-view point cloud registration method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Livestock farming plays a vital role in my country's animal husbandry industry. Animal body shape, weight parameters, and body condition scores are crucial for guiding animal health. However, traditional measurement methods rely heavily on manual observation, which is detrimental to welfare-oriented livestock farming. Three-dimensional point clouds contain information about the surface texture of the object being measured. Reconstructing livestock 3D point clouds enables non-contact measurement, reducing manual workload and minimizing stress on livestock, thus promoting more refined, intelligent, and welfare-oriented livestock farming.

[0003] To achieve 3D point cloud reconstruction of livestock, point cloud data is typically acquired from multiple perspectives (e.g., top and side views). Therefore, it is necessary to register point clouds from multiple perspectives, which is the foundation of 3D reconstruction. Current livestock point cloud registration methods are prone to getting trapped in local optima, leading to registration failures and requiring manual adjustments. Furthermore, these methods have long running times and low registration accuracy. Summary of the Invention

[0004] This invention provides a multi-view point cloud registration method, apparatus, device, and computer-readable storage medium to solve the technical problem of low registration accuracy in existing livestock point cloud registration methods.

[0005] This invention provides a multi-view point cloud registration method, comprising:

[0006] Acquire multi-view depth video of the object to be detected, crop the multi-view depth video into depth image frames, and convert the depth image frames into point cloud data;

[0007] The point cloud data is extracted to obtain a multi-view point cloud of the object to be detected, which includes a top-view point cloud and a side-view point cloud.

[0008] The top-view point cloud and the completed side-view point cloud are coarsely registered, and the coarsely registered point cloud is sliced.

[0009] The top-view point cloud and the completed side-view point cloud are precisely registered based on the centroid of the overlapping regions of each slice.

[0010] According to a multi-view point cloud registration method provided by the present invention, the step of extracting the multi-view point cloud of the object to be detected from the point cloud data includes:

[0011] Remove device point clouds from the point cloud data based on a preset region of interest;

[0012] Ground point cloud data is removed from the point cloud data based on ground point cloud planar features and a random sampling consistency algorithm.

[0013] The point cloud data after removing the device point cloud and the ground point cloud is extracted according to the preset clustering algorithm to obtain the multi-view point cloud of the object to be detected.

[0014] According to a multi-view point cloud registration method provided by the present invention, the step of extracting the point cloud data to obtain the multi-view point cloud of the object to be detected includes:

[0015] Determine the point interval and extreme value of the point coordinates corresponding to the side view point cloud, and slice the side view point cloud based on the point interval and extreme value of the point coordinates to obtain point cloud slices;

[0016] The missing region is determined based on the distance between adjacent points corresponding to the point cloud slice;

[0017] Projecting the point cloud slices yields a projected point sequence, and curve fitting is performed on the projected point sequence to obtain a fitted point sequence;

[0018] Based on the coordinates of the missing region, extract the target point column of the region that is the same as the missing region from the fitted point column;

[0019] The target point sequence is restored to the projected point sequence to obtain the completed side view point cloud.

[0020] According to a multi-view point cloud registration method provided by the present invention, the coarse registration of the top-view point cloud and the completed side-view point cloud includes:

[0021] Identify the non-collinear points in the point cloud of the railings;

[0022] The rotation and translation matrix between the top-view point cloud and the completed side-view point cloud is determined based on the singular value decomposition algorithm.

[0023] Based on the non-collinear points of the same position and the rotation and translation matrix, coarse registration is performed on the top-view point cloud and the completed side-view point cloud.

[0024] According to a multi-view point cloud registration method provided by the present invention, the step of finely registering the top-view point cloud and the completed side-view point cloud based on the centroid of the overlapping regions of each slice includes:

[0025] Determine the first registration region for the top view slice and the second registration region for the side view slice in each slice;

[0026] The fitness function is determined based on the centroids of the first and second registration regions.

[0027] The fitness function is optimized based on a preset dolphin echolocation algorithm, and the top-view point cloud and the completed side-view point cloud are finely registered according to the optimization results.

[0028] According to the multi-view point cloud registration method provided by the present invention, the step of optimizing the fitness function based on a preset dolphin echolocation algorithm and performing fine registration of the top-view point cloud and the completed side-view point cloud based on the optimization result includes:

[0029] Determine the first matrix, the second matrix, and the optimal solution for each round corresponding to the preset dolphin echolocation algorithm;

[0030] Calculate the fitness of each solution in the second matrix, and save the possible values ​​of the optimal solutions in each round and the fitness into a memory matrix;

[0031] The third matrix is ​​determined based on the possible values ​​of each variable in the second matrix, and the probability value of each possible value in the first matrix is ​​calculated based on the third matrix.

[0032] Based on the probability values, the second matrix is ​​redefined, and the top-view point cloud and the completed side-view point cloud are finely registered.

[0033] The present invention also provides a multi-view point cloud registration device, comprising:

[0034] The point cloud data conversion module is used to acquire multi-view depth video of the object to be detected, crop the multi-view depth video into depth image frames, and convert the depth image frames into point cloud data.

[0035] A multi-view point cloud extraction module is used to extract the point cloud data to obtain a multi-view point cloud of the object to be detected, wherein the multi-view point cloud includes a top-view point cloud and a side-view point cloud.

[0036] The point cloud coarse registration module is used to coarsely register the top-view point cloud and the completed side-view point cloud, and to slice the coarsely registered point cloud.

[0037] The point cloud fine registration module is used to fine register the top-view point cloud and the completed side-view point cloud based on the centroid of the overlapping regions of each slice.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-view point cloud registration method as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-view point cloud registration method as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the multi-view point cloud registration method as described above.

[0041] The multi-view point cloud registration method, apparatus, device, and computer-readable storage medium provided by this invention acquire multi-view depth videos of the object to be detected, crop the acquired multi-view depth videos into depth image frames, further convert the depth image frames into point cloud data, and then extract the point cloud data to obtain the multi-view point cloud of the object to be detected. The multi-view point cloud includes top-view point cloud and side-view point cloud. Then, coarse registration is performed on the top-view point cloud and the completed side-view point cloud, and the coarsely registered point cloud is sliced. Finally, fine registration is performed on the top-view point cloud and the completed side-view point cloud based on the centroid of the overlapping areas of each slice. This improves the accuracy of point cloud registration, provides technical support for subsequent 3D point cloud reconstruction, and is conducive to realizing non-contact measurement of livestock body size and weight and body condition assessment, reducing the occurrence of livestock stress response. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is one of the flowcharts illustrating the multi-view point cloud registration method provided by the present invention;

[0044] Figure 2 This is a schematic diagram of the cow's top-down point cloud in the multi-view point cloud registration method provided by the present invention;

[0045] Figure 3 This is a schematic diagram of the cow side view point cloud in the multi-view point cloud registration method provided by the present invention;

[0046] Figure 4 This is the second flowchart of the multi-view point cloud registration method provided by the present invention;

[0047] Figure 5 This is a diagram showing the effect of completing the cow side view point cloud in the multi-view point cloud registration method provided by this invention.

[0048] Figure 6 This is a schematic diagram of the structure of the multi-view point cloud registration device provided by the present invention;

[0049] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] The following is combined Figures 1-5 The multi-view point cloud registration method of the present invention is described.

[0052] Please refer to Figure 1 This invention provides a multi-view point cloud registration method, comprising:

[0053] Step 100: Acquire multi-view depth video of the object to be detected, crop the multi-view depth video into depth image frames, and convert the depth image frames into point cloud data.

[0054] Specifically, a one-way walking acquisition channel is set up for livestock (i.e., the object to be detected in this embodiment, hereinafter referred to as a cow). Depth cameras can be arranged above and to the side of the channel, including the left and right sides, so as to acquire the cow's top-view depth video and side-view depth video (i.e., the multi-view depth video in this embodiment). After acquiring the cow's top-view depth video and side-view depth video, the acquired cow's top-view depth video and side-view depth video are cropped into depth image frames, and then the depth image frames are further converted into point cloud data.

[0055] Step 200: Extract the point cloud data to obtain the multi-view point cloud of the object to be detected, the multi-view point cloud including top view point cloud and side view point cloud;

[0056] Specifically, the point cloud data obtained above includes irrelevant background point clouds such as the ground, fences, and dairy farm equipment, which need to be removed for different types of irrelevant background point clouds. By setting regions of interest, a large number of irrelevant point clouds such as those of dairy farm equipment are removed, leaving point clouds containing ground and farm fence points. Based on the planar features of the ground point cloud, the RANSAC (Random Sample Consensus) algorithm is used to fit the ground point cloud for removal. For the fence point cloud, clustering methods are used to obtain cow top-view and cow side-view point clouds, such as... Figure 2 and Figure 3 As shown, where, Figure 2 A top-down view of the point cloud of a dairy cow. Figure 3 A schematic diagram of the point cloud from the side view of a dairy cow.

[0057] Step 300: Perform coarse registration on the top-view point cloud and the completed side-view point cloud, and slice the coarsely registered point cloud.

[0058] Specifically, the cow top-view point cloud and the completed cow side-view point cloud are located in two different coordinate systems. It is necessary to unify the cow top-view point cloud and the completed cow side-view point cloud into one coordinate system, that is, to register the cow top-view point cloud and the completed cow side-view point cloud. This includes: since the fence point cloud has obvious features, select multiple (e.g., 3) non-collinear points with the same position on the fence point cloud according to the dairy farm fence. Calculate the rotation and translation matrix between the top-view point cloud and the completed side-view point cloud using the SVD (singular value decomposition) algorithm to complete the coarse registration of the top-view point cloud and the completed side-view point cloud.

[0059] Step 400: Perform fine registration of the top-view point cloud and the completed side-view point cloud based on the centroid of the overlapping regions of each slice.

[0060] Specifically, the point cloud after coarse registration is sliced ​​to obtain several point cloud slices. The highest point of the top-view slice is located at the cow's spine. The registration region of the top-view slice can be extracted through the spine. The registration region of the side-view slice is determined by the registration region of the top-view slice. The centroids of the two overlapping registration regions are extracted, and a fitness function is established based on the centroids, as shown in Formula 1, where M is the number of slices, R is the rotation matrix, T is the translation matrix, and point... t To determine the centroid of the registration region for the top-view slice, point s The centroid of the registration region for the side-view slice is used. The fitness function is further optimized based on the improved dolphin echolocation algorithm to complete the fine registration between the top-view point cloud and the completed side-view point cloud, which will be described in detail below.

[0061]

[0062] This embodiment acquires multi-view depth videos of the object to be detected, crops the acquired multi-view depth videos into depth image frames, further converts the depth image frames into point cloud data, and then extracts the point cloud data to obtain multi-view point clouds of the object to be detected. The multi-view point clouds include top-view point clouds and side-view point clouds. Then, coarse registration is performed on the top-view point clouds and the completed side-view point clouds, and the coarsely registered point clouds are sliced. Finally, fine registration is performed on the top-view point clouds and the completed side-view point clouds based on the centroid of the overlapping areas of each slice, which improves the accuracy of point cloud registration and provides technical support for subsequent 3D point cloud reconstruction. This is conducive to realizing non-contact measurement of livestock body size and weight and body condition assessment, and reducing the occurrence of livestock stress response.

[0063] In one embodiment, the multi-view point cloud registration method provided in this application may further include:

[0064] Step 201: Remove device point clouds from the point cloud data based on a preset region of interest;

[0065] Step 202: Remove ground point cloud data from the point cloud data based on ground point cloud planar features and random sampling consistency algorithm;

[0066] Step 203: Extract the point cloud data after removing the device point cloud and the ground point cloud according to the preset clustering algorithm to obtain the multi-view point cloud of the object to be detected.

[0067] Specifically, the point cloud data obtained from the conversion of depth image frames includes irrelevant background point clouds such as the ground, fences, and dairy farm equipment. Different types of irrelevant background point clouds need to be removed. By setting a region of interest (i.e., the preset region of interest in this embodiment), a large number of irrelevant point clouds, such as those for dairy farm equipment (i.e., the equipment point clouds corresponding to the objects to be detected in this embodiment), are removed. The remaining point cloud includes point clouds of the ground and dairy farm fences. Based on the planar features of the ground point cloud, the RANSAC algorithm is used to fit the ground point cloud for removal. For the fence point cloud, a clustering method is used to extract the cow's top-view point cloud and cow's side-view point cloud, as shown below. Figure 2 and Figure 3 As shown.

[0068] This embodiment obtains multi-view point clouds of the object to be detected by removing irrelevant background point clouds, which lays the technical support for subsequent point cloud registration.

[0069] Please refer to Figure 4 In one embodiment, the multi-view point cloud registration method provided in this application may further include:

[0070] Step 210: Determine the point interval and extreme value of the point coordinates corresponding to the side view point cloud, and slice the side view point cloud based on the point interval and extreme value of the point coordinates to obtain point cloud slices;

[0071] Step 220: Determine the missing region based on the distance between adjacent points corresponding to the point cloud slice;

[0072] Step 230: Project the point cloud slice to obtain a projection point sequence, and perform curve fitting on the projection point sequence to obtain a fitted point sequence;

[0073] Step 240: Based on the coordinates of the missing region, extract the target point column of the region that is the same as the missing region from the fitted point column;

[0074] Step 250: Restore the target point sequence to the projected point sequence to obtain the completed side view point cloud.

[0075] Specifically, in the cow side-view point cloud, some point cloud areas are missing due to obstruction by the dairy farm fences, affecting subsequent point cloud processing. Therefore, it is necessary to complete the missing areas in the side-view point cloud. The missing area completion process includes: First, determining the minimum interval between points in the cow side-view point cloud (i.e., the point interval in this embodiment), the maximum and minimum values ​​of the X-axis, and the maximum and minimum values ​​of the Y-axis (i.e., the extreme values ​​of the point coordinates in this embodiment); Slicing the cow side-view point cloud based on a thickness multiple (e.g., twice) of the point interval to obtain side-view point cloud slices; Determining the missing areas based on the distance between adjacent points in the side-view point cloud slices and recording the coordinates of the missing areas; Projecting the side-view point cloud slices, performing curve fitting on the projected point sequence to obtain a fitted point sequence, extracting the point sequence of the same region as the missing area in the fitted point sequence based on the coordinates of the missing area (i.e., the target point sequence in this embodiment), and restoring the target point sequence to the projected point sequence to obtain the completed side-view point cloud. The completion effect is as follows: Figure 5 As shown.

[0076] This embodiment completes the missing point cloud region, laying the technical support for subsequent point cloud registration.

[0077] In one embodiment, the multi-view point cloud registration method provided in this application may further include:

[0078] Step 310: Determine the non-collinear points in the railing point cloud;

[0079] Step 320: Determine the rotation and translation matrix between the top-view point cloud and the completed side-view point based on the singular value decomposition algorithm;

[0080] Step 330: Based on the non-collinear points of the same position and the rotation and translation matrix, perform coarse registration on the top-view point cloud and the completed side-view point cloud.

[0081] Specifically, the cow top-view point cloud and the completed cow side-view point cloud are located in two different coordinate systems. It is necessary to unify the cow top-view point cloud and the completed cow side-view point cloud into a single coordinate system, i.e., to register the cow top-view point cloud and the completed cow side-view point cloud. The registration process specifically includes: since the fence point cloud has obvious features, multiple (e.g., 3) non-collinear points with the same position on the fence point cloud are selected based on the dairy farm fence (i.e., non-collinear points with the same position in this embodiment). The rotation and translation matrix between the top-view point cloud and the completed side-view point cloud is calculated using the SVD algorithm (i.e., the singular value decomposition algorithm in this embodiment). Based on the non-collinear points with the same position and the rotation and translation matrix, coarse registration is performed on the top-view point cloud and the completed side-view point cloud, completing the coarse registration of the top-view point cloud and the completed side-view point cloud.

[0082] This embodiment calculates the rotation and translation matrix between the top-view point cloud and the completed side-view point cloud using the singular value decomposition algorithm, thus completing the coarse registration between the top-view point cloud and the completed side-view point cloud and initially realizing the coarse registration of the point cloud.

[0083] In one embodiment, the multi-view point cloud registration method provided in this application may further include:

[0084] Step 410: Determine the first registration region of the top view slice and the second registration region of the side view slice in each slice;

[0085] Step 420: Determine the fitness function based on the centroids of the first registration region and the second registration region;

[0086] Step 430: Optimize the fitness function based on the preset dolphin echolocation algorithm, and perform fine registration on the top-view point cloud and the completed side-view point cloud according to the optimization results.

[0087] Specifically, the point cloud after coarse registration is sliced ​​to obtain several point cloud slices. The highest point of the top-view slice is located at the cow's spine. The registration region of the top-view slice (i.e., the first registration region in this embodiment) can be extracted through the spine. The registration region of the side-view slice (i.e., the second registration region in this embodiment) is determined through the registration region of the top-view slice. The centroids of the two overlapping registration regions are extracted, and a fitness function is established based on the centroids, as shown in Formula 1, where M is the number of slices of the point cloud after coarse registration, R is the rotation matrix, and T is the translation matrix (R and T are the rotation and translation matrices in this embodiment). t The centroid of the first registration region, point s Using the centroid of the second registration region as the reference, Formula 1 is further optimized based on the improved dolphin echolocation algorithm. Based on the optimization results, the top-view point cloud and the completed side-view point cloud are finely registered to complete the fine registration between the top-view point cloud and the completed side-view point cloud.

[0088] This embodiment optimizes the fitness function by using a preset dolphin echolocation algorithm. Based on the optimization results, it performs fine registration of the top-view point cloud and the completed side-view point cloud, which improves the accuracy of point cloud registration and provides technical support for subsequent 3D point cloud reconstruction. This facilitates non-contact measurement of livestock body size and weight, as well as body condition assessment, and reduces the occurrence of livestock stress.

[0089] In one embodiment, the multi-view point cloud registration method provided in this application may further include:

[0090] Step 431: Determine the first matrix, the second matrix, and the optimal solution for each round corresponding to the preset dolphin echolocation algorithm;

[0091] Step 432: Calculate the fitness of each solution in the second matrix, and save the possible values ​​of the optimal solutions in each round and the fitness into the memory matrix;

[0092] Step 433: Determine the third matrix based on the possible values ​​of each variable in the second matrix, and calculate the probability value of each possible value in the first matrix based on the third matrix;

[0093] Step 434: Based on the probability values, redetermine the second matrix and perform fine registration on the top-view point cloud and the completed side-view point cloud.

[0094] Specifically, an improved dolphin echolocation algorithm is used to optimize Formula 1, achieving precise registration. The specific process is as follows:

[0095] 1. Determine the range of values ​​for six variables: X-axis rotation angle, Y-axis rotation angle, Z-axis rotation angle, X-axis translation distance, Y-axis translation distance, and Z-axis translation distance. Discretize each variable uniformly to obtain six ordered vectors, forming a matrix PV. N*6 (i.e., the first matrix in this embodiment), select any possible value of each variable to form matrix L. M*6 (That is, the second matrix in this embodiment, where M is the population size).

[0096] 2. Calculate the probability that the optimal solution in each round is the global optimal solution based on the convergence factor in the original dolphin echolocation algorithm, as shown in Formula 2, where i loop I is the current loop count. N Let PP1 be the total number of iterations, PP(i) be the probability that the optimal solution in the first iteration is the global optimal solution, and PP(i) be the probability that the optimal solution in the first iteration is the global optimal solution. loop ) is the i-th loop The probability that the optimal solution of the next iteration is the global optimal solution, where d is the degree of the curve.

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] 3. Calculate the fitness of each solution in the second matrix, and store the possible values ​​of the optimal solution and the fitness value in the memory matrix. middle.

[0103] 4. Change the symmetric linear function in the original dolphin echolocation algorithm to a nonlinear function, as shown in Formula 3, where k∈[-R e ,R e ], and k is an integer, R e To affect the radius, R is dynamically adjusted based on the number of iterations. e In the early stages of the algorithm, a larger value is selected, and in the later stages, a smaller value is selected, i.e., R. e = N / 4δ, where δ is the coefficient of variation, δ>1. Based on the possible values ​​of each variable in the second matrix, calculate the cumulative fitness of the possible values ​​in the sequence vectors that affect their neighborhoods, and form the matrix AF. N*6 (i.e., the third matrix in this embodiment), as shown in Formula 4, where L i Let be the i-th solution in the second matrix, and s be the index of the j-th variable in the i-th solution within the sequence vector. For the part that exceeds the boundary, the mirror principle is used for processing, as shown in Formula 5.

[0104] 5. Based on each possible value of the optimal solution, set its corresponding cumulative fitness to 0.

[0105] 6. Calculate the probability values ​​of each possible value in the first matrix based on the third matrix, as shown in Formula 6.

[0106] 7. Calculate the probability of each possible value in the first matrix appearing in the next iteration. Let the probability of each possible value for the optimal solution be PP(i). loop The probabilities of the remaining possible values ​​are P(i,j)*(1-PP(i)). loop )).

[0107] 8. Generate a new matrix L based on the probability. When the fitness difference of the optimal solution is less than a certain value, determine the corresponding possible values ​​based on the maximum fitness value in the memory matrix. Find the position corresponding to each possible value in the first matrix, select a value range within a certain range and perform secondary discretization. Repeat the above steps until the number of iterations reaches a certain number or the fitness no longer changes, thus completing the fine registration between the top-view point cloud and the completed side-view point cloud.

[0108] This embodiment optimizes the fitness function by using a preset dolphin echolocation algorithm. Based on the optimization process and results, it performs fine registration of the top-view point cloud and the completed side-view point cloud, which improves the accuracy of point cloud registration and provides technical support for subsequent 3D point cloud reconstruction. This facilitates non-contact measurement of livestock body size and weight, as well as body condition assessment, and reduces the occurrence of livestock stress.

[0109] The multi-view point cloud registration device provided by the present invention is described below. The multi-view point cloud registration device described below and the multi-view point cloud registration method described above can be referred to in correspondence.

[0110] Please refer to Figure 6 The present invention also provides a multi-view point cloud registration device, comprising:

[0111] The point cloud data conversion module 601 is used to acquire multi-view depth video of the object to be detected, crop the multi-view depth video into depth image frames, and convert the depth image frames into point cloud data.

[0112] The multi-view point cloud extraction module 602 is used to extract the point cloud data to obtain the multi-view point cloud of the object to be detected, wherein the multi-view point cloud includes top-view point cloud and side-view point cloud.

[0113] The point cloud coarse registration module 603 is used to coarsely register the top-view point cloud and the completed side-view point cloud, and to slice the coarsely registered point cloud.

[0114] The point cloud fine registration module 604 is used to fine register the top-view point cloud and the completed side-view point cloud according to the centroid of the overlapping areas of each slice.

[0115] Optionally, the point cloud data conversion module includes:

[0116] The device point cloud removal unit is used to remove device point clouds from the point cloud data based on a preset region of interest.

[0117] The ground point cloud removal unit is used to remove ground point clouds from the point cloud data based on the planar features of the ground point cloud and the random sampling consistency algorithm.

[0118] The multi-view point cloud extraction unit is used to extract point cloud data after removing the device point cloud and the ground point cloud according to a preset clustering algorithm to obtain the multi-view point cloud of the object to be detected.

[0119] Optionally, the multi-view point cloud registration device includes:

[0120] The point cloud slicing module is used to determine the point interval and extreme value of the point coordinates corresponding to the side view point cloud, and to slice the side view point cloud based on the point interval and extreme value of the point coordinates to obtain point cloud slices.

[0121] The missing region determination module is used to determine the missing region based on the distance between adjacent points corresponding to the point cloud slice;

[0122] The curve fitting module is used to project the point cloud slices to obtain a projection point sequence, and to perform curve fitting on the projection point sequence to obtain a fitted point sequence.

[0123] The target point extraction module is used to extract the target point column of the fitted point column that is the same as the missing region based on the coordinates of the missing region.

[0124] The point column restoration module is used to restore the target point column to the projected point column to obtain the completed side view point cloud.

[0125] Optionally, the point cloud coarse registration module includes:

[0126] The unit for determining points with the same position is used to determine non-collinear points with the same position in the point cloud of railings;

[0127] The rotation and translation matrix determination unit is used to determine the rotation and translation matrix between the top-view point cloud and the completed side-view point based on the singular value decomposition algorithm.

[0128] The coarse registration unit is used to perform coarse registration on the top-view point cloud and the completed side-view point cloud based on the non-collinear points with the same position and the rotation and translation matrix.

[0129] Optionally, the point cloud fine registration module includes:

[0130] The registration region determination unit is used to determine the first registration region of the top view slice and the second registration region of the side view slice in each slice;

[0131] The fitness function determination unit is used to determine the fitness function based on the centroid of the first registration region and the centroid of the second registration region.

[0132] The fine registration unit is used to optimize the fitness function based on a preset dolphin echolocation algorithm, and to perform fine registration on the top-view point cloud and the completed side-view point cloud according to the optimization result.

[0133] Optionally, the fine registration unit includes:

[0134] The matrix determination unit is used to determine the first matrix, the second matrix, and the optimal solution for each round of the preset dolphin echolocation algorithm.

[0135] The memory matrix storage unit is used to calculate the fitness of each solution in the second matrix and save each possible value of the optimal solution in each round and the fitness to the memory matrix;

[0136] The probability value calculation unit is used to determine the third matrix based on the possible values ​​of each variable in the second matrix, and to calculate the probability value of each possible value in the first matrix according to the third matrix.

[0137] The registration unit is used to redetermine the second matrix based on the probability value and perform fine registration on the top-view point cloud and the completed side-view point cloud.

[0138] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740. The processor 710, communications interface 720, and memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions from the memory 730 to execute a multi-view point cloud registration method.

[0139] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0140] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the multi-view point cloud registration method provided by the above methods.

[0141] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-view point cloud registration method provided by the above methods.

[0142] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0143] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0144] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-view point cloud registration method, characterized in that, include: Acquire multi-view depth video of the object to be detected, crop the multi-view depth video into depth image frames, and convert the depth image frames into point cloud data; The point cloud data is extracted to obtain a multi-view point cloud of the object to be detected, which includes a top-view point cloud and a side-view point cloud. Identify the non-collinear points in the point cloud of the railings; The rotation and translation matrix between the top-view point cloud and the completed side-view point cloud is determined based on the singular value decomposition algorithm. Based on the non-collinear points of the same position and the rotation and translation matrix, coarse registration is performed on the top-view point cloud and the completed side-view point cloud. Slice the point cloud after coarse registration; Determine the first registration region for the top view slice and the second registration region for the side view slice in each slice; The fitness function is determined based on the centroids of the first and second registration regions; the fitness function is expressed as: ； Where M is the number of slices, R is the rotation matrix, and T is the translation matrix. To locate the centroid of the registration region for the top-view slice, The centroid of the registration region for the side view slice; The first matrix, the second matrix, and the optimal solution for each round are determined according to the preset dolphin echolocation algorithm. The first matrix is ​​composed of six ordered vectors obtained by uniformly discretizing each of the six variables, namely, the rotation angle of the X-axis, the rotation angle of the Y-axis, the rotation angle of the Z-axis, the translation distance of the X-axis, the translation distance of the Y-axis, and the translation distance of the Z-axis. The second matrix is ​​composed by selecting any possible value of each variable. Calculate the fitness of each solution in the second matrix, and save the possible values ​​of the optimal solutions in each round and the fitness into a memory matrix; The third matrix is ​​determined based on the possible values ​​of each variable in the second matrix, and the probability value of each possible value in the first matrix is ​​calculated based on the third matrix; the third matrix is ​​composed of the cumulative fitness of each possible value in a sequence vector, calculated based on the possible values ​​of each variable in the second matrix. Based on the probability values, the second matrix is ​​redefined, and the top-view point cloud and the completed side-view point cloud are finely registered.

2. The multi-view point cloud registration method according to claim 1, characterized in that, The step of extracting the multi-view point cloud of the object to be detected from the point cloud data includes: Remove device point clouds from the point cloud data based on a preset region of interest; Ground point cloud data is removed from the point cloud data based on ground point cloud planar features and a random sampling consistency algorithm. The point cloud data after removing the device point cloud and the ground point cloud is extracted according to the preset clustering algorithm to obtain the multi-view point cloud of the object to be detected.

3. The multi-view point cloud registration method according to claim 1, characterized in that, After extracting the point cloud data to obtain the multi-view point cloud of the object to be detected, the process includes: Determine the point interval and extreme value of the point coordinates corresponding to the side view point cloud, and slice the side view point cloud based on the point interval and extreme value of the point coordinates to obtain point cloud slices; The missing region is determined based on the distance between adjacent points corresponding to the point cloud slice; Projecting the point cloud slices yields a projected point sequence, and curve fitting is performed on the projected point sequence to obtain a fitted point sequence; Based on the coordinates of the missing region, extract the target point column of the region that is the same as the missing region from the fitted point column; The target point sequence is restored to the projected point sequence to obtain the completed side view point cloud.

4. A multi-view point cloud registration device, characterized in that, include: The point cloud data conversion module is used to acquire multi-view depth video of the object to be detected, crop the multi-view depth video into depth image frames, and convert the depth image frames into point cloud data. A multi-view point cloud extraction module is used to extract the point cloud data to obtain a multi-view point cloud of the object to be detected, wherein the multi-view point cloud includes a top-view point cloud and a side-view point cloud. The point cloud coarse registration module is used to determine the non-collinear points of common position in the railing point cloud; to determine the rotation and translation matrix between the top view point cloud and the completed side view point cloud based on the singular value decomposition algorithm; and to perform coarse registration on the top view point cloud and the completed side view point cloud according to the non-collinear points of common position and the rotation and translation matrix. Slice the point cloud after coarse registration; The point cloud fine registration module is used to determine the first registration region of the top-view slice and the second registration region of the side-view slice in each slice; based on the centroid of the first registration region and the centroid of the second registration region, a fitness function is determined; the fitness function is expressed as: ； Where M is the number of slices, R is the rotation matrix, and T is the translation matrix. To locate the centroid of the registration region for the top-view slice, The centroid of the registration region for the side view slice; The first matrix, the second matrix, and the optimal solution for each round are determined according to the preset dolphin echolocation algorithm. The first matrix is ​​composed of six ordered vectors obtained by uniformly discretizing each of the six variables, namely, the rotation angle of the X-axis, the rotation angle of the Y-axis, the rotation angle of the Z-axis, the translation distance of the X-axis, the translation distance of the Y-axis, and the translation distance of the Z-axis. The second matrix is ​​composed by selecting any possible value of each variable. Calculate the fitness of each solution in the second matrix, and save the possible values ​​of the optimal solutions in each round and the fitness into a memory matrix; The third matrix is ​​determined based on the possible values ​​of each variable in the second matrix, and the probability value of each possible value in the first matrix is ​​calculated based on the third matrix; the third matrix is ​​composed of the cumulative fitness of each possible value in a sequence vector, calculated based on the possible values ​​of each variable in the second matrix. Based on the probability values, the second matrix is ​​redefined, and the top-view point cloud and the completed side-view point cloud are finely registered.

5. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-view point cloud registration method as described in any one of claims 1 to 3.

6. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-view point cloud registration method as described in any one of claims 1 to 3.

7. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-view point cloud registration method as described in any one of claims 1 to 3.

Images