Point cloud conversion method, device and storage medium based on mobile luggage

Abstract

The present application relates to the field of target detection technology, and discloses a three-dimensional modeling method, device and storage medium based on moving luggage. The method includes: collecting multiple depth images during the movement of the target luggage, and determining the initial depth image and the depth image to be aggregated from them to extract the initial point cloud and the point cloud to be aggregated; determining the alignment coordinate system according to the initial point cloud, and extracting the initial voxel set according to the initial point cloud; calculating the horizontal displacement and normal vector of the target luggage corresponding to the depth image to be aggregated, so as to convert the point cloud to be aggregated into a conversion point cloud under the alignment coordinate system, and extracting the voxel set to be aggregated according to the conversion point cloud; and constructing a three-dimensional grid model using the initial voxel set and each voxel set to be aggregated. The present application utilizes the characteristic of unidirectional movement of luggage on a plane, and applies it as a constraint condition to the posture calculation of the depth camera, thereby reducing the error of the relative posture estimation between the depth camera and the moving object, and solving the problem of three-dimensional modeling of moving luggage.

Description

[0001] This application is a divisional application with application number 202210289591.3, application date March 23, 2022, and invention name “Three-dimensional modeling method, device and storage medium based on mobile luggage”. Technical Field

[0002] The present application relates to the field of target detection technology, and in particular to a point cloud conversion method, device and storage medium based on mobile luggage. Background Art

[0003] At present, the method of detecting whether luggage is damaged is usually to use 3D modeling technology. Usually, the target luggage is stationary, and then the depth camera is rotated around the target luggage to collect point cloud data corresponding to different depth images, and each depth image is converted to the same coordinate system through the ICP (Iterative Closest Point) algorithm to achieve point cloud aggregation and build a 3D grid model. The constructed 3D model can then be used to determine whether the luggage is damaged.

[0004] Therefore, to build a 3D model, it is necessary to use the point cloud conversion method to convert each depth image to the same coordinate system. However, since the surrounding environment of the stationary luggage collected in the relevant technology is unchanged, it is relatively simple to achieve point cloud aggregation. For moving luggage, since the surrounding environment of the moving luggage is changing, this will cause the point cloud shape to be inconsistent between frames, that is, the point cloud data of the moving luggage is quite different. Therefore, for moving luggage, no matter how the traditional ICP algorithm is calculated, the translation matrix and rotation matrix calculated by it cannot make the point cloud overlap, which will cause the point cloud data collected at different times to be unable to be aggregated, resulting in the failure of building the 3D network model. Summary of the invention

[0005] The main purpose of this application is to provide a point cloud conversion method, device and storage medium based on mobile luggage, aiming to realize the conversion of the point cloud of mobile luggage to achieve point cloud aggregation, thereby performing rapid three-dimensional modeling.

[0006] In a first aspect, the present application provides a point cloud conversion method based on mobile luggage, characterized in that the method comprises:

[0007] During the movement of the target luggage, an initial point cloud and a point cloud to be aggregated corresponding to the target luggage are acquired, wherein the acquisition time of the initial point cloud and the point cloud to be aggregated are different;

[0008] Determine the number of points in the initial point cloud as a first number, and determine the number of points in the to-be-aggregated point cloud as a second number;

[0009] Selecting a minimum value from the first number and the second number as a matching number, and selecting a first point set from the initial point cloud according to the matching number;

[0010] Filtering the nearest points from the point cloud to be aggregated according to the first point set to obtain a second point set;

[0011] Constructing a first matrix according to the horizontal displacement of the target luggage;

[0012] Constructing a rotation matrix, and constructing a second matrix according to the rotation matrix and the first matrix;

[0013] Performing matrix transformation on each point in the first point set according to the second matrix to obtain a third point set;

[0014] Calculating the translation matrix and the rotation matrix corresponding to the third point set and the second point set when the result value of a preset deviation function is minimum, wherein the preset deviation function is related to the second point set, the third point set, the normal vector, the translation matrix, and the rotation matrix;

[0015] Performing a rotation transformation on the second point set using the rotation matrix to obtain a fourth point set, and performing a translation transformation on the fourth point set using the translation matrix to obtain a fifth point set;

[0016] Using the rotation matrix to perform a rotation transformation on the point cloud to be aggregated to obtain a first temporary point cloud, and using the translation matrix to perform a translation transformation on the first temporary point cloud to obtain a second temporary point cloud;

[0017] An average distance between the fifth point set and the first point set is calculated, and when the average distance is less than a preset distance, the second temporary point cloud is determined to be a converted point cloud.

[0018] In a second aspect, the present application also provides a terminal device, comprising a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein when the computer program is executed by the processor, the steps of the point cloud conversion method based on mobile luggage as described above are implemented.

[0019] In a third aspect, the present application further provides a storage medium, on which a computer program is stored, wherein when the computer program is executed by a processor, the steps of the point cloud conversion method based on moving luggage as described above are implemented.

[0020] The present application provides a point cloud conversion method, device and storage medium based on moving luggage. In the present application, multiple depth images at multiple angles are collected during the movement of the target luggage. During the aggregation process of the depth images, the horizontal displacement and normal vector of the target luggage are calculated, so as to combine the horizontal displacement and normal vector to achieve point cloud overlap when converting the coordinate system of the depth image, thereby constructing a target three-dimensional model. Through the technical solution provided by the present application, the horizontal displacement and normal vector of the target luggage between the depth images are combined and applied as constraints to the posture calculation of the depth camera, which reduces the error of the relative posture estimation between the depth camera and the target object, and realizes accurate aggregation between multiple depth images; in addition, the present application collects depth images while aggregating the collected depth images during the movement of the target luggage, and the three-dimensional model construction time is short, realizing rapid three-dimensional modeling of moving luggage. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0022] Figure 1 A schematic diagram of the steps of a three-dimensional modeling method based on mobile luggage provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of an application scenario of a three-dimensional modeling method based on mobile luggage provided in an embodiment of the present application;

[0024] Figure 3 A schematic block diagram of the structure of a terminal device provided in an embodiment of the present application.

[0025] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] The flowcharts shown in the accompanying drawings are only examples and do not necessarily include all the contents and operations / steps, nor do they have to be executed in the order described. For example, some operations / steps can also be decomposed, combined or partially merged, so the actual execution order may change according to actual conditions. In addition, although the functional modules are divided in the device schematic, in some cases, the module division can be different from that in the device schematic.

[0028] Embodiments of the present application provide a three-dimensional modeling method, device, and storage medium based on mobile luggage.

[0029] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0030] Please refer to Figure 1 , Figure 1 A schematic flowchart of the steps of a three-dimensional modeling method based on mobile luggage provided in an embodiment of the present application.

[0031] like Figure 1 As shown, the three-dimensional modeling method based on moving luggage includes steps S10 to S16.

[0032] Step S10: During the movement of the target luggage, a plurality of depth images of the target luggage at a plurality of angles are collected.

[0033] It can be understood that depth images are images captured by a depth camera, also known as range images.

[0034] The present application is applied to a three-dimensional modeling system for luggage. When the target luggage is placed on a conveyor belt, the conveyor belt drives the target luggage to move in a preset direction. During the movement of the target luggage, multiple depth cameras arranged above the conveyor belt continuously collect multiple depth images of the target luggage from multiple angles.

[0035] In some embodiments, the plurality of angles include a first angle, a second angle, and a third angle;

[0036] The depth image corresponding to the first angle includes at least image information of a first side surface and a second side surface of the target luggage;

[0037] The depth image corresponding to the second angle includes at least image information of a third side surface of the target luggage opposite to the first side surface, and a fourth side surface opposite to the second side surface;

[0038] The depth image corresponding to the third angle includes at least image information of an upper surface of the target luggage, wherein the upper surface is connected to the first side surface, the second side surface, the third side surface, and the fourth side surface.

[0039] It can be understood that the first angle, the second angle and the third angle correspond to different image acquisition areas.

[0040] like Figure 2 As shown, the three-dimensional modeling method based on moving luggage provided in the present application is applied to a luggage conveying system and is executed by a processor in a terminal device of the luggage conveying system.

[0041] Specifically, the baggage transfer system includes a terminal device 20 , a conveying device 21 , and an image acquisition device 22 .

[0042] The conveying device includes a conveyor belt 210, a driving mechanism 211, a first guardrail 212 and a second guardrail 213. The first guardrail 212 and the second guardrail 213 are respectively installed on the driving mechanism 211 on both sides of the conveyor belt 210.

[0043] The image acquisition device 22 includes a first vertical rod 220, a second vertical rod 221, a cross rod 222, a first depth camera 223, a second depth camera 224, and a third depth camera 225. The first vertical rod 220 is installed on the driving mechanism 211 near the first guardrail 212, the second vertical rod 221 is installed on the driving mechanism 211 near the second guardrail 213, the first end of the cross rod 222 is connected to the end of the first vertical rod 220 away from the driving mechanism 211, and the second end of the cross rod 222 away from the first end is connected to the end of the second vertical rod 221 away from the driving mechanism 211.

[0044] The first depth camera 223 is installed on the side of the first crossbar 222 near the first end, and the image acquisition angle of the first depth camera 223 corresponds to the first angle 2230, which is used to collect the depth picture of the starting conveying area of ​​the conveyor belt 210.

[0045] The second depth camera 224 is installed on the second crossbar side surface of the crossbar 222 near the second end and opposite to the first crossbar side surface, and the image acquisition angle of the second depth camera 224 corresponds to the second angle 2240, which is used to capture the depth picture of the end conveying area of ​​the conveyor belt 210.

[0046] The third depth camera 225 is installed at the middle position of the third crossbar side of the crossbar 222 close to the conveyor belt 210, and the image acquisition angle of the third depth camera 225 corresponds to the third angle 2250, which is used to capture the depth image of the middle conveying area of ​​the conveyor belt 210.

[0047] The terminal device 20 is electrically connected to the first depth camera 223 , the second depth camera 224 and the third depth camera 225 , respectively, and is configured to receive and process depth images collected by the first depth camera 223 , the second depth camera 224 and the third depth camera 225 .

[0048] When the target baggage 23 moves in the starting conveying area of ​​the conveyor belt 210, the first depth camera 223 can collect depth image information including the first side 230 and the second side 231 of the target baggage 23; when the target baggage 23 moves in the middle conveying area of ​​the conveyor belt 210, the third depth camera 225 can collect depth image information including the top area of ​​the target baggage 23, that is, image information of the upper surface 232 of the target baggage 23; when the target baggage 23 moves in the ending conveying area of ​​the conveyor belt 210, the second depth camera can collect depth image information including the third side opposite to the first side 230 and the fourth side opposite to the second side 231 of the target baggage 23.

[0049] By combining multiple depth images corresponding to the first angle 2230, the second angle 2240 and the third angle 2250 collected, image information of the first side 230, the second side 231, the third side, the fourth side and the upper surface 232 of the target luggage 23 can be obtained, and then a three-dimensional model of the target luggage 23 can be constructed.

[0050] Step S11: determine the depth picture with the earliest acquisition time as the initial depth picture, and determine the depth pictures with acquisition time after the acquisition time of the initial depth picture as the depth pictures to be aggregated.

[0051] It can be understood that among the multiple depth images corresponding to the target baggage 23 collected, each depth image has its corresponding image collection time. The depth image with the earliest collection time is the initial depth image, and other depth images except the initial depth image, that is, the depth images collected after the collection time of the initial depth image, are all depth images to be aggregated.

[0052] In some embodiments, when the target baggage 23 is placed on the conveyor belt 210, the conveyor belt 210 drives the target baggage 23 to move from the starting conveying area to the ending conveying area. The first depth image of the target baggage 23 collected in the starting conveying area is the initial depth image, and the depth images collected during the subsequent movement are the depth images to be aggregated.

[0053] Step S12: acquiring a point cloud corresponding to the target luggage from the initial depth image as an initial point cloud, and acquiring a point cloud corresponding to the target luggage from the to-be-aggregated depth image as a to-be-aggregated point cloud.

[0054] It can be understood that the initial depth image can be converted into first overall point cloud data according to the attribute parameters of the depth camera that collects the initial depth image, wherein the first overall point cloud data includes not only the point cloud of the target luggage 23, but also the point cloud of the background part other than the target luggage 23. The point cloud corresponding to the target luggage 23 is segmented from the first overall point cloud data to obtain the initial point cloud.

[0055] Similarly, the depth image to be aggregated can be converted into second overall point cloud data according to the attribute parameters of the depth camera that collects the depth image to be aggregated, and the point cloud data corresponding to the target luggage 23 can be extracted from the second overall point cloud data to obtain the point cloud to be aggregated.

[0056] In some implementations, obtaining a point cloud corresponding to the target luggage from the initial depth image as an initial point cloud includes:

[0057] Determine the target horizontal area range and the target depth range according to the acquisition angle corresponding to the initial depth image;

[0058] Acquire a point cloud of the initial depth image as a first point cloud;

[0059] Extracting the point cloud whose projection area is within the target horizontal area from the first point cloud to obtain a second point cloud;

[0060] A point cloud whose depth value is within the target depth range is extracted from the second point cloud to obtain an initial point cloud corresponding to the target luggage.

[0061] It can be understood that in the present application, the distance between the depth camera responsible for collecting depth images and the conveyor belt 210 responsible for carrying and moving the target luggage 23 is fixed, and the depth camera is fixedly mounted on the cross bar 222 above the conveyor belt 210. On this basis, the image collection area range of the depth camera is also fixed.

[0062] In addition, the image acquisition area of ​​the depth camera is also larger than that of the target baggage 23. The depth image captured by the depth camera includes not only the image elements of the target baggage 23, but also image elements other than the target baggage 23, such as the conveyor belt 210, the first guardrail 212, and the second guardrail 213. Moreover, the distance between the target baggage 23 and the depth camera is necessarily closer than the distance between the conveyor belt 210 and the depth camera.

[0063] Therefore, the corresponding target horizontal area range and target depth range can be pre-set according to the acquisition angle corresponding to the depth camera that acquires the depth image, so as to extract the point cloud whose projection area of ​​the target luggage 23 is within the target horizontal area range and whose depth value is within the target depth range from the first point cloud, so as to obtain the initial point cloud of the target luggage 23, wherein the depth value corresponds to the z-axis coordinate value in the three-dimensional coordinates of the point cloud.

[0064] In some implementations, obtaining a point cloud corresponding to the target luggage from the initial depth image as an initial point cloud includes:

[0065] Acquire a comparison depth picture according to the acquisition angle corresponding to the initial depth picture, wherein the comparison depth picture is an empty depth picture in which the camera angle corresponds to the acquisition angle and the picture elements do not contain the target luggage;

[0066] Acquire the point cloud of the initial depth image as a first point cloud, and acquire the point cloud of the comparison depth image as a comparison point cloud;

[0067] The point cloud that matches the comparison point cloud is removed from the first point cloud to obtain an initial point cloud corresponding to the target baggage.

[0068] It can be understood that the depth camera is fixedly installed in the air area above the conveyor belt 210, and the depth camera that collects the initial depth image can be determined according to the acquisition angle corresponding to the initial depth image. When there is no luggage on the conveyor belt 210, the conveyor belt 210 is in an empty state. At this time, the depth image collected by the depth camera that does not contain the image elements of the target luggage 23 is the comparison depth image. The point cloud extracted from the comparison depth image is the comparison point cloud.

[0069] The point clouds with the same or similar three-dimensional coordinates and color values ​​as the comparison point cloud in the first point cloud are removed, so as to obtain the initial point cloud corresponding to the target baggage 23 in the initial depth image.

[0070] Step S13: determine the world coordinate system corresponding to the initial point cloud as the alignment coordinate system, and obtain the voxels of the initial point cloud as the initial voxel set.

[0071] It can be understood that the world coordinate system corresponding to the initial point cloud is the alignment coordinate system. After the alignment coordinate system is determined, other point clouds to be aggregated can be converted to the alignment coordinate system to achieve aggregation with the initial point cloud, and then build a three-dimensional mesh model. At this time, the voxels extracted using the initial point cloud are the initial voxel set.

[0072] Step S14: Calculate the horizontal displacement of the target luggage according to the depth picture to be aggregated and the initial depth picture, and calculate the normal vector of the plane where the target luggage is located according to the depth picture to be aggregated.

[0073] It can be understood that the initial depth image and the depth image to be aggregated are taken by the depth camera during the movement of the target luggage 23, that is, the acquisition time of the initial depth image and the depth image to be aggregated is different, and the position of the target luggage 23 in the initial depth image and the depth image to be aggregated is also different. According to the position of the target luggage 23 in the initial depth image and the depth image to be aggregated, the horizontal displacement of the target luggage 23 when it moves from the position in the initial depth image to the position in the depth image to be aggregated can be calculated.

[0074] In addition, the plane where the conveyor belt 210 responsible for carrying and moving the target baggage 23 is located is the plane where the target baggage 23 is located, and the vector represented by the straight line perpendicular to the plane where the conveyor belt 210 is located is the normal vector of the plane. In some embodiments, the normal vector can be calculated by using the RANSAC (random sample consensus) algorithm to act on the depth image to be aggregated, and the normal vector can also be calculated by using the least squares method to act on the depth image to be aggregated, which is not limited here.

[0075] In some implementations, the calculating the horizontal displacement of the target luggage according to the depth picture to be aggregated and the initial depth picture includes:

[0076] Acquire the horizontal moving speed of the target luggage, and acquire the collection time interval between the point cloud to be aggregated and the initial point cloud;

[0077] The horizontal displacement of the target luggage is obtained according to the product of the horizontal moving speed and the collection time interval.

[0078] It can be understood that the target baggage 23 is placed on the conveyor belt 210, and the conveyor belt 210 drives the target baggage 23 to move at a constant speed, and the horizontal moving speed of the target baggage 23 is obtained. On this basis, the collection time interval can also be determined according to the collection time difference between the point cloud to be aggregated and the initial point cloud. Then, the product of the horizontal moving speed and the collection time interval is calculated, and the horizontal displacement of the target baggage 23 is obtained.

[0079] Step S15: transform the point cloud to be aggregated into a conversion point cloud corresponding to the alignment coordinate system by using the horizontal displacement and the normal vector, and obtain voxels of the conversion point cloud as a voxel set to be aggregated.

[0080] It can be understood that before aggregating the point cloud to be aggregated into the initial point cloud to construct a three-dimensional mesh model, the point cloud to be aggregated needs to be converted into an aligned coordinate system.

[0081] In the prior art, the target luggage 23 is usually kept stationary, and then a depth camera is used to rotate around the target luggage 23 to collect depth images, and each depth image is converted to the same coordinate system through an ICP (Iterative Closest Point) algorithm to achieve point cloud aggregation and construct a three-dimensional mesh model.

[0082] However, in the present application, the target luggage 23 is placed on the conveyor belt 210, and the depth image is collected in the process of the conveyor belt 210 driving the target luggage 23 to move, and the camera that collects the depth image is in a stationary state. That is, in each collected depth image, the target luggage 23 is in a moving state. In this case, when using the traditional ICP algorithm to convert the coordinate system of multiple depth images, there will be a situation where convergence does not occur, resulting in calculation failure. This is because the surrounding environment of the moving object is changing, which will cause the point cloud shape to be inconsistent between frames. No matter how the traditional ICP algorithm calculates, the translation matrix and rotation matrix calculated by it cannot make the point cloud overlap, which will cause multiple depth images to fail to aggregate, resulting in the failure of building a three-dimensional network model.

[0083] In the present application, the characteristic that the target luggage 23 moves on a plane is utilized, and the horizontal displacement and the normal vector are used as constraints to realize the conversion of the point cloud to be aggregated into a transformed point cloud in the corresponding alignment coordinate system, and then voxels are extracted from the transformed point cloud to obtain a voxel set to be aggregated.

[0084] Through the technical solution provided in the present application, the linear motion of the target luggage 23 can be converted into the relative motion of the depth camera, and the coordinate system of the target luggage 23 can be kept unchanged, thereby simplifying the subsequent three-dimensional grid model construction process, and further achieving the technical effect of realizing rapid three-dimensional modeling of mobile luggage using only a simple depth image acquisition device.

[0085] In some implementations, the converting the to-be-aggregated point cloud into a converted point cloud corresponding to the alignment coordinate system by using the horizontal displacement and the normal vector includes:

[0086] Determine the number of points in the initial point cloud as a first number, and determine the number of points in the to-be-aggregated point cloud as a second number;

[0087] Selecting a minimum value from the first number and the second number as a matching number, and selecting a first point set from the initial point cloud according to the matching number;

[0088] Filtering the nearest points from the point cloud to be aggregated according to the first point set to obtain a second point set;

[0089] constructing a first matrix according to the horizontal displacement;

[0090] Constructing a rotation matrix, and constructing a second matrix according to the rotation matrix and the first matrix;

[0091] Performing matrix transformation on each point in the first point set according to the second matrix to obtain a third point set;

[0092] Calculating the translation matrix and the rotation matrix corresponding to the third point set and the second point set when the result value of a preset deviation function is minimum, wherein the preset deviation function is related to the second point set, the third point set, the normal vector, the translation matrix, and the rotation matrix;

[0093] Performing a rotation transformation on the second point set using the rotation matrix to obtain a fourth point set, and performing a translation transformation on the fourth point set using the translation matrix to obtain a fifth point set;

[0094] Using the rotation matrix to perform a rotation transformation on the point cloud to be aggregated to obtain a first temporary point cloud, and using the translation matrix to perform a translation transformation on the first temporary point cloud to obtain a second temporary point cloud;

[0095] An average distance between the fifth point set and the first point set is calculated, and when the average distance is less than a preset distance, the second temporary point cloud is determined to be a converted point cloud.

[0096] It can be understood that the initial point cloud and the point cloud to be aggregated are point clouds corresponding to the target baggage 23 extracted from the initial depth image and the depth image to be aggregated, respectively. The number of points in the initial point cloud is the first number, and the number of points in the point cloud to be aggregated is the second number. When the first number is smaller than the second number, the first number is determined to be the matching number; when the first number is greater than or equal to the second number, the second number is determined to be the matching number.

[0097] The first point set is a set of points selected from the initial point cloud, and the number of points in the first point set is the same as the number of matches. After determining the first point set, the nearest points of each point in the first point set can be obtained one by one from the point cloud to be aggregated to obtain the second point set. Similarly, the number of points in the second point set is also the same as the number of matches, and corresponds one-to-one with each point in the first point set.

[0098] In this embodiment, assuming that the horizontal displacement is d, the first matrix constructed according to the horizontal displacement is t prior =[d,0,0] T .

[0099] Assuming that the constructed rotation matrix is ​​R, the second matrix constructed according to the rotation matrix R and the first matrix is: Wherein, R is the 3×3 rotation matrix to be determined.

[0100] Assume that the first point set is: X = {xl, x2, ..., xi}, and assume that the second point set is: P = {pl, p2, ..., pi}.

[0101] Then, matrix transformation is performed on each point in the first point set according to the second matrix, and the obtained third point set is: Y = {yl, y2,…, yi}.

[0102] Assume the normal vector is n, and the number of matches is N P , the preset deviation function constructed in this application is:

[0103] After calculating the optimal matching rotation matrix R and translation matrix t so that the result value of the preset deviation function is minimized, the rotation matrix R and translation matrix t can be used to perform rotation transformation and translation transformation on the second point set respectively to obtain the fifth point set; and the rotation matrix R and translation matrix t can be used again to perform rotation transformation and translation transformation on the aggregated point cloud respectively to obtain the second temporary point cloud.

[0104] At this time, if the average distance between the fifth point set and the first point set is less than the preset distance, it means that the second temporary point cloud obtained by using the rotation matrix R and the translation matrix t has been transformed into the same alignment coordinate system as the initial point cloud. At this time, the second temporary point cloud can be directly determined as the converted point cloud.

[0105] In some embodiments, the method further comprises:

[0106] When the average distance is greater than or equal to the preset distance, the temporary point cloud is used as the point cloud to be aggregated, and the step of selecting the nearest points from the point cloud to be aggregated according to the first point set is re-executed.

[0107] It can be understood that if the average distance between the fifth point set and the first point set is greater than or equal to the preset distance, it means that the second temporary point cloud has not yet been converted to the aligned coordinate system. At this time, the temporary point cloud can be used as the point cloud to be aggregated, and the step of screening the nearest points from the point cloud to be aggregated according to the first point set is re-executed until the appropriate rotation matrix and translation matrix t are matched, so that the average distance between the fifth point set and the first point set is less than the preset distance.

[0108] Step S16: construct a three-dimensional grid model according to the initial voxel set and each of the voxel sets to be aggregated, and perform mapping processing on the three-dimensional grid model to obtain a target three-dimensional model corresponding to the target luggage.

[0109] It can be understood that after obtaining the voxel sets to be aggregated of each point cloud to be aggregated in the aligned coordinate system, the initial voxel set and each voxel set to be aggregated can be aggregated to construct a three-dimensional mesh model. After mapping the three-dimensional mesh model, the target three-dimensional model is obtained.

[0110] In some implementations, an MVS (Multi-View Stereo) algorithm may be used to implement mapping of a three-dimensional mesh model, or other methods may be used as needed to implement mapping of a three-dimensional mesh model, which is not limited here.

[0111] In some embodiments, constructing a three-dimensional mesh model according to the initial voxel set and each of the voxel sets to be aggregated includes:

[0112] Calculating the TSDF value and weight value of each voxel in the initial voxel set and each voxel set to be aggregated;

[0113] Aggregating each of the to-be-aggregated voxel sets into the initial voxel set according to the TSDF value and the weight value to obtain an aggregated result voxel set;

[0114] Identify the isosurface of the aggregation result voxel set, and construct a three-dimensional mesh model according to the isosurface.

[0115] In some embodiments, a TSDF (truncated signed distance function) algorithm may be used to calculate the TSDF values ​​and weight values ​​corresponding to each voxel in the initial voxel set and each voxel set to be aggregated. And according to the TSDF values ​​and weight values ​​corresponding to each voxel, each voxel set to be aggregated is aggregated into the initial voxel set to obtain an aggregated result voxel set.

[0116] After obtaining the aggregated voxel set, the isosurface of the voxel set can be identified according to the MC (Marching Cube) algorithm to construct a three-dimensional mesh model.

[0117] In the present application, multiple depth images at multiple angles are collected during the movement of the target luggage 23. During the aggregation process of the depth images, the horizontal displacement and normal vector of the target luggage 23 are calculated, so that when the coordinate system of the depth images is converted, the horizontal displacement and normal vector are combined to achieve point cloud overlap, thereby constructing a target three-dimensional model. Through the technical solution provided by the present application, the horizontal displacement and normal vector of the target luggage 23 between the depth images are combined to achieve accurate aggregation between multiple depth images, and realize rapid three-dimensional modeling of moving luggage.

[0118] like Figure 3As shown, the terminal device 301 includes a processor 3011, a memory and a network interface connected via a system bus, wherein the memory may include a storage medium 3012 and an internal memory 3015, and the storage medium 3012 may be non-volatile or volatile.

[0119] The storage medium 3012 can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor 3011 can execute any three-dimensional modeling method based on moving luggage.

[0120] Processor 3011 is used to provide computing and control capabilities to support the operation of the entire terminal device.

[0121] The internal memory 3015 provides an environment for the operation of the computer program in the storage medium 3012. When the computer program is executed by the processor 3011, the processor 3011 can execute any three-dimensional modeling method based on moving luggage.

[0122] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 3 The structure shown in the figure is only a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the terminal device to which the scheme of the present application is applied. The specific terminal device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0123] It should be understood that the processor 3011 may be a central processing unit (CPU), and the processor 3011 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0124] In some implementations, the processor 3011 is used to run a computer program stored in the memory to implement the following steps:

[0125] During the movement of the target luggage, multiple depth images of the target luggage at multiple angles are collected;

[0126] Determine the depth picture with the earliest acquisition time as the initial depth picture, and determine the depth picture with an acquisition time after the acquisition time of the initial depth picture as the depth picture to be aggregated;

[0127] Acquire the point cloud corresponding to the target baggage from the initial depth image as the initial point cloud, and acquire the point cloud corresponding to the target baggage from the depth image to be aggregated as the point cloud to be aggregated;

[0128] Determine a world coordinate system corresponding to the initial point cloud as an alignment coordinate system, and obtain voxels of the initial point cloud as an initial voxel set;

[0129] Calculating a horizontal displacement of the target luggage according to the depth picture to be aggregated and the initial depth picture, and calculating a normal vector of a plane where the target luggage is located according to the depth picture to be aggregated;

[0130] The point cloud to be aggregated is converted into a transformed point cloud corresponding to the alignment coordinate system by using the horizontal displacement and the normal vector, and voxels of the transformed point cloud are obtained as a voxel set to be aggregated;

[0131] A three-dimensional grid model is constructed according to the initial voxel set and each of the voxel sets to be aggregated, and mapping is performed on the three-dimensional grid model to obtain a target three-dimensional model corresponding to the target luggage.

[0132] In some implementations, when the processor 3011 obtains the point cloud corresponding to the target luggage from the initial depth image as the initial point cloud, it is configured to implement:

[0133] Determine the target horizontal area range and the target depth range according to the acquisition angle corresponding to the initial depth image;

[0134] Acquire a point cloud of the initial depth image as a first point cloud;

[0135] Extracting the point cloud whose projection area is within the target horizontal area from the first point cloud to obtain a second point cloud;

[0136] A point cloud whose depth value is within the target depth range is extracted from the second point cloud to obtain an initial point cloud corresponding to the target luggage.

[0137] In some implementations, when the processor 3011 obtains the point cloud corresponding to the target luggage from the initial depth image as the initial point cloud, it is configured to implement:

[0138] Acquire a comparison depth picture according to the acquisition angle corresponding to the initial depth picture, wherein the comparison depth picture is an empty depth picture in which the camera angle corresponds to the acquisition angle and the picture elements do not contain the target luggage;

[0139] Acquire the point cloud of the initial depth image as a first point cloud, and acquire the point cloud of the comparison depth image as a comparison point cloud;

[0140] The point cloud that matches the comparison point cloud is removed from the first point cloud to obtain an initial point cloud corresponding to the target baggage.

[0141] In some implementations, when the processor 3011 calculates the horizontal displacement of the target luggage according to the depth picture to be aggregated and the initial depth picture, it is configured to implement:

[0142] Acquire the horizontal moving speed of the target luggage, and acquire the collection time interval between the point cloud to be aggregated and the initial point cloud;

[0143] The horizontal displacement of the target luggage is obtained according to the product of the horizontal moving speed and the collection time interval.

[0144] In some implementations, when the processor 3011 uses the horizontal displacement and the normal vector to convert the point cloud to be aggregated into a converted point cloud corresponding to the alignment coordinate system, it is configured to implement:

[0145] Determine the number of points in the initial point cloud as a first number, and determine the number of points in the to-be-aggregated point cloud as a second number;

[0146] Selecting a minimum value from the first number and the second number as a matching number, and selecting a first point set from the initial point cloud according to the matching number;

[0147] Filtering the nearest points from the point cloud to be aggregated according to the first point set to obtain a second point set;

[0148] constructing a first matrix according to the horizontal displacement;

[0149] Constructing a rotation matrix, and constructing a second matrix according to the rotation matrix and the first matrix;

[0150] Performing matrix transformation on each point in the first point set according to the second matrix to obtain a third point set;

[0151] Calculating the translation matrix and the rotation matrix corresponding to the third point set and the second point set when the result value of a preset deviation function is minimum, wherein the preset deviation function is related to the second point set, the third point set, the normal vector, the translation matrix, and the rotation matrix;

[0152] Performing a rotation transformation on the second point set using the rotation matrix to obtain a fourth point set, and performing a translation transformation on the fourth point set using the translation matrix to obtain a fifth point set;

[0153] Using the rotation matrix to perform a rotation transformation on the point cloud to be aggregated to obtain a first temporary point cloud, and using the translation matrix to perform a translation transformation on the first temporary point cloud to obtain a second temporary point cloud;

[0154] An average distance between the fifth point set and the first point set is calculated, and when the average distance is less than a preset distance, the second temporary point cloud is determined to be a converted point cloud.

[0155] In some implementations, the processor 3011 is further configured to implement:

[0156] When the average distance is greater than or equal to the preset distance, the temporary point cloud is used as the point cloud to be aggregated, and the step of selecting the nearest points from the point cloud to be aggregated according to the first point set is re-executed.

[0157] In some implementations, when constructing a three-dimensional mesh model according to the initial voxel set and each of the voxel sets to be aggregated, the processor 3011 is configured to implement:

[0158] Calculating the TSDF value and weight value of each voxel in the initial voxel set and each voxel set to be aggregated;

[0159] Aggregating each of the to-be-aggregated voxel sets into the initial voxel set according to the TSDF value and the weight value to obtain an aggregated result voxel set;

[0160] Identify the isosurface of the aggregation result voxel set, and construct a three-dimensional mesh model according to the isosurface.

[0161] In some implementations, when the processor 3011 collects multiple depth images of the target luggage at multiple angles, the multiple angles include a first angle, a second angle, and a third angle;

[0162] The depth image corresponding to the first angle includes at least image information of a first side surface and a second side surface of the target luggage;

[0163] The depth image corresponding to the second angle includes at least image information of a third side surface of the target luggage opposite to the first side surface, and a fourth side surface opposite to the second side surface;

[0164] The depth image corresponding to the third angle includes at least image information of an upper surface of the target luggage, wherein the upper surface is connected to the first side surface, the second side surface, the third side surface, and the fourth side surface.

[0165] It should be noted that technicians in the relevant field can clearly understand that for the convenience and simplicity of description, the specific working process of the terminal device described above can refer to the corresponding process in the aforementioned embodiment of the three-dimensional modeling method based on mobile luggage, and will not be repeated here.

[0166] An embodiment of the present application further provides a storage medium, which is a computer-readable storage medium, on which a computer program is stored, wherein the computer program includes program instructions, and the method implemented when the program instructions are executed can refer to the various embodiments of the three-dimensional modeling method based on mobile luggage in the present application.

[0167] The computer-readable storage medium may be an internal storage unit of the terminal device described in the foregoing embodiment, such as a hard disk or memory of the terminal device. The computer-readable storage medium may also be an external storage device of the terminal device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc., equipped on the terminal device.

[0168] It should be understood that the terms used in this application specification are only for the purpose of describing specific embodiments and are not intended to limit the application. As used in this application specification and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0169] It should also be understood that the term "and / or" used in the specification of this application and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations. It should be noted that, in this article, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "including a..." does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0170] The serial numbers of the embodiments of the present application are for description only and do not represent the advantages and disadvantages of the embodiments. The above description is only a specific implementation mode of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A point cloud conversion method based on mobile luggage, characterized in that: The method comprises: During the movement of the target luggage, an initial point cloud and a point cloud to be aggregated corresponding to the target luggage are acquired, wherein the acquisition time of the initial point cloud and the point cloud to be aggregated are different; Determine the number of points in the initial point cloud as a first number, and determine the number of points in the to-be-aggregated point cloud as a second number; Selecting a minimum value from the first number and the second number as a matching number, and selecting a first point set from the initial point cloud according to the matching number; Filtering the nearest points from the point cloud to be aggregated according to the first point set to obtain a second point set; Constructing a first matrix according to the horizontal displacement of the target luggage; Constructing a rotation matrix, and constructing a second matrix according to the rotation matrix and the first matrix; Performing matrix transformation on each point in the first point set according to the second matrix to obtain a third point set; Calculating the translation matrix and the rotation matrix corresponding to the third point set and the second point set when the result value of a preset deviation function is minimum, wherein the preset deviation function is related to the second point set, the third point set, the normal vector, the translation matrix, and the rotation matrix; Performing a rotation transformation on the second point set using the rotation matrix to obtain a fourth point set, and performing a translation transformation on the fourth point set using the translation matrix to obtain a fifth point set; Using the rotation matrix to perform a rotation transformation on the point cloud to be aggregated to obtain a first temporary point cloud, and using the translation matrix to perform a translation transformation on the first temporary point cloud to obtain a second temporary point cloud; An average distance between the fifth point set and the first point set is calculated, and when the average distance is less than a preset distance, the second temporary point cloud is determined to be a converted point cloud.

2. The method according to claim 1, characterized in that The method further comprises: Obtaining a depth image to be aggregated corresponding to the point cloud to be aggregated; The RANSAC algorithm or the least squares method is used to calculate the depth image to be aggregated to obtain the normal vector of the plane where the target luggage is located.

3. The method according to claim 1, characterized in that The method further comprises: When the average distance is greater than or equal to the preset distance, the temporary point cloud is used as the point cloud to be aggregated, and the step of selecting the nearest points from the point cloud to be aggregated according to the first point set is re-executed.

4. The method according to claim 1, characterized in that The method further comprises: The horizontal displacement of the target luggage is calculated according to the point cloud to be aggregated and the initial point cloud.

5. The method according to claim 4, characterized in that The step of calculating the horizontal displacement of the target luggage according to the point cloud to be aggregated and the initial point cloud comprises: Acquire the horizontal moving speed of the target luggage, and acquire the collection time interval between the point cloud to be aggregated and the initial point cloud; The horizontal displacement of the target luggage is obtained according to the product of the horizontal moving speed and the collection time interval.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining an initial depth image corresponding to the initial point cloud and a depth image to be aggregated corresponding to the point cloud to be aggregated; The horizontal displacement of the target luggage is calculated according to the position of the target luggage in the initial depth image and the depth image to be aggregated.

7. The method according to claim 1, characterized in that After determining that the second temporary point cloud is a conversion point cloud, the method further includes: Acquire voxels of the initial point cloud as an initial voxel set, and acquire voxels of the converted point cloud as a voxel set to be aggregated; A three-dimensional mesh model is constructed according to the initial voxel set and each of the voxel sets to be aggregated.

8. The method according to claim 7, characterized in that The constructing of a three-dimensional mesh model according to the initial voxel set and each of the voxel sets to be aggregated comprises: Calculating the TSDF value and weight value of each voxel in the initial voxel set and each voxel set to be aggregated; Aggregating each of the to-be-aggregated voxel sets into the initial voxel set according to the TSDF value and the weight value to obtain an aggregated result voxel set; The isosurface of the aggregation result voxel set is identified according to the MC algorithm, and a three-dimensional mesh model is constructed according to the isosurface.

9. A terminal device, characterized in that: The terminal device includes a processor, a memory, a computer program stored in the memory and executable by the processor, and a data bus for realizing connection and communication between the processor and the memory, wherein when the computer program is executed by the processor, the steps of the point cloud conversion method based on moving luggage as described in any one of claims 1 to 8 are realized.

10. A storage medium for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the point cloud conversion method based on moving luggage according to any one of claims 1 to 8.

Images