Target three-dimensional reconstruction system and method for constructing three-dimensional contour of moving target

Abstract

The present application discloses a target three-dimensional reconstruction system and a method for constructing a three-dimensional contour of a moving target. The target three-dimensional reconstruction system includes: a gantry, a lidar, a line array camera, a microwave radar, a wheel sensor, and a control module; wherein, an operating track of the target is arranged below the gantry; the control module is configured to: during the process of the target running on the operating track, preprocess the data collected by each lidar and each line array camera to obtain three-dimensional coordinate data of the target; calculate the movement amount of the target in three-dimensional directions according to the three-dimensional coordinate data and the moving speed of the target; reconstruct the three-dimensional contour of the target according to the movement amount of the target in three-dimensional directions, the data collected by the microwave radar and each lidar. In the above manner, by fusing the independent speed measurement or cooperative speed measurement results of multiple sensors, the present solution can achieve high-precision three-dimensional contour reconstruction and dynamic analysis of the target running on the track, and significantly improve the accuracy of the reconstructed contour and the reliability of the system.

Description

Technical Field

[0001] This application belongs to the technical field of intelligent detection and three-dimensional reconstruction, and particularly relates to a target three-dimensional reconstruction system and a method for constructing a three-dimensional contour of a moving target. Background Art

[0002] With the continuous speed increase of railway freight transportation, the issues of the safety and maintenance efficiency of freight trains have become increasingly prominent. This is not only related to the safe delivery of goods, but also to the overall efficiency and reliability of railway transportation. Traditional freight detection relies on the two-dimensional imaging technology of a single linear array camera and requires manual visual analysis. Such solutions have the following defects: (1) Visible light imaging is easily interfered by lighting conditions (such as low light and strong light), resulting in a decrease in the signal-to-noise ratio of the image; (2) The two-dimensional projection loses depth information and cannot quantify the three-dimensional deformation of the target, thus affecting the accuracy and integrity of the analysis results. Moreover, the imaging mode of the linear array camera is a 2D plane mode, which can only capture the projection information of the object to be measured on the two-dimensional plane. The spatial scale information such as the depth and height of the target to be measured in the three-dimensional space is completely lost in the 2D imaging, resulting in the inability to perform quantitative analysis on the spatial scale. Summary of the Invention

[0003] To solve the above problems, this application provides a target three-dimensional reconstruction system and a method for constructing a three-dimensional contour of a moving target, which can perform three-dimensional reconstruction and analysis quantification on the target to be measured.

[0004] One technical solution adopted by this application is: to provide a target three-dimensional reconstruction system, which includes: a gantry, where lidars and linear array cameras are provided at the crossbeam, left and right columns of the gantry, and at the two included angles formed by the crossbeam and the left and right columns of the gantry; wherein, a running track of the target is provided below the gantry; a plurality of wheel sensors are deployed on both sides of the running track; at least one microwave radar, which is deployed on the crossbeam of the gantry; a control module, which is connected to each lidar, microwave radar, each linear array camera, and each wheel sensor, and the control module is configured to: during the process of the target running on the running track, preprocess the data collected by each lidar and each linear array camera to obtain the three-dimensional coordinate data of the target; calculate the movement amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; reconstruct the three-dimensional contour of the target according to the movement amount of the target in the three-dimensional direction, the data collected by the microwave radar and each lidar.

[0005] In one embodiment, the control module is further configured to: perform time synchronization on the data collected by each lidar and each line array camera; obtain the attitude information of each lidar based on the acceleration and angular velocity of each lidar within a preset scanning period, and preprocess the data collected by each lidar according to the attitude information to obtain the three-dimensional coordinate data of the target; wherein, the attitude information includes the displacement information and rotation information of the lidar within the preset scanning period; wherein, the acceleration and angular velocity are collected by the inertial measurement unit on the lidar.

[0006] In one embodiment, the control module is further configured to: integrate the acceleration and angular velocity of each lidar within a single scanning period respectively to obtain the attitude information; and construct a first transformation matrix based on the integration result, and compensate each point within the first transformation matrix to perform motion distortion correction processing on the data collected by each lidar.

[0007] In one embodiment, the control module is further configured to: determine a reference coordinate system, and calculate the calibration parameters of each lidar, microwave radar, each line array camera, and each wheel sensor relative to the reference coordinate system; wherein, the calibration parameters include a rotation matrix and a translation vector; convert the data collected by each lidar, microwave radar, each line array camera, and each wheel sensor into the reference coordinate system according to the calibration parameters; integrate the acceleration and angular velocity of each lidar within two scanning periods respectively to obtain the attitude information, and construct a second transformation matrix according to the integration result to obtain the attitude trajectory of the lidar.

[0008] In one embodiment, the control module is further configured to: match the data collected by each lidar with a preset map; construct a third transformation matrix according to the matching result, and update the attitude trajectory of the lidar according to the third transformation matrix and the acceleration and angular velocity of each lidar within two scanning periods; obtain the odometer corresponding to each lidar according to the update result to obtain the odometer of the target, and obtain the three-dimensional coordinate data of the target.

[0009] In one embodiment, the control module is further configured to: obtain the first moving speed of the target based on the odometer of the target; calculate the moving amount in the three-dimensional direction of the target according to the three-dimensional coordinate data and the first moving speed.

[0010] In one embodiment, the control module is further configured to: obtain the second moving speed based on the data collected by the wheel sensor; obtain the third moving speed based on the data collected by the microwave radar; fuse the first moving speed, the second moving speed, and the third moving speed to obtain the fourth moving speed of the target; calculate the moving amount in the three-dimensional direction of the target according to the three-dimensional coordinate data and the fourth moving speed.

[0011] In one embodiment, the control module is further configured to: obtain a state prediction equation, fuse the first moving speed, the second moving speed, and the third moving speed according to the state prediction equation; obtain the odometer of the target according to the fusion result, and obtain the fourth moving speed based on the odometer of the target.

[0012] The present application also provides a method for constructing a three-dimensional contour of a moving target. When the target runs on the running track, it passes through a gantry. Lidar and line array cameras are arranged at the crossbeam, the left and right columns of the gantry, and at the two included angles formed by the crossbeam and the left and right columns of the gantry; wherein, the running track of the target is arranged below the gantry; a plurality of wheel sensors are deployed on both sides of the running track; at least one microwave radar is deployed on the crossbeam of the gantry; each lidar, microwave radar, each line array camera, and each wheel sensor are respectively connected to the control module. The method includes: preprocessing the data collected by each lidar and each line array camera to obtain the three-dimensional coordinate data of the target; calculating the moving amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; reconstructing the three-dimensional contour of the target according to the moving amount of the target in the three-dimensional direction, the data collected by the microwave radar, and each lidar.

[0013] A technical solution adopted by the present application is: to provide a three-dimensional reconstruction system for a target. The three-dimensional reconstruction system for the target includes: a gantry, where lidar and line array cameras are arranged at the crossbeam, the left and right columns of the gantry, and at the two included angles formed by the crossbeam and the left and right columns of the gantry; wherein, the running track of the target is arranged below the gantry; a plurality of wheel sensors are deployed on both sides of the running track; at least one microwave radar, which is deployed on the crossbeam of the gantry; a control module, which is connected to each lidar, microwave radar, each line array camera, and each wheel sensor. The control module is configured to: during the process of the target running on the running track, preprocess the data collected by each lidar and each line array camera to obtain the three-dimensional coordinate data of the target; calculate the moving amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; reconstruct the three-dimensional contour of the target according to the moving amount of the target in the three-dimensional direction, the data collected by the microwave radar, and each lidar. In the above manner, the data collected by multiple sensors, the results of independent speed measurement or fused speed measurement of the microwave radar, wheel sensors, and lidar can be used to perform high-precision three-dimensional contour reconstruction and in-depth analysis on the moving target, improving the accuracy and reliability of reconstructing the three-dimensional contour of the target. By fusing the independent speed measurement or collaborative speed measurement results of multiple sensors (including microwave radar, wheel sensors, and lidar), this solution can achieve high-precision three-dimensional contour reconstruction and dynamic analysis of the target running on the track, significantly improving the accuracy of the reconstructed contour and the reliability of the system. Description of the Drawings

[0014] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings. Among them:

[0015] Figure 1 is the first structural schematic diagram of the target three-dimensional reconstruction system provided by some embodiments of the present application;

[0016] Figure 2 is the first flowchart of the method for the target three-dimensional contour provided by some embodiments of the present application;

[0017] Figure 3 is provided by some embodiments of the present application Figure 2 is the sub-flowchart of step S21 therein;

[0018] Figure 4 is provided by some embodiments of the present application Figure 3 is the first sub-flowchart of step S212 therein;

[0019] Figure 5 is provided by some embodiments of the present application Figure 3 is the second sub-flowchart of step S212 therein;

[0020] Figure 6 is provided by some embodiments of the present application Figure 3 is the third sub-flowchart of step S212 therein;

[0021] Figure 7 is provided by some embodiments of the present application Figure 2 is the first sub-flowchart of step S22 therein;

[0022] Figure 8 is provided by some embodiments of the present application Figure 2 is the second sub-flowchart of step S22 therein;

[0023] Figure 9 is provided by some embodiments of the present application Figure 8 is the sub-flowchart of step S225 therein. Detailed implementation manners

[0024] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application. It can be understood that the specific embodiments described herein are only used to explain the present application, rather than limiting the present application. In addition, it should be noted that for the convenience of description, only parts related to the present application are shown in the drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0025] The terms "first", "second", etc. in the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally further include steps or units not listed, or may optionally further include other steps or units inherent to these processes, methods, products, or devices.

[0026] Referring to "embodiment" herein means that a specific feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The phrase appears at various positions in the specification and does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments. Those skilled in the art will explicitly and implicitly understand that the embodiments described herein may be combined with other embodiments.

[0027] Refer to Figure 1 , Figure 1 which is a schematic structural diagram of the first embodiment of the target three-dimensional reconstruction system provided by the present application. The target three-dimensional reconstruction system 100 includes: a gantry 10, a lidar 20, a line array camera 30, a wheel sensor 40, at least one microwave radar 50, and a control module (not shown in the figure).

[0028] Among them, lidars 20 and line array cameras 30 are provided at the cross beam, the left and right columns of the gantry 10, and at the two included angles formed by the cross beam and the left and right columns of the gantry 10; a running track of the target is provided below the gantry 10; and a plurality of wheel sensors 40 are deployed on both sides of the running track.

[0029] Among them, the microwave radar 50 is deployed on the cross beam of the gantry 10.

[0030] Among them, the control module is connected to each lidar 20, microwave radar 50, each line array camera 30, and each wheel sensor 40, and the control module is configured to: during the process of the target running on the running track, preprocess the data collected by each lidar 20 and each line array camera 30 to obtain the three-dimensional coordinate data of the target; calculate the movement amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; reconstruct the three-dimensional contour of the target according to the movement amount of the target in the three-dimensional direction, the data collected by the microwave radar 50 and each lidar 20.

[0031] Specifically, multiple lidars 20 are installed at different positions of the gantry 10, which can realize more comprehensive acquisition of spatial information of the target and high-precision positioning. After the point cloud data of the target collected by the lidar 20, these data are usually processed such as noise filtering, duplicate removal, registration, etc., so as to remove the stray signals introduced by environmental factors (such as atmospheric scattering, equipment errors, etc.) and eliminate the points of repeated measurements. During the process of the target running on the running track, due to the vibration or sway of the gantry 10, the collected data will be distorted, so preprocessing is required to eliminate the distortion of the data; at the same time, the data collected by the lidar 20 from different perspectives, due to the differences in the collection position, angle and time, their coordinate systems are often independent of each other. When preprocessing, it is also necessary to unify the data collected by the lidar 20 from different perspectives into the same coordinate system to form a complete three-dimensional point cloud map.

[0032] Specifically, when the line array camera 30 is selected as an auxiliary imaging means, it is usually necessary to process the image data collected by it, such as preprocessing the image data such as image enhancement, correction and segmentation, so as to improve the visual features of the image, facilitate subsequent feature extraction, eliminate problems such as lens distortion and perspective distortion, ensure the accuracy of the image geometric information, and extract the image area related to the target to be reconstructed, reducing the interference of irrelevant information.

[0033] Specifically, based on the three-dimensional coordinate data of the target, the movement amount of the target in the three-dimensional direction within a continuous time interval can be calculated. For example, by comparing the three-dimensional coordinates at different time points, the displacement changes of the target in the X, Y, and Z directions can be deduced. Fuse the data collected by sensors such as the microwave radar 50, line array camera 30, and lidar 20, and combine the movement amount of the target to update the three-dimensional contour of the target in real time to achieve dynamic modeling.

[0034] In the above solution, by combining the data of multiple sensors and the dynamic information of the target, the shape and position changes of the target can be accurately captured, and high-precision and real-time three-dimensional reconstruction can be realized. Through data fusion and dynamic modeling, the stability and reliability of the three-dimensional contour of the reconstructed target can be effectively improved.

[0035] In some embodiments, the control module is further configured to: perform time synchronization on the data collected by each lidar 20 and each line array camera 30; obtain the attitude information of each lidar 20 based on the acceleration and angular velocity of each lidar 20 within a preset scanning period, and preprocess the data collected by each lidar 20 according to the attitude information to obtain the three-dimensional coordinate data of the target; wherein, the attitude information includes the displacement information and rotation information of the lidar 20 within the preset scanning period; wherein, the acceleration and angular velocity are collected by the inertial measurement unit on the lidar 20.

[0036] Specifically, since the lidar 20 and the line array camera 30 are two different sensors, and they collect data at different frequencies and times, therefore, in order to ensure that these data can accurately reflect the scene information at the same moment, it is necessary to perform time synchronization on them. In one embodiment, time synchronization is achieved by using PTP (Precision Time Protocol). During the data collection process of the lidar 20 and the line array camera 30, each radar data frame and camera data frame will be attached with a timestamp to identify the time when the frame data is collected. To achieve multi-sensor time synchronization, the system clock or external GNSS (Global Navigation Satellite System) time can be used as the global reference time source through PTP. Specifically, based on the GNSS timing module or the system clock, synchronization signals are distributed to the lidar and the line array camera through PTP to force the alignment of the local clocks of each sensor. Under this mechanism, both the lidar and the line array camera achieve sub-microsecond-level time alignment accuracy with a unified time reference, thereby eliminating the clock drift error between devices and ensuring that all data is strictly aligned on the time axis.

[0037] Among them, the inertial measurement unit (Inertial Navigation Unit, IMU) is an autonomous navigation system that can provide the motion state information of the carrier itself. In one embodiment, taking the lidar 20 as the carrier, through the accelerometer and gyroscope inside the inertial measurement unit, the acceleration and angular velocity of the lidar 20 in three-dimensional space can be measured in real time, so as to obtain the attitude information of the lidar 20.

[0038] In the above solution, time synchronization ensures that data from different sensors can be aligned on the same time reference, and the combination of the inertial measurement unit and the lidar 20 combines the attitude information of the lidar 20 with the point cloud data collected by the lidar 20 for motion distortion correction, thereby improving the accuracy and reliability of target three-dimensional reconstruction.

[0039] In some embodiments, the control module is further configured to: integrate the acceleration and angular velocity of each lidar 20 within a single scan cycle respectively to obtain attitude information; and construct a first transformation matrix based on the integration result, and compensate each point within the first transformation matrix to perform motion distortion correction processing on the data collected by each lidar 20.

[0040] Specifically, the accelerometer inside the inertial measurement unit measures the linear acceleration of the lidar 20 in the X, Y, and Z axes, while the gyroscope measures the angular velocity in the three axes. By pre-integrating these acceleration and angular velocity data respectively, the linear displacement change and rotation angle change of the lidar 20 within the preset scan cycle can be obtained, that is, the attitude information. A first transformation matrix is constructed according to the integration result. The first transformation matrix describes the displacement change and rotation angle change of the lidar 20 within a single scan cycle. According to this change, compensation is realized between the remaining points and the first point (arranged in chronological order or data recording order) in a frame of data formed by the lidar 20 scanning (that is, all point cloud data collected within a single scan cycle), and the coordinates of the remaining points in each frame are compensated to the coordinate system at the moment when the first point is collected, eliminating the distortion caused by the movement of the carrier.

[0041] In an application scenario, first, pre-integrate the acceleration of each lidar 20 within a single scan cycle: obtain the acceleration through the inertial measurement unit and obtain the transformation of the velocity within the time window:

[0042]

[0043] where represents the acceleration measured by the inertial measurement unit, represents the gravitational acceleration, and are the start time and end time of a single scan of the lidar 20 respectively. The velocity and can be obtained through the above formula.

[0044] Integrate the velocity within the time window, which is expressed by the following formula:

[0045]

[0046] Assume that within the time window, the velocity change is relatively stable or uniform, usually can be simplified to: , where is the average velocity, is and The time interval between represents the average speed, usually expressed as: .

[0047] That is, the time window The displacement within is expressed as:

[0048]

[0049] Wherein, is The average acceleration within the time window, that is, the mean value of the linear acceleration measured by the inertial measurement unit, is the initial velocity.

[0050] Then, the angular velocity of each lidar 20 within a single scan cycle is pre-integrated: For The angular velocity measured by the inertial measurement unit within the time window is pre-integrated and represented using quaternions. The relative rotation angle of the lidar 20 within the time window can be obtained through the following formula:

[0051]

[0052] Wherein, and represent the quaternions of the rotation angles at the initial and end moments of a single scan cycle, represents the angular velocity.

[0053] Finally, according to the pre-integration results within the time window and a first transformation matrix is constructed. Each point within each frame of the lidar 20 is traversed, and then according to the time difference between the i-th point and the initial point or the end point, linear interpolation is performed to obtain the transformation from the i-th point to the first point in each frame. The coordinates of each point within the frame are compensated to the coordinate form of the first point through the following formula, thereby eliminating the distortion of the point cloud:

[0054]

[0055] Wherein, is the original coordinate of the i-th point in each frame of the lidar 20, is the transformation from the i-th point to the first point after interpolation, represents the representation of the i-th point relative to the coordinate system of the first point after motion distortion correction.

[0056] In the above solution, the precise compensation transformation can preserve the detailed information in the point cloud data. The point cloud data after motion distortion correction by the above method can reduce error accumulation and improve the accuracy and stability of target three-dimensional reconstruction.

[0057] In some embodiments, the control module is further configured to: determine a reference coordinate system, and calculate the calibration parameters of each lidar 20, microwave radar 50, each line array camera 30, and each wheel sensor 40 relative to the reference coordinate system; wherein, the calibration parameters include a rotation matrix and a translation vector; convert the data collected by each lidar 20, microwave radar 50, each line array camera 30, and each wheel sensor 40 into the reference coordinate system according to the calibration parameters; integrate the acceleration and angular velocity of each lidar 20 within two scanning periods respectively to obtain attitude information, and construct a second transformation matrix according to the integration result to obtain the attitude trajectory of the lidar 20.

[0058] In one embodiment, the target can be regarded as a fixed reference object, and the lidar 20 installed on the gantry 10 itself can be used as a dynamic sensor, so as to estimate the relative motion odometer of the target by using the coordinate change and the change of direction or orientation of the lidar 20 in space relative to the reference coordinate system.

[0059] Specifically, taking the coordinate system of a certain lidar 20 as the reference coordinate system, through the calibration parameters, the coordinate conversion relationship between different sensors can be established, so as to convert the data collected by each lidar 20, microwave radar 50, each line array camera 30, and each wheel sensor 40 into the same reference system for fusion. For different types of sensors, the calculation methods and contents of the calibration parameters are also different, which are not limited herein.

[0060] In one embodiment, within two scanning times, the acceleration and angular velocity measured by the inertial measurement unit are pre-integrated respectively to obtain a series of attitude estimations of the lidar 20, and these attitude estimations can be represented as a second transformation matrix, which contains the coordinate and orientation (or direction) information of the lidar 20 at each discrete time point within two scanning times. Since the integration operation is based on the measured values at discrete time points, the obtained attitude estimations are also discrete, and these discrete points are connected to form a rough discrete time trajectory describing the attitude trajectory of the lidar 20. Due to factors such as cumulative error and sensor noise in the integration process, there are certain errors and uncertainties in this rough discrete time trajectory, so refinement operations are required.

[0061] Specifically, a set of continuous-time equations are constructed (these equations may include the kinematic model, dynamic model, and possible deformation model of the object, etc.), and these time equations describe the continuous change process of the lidar 20 from the previous pose to the current pose; by solving these time equations, a more accurate and smooth pose trajectory can be obtained. During the process of refining the pose trajectory, it is also necessary to consider the possible deformation or inclination changes that may occur during the relative movement of the lidar 20, and these changes may cause the positions of some points on the lidar 20 to shift in three-dimensional space; in order to restore the accurate positions of these points, a specific de-skewing transformation needs to be applied from the nearest previous transformation (i.e., the previous pose estimation) to each current point, and this de-skewing transformation may include operations such as rotation, translation, and possible scaling.

[0062] In the above solution, by refining the trajectory and restoring the de-skewing transformation, the pose trajectory of the lidar 20 within a preset scanning period can be obtained, thereby significantly improving the accuracy of target three-dimensional reconstruction.

[0063] In some embodiments, the control module is further configured to: match the data collected by each lidar 20 with a preset map; construct a third transformation matrix according to the matching result, and update the pose trajectory of the lidar 20 according to the third transformation matrix and the acceleration and angular velocity of each lidar 20 within two scanning periods; obtain the odometer corresponding to each lidar 20 according to the update result to obtain the odometer of the target, and obtain the three-dimensional coordinate data of the target.

[0064] Specifically, after performing motion distortion correction processing on the data collected by each lidar 20, the data collected by each lidar 20 is matched with a preset map, and the preset map is a high-precision three-dimensional environment model in the world coordinate system. By calculating the distance between the point cloud data collected by the lidar 20 and the corresponding points or planes in the preset map, the optimal transformation relationship is found, and the point cloud data collected by the lidar 20 will be converted into the world coordinate system and matched and updated with the features in the preset map. The above matching result can obtain a third transformation matrix, which reflects information such as the three-dimensional coordinates of the point cloud data collected by the lidar 20 in the world coordinate system. By fusing the third transformation matrix with the acceleration and angular velocity of the lidar 20 within two scanning periods, a more complete and accurate pose trajectory of the lidar 20 in the world coordinate system can be generated.

[0065] After obtaining the complete and accurate pose trajectory of the lidar 20 in the world coordinate system, the odometer of the lidar 20 in the world coordinate system can be obtained according to the pose trajectory, and then converted into the odometer of the target in the world coordinate system, that is, the accurate three-dimensional coordinate data of the target in the world coordinate system and information such as the orientation or direction of the target relative to the world coordinate system are obtained.

[0066] Among them, during the matching process, to improve the accuracy and efficiency of matching, the Generalized Iterative Closest Point (GICP) optimization prior can be introduced. The GICP algorithm minimizes the distance between corresponding points or planes through iterative optimization, thereby obtaining a more accurate transformation relationship. By incorporating prior knowledge (such as the previous matching result), the convergence speed of the matching process can be further accelerated, and the stability and accuracy of the matching can be improved.

[0067] In the above solution, by matching the data collected by the lidar 20 with a preset map, the point cloud data collected by the lidar 20 can be converted into the world coordinate system. Combining the acceleration and angular velocity of the lidar 20, the attitude trajectory of the lidar 20 can be estimated and updated in real time, thereby obtaining the odometer of the target during relative movement, improving the positioning accuracy of the target's three-dimensional contour and the accuracy of constructing the target's three-dimensional contour.

[0068] In some embodiments, the control module is further configured to: obtain the first moving speed of the target based on the odometer of the target; calculate the moving amount in the three-dimensional direction of the target according to the three-dimensional coordinate data and the first moving speed.

[0069] In one embodiment, the speed can be measured by the lidar 20, the wheel sensor 40, and the microwave radar 50 respectively, or the speed measurement results of the three can be fused to obtain the final moving speed. The method of measuring speed by the lidar 20 will be mainly described below.

[0070] Among them, after the lidar 20 performs the above-mentioned coordinate system conversion of multiple sensor data, pre-integration of inertial measurement unit measurements, point cloud data matching, and estimation and update of the attitude trajectory, it outputs odometer information. After converting the odometer output by the lidar 20 into the odometer of the target, the odometer contains the three-dimensional coordinate data of the target within a preset scanning period.

[0071] In one application scenario, the three-dimensional coordinate data of the target within two scanning periods is obtained, and the first moving speed is calculated based on the target's three-dimensional coordinates: First, calculate the speed of the target in the three-dimensional direction, which is represented by the following formula:

[0072]

[0073] Among them, , , respectively represent the speeds in the X, Y, and Z directions, , , respectively represent the changes in coordinates in the X, Y, and Z directions, respectively represent the start and end times of the scan. Then, , , The first moving speed is obtained by combining, and the moving amount in the target three-dimensional direction is calculated according to the three-dimensional coordinate data and the first moving speed.

[0074] In the above solution, the odometer output by the lidar 20 can provide accurate position and orientation information. By obtaining the moving speed of the target through the lidar 20 odometer, the adaptability to the dynamic environment can be enhanced, and the accuracy and efficiency of the target three-dimensional reconstruction can be improved.

[0075] In some embodiments, the control module is further configured to: obtain a second moving speed based on the data collected by the wheel sensor 40; obtain a third moving speed based on the data collected by the microwave radar 50; fuse the first moving speed, the second moving speed and the third moving speed to obtain a fourth moving speed of the target; calculate the moving amount in the target three-dimensional direction according to the three-dimensional coordinate data and the fourth moving speed.

[0076] Specifically, the microwave radar 50 can directly output a speed measurement result; for the wheel sensor 40, by using two groups of wheel sensors 40 with known distances, when the target passes through the two groups of wheel sensors 40 respectively during the operation on the running track, two times are respectively recorded for the signals triggered by passing through the first group of wheel sensors 40 and the second group of wheel sensors 40, and thus the speed measurement result can be obtained through the distance and the time difference.

[0077] Among them, for the speed measurement outputs of the wheel sensor 40 and the microwave radar 50, due to their low output frequencies, it is necessary to use the uniform linear motion model with the time stamps similar to those in the lidar 20 speed measurement result as the reference, first interpolate the obtained speeds respectively, and then integrate the speeds to obtain the final second moving speed and third moving speed.

[0078] The Kalman filtering algorithm is used to fuse the data obtained from the three speed measurement methods, that is, fuse the first moving speed, the second moving speed and the third moving speed to obtain the fourth moving speed of the target, and calculate the moving amount in the target three-dimensional direction according to the three-dimensional coordinate data and the fourth moving speed.

[0079] In some embodiments, the control module is further configured to: obtain a state prediction equation, fuse the first moving speed, the second moving speed and the third moving speed according to the state prediction equation; obtain the odometer of the target according to the fusion result, and obtain the fourth moving speed based on the odometer of the target.

[0080] Specifically, the state prediction equation is used to estimate the current state according to the previous state, and usually includes a state transition matrix. Assuming that the target is in uniform variable linear motion, the state prediction equation can be simplified as:

[0081]

[0082] Among them, is the prediction of the state at time k based on the state at time k-1, and A is the state transition matrix. is the optimal state estimate at time k-1, that is, at time k-1, based on all available measurement information (including the first moving speed, the second moving speed, and the third moving speed) and the dynamic model of the system, the most accurate estimate of the target speed. The Kalman gain determines the weights of the predicted value and the measured value in the final state estimate, and the calculation of the Kalman gain is based on the process noise covariance and the measurement noise covariance. The first moving speed, the second moving speed, and the third moving speed are weighted and averaged, and the weights are determined by the Kalman gain to obtain the final state estimate , that is, the estimate of the target speed at time k. The state estimate is converted into an odometer, and the odometer contains the three-dimensional coordinate data of the target within a preset scanning period, that is, the three-dimensional coordinate data of the target within the preset scanning period is obtained, and the speed of the target in the three-dimensional direction is calculated according to the transformation of the target three-dimensional coordinates and combined to finally obtain the fourth moving speed.

[0083] In the above solution, the option of multiple speed measurement methods can improve the reliability of the overall speed measurement system. Fusing the results of different speed measurement methods can improve the accuracy and real-time performance of the speed measurement results in a dynamic environment, thereby improving the accuracy, reliability, and adaptability of the target three-dimensional reconstruction.

[0084] Refer to Figure 2 , Figure 2 is a schematic flowchart of the first embodiment of the method for constructing the three-dimensional contour of the target provided by this application. When the target runs on the running track, it passes through the gantry. Laser radars and line array cameras are arranged at the crossbeam, the left and right columns of the gantry, and at the two included angles formed by the crossbeam and the left and right columns of the gantry; among them, the running track of the target is arranged below the gantry; a plurality of wheel sensors are deployed on both sides of the running track; at least one microwave radar is deployed on the crossbeam of the gantry; each laser radar, microwave radar, each line array camera, and each wheel sensor are respectively connected to the control module. The method includes:

[0085] Step S21: Preprocess the data collected by each laser radar and each line array camera to obtain the three-dimensional coordinate data of the target.

[0086] In some embodiments, refer to Figure 3 , the above step S21 may include the following steps:

[0087] Step S211: Synchronize the time of the data collected by each laser radar and each line array camera.

[0088] Step S212: Obtain the attitude information of each lidar based on the acceleration and angular velocity within a preset scanning period, and preprocess the data collected by each lidar according to the attitude information to obtain the three-dimensional coordinate data of the target; wherein, the attitude information includes displacement information and rotation information of the lidar within the preset scanning period.

[0089] In some embodiments, referring to Figure 4 , the above step S212 may include the following steps:

[0090] Step S2121: Integrate the acceleration and angular velocity of each lidar within a single scanning period respectively to obtain the attitude information.

[0091] Step S2122: Construct a first transformation matrix based on the integration result, and compensate each point within the first transformation matrix to perform motion distortion correction processing on the data collected by each lidar.

[0092] In some embodiments, referring to Figure 5 , the above step S212 may further include the following steps:

[0093] Step S2123: Determine the reference coordinate system, and calculate the calibration parameters of each lidar, microwave radar, each linear array camera, and each wheel sensor relative to the reference coordinate system; wherein, the calibration parameters include a rotation matrix and a translation vector.

[0094] Step S2124: Convert the data collected by each lidar, microwave radar, each linear array camera, and each wheel sensor into the reference coordinate system according to the calibration parameters.

[0095] Step S2125: Integrate the acceleration and angular velocity of each lidar within two scanning periods respectively to obtain the attitude information, and construct a second transformation matrix according to the integration result to obtain the attitude trajectory of the lidar.

[0096] In some embodiments, referring to Figure 6 , the above step S212 may further include the following steps:

[0097] Step S2126: Match the data collected by each lidar with a preset map.

[0098] Step S2127: Construct a third transformation matrix according to the matching result, and update the attitude trajectory of the lidar according to the third transformation matrix and the acceleration and angular velocity of each lidar within two scanning periods.

[0099] Step S2128: Obtain the odometer corresponding to each lidar according to the update result to obtain the odometer of the target and obtain the three-dimensional coordinate data of the target.

[0100] Step S22: Calculate the movement amount in the target three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target.

[0101] In some embodiments, referring to Figure 7 , the above step S22 may include the following steps:

[0102] Step S221: Obtain the first moving speed of the target based on the odometer of the target.

[0103] Step S222: Calculate the movement amount in the target three-dimensional direction according to the three-dimensional coordinate data and the first moving speed.

[0104] In some embodiments, referring to Figure 8 , the above step S22 may further include the following steps:

[0105] Step S223: Obtain the second moving speed based on the data collected by the wheel sensor.

[0106] Step S224: Obtain the third moving speed based on the data collected by the microwave radar.

[0107] Step S225: Fuse the first moving speed, the second moving speed and the third moving speed to obtain the fourth moving speed of the target.

[0108] Step S226: Calculate the movement amount in the target three-dimensional direction according to the three-dimensional coordinate data and the fourth moving speed.

[0109] In some embodiments, referring to Figure 9 , the above step S225 may include the following steps:

[0110] Step S2251: Obtain the state prediction equation, and fuse the first moving speed, the second moving speed and the third moving speed according to the state prediction equation.

[0111] Step S2252: Obtain the odometer of the target according to the fusion result, and obtain the fourth moving speed based on the odometer of the target.

[0112] Step S23: Reconstruct the three-dimensional contour of the target according to the movement amount in the target three-dimensional direction, the data collected by the microwave radar and each lidar.

[0113] One technical solution adopted in this application is: to provide a target three-dimensional reconstruction system, which includes: a gantry, where lidar and line array cameras are provided at the crossbeam, left and right columns of the gantry, and at the two included angles formed by the crossbeam and the left and right columns of the gantry; among them, a running track of the target is provided below the gantry; a plurality of wheel sensors are deployed on both sides of the running track; at least one microwave radar, which is deployed on the crossbeam of the gantry; a control module, connected to each lidar, microwave radar, each line array camera and each wheel sensor, and the control module is configured to: during the process of the target running on the running track, preprocess the data collected by each lidar and each line array camera to obtain the three-dimensional coordinate data of the target; calculate the movement amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; reconstruct the three-dimensional contour of the target according to the movement amount of the target in the three-dimensional direction, the data collected by the microwave radar and each lidar. In the above manner, the data collected by multiple sensors, fusing the results of independent speed measurement or fusion speed measurement of the microwave radar, wheel sensors, and lidar, can perform high-precision three-dimensional contour reconstruction and in-depth analysis on the moving target, improving the accuracy and reliability of reconstructing the three-dimensional contour of the target.

[0114] The specific implementation manners of the present invention are only examples, and those skilled in the art can make equivalent substitutions or optimizations for the sensor type, data fusion algorithm, etc. within the scope of the claims, and such variations all belong to the protection scope of the present invention. For example, the device implementation manners described above are only illustrative. For example, the division of the modules or units is only a logical function division, and there may be other division manners in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.

[0115] The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0116] In addition, the functional units in each embodiment of this application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above integrated units can be implemented in the form of hardware or in the form of software functional units.

[0117] The above are only the embodiments of the present application, and do not thus limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made by using the content of the specification and drawings of the present application, or directly or indirectly applied in other related technical fields, shall similarly be included within the patent protection scope of the present application.

Claims

1. A target three-dimensional reconstruction system, characterized in that: The target three-dimensional reconstruction system comprises: A gantry, wherein laser radars and linear array cameras are arranged at the crossbeam, left and right columns of the gantry and the two angles formed by the crossbeam and left and right columns of the gantry; wherein a running track of the target is arranged below the gantry; and a plurality of wheel sensors are deployed on both sides of the running track; at least one microwave radar, wherein the microwave radar is deployed on a crossbeam of the gantry; A control module is connected to each of the laser radars, the microwave radars, each of the linear array cameras and each of the wheel sensors, and is configured to: when the target is running on the running track, pre-process the data collected by each of the laser radars and each of the linear array cameras to obtain the three-dimensional coordinate data of the target; calculate the movement amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; and reconstruct the three-dimensional contour of the target according to the movement amount of the target in the three-dimensional direction, the data collected by the microwave radar and each of the laser radars; Among them, the control module is also configured to: match the data collected by each of the laser radars with a preset map; construct a third transformation matrix based on the matching results, and update the attitude trajectory of the laser radar based on the third transformation matrix and the acceleration and angular velocity of each of the laser radars within two scanning cycles; obtain the odometer corresponding to each of the laser radars according to the update results to obtain the odometer of the target and obtain the three-dimensional coordinate data of the target; obtain the first moving speed of the target based on the odometer of the target; calculate the movement amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the first moving speed.

2. The target three-dimensional reconstruction system according to claim 1, characterized in that: The control module is further configured to: The data collected by each of the laser radars and each of the linear array cameras are time synchronized; the posture information of each of the laser radars is obtained according to the acceleration and angular velocity of each of the laser radars within a preset scanning period, and the data collected by each of the laser radars is preprocessed according to the posture signal to obtain the three-dimensional coordinate data of the target; wherein the posture information includes the displacement information and rotation information of the laser radar within the preset scanning period; wherein the acceleration and the angular velocity are obtained by collecting the inertial measurement unit on the laser radar.

3. The target three-dimensional reconstruction system according to claim 2, characterized in that: The control module is further configured to: The acceleration and angular velocity of each of the laser radars in a single scanning cycle are respectively integrated to obtain the posture information; a first transformation matrix is ​​constructed based on the integration results, and each point in the first transformation matrix is ​​compensated to perform motion distortion correction processing on the data collected by each of the laser radars.

4. The target three-dimensional reconstruction system according to claim 2, characterized in that: The control module is further configured to: Determine a reference coordinate system, and calculate calibration parameters of each of the laser radars, the microwave radars, each of the linear array cameras, and each of the wheel sensors relative to the reference coordinate system; wherein the calibration parameters include a rotation matrix and a translation vector; Converting data collected by each of the laser radars, the microwave radars, each of the linear array cameras, and each of the wheel sensors into the reference coordinate system according to the calibration parameters; The acceleration and angular velocity of each laser radar in two scanning cycles are respectively integrated to obtain the posture information, and a second transformation matrix is ​​constructed according to the integration results to obtain the posture trajectory of the laser radar.

5. The target three-dimensional reconstruction system according to claim 1, characterized in that: The control module is further configured to: Acquire a second moving speed based on the data collected by the wheel sensor; Acquire a third moving speed based on the data collected by the microwave radar; fusing the first moving speed, the second moving speed and the third moving speed to obtain a fourth moving speed of the target; The movement amount of the target in the three-dimensional direction is calculated according to the three-dimensional coordinate data and the fourth movement speed.

6. The target three-dimensional reconstruction system according to claim 5, characterized in that: The control module is further configured to: A state prediction equation is obtained, and the first moving speed, the second moving speed, and the third moving speed are fused according to the state prediction equation; an odometer of the target is obtained according to the fusion result, and the fourth moving speed is obtained based on the odometer of the target.

7. A method for constructing a three-dimensional contour of a target, characterized in that: The target passes through the gantry during the running track, and the crossbeam, left and right columns of the gantry and the two angles formed by the crossbeam and left and right columns of the gantry are all provided with laser radars and linear array cameras; wherein the running track of the target is arranged below the gantry; a plurality of wheel sensors are arranged on both sides of the running track; at least one microwave radar is arranged on the crossbeam of the gantry; each of the laser radars, the microwave radars, each of the linear array cameras and each of the wheel sensors are respectively connected to a control module, and the method comprises: Preprocessing the data collected by each of the laser radars and each of the linear array cameras to obtain three-dimensional coordinate data of the target; Calculating the movement amount of the target in the three-dimensional direction according to the three-dimensional coordinate data and the moving speed of the target; Reconstructing a three-dimensional profile of the target according to the movement amount of the target in the three-dimensional direction and the data collected by the microwave radar and each of the laser radars; Wherein, the preprocessing of the data collected by each of the laser radars and each of the linear array cameras to obtain the three-dimensional coordinate data of the target includes: matching the data collected by each of the laser radars with a preset map; constructing a third transformation matrix according to the matching result, and updating the attitude trajectory of the laser radar according to the third transformation matrix and the acceleration and angular velocity of each of the laser radars in two scanning cycles; obtaining the odometer corresponding to each of the laser radars according to the update result to obtain the odometer of the target and obtain the three-dimensional coordinate data of the target; Calculating the movement amount of the target in the three-dimensional direction based on the three-dimensional coordinate data and the moving speed of the target includes: obtaining a first moving speed of the target based on the odometer of the target; and calculating the movement amount of the target in the three-dimensional direction based on the three-dimensional coordinate data and the first moving speed.

8. The method for constructing a three-dimensional contour of a target according to claim 7, characterized in that: The preprocessing of the data collected by each of the laser radars and each of the linear array cameras to obtain the three-dimensional coordinate data of the target includes: Performing time synchronization on the data collected by each of the laser radars and each of the linear array cameras; The posture information of each laser radar is obtained according to the acceleration and angular velocity of each laser radar within a preset scanning cycle, and the data collected by each laser radar is preprocessed according to the posture information to obtain the three-dimensional coordinate data of the target; wherein the posture information includes the displacement information and rotation information of the laser radar within the preset scanning cycle.

Images