Millimeter wave radar point cloud imaging network model automatic labeling method and device based on binocular camera

Abstract

The invention relates to an automatic marking method and device for a millimeter wave radar point cloud imaging network model based on a binocular camera, and belongs to the technical field of image processing. According to the method, image data are collected through a binocular camera, human body key points in an image are automatically identified by using a deep learning model, and then world coordinates of the key points are calculated. Meanwhile, the point cloud data collected by the millimeter wave radar are preprocessed and clustered to obtain high-quality point cloud data. And finally, fusing the camera key point coordinate data and the point cloud data into the same coordinate system, and performing registration by establishing a human skeleton model to realize automatic labeling. According to the method, the advantages of the binocular camera and the millimeter wave radar are fused, the precision and robustness of motion capture are improved, the labor cost is reduced through automatic labeling, the labeling efficiency is improved, and the method has a wide application prospect.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and relates to a method and a device for automatically labeling a millimeter wave radar point cloud imaging network model based on a binocular camera. Background Art

[0002] With the development of autonomous driving, robotics and other fields, millimeter wave radar and camera, as two important perception sensors, play an important role in target detection, tracking and recognition. Millimeter wave radar has the advantages of strong penetration and strong anti-interference ability, while camera has the advantages of high resolution and rich information. The combined application of millimeter wave radar and camera can give full play to the advantages of both sensors and improve the performance and reliability of the perception system.

[0003] The existing technologies mainly include the following technical solutions for the combined application of millimeter wave radar and camera:

[0004] Patent application number 202310718584.5: Through the joint calibration of external parameters (internal parameters are parameters related to the camera's own characteristics, such as the camera's focal length, pixel size, etc. The external parameters are the rotation and translation transformation relationship of the camera relative to the world coordinate system), the camera, lidar, and millimeter-wave radar rely on preset positions and feature targets to achieve precise transformation of multi-sensor external parameters, mainly solving the problem of coordinate system synchronization between multiple sensors.

[0005] Patent application number 202311121008.9: uses millimeter-wave radar, camera, and ultrasonic radar to integrate and perceive obstacles and sign line information in different areas, and dynamically updates the parking path, mainly to solve the problems of perception range and recognition ability during parking path planning.

[0006] Patent application number 202311215756.3: Candidate targets detected by millimeter-wave radar are screened through cameras and lidar for training classifiers, thereby achieving automatic elimination of false targets in multiple scenarios.

[0007] Patent application number 202311761613.2: By fusing the millimeter-wave radar sparse point cloud with a single-frame image, the problem of blur and detail loss during radar and image fusion is solved, and the depth estimation accuracy and stability under complex meteorological conditions are improved.

[0008] Although existing technologies have achieved certain results in the joint application of millimeter-wave radar and cameras, there are still some problems:

[0009] Lack of joint calibration of human motion information: Existing technologies focus more on the calibration of external parameters between sensors and the detection and tracking of static objects, but do not involve high-precision calculation and labeling of dynamic human motion information.

[0010] Insufficient tracking of dynamic targets: Existing technologies focus on the static information of objects or obstacles in target detection and classification, and lack accurate tracking and synchronization of dynamic human motion information.

[0011] Information distortion in multi-sensor fusion: Existing technologies may cause blurring and loss of details when fusing radar and image data, affecting annotation accuracy and consistency. Summary of the invention

[0012] In view of this, the object of the present invention is to provide a method and device for automatic annotation of millimeter-wave radar point cloud imaging network model based on a binocular camera.

[0013] In order to achieve the above object, the present invention provides the following technical solutions:

[0014] A method for automatically labeling a millimeter wave radar point cloud imaging network model based on a binocular camera, the method comprising the following steps:

[0015] Data collection and preprocessing, including:

[0016] Collect binocular camera image data in real time, and encode, display and save it;

[0017] Collect millimeter-wave radar point cloud data and remove low-confidence parts to reduce noise interference;

[0018] Apply clustering algorithm to cluster point cloud data to obtain high-quality point cloud data;

[0019] Human body key point recognition and coordinate calculation, including:

[0020] Call the human skeleton detection deep learning model to identify the key points of the human body in the image and obtain the pixel coordinates of the key points;

[0021] Calibrate the binocular camera to obtain the intrinsic parameter matrix and extrinsic parameter matrix;

[0022] Use the binocular matching algorithm to calculate the disparity map, and combine the internal and external parameter matrices to calculate the world coordinates of the key points;

[0023] Key point data smoothing and calibration, including:

[0024] Use the breadth-first search BFS algorithm to predict invalid disparity values ​​and fill in missing data;

[0025] Use Kalman filter algorithm to predict invalid key point coordinate data to ensure data continuity;

[0026] Estimate the horizontal, pitch and roll angles of the human head and upper limbs through the displacement changes of the pixel coordinates of the key points;

[0027] Use the rotation angle data to recalibrate the world coordinates of key points to reduce measurement errors;

[0028] Multi-sensor data fusion and automatic annotation, including:

[0029] The millimeter-wave radar point cloud data and the camera key point coordinate data are synchronized to the same coordinate system through rigid transformation;

[0030] Establish a human skeleton model and align the point cloud data with the skeleton coordinates;

[0031] By combining the point cloud with the skeleton coordinates to form a labeling matrix, automatic labeling is achieved.

[0032] Furthermore, the clustering algorithm is a DBSCAN algorithm.

[0033] Furthermore, the binocular matching algorithm is an SGBM algorithm.

[0034] Furthermore, the method for calculating the rotation angle comprises the following steps:

[0035] Calculate the difference between the left eye and the left ear, the left eye and the right eye, and the right eye and the right ear in the x-axis and y-axis directions, and perform smoothing;

[0036] Based on the difference ratio, the head horizontal angle, pitch angle and roll angle are determined using relevant mathematical relationships;

[0037] Cross-correct the angles of the head and upper limbs to reduce data oscillation.

[0038] Further, the rigid transformation comprises the following steps:

[0039] Measuring the difference between the camera device and the millimeter wave radar device in the z-axis direction;

[0040] Synchronize the camera key point coordinate data and point cloud data to the same coordinate system.

[0041] A device for automatically labeling a millimeter wave radar point cloud imaging network model based on a binocular camera, the device comprising:

[0042] The data acquisition module is responsible for collecting binocular camera image data and millimeter-wave radar point cloud data in real time;

[0043] The preprocessing module receives the raw data from the data acquisition module, performs preprocessing on it, including noise removal and clustering operations, and outputs the processed data to the key point recognition and coordinate calculation module;

[0044] The key point recognition and coordinate calculation module receives the data from the preprocessing module, uses the human skeleton detection deep learning model to recognize the key points of the human body in the image, and uses the binocular matching algorithm and the coordinate calculation module to calculate the world coordinates of the key points;

[0045] The key point data smoothing and calibration module receives the key point coordinate data from the key point identification and coordinate calculation module, and uses the BFS algorithm, Kalman filter algorithm and angle estimation module to smooth and calibrate the key point data;

[0046] The multi-sensor data fusion module receives the key point coordinate data from the key point data smoothing and calibration module and the point cloud data from the data acquisition module, fuses the two into the same coordinate system using the rigid transformation module, and performs registration using the registration module;

[0047] The automatic labeling module receives the fusion data from the multi-sensor data fusion module, generates a labeling matrix using the labeling matrix generation module, and realizes automatic labeling;

[0048] The result output module receives the annotation matrix from the automatic annotation module and outputs it to an external device or storage device.

[0049] Furthermore, the preprocessing module includes a clustering algorithm module, and the clustering algorithm is a DBSCAN algorithm.

[0050] The beneficial effects of the present invention are:

[0051] 1. Real-time image acquisition, automatic key point recognition and spatial coordinate calculation

[0052] The present invention proposes a technical route for automatic key point spatial coordinate calculation based on images, which can collect images in real time and automatically identify the key points of the human body in the images through a deep learning model. On this basis, the world coordinates of the key points are automatically obtained. Although image acquisition and key point recognition are existing technologies, the present invention combines them in a complete system to achieve efficient fusion of multi-sensor data.

[0053] 2. Invalid key point prediction with Breadth First Search (BFS) and Kalman Filter

[0054] The present invention proposes to use the BFS algorithm to effectively predict invalid key point disparity values ​​within a single frame. Furthermore, the Kalman filter algorithm is used to predict invalid key point coordinates between frames to ensure that the key point coordinates in the time dimension are always valid, thereby achieving smooth and continuous key point data.

[0055] 3. Three-DOF human motion angle estimation and key point calibration

[0056] The present invention estimates the rotation angles of the three degrees of freedom of the human head and upper limbs, namely, horizontal, pitch and roll, through the displacement changes of the pixel coordinates of key points between frames. This technology can not only provide detailed human motion information, but also calibrate the spatial coordinates of key points through the rotation angle, reduce measurement errors, and make the presentation of the human skeleton in three-dimensional space more accurate and natural.

[0057] 4. Synchronization and automatic annotation of radar point cloud and key point coordinates

[0058] This paper proposes a technical route to perform high-quality clustering of point clouds collected by millimeter-wave radar using the DBSCAN clustering algorithm (determining parameters through K-distance). Then, the key point coordinates obtained by the camera side and the radar point cloud data are synchronized to the same coordinate system through rigid transformation, realizing automatic annotation of radar point clouds.

[0059] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0061] Figure 1 It is the system flow chart;

[0062] Figure 2 It is a device for camera data collection and radar automatic labeling;

[0063] Figure 3 is the conversion relationship between the world coordinate system and the pixel coordinate system;

[0064] Figure 4 is the parallax relationship of the binocular camera’s view to the same target;

[0065] Figure 5 Calculate the horizontal angle diagram;

[0066] Figure 6 Calculate the pitch angle diagram;

[0067] Figure 7 Calculation diagram of the roll angle. DETAILED DESCRIPTION

[0068] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0069] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0070] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0071] 1. System architecture and overall process Figure 1 shown.

[0072] The low-confidence parts of the input millimeter-wave radar point cloud data are removed to reduce the impact of radar echo noise; the clustering algorithm is applied to further suppress the noise and obtain high-quality point cloud data.

[0073] At the same time, binocular camera data is collected and sent to a computer, where an algorithm is used to encode the two-way image data for real-time display and storage.

[0074] Call the human skeleton detection deep learning model to obtain the data of each human key point in the image.

[0075] Calibrate the binocular camera to achieve image distortion correction and obtain internal and external parameters.

[0076] Use binocular matching and reprojection algorithm to obtain the world coordinates corresponding to the joint points.

[0077] The data is smoothed by the Kalman filter algorithm, and valid data is predicted and filled in at the invalid data point locations.

[0078] The horizontal angle, pitch angle and roll angle of the human body are estimated by smoothing the lateral and longitudinal differences of different joints in the image. Then the world coordinates of each key point are recalibrated using the rotation angle data.

[0079] The radar point cloud and camera joint point coordinates are synchronized to the same coordinate system through rigid transformation, and then the human skeleton is established. By strictly aligning the spatial positions of the key points of the human skeleton, the skeleton framework can "envelope" or "cover" the clustered point cloud data in three-dimensional space. Finally, by combining the point cloud and skeleton coordinates to form a labeling matrix, automatic labeling is achieved.

[0080] 2. Binocular Vision and Human Motion Information Extraction

[0081] The present invention mainly proposes a new technical route: using a binocular camera element to collect images in real time, and using a human skeleton detection deep learning model to automatically identify multiple human key points in the image, and automatically obtain the world coordinates (spatial coordinates) corresponding to the key point positions in the image. However, the technology of each step itself is used in existing research. The schematic diagram of the relevant device is shown in the figure below. Figure 2 shown.

[0082] (1) Data collection by binocular camera

[0083] This solution collects the image data of the binocular camera in real time through an external input source and transmits it to the computer. Through the computer-side algorithm, the dual-channel image data is encoded, displayed and saved in real time to ensure the efficiency and synchronization of data collection, providing a basis for subsequent image processing and analysis.

[0084] (2) Human skeleton key point recognition

[0085] The present invention calls the human skeleton detection deep learning model to identify the key point positions of the human skeleton in the image in real time. Since the position is actually the pixel point position in the image, the coordinate system can be identified as a pixel coordinate system, with the horizontal coordinate marked as u and the vertical coordinate marked as v. The key points include the following positions: left ear, right ear, nose, left eye, right eye, left shoulder, right shoulder, left hip, right hip, left arm, right arm, left knee and right knee.

[0086] (3) Distortion correction processing of binocular camera

[0087] The present invention calibrates the internal and external parameter matrices of the camera and corrects the distortion to ensure the accuracy and consistency of the image data. In order to complete this process, a method based on chessboard calibration is adopted. By shooting multiple groups of chessboard images, the camera's internal parameter matrix (including focal length and optical center) and external parameter matrix (the transformation relationship between the camera and the world coordinate system) can be effectively obtained, and the radial and tangential distortion parameters of the image can be calculated at the same time.

[0088] (4) Binocular matching and key point world coordinate calculation

[0089] The mapping from pixel coordinates to world coordinates involves three coordinate system transformations:

[0090] Pixel coordinate system: The upper left corner of the camera image is used as the coordinate origin. The number of pixels on the long side and the number of pixels on the short side of the object on the image are used as coordinates. The unit is pixel. The following is notated as (u, v).

[0091] Camera coordinate system: The center of the camera image is used as the coordinate origin. The coordinates of the actual scene are described according to the camera's viewing angle. The unit is the international unit. The present invention sets it to millimeters. c ,y c ,z c ) logo.

[0092] World coordinate system: artificially defined, in the present invention, the origin of the world coordinate system is set at the center of the world plane where the calibration plate is located. The difference from the camera coordinate system is only the relative position of the depth coordinate. The unit is the international unit, which is set to millimeters in the present invention. The following is based on (x w ,y w ,z w ) logo.

[0093] 1) Transformation relationship from world coordinate system to pixel coordinate system

[0094] Set the center of the plane where the pixel coordinate system is located as the center of the image, and establish a two-dimensional coordinate system, set as xOy, then the conversion relationship between the world coordinate system and the pixel coordinate system is as follows: Figure 3 As shown:

[0095] The mathematical relationship between them satisfies:

[0096]

[0097] Among them, the matrix:

[0098]

[0099] That is the internal parameter matrix:

[0100]

[0101] It is the external parameter matrix, which is composed of the rotation matrix R and the translation matrix T.

[0102] Calculate the unit normal vector of the chessboard in the camera coordinate system and the distance from the origin of the camera coordinate system to the chessboard, and obtain the optimal intrinsic parameter matrix and extrinsic parameter matrix corresponding to the depth position by mean processing or comparing the minimum reprojection error.

[0103] If you

[0104]

[0105] Then, when the internal and external parameter matrices, depth coordinates, and image pixel coordinates are known, we have

[0106]

[0107] At this time w =z c .

[0108] 2) Binocular matching and calculation of depth z c

[0109] From formula (3), we can know that if we want to know the spatial position (x w ,y w ,z w ) then we must first find z w Suppose there is a point P in space, whose projections on the first camera and the second camera are as follows: Figure 4 shown.

[0110] According to the principle of similar triangles:

[0111]

[0112] Where f is the optical focal length of the camera, in pixels per millimeter; b is the baseline distance of the binocular cameras, that is, the physical distance between the binocular cameras, in millimeters. So, the depth z = z c =z w for

[0113]

[0114] Where d represents the binocular disparity pixel value. Now let's consider how to obtain d.

[0115] The essential task of binocular matching is to find the position corresponding to the specified point in the first camera from the second camera. The present invention adopts SGBM (Semi-Global Block Matching) to perform binocular matching. SGBM first performs block matching in the local area, that is, a fixed-size block (window) is selected in the left and right images, and the best matching block for each pixel is found by traversing different disparity values. The matching metric is the similarity between the two blocks. In order to overcome the errors that may be caused by pure local block matching (such as insufficient texture, occlusion, etc.), SGBM is optimized on a global scale. It uses a semi-global method, that is, consistency constraints are performed in multiple directions of the image (horizontal, vertical, diagonal) to minimize the situation where the disparity changes are too large.

[0116] Finally, SGBM will output a disparity map for each pixel. The disparity value reflects the horizontal displacement d of the pixel in the left and right images. According to the binocular geometric relationship of formula (5), the depth information of the object can be estimated. Combining the internal and external parameters of the camera and the disparity value, as shown in formula (3), the world coordinates (spatial coordinates) corresponding to the specified image position can be calculated.

[0117] 3. Smoothing and calibration of human motion data

[0118] In this section, the present invention proposes a new technical route: using the Breadth First Search (BFS) algorithm to effectively predict the invalid key point disparity values ​​within the frame; using the Kalman filter algorithm to effectively predict the invalid key point coordinate data between frames to ensure that the key point (x w ,y w ,z w ) have valid values.

[0119] At the same time, the present invention also proposes a new technology: to estimate the rotation angles of the three degrees of freedom of the human head and upper limbs, namely, horizontal, pitch and roll, by the displacement of the pixel coordinates of the key points between frames. And this technology is used to assist in calibrating the spatial coordinates of the key points (x w ,y w ,z w ), which can suppress the influence of measurement errors and make the drawn human skeleton present a better human body shape in the three-dimensional image.

[0120] (1) Data prediction processing of invalid disparity values ​​using breadth-first search algorithm

[0121] In the SGBM algorithm, since the algorithm performs image matching based on the energy method, it may be affected by light conditions and color, which may cause some areas to fail to match appropriate points (for example, when the light is dark or the object is dark in color), thereby generating invalid disparity values, which are usually manifested as infinite disparity values. In order to solve this problem, the present invention introduces the BFS algorithm to process invalid disparity values.

[0122] BFS is a classic graph search algorithm. Its basic idea is to start from the starting point, expand outward layer by layer, and give priority to visiting the nodes closest to the starting point. Therefore, in the case of invalid disparity values, BFS will search for nearby points with valid disparity values ​​in the depth map generated by SGBM. Since the BFS algorithm always gives priority to visiting the nodes closest to the starting node, the new matching points found are usually located near the original invalid points, ensuring the reliability and accuracy of disparity compensation. In this way, the invalid disparity data generated by the SGBM algorithm in specific scenarios can be effectively filled, further improving the quality of stereo matching.

[0123] (2) Kalman filter data prediction processing for invalid frames

[0124] When the deep learning model detects key points of the human body in an image, it may take prediction processing for key points outside the image, which will cause the output pixel coordinates of the corresponding key points to exceed the image range. In this case, applying formula (3) will result in invalid output values.

[0125] However, the displacement of the same key point must be continuous in time (i.e., between frames), so the present invention uses the Kalman filter algorithm to predict the value of the next frame of invalid data for the time series based on the two-dimensional state variables of the coordinates and velocity information of the previous frame. The Kalman filter algorithm is a classic recursive estimation algorithm, which is commonly used for smoothing and predicting time series data. The algorithm is based on a dynamic model of the system state and estimates the state at the current moment by combining the state information of the previous frame (such as position and velocity). Through the Kalman filter algorithm, the present invention can smooth and compensate for invalid key point coordinates between frames, ensure the continuity and accuracy of the key point position in the time dimension, and avoid the problem of invalid key point coordinates caused by inaccurate predictions of deep learning models.

[0126] (3) Angle estimation of key points

[0127] For the estimation of the rotation angle of the head and body, the entire algorithm flow is as follows:

[0128] 1) By calculating the difference between the left eye and the left ear, the left eye and the right eye, and the right eye and the right ear in the x-axis and y-axis directions, the algorithm is applied to smooth the time series;

[0129] 2) Based on the above difference ratio, three sets of angle data θ1, θ2, θ3 are determined using relevant mathematical relationships, such as Figure 5 As shown;

[0130] 3) Since the output range of the inverse trigonometric function is 0°-180°, the algorithm expands it to 0°-359° by splitting the angle. The angle calculated in each frame is compared with the previous frame, and the angle with the smallest difference is selected as the horizontal angle of the frame;

[0131] 4) Head pitch angle γ and roll angle There is no need to deal with the problem of spanning 180°. The inverse trigonometric function is used to solve the problem directly, and the calculation results of the ears and eyes on both sides are weighted averaged to reduce the error. The measurement principle is as follows: Figure 6 and Figure 7 As shown;

[0132] 5) Cross-correct the head's roll angle and horizontal angle to reduce the angle oscillation near 0° and 180°;

[0133] 6) Calculate the difference between the x-axis and y-axis directions of the key points of the shoulder and hip, and smooth the time series;

[0134] 7) Similar to the horizontal angle calculation of the head, the horizontal rotation angle data of the upper limbs is obtained by inverse trigonometric functions, and the problems spanning 179° and 180° are processed;

[0135] 8) The roll angle and pitch angle of the upper limb are directly obtained by inverse trigonometric functions, and the angles on the left and right sides are weighted averaged to obtain the final result;

[0136] 9) The algorithm cross-checks the angles of the head and upper limbs to ensure that the relative angles between the two do not exceed, thereby further reducing data oscillation. Finally, the horizontal angle, pitch angle, and roll angle of the head and upper limbs are output and arranged into a matrix according to the number of frames.

[0137] (4) Recalibration of key points based on rotation angle data

[0138] In step (3), the preliminary world coordinate data of the key points are obtained, but there is a large measurement error in the depth direction. Using the angle information, the corresponding world coordinate data constraints of the key points can be established: the center of the upper limb is used as the main point, and the center of the head is used as the secondary point, where the center coordinates are obtained by weighted average of each key point. If the center of the head is far from the vertical line passing through the center of the upper limb, it is calibrated to the vertical line. Then, based on the position of the main point and the upper limb angle data of the current frame, the coordinates of the left shoulder, right shoulder, left hip, and right hip are recalibrated; based on the position of the secondary point and the head angle data of the current frame, the coordinates of the left ear and right ear are recalibrated.

[0139] 4. Radar point cloud data processing and optimization

[0140] The millimeter wave radar outputs the point cloud data directly to the computer side. By encoding the data and filtering out the point cloud data with low signal-to-noise ratio, the interference of radar echo noise can be reduced, and then the DBSCAN algorithm is applied to the point cloud data to cluster the point cloud into clusters, further reducing the noise. At this time, high-quality human motion point cloud data can be obtained. Based on the distance matrix between points, the present invention uses the inflection point of the K-distance graph to automatically estimate the key parameters in the DBSCAN clustering algorithm, so that the point cloud data can be effectively clustered.

[0141] 5. Multi-sensor data fusion and automatic labeling

[0142] The point cloud data obtained in step 4 is in the world coordinate system of Euclidean coordinates, with the unit of millimeter; and the key point coordinate data obtained in step (4) of step 3 is also in the world coordinate system of Euclidean coordinates, with the unit of millimeter. Therefore, when the camera device and the millimeter-wave radar device are placed vertically and horizontally (that is, the origin in the z direction is the same), the origins of the two coordinate systems differ only in the y direction. diff By measuring multiple sets of data, y can be determined diff , the two sets of coordinate data can be integrated into the same world coordinate system.

[0143] The human skeleton drawn with the human key point data calibrated in step 3 will not have sudden posture changes or misalignments. By strictly aligning the spatial positions of the key points of the human skeleton, the skeleton framework can "envelope" or "cover" the clustered point cloud data in three-dimensional space. Specifically, the spatial region where each key point (such as the head, torso, limbs, etc.) is located will impose constraints on its corresponding point cloud cluster, so that the point cloud in the region can be accurately classified into the corresponding human body part.

[0144] This fusion method of skeleton and point cloud can ensure the accurate matching of point cloud data with each key point of the human body, thus achieving complete coverage of point clouds of various parts of the human body. Finally, by combining the point cloud with the skeleton coordinates, a labeling matrix is ​​formed, which not only contains the coordinate information of each key part of the human body, but also provides automatic labels for each point cloud cluster, accurately identifying the key points and parts to which each point cloud point belongs. In this way, the fusion process of point cloud data and skeleton key points effectively improves the accuracy of labeling and the interpretability of data.

[0145] 1. Accuracy and stability of data annotation

[0146] The accuracy of point cloud data annotation in the prior art is limited by simple geometric alignment methods and lacks multi-dimensional calibration, especially in human posture estimation and skeleton annotation, which makes it difficult to ensure the precise alignment of key points. The present invention achieves high-precision annotation through multi-dimensional key point calibration and posture estimation. The depth information is calculated by combining the binocular matching algorithm (SGBM) with the reprojection matrix, the key point data is smoothed using Kalman filtering, and the data is further calibrated in combination with the angle estimation of the human posture (horizontal, pitch and roll angles of the head and body), which significantly reduces the measurement noise and instability. The present invention can effectively reduce the fluctuation and noise of key point coordinates, ensure the accurate fusion of skeleton and point cloud data, and provide more stable and accurate annotation results.

[0147] 2. Improvement of automatic labeling accuracy

[0148] The prior art usually annotates point clouds and camera data manually or semi-automatically, especially in dynamic scenes, where it is difficult to ensure consistency and accuracy with manual annotation. The present invention automatically integrates the key points of the human skeleton acquired by the binocular camera with the radar point cloud data, and can automatically align the point cloud data with the human skeleton in real time and accurately annotate. The key points of the image identified by the deep learning model are used as a reference to reduce the system error in the annotation process. This solution can significantly improve the automation level and accuracy of the annotation of point clouds and human motion data, is suitable for real-time scenes, and improves the annotation efficiency and consistency of the system.

[0149] 3. High compatibility and high data processing efficiency of multi-sensor fusion

[0150] When processing multi-sensor data in complex scenes, the existing technology usually faces the problems of low processing efficiency and poor environmental adaptability, especially the lack of real-time data processing capabilities in dynamic scenes. The present invention optimizes the processing flow of the overall data flow, adopts efficient data acquisition and transmission technology, and combines the algorithm architecture of multi-sensor data fusion to greatly improve the processing speed and real-time performance of the system. The system realizes the seamless fusion of image sensor and millimeter wave radar data, and synchronizes the coordinate system using rigid transformation between multi-source sensors. By suppressing the noise and clustering the radar point cloud, and combining it with the key point data of the camera, the system can maintain high stability in complex environments. In addition, based on a unified algorithm interface, the system can flexibly call different data processing modules and show a high degree of compatibility in different hardware environments. This system architecture not only improves the efficiency of real-time annotation, but also has a strong scene adaptability, so that it still performs well in dynamic or harsh environments, ensuring the stability and reliability of data annotation. .

[0151] 4. System versatility and scalability

[0152] Systems in the prior art are often designed for specific tasks or scenarios, lacking flexibility and difficult to adapt to different application requirements. The present invention is compatible with a variety of sensor data inputs through a modular design and can adapt to a variety of application scenarios. For example, by collecting camera data and radar point clouds through FPGA, the system can be expanded to other types of sensors (such as lidar, depth camera, etc.); at the same time, the software algorithm also has good compatibility, including but not limited to MATLAB and Python implementations, and can be flexibly adjusted according to different task requirements. The system has high versatility and scalability, can be applied to automatic labeling needs in a variety of scenarios, and provides guarantees for future functional expansion and system upgrades.

[0153] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for automatically labeling a millimeter-wave radar point cloud imaging network model based on a binocular camera, characterized in that: The method comprises the following steps: Data collection and preprocessing, including: Collect binocular camera image data in real time, and encode, display and save it; Collect millimeter-wave radar point cloud data and remove low-confidence parts to reduce noise interference; Apply clustering algorithm to cluster point cloud data to obtain high-quality point cloud data; Human body key point recognition and coordinate calculation, including: Call the human skeleton detection deep learning model to identify the key points of the human body in the image and obtain the pixel coordinates of the key points; Calibrate the binocular camera to obtain the intrinsic parameter matrix and extrinsic parameter matrix; Use the binocular matching algorithm to calculate the disparity map, and combine the internal and external parameter matrices to calculate the world coordinates of the key points; Key point data smoothing and calibration, including: Use the breadth-first search BFS algorithm to predict invalid disparity values ​​and fill in missing data; Use Kalman filter algorithm to predict invalid key point coordinate data to ensure data continuity; Estimate the horizontal, pitch and roll angles of the human head and upper limbs through the displacement changes of the pixel coordinates of the key points; Use the rotation angle data to recalibrate the world coordinates of key points to reduce measurement errors; Multi-sensor data fusion and automatic annotation, including: The millimeter-wave radar point cloud data and the camera key point coordinate data are synchronized to the same coordinate system through rigid transformation; Establish a human skeleton model and align the point cloud data with the skeleton coordinates; By combining the point cloud with the skeleton coordinates to form a labeling matrix, automatic labeling is achieved.

2. The method for automatic annotation of a millimeter wave radar point cloud imaging network model based on a binocular camera according to claim 1 is characterized in that: The clustering algorithm is the DBSCAN algorithm.

3. The method for automatic annotation of a millimeter wave radar point cloud imaging network model based on a binocular camera according to claim 1 is characterized in that: The binocular matching algorithm is the SGBM algorithm.

4. The method for automatic annotation of a millimeter wave radar point cloud imaging network model based on a binocular camera according to claim 1 is characterized in that: The method for calculating the rotation angle comprises the following steps: Calculate the difference between the left eye and the left ear, the left eye and the right eye, and the right eye and the right ear in the x-axis and y-axis directions, and perform smoothing; Based on the difference ratio, the head horizontal angle, pitch angle and roll angle are determined using relevant mathematical relationships; Cross-correct the angles of the head and upper limbs to reduce data oscillation.

5. The method for automatic annotation of a network model of a millimeter wave radar point cloud imaging based on a binocular camera according to claim 1, characterized in that: The rigid transformation comprises the following steps: Measuring the difference between the camera device and the millimeter wave radar device in the z-axis direction; Synchronize the camera key point coordinate data and point cloud data to the same coordinate system.

6. A device for automatically labeling a millimeter wave radar point cloud imaging network model based on a binocular camera, characterized in that: The device includes: The data acquisition module is responsible for collecting binocular camera image data and millimeter-wave radar point cloud data in real time; The preprocessing module receives the raw data from the data acquisition module, performs preprocessing on it, including noise removal and clustering operations, and outputs the processed data to the key point recognition and coordinate calculation module; The key point recognition and coordinate calculation module receives the data from the preprocessing module, uses the human skeleton detection deep learning model to recognize the key points of the human body in the image, and uses the binocular matching algorithm and the coordinate calculation module to calculate the world coordinates of the key points; The key point data smoothing and calibration module receives the key point coordinate data from the key point identification and coordinate calculation module, and uses the BFS algorithm, Kalman filter algorithm and angle estimation module to smooth and calibrate the key point data; The multi-sensor data fusion module receives the key point coordinate data from the key point data smoothing and calibration module and the point cloud data from the data acquisition module, fuses the two into the same coordinate system using the rigid transformation module, and performs registration using the registration module; The automatic labeling module receives the fusion data from the multi-sensor data fusion module, generates a labeling matrix using the labeling matrix generation module, and realizes automatic labeling; The result output module receives the annotation matrix from the automatic annotation module and outputs it to an external device or storage device.

7. The device for automatically labeling network models of millimeter wave radar point cloud imaging based on binocular cameras according to claim 6 is characterized in that: The preprocessing module includes a clustering algorithm module, and the clustering algorithm is a DBSCAN algorithm.

Images