Aircraft dragging risk early warning method based on head-mounted display equipment and laser radar

Abstract

The invention discloses an aircraft dragging risk early warning method based on head-mounted display equipment and a laser radar. The method comprises the following steps: establishing an aircraft dragging process risk early warning system; calibrating an RGB camera and a laser radar; training the two-dimensional target detection model to realize two-dimensional detection; determining a target point cloud and an obstacle point cloud; determining a target collision area point cloud and a safety distance; determining the curved surface type of the risk surface; correcting the risk patterns in a partitioned manner according to the curved surface type; generating a grid; and fitting the corrected risk image to the grid, and the like. According to the aircraft dragging risk early warning method based on the head-mounted display equipment and the laser radar, a system which takes the head-mounted display equipment as a carrier, integrates multiple sensors such as the laser radar, an RGB camera and a TOF camera and integrates multiple technologies such as augmented reality, deep learning, point cloud processing and distortion correction is designed; the method has stronger visibility, real-time performance and accuracy, and can be used for monitoring and early warning various risks in a complex environment.

Description

Technical Field

[0001] The present invention belongs to the technical field of collision risk detection at complex aircraft maintenance sites, and specifically relates to an aircraft dragging risk warning method based on a head-mounted display device and a laser radar. Background Art

[0002] During repair and maintenance, aircraft are frequently towed between maintenance areas. Due to the complex environment within aircraft repair shops, limited space, and the dense distribution of obstacles such as maintenance equipment, ground tractors, and other parked aircraft, the risk of collision during towing is high. Especially in confined areas or with limited visibility, aircraft parts such as wings, tails, and engine covers are prone to scratching against surrounding equipment or structures, resulting in significant safety risks and financial losses.

[0003] Currently, the industry primarily relies on manual visual inspection to monitor collision risks during aircraft towing. Typically, six to eight ground crew members closely follow the aircraft during towing, visually observing its relative position to surrounding obstacles and directing the towing vehicle to adjust its path via intercom or hand gestures. This method not only relies on the staff's experience and judgment, which is subject to significant subjective errors, but also makes it difficult to detect potential collision risks in certain situations with limited vision. Furthermore, prolonged, high-intensity monitoring can easily lead to fatigue, further reducing detection reliability. Therefore, this traditional manual monitoring method no longer meets the needs of modern aviation maintenance. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide an aircraft dragging risk warning method based on a head-mounted display device and a laser radar.

[0005] To achieve the above-mentioned objectives, the present invention provides an aircraft dragging risk warning method based on a head-mounted display device and a laser radar, comprising the following steps performed in sequence:

[0006] 1) Establish an aircraft towing process risk warning system based on head-mounted display devices and lidar;

[0007] 2) Calibrate the RGB camera and lidar in the aircraft towing process risk warning system based on the head-mounted display device and lidar to obtain the intrinsic and extrinsic parameters of the RGB camera and the extrinsic parameters of the lidar;

[0008] 3) Collecting original target images, screening them, and annotating them. All annotated target images form a target image sample dataset and divide it into a training set and a test set in proportion. Build an original two-dimensional target detection model and train and evaluate it using the training set and test set respectively to obtain the final two-dimensional target detection model.

[0009] 4) Using the calibrated RGB camera in the head-mounted display device to continuously capture a series of real-time images of the target to be detected and input them into the final two-dimensional target detection model obtained in step 3), the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; using the calibrated lidar to capture the point cloud of the target to be detected in real time, combining the RGB camera intrinsic parameters and the RGB camera and lidar extrinsic parameters obtained by calibration in step 2), pre-processing the target point cloud corresponding to the target image at the same time, including depth screening, camera field of view screening, and ground point cloud removal, then using the above-mentioned two-dimensional detection frame of the target to be detected combined with the intrinsic parameters of the calibrated RGB camera to calculate the target depth, construct a three-dimensional point cloud viewing cone based on the target depth, and then cluster and segment the pre-processed target point cloud based on the three-dimensional point cloud viewing cone to determine the target point cloud and obstacle point cloud, thereby completing three-dimensional target detection;

[0010] 5) Find the closest point pair between the target point cloud and the obstacle point cloud identified in step 4). The distance between the point pairs is the actual distance between the aircraft surface and the obstacle. Then, the target point cloud is magnified along the normal direction. Within the collision detection area, the bounding box detection method is first used to perform a rough collision detection on the target point cloud and the obstacle point cloud. If the bounding box is judged to have collided, the voxel detection method is then used to perform a precise collision detection. In this way, the target collision area point cloud, collision risk point set, and area are determined.

[0011] 6) Using the target point cloud obtained in step 4), calculate the principal curvature, Gaussian curvature, and mean curvature of the surface where the collision risk area is located, thereby determining the surface type of the surface where the collision risk area is located. Then, based on its geometric characteristics, divide the surface where the collision risk area is located into five different planes;

[0012] 7) Calculate the convex hull of the collision risk point set obtained in step 5). Then, project the convex hull onto a normalized plane based on the lidar extrinsic parameters obtained by calibration in step 2) to obtain the approximate shape of the collision risk area. Based on the collision risk area surfaces of different surface types obtained in step 6), perform corresponding partitioning, making each partition as close to a small plane as possible. Perform perspective transformation correction on each partition separately, and then stitch the corrected partitions together to ultimately generate a complete warning area image.

[0013] 8) Using the TOF camera in the head-mounted display device to collect the TOF point cloud, then downsampling the TOF point cloud based on geometric features and generating a mesh to ensure that the projected image can match the geometry of the real environment;

[0014] 9) Using the internal and external parameters of the lidar and RGB camera obtained in step 2), the collision risk point set obtained in step 5) is converted to the TOF camera coordinate system to obtain the correct display position of the warning area image in the head-mounted display device; then, the warning area image containing the red pattern and the actual distance digital pattern obtained in step 7) is fitted into the grid generated in step 8) to achieve the effect that the staff can directly see the warning area image overlaid on the real collision risk area in the augmented reality environment of the head-mounted display device.

[0015] 2. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: in step 1), the aircraft dragging process risk warning system based on a head-mounted display device and a laser radar comprises a head-mounted display device and a laser radar; wherein the head-mounted display device comprises a headband, an industrial computer, an LCD display screen, an RGB camera, and a depth camera; the two LCD display screens are respectively mounted on both sides of the front side of the headband; the RGB camera and the depth camera are mounted in the middle of the front side of the headband; the industrial computer is mounted on the headband and is electrically connected to the LCD display screen, the RGB camera, and the depth camera respectively, and is also connected to the laser radar mounted on the upper end of the headband via an Ethernet interface;

[0016] The head-mounted display device uses Microsoft HoloLens2, which is a head-mounted device for augmented reality and mixed reality; the laser radar is a point cloud acquisition device, using a 96-line Airy laser radar with a diameter of 60mm and a weight of 230g.

[0017] In step 2), the method for calibrating the RGB camera and the lidar in the aircraft towing process risk warning system based on the head-mounted display device and the lidar to obtain the intrinsic and extrinsic parameters of the RGB camera and the extrinsic parameters of the lidar is:

[0018] The staff wore a head-mounted display device equipped with a lidar. First, they used a checkerboard calibration plate. Under the control of an industrial computer, they used an RGB camera to collect images of the checkerboard calibration plate at different angles and distances through the principle of homography matrix mapping. Combined with the corner point positions in the image, the internal parameters of the RGB camera, including focal length, principal point position, and distortion coefficient, were calculated through a nonlinear optimization algorithm. Then, the RGB camera and lidar with calibrated internal parameters were used to collect multiple sets of images and point cloud data of the checkerboard calibration plate at different angles and distances. Then, based on the correspondence between the checkerboard corner points, the rotation matrix and translation vector of the two were calculated as external parameters to obtain the spatial position relationship between the RGB camera and the lidar.

[0019] In step 3), the original target images are collected and screened and annotated, and a target image sample data set is formed by all the annotated target images and divided into a training set and a test set in proportion; an original two-dimensional target detection model is built and trained and evaluated using the training set and the test set respectively, and the method for obtaining the final two-dimensional target detection model is:

[0020] Use any camera to capture original target images at different angles and depths, including aircraft wings, tail fins, and engines. Then, filter the original target images to retain high-quality images. Then, use the labellmg.exe annotation method to manually mark the target locations in square annotation boxes on the filtered target images to obtain labeled target images. A target image sample dataset is constructed from all labeled target images, which is then divided into a training set and a test set in an 8:2 ratio.

[0021] In the industrial computer, the deep learning YOLOv9 algorithm model is used to build the original two-dimensional target detection model, and then the above training set is input into the original two-dimensional target detection model for training. By iteratively optimizing the loss function, the model performance is continuously adjusted. After that, the test set is input into the trained two-dimensional target detection model. The accuracy of the two-dimensional target detection model is evaluated by calculating the detection accuracy and average precision. Finally, the two-dimensional target detection model is optimized according to the evaluation results, and the two-dimensional target detection model with the best detection performance is selected as the final two-dimensional target detection model.

[0022] In step 4), the calibrated RGB camera in the head-mounted display device is used to continuously capture a series of real-time images of the target to be detected and input them into the final two-dimensional target detection model obtained in step 3), and the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; the calibrated lidar is used to capture the point cloud of the target to be detected in real time, and the RGB camera intrinsic parameters and the RGB camera and lidar extrinsic parameters obtained by calibration in step 2) are combined to perform preprocessing including depth screening, camera field of view screening, and ground point cloud removal on the point cloud of the target to be detected at the same time corresponding to the target image to be detected. Then, the target depth is calculated using the above-mentioned two-dimensional detection frame of the target to be detected in combination with the intrinsic parameters of the calibrated RGB camera, and a three-dimensional point cloud viewing cone is constructed based on the target depth. Then, the preprocessed point cloud of the target to be detected is clustered and segmented based on the three-dimensional point cloud viewing cone to determine the target point cloud and the obstacle point cloud. The method for completing three-dimensional target detection is as follows:

[0023] A series of real-time images of the target to be detected are continuously captured using the calibrated RGB camera in the head-mounted display device and input into the final two-dimensional target detection model obtained in step 3), and the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; a point cloud of the target to be detected is captured in real time using the calibrated lidar in step 2) and input into the industrial computer, a depth threshold of the target region of interest is set, and point clouds exceeding the depth threshold range are eliminated to obtain a depth-filtered target point cloud; combined with the internal parameters of the RGB camera obtained in step 2), the depth-filtered target point cloud is projected onto the RGB camera image plane, and valid pixels within the field of view of the RGB camera are extracted to obtain a target point cloud within the field of view of the RGB camera; finally, a RANSAC algorithm is used to fit the ground point cloud and eliminate it, thereby achieving preprocessing of the target point cloud to be detected;

[0024] Then use the intrinsic parameters of the RGB camera obtained in step 2):

[0025]

[0026] Where dx is the pixel size of the RGB camera, which is obtained from the factory parameters of the RGB camera. Let f / dx be f x , calculate the camera focal length f by the following formula:

[0027] f=f x ·dx

[0028] Then, the horizontal pixel length of the two-dimensional detection frame of the target to be detected obtained in step 4) is recorded as the pixel width w, and the image width W of the target to be detected is expressed as follows:

[0029] W=w·dx

[0030] The actual length of the target to be detected is measured using a laser rangefinder and used as its prior length L. Based on the camera imaging principle and the relationship between the camera focal length f, image width W, and pixel size dx, the target depth calculation formula is obtained:

[0031]

[0032] Using the intrinsic and extrinsic parameters of the RGB camera obtained in step 2), the above-mentioned two-dimensional detection frame of the target to be detected and the preprocessed point cloud of the target to be detected are converted to the camera coordinate system. The three-dimensional point cloud cone containing the target to be detected is selected from the two-dimensional detection frame of the target to be detected. Then, according to the above-mentioned target depth d and the prior three-dimensional size of the target to be detected, a depth threshold is set. The range of the three-dimensional point cloud cone is narrowed according to the set depth threshold. Within this range, the point cloud in the narrowed three-dimensional point cloud cone is clustered and segmented using the DBSCAN clustering algorithm. The target point cloud is determined in the cluster cluster according to the prior length L of the target to be detected. Other clusters outside the target point cloud are regarded as obstacle point clouds, thereby completing three-dimensional target detection.

[0033] In step 5), the closest point pair between the target point cloud and the obstacle point cloud identified in step 4) is found. The distance between the point pairs is the actual distance between the aircraft surface and the obstacle. Then, the target point cloud is magnified along the normal direction. Within the collision detection area, a bounding box detection method is first used to perform a rough collision detection on the target point cloud and the obstacle point cloud. If it is determined that a collision has occurred within the bounding box, a voxel detection method is then used to perform a precise collision detection. The method for determining the target collision area point cloud, the collision risk point set, and the area is as follows:

[0034] A KD tree-accelerated nearest neighbor search method is used to find the nearest neighbor point in the obstacle point cloud for each point in the target point cloud and calculate the Euclidean distance between the two points. Finally, the minimum value among all point pairs is selected as the actual distance between the aircraft surface and the obstacle.

[0035] Before performing collision detection, in order to ensure that no actual collision has occurred when the bounding box collision is detected, the target point cloud needs to be enlarged along the normal direction to create a certain safety distance; then, the AABB bounding box of the corresponding point cloud is obtained by calculating the maximum and minimum values ​​of the target point cloud and the obstacle point cloud on the x, y, and z axes. After that, in the camera coordinate system, it is determined whether the projections of the target point cloud AABB bounding box and the obstacle point cloud AABB bounding box on the x, y, and z axes intersect. As long as the projections on at least one axis do not intersect, it is determined that there is no collision risk. Otherwise, it is determined that there may be a collision risk and more refined voxel detection is required;

[0036] In order to exclude the situation where the bounding boxes collide but the point clouds do not, the target point cloud and the obstacle point cloud are divided into voxel grids of equal size. In the camera coordinate system, if the projections of the target point cloud voxel grid and the obstacle point cloud voxel grid on the x, y, and z axes intersect, a collision is determined. Otherwise, no collision occurs, and the target collision area point cloud is determined.

[0037] In order to obtain the collision risk area, the overlapping area of ​​the target point cloud voxel grid and the obstacle voxel grid is calculated by voxel indexing. The method is to first extract the voxel index sets corresponding to the target point cloud and the obstacle point cloud, and use set operations to calculate the intersection of the two to determine the voxel index of the collision risk area; then, in the target point cloud to be detected, the points whose indexes belong to the intersection part are screened out as the collision risk point set, and this point set is the collision risk area.

[0038] In step 6), the target point cloud obtained in step 4) is used to calculate the principal curvature, Gaussian curvature, and mean curvature of the surface where the collision risk area is located, thereby determining the surface type of the surface where the collision risk area is located. Then, based on its geometric characteristics, the method for dividing the surface where the collision risk area is located into five different planes is as follows:

[0039] The surface morphology is determined by the maximum and minimum principal curvatures k1 and k2 of the point cloud. Based on the target point cloud obtained in step 4), the neighborhood point set of one point is taken. Its normal vector is obtained by calculating the covariance matrix and finding its eigenvalues. Then, the curvature tensor is constructed in the tangent plane of the normal vector and its eigenvalues ​​are solved to finally obtain the maximum and minimum principal curvatures k1 and k2.

[0040] Then, the Gaussian curvature K and mean curvature H are calculated based on the maximum and minimum principal curvatures k1 and k2. The Gaussian curvature reflects the overall curvature of the surface at that point. If K = 0, it indicates that the area is a cylinder or plane. If K > 0, it indicates that the area is a sphere. The calculation formula is as follows:

[0041] K=k1k2

[0042] The mean curvature H reflects the local concave-convex characteristics of the surface at that point. If H < 0, it indicates that the area is convex, and if H > 0, it indicates that the area is concave. The calculation formula is as follows

[0043]

[0044] Using curvature analysis, the surface where the collision risk area is located is divided into five types: when k1=k2=0, the surface is a plane; when one of the principal curvatures is zero and the other is non-zero, the surface is a cylindrical surface, where H<0 is a convex cylindrical surface and H>0 is a concave cylindrical surface; when both principal curvatures are non-zero and K>0, the area is a spherical surface, where H<0 is a convex spherical surface and H>0 is a concave spherical surface.

[0045] In step 7), the convex hull is calculated for the collision risk point set obtained in step 5), and then the convex hull is projected onto a normalized plane based on the laser radar extrinsic parameters obtained by calibration in step 2) to obtain the approximate shape of the collision risk area. The surfaces of the collision risk areas of different surface types obtained in step 6) are partitioned accordingly, so that each partition is as close to a small plane as possible. Perspective transformation correction is performed on each partition separately, and then the corrected partitions are spliced ​​to finally generate a complete warning area image. The method is as follows:

[0046] The collision risk point set obtained in step 5) is processed using the convex hull method. The RGB of the convex hull is set to (255, 0, 0), i.e., red. The extrinsic parameters of LiDAR 2 obtained in step 2) are used to project the convex hull onto a normalized plane. The actual distance between the aircraft surface and the obstacle obtained in step 5) is projected as a pattern onto the normalized plane where the convex hull is located.

[0047] Subsequently, the pattern contained in the above-mentioned normalized plane is partitioned, and the partitioning method is designed according to the surface type obtained in step 6); no partitioning is required for the plane area; for the vertical cylindrical area, the collision risk area is divided into multiple longitudinal rectangles in the vertical direction, and its surface topological relationship is kept unchanged during the final mapping; for the horizontal cylindrical area, the risk area is divided into multiple horizontal strips in the horizontal direction, so that the expanded projection image maintains the original surface ratio; for the spherical area, since a single expansion method is difficult to ensure low distortion, a quadrilateral segmentation method is used to divide the spherical area into multiple regular quadrilateral areas; after partitioning, each partition is subjected to perspective transformation correction, and finally spliced ​​back to the surface through mapping, thereby obtaining a complete warning area image, which can accurately adapt to different surface shapes.

[0048] In step 8), the TOF point cloud is collected by using the TOF camera in the head-mounted display device, and then the TOF point cloud is downsampled based on geometric features and a grid is generated to ensure that the projected image can match the geometry of the real environment:

[0049] The TOF point cloud is collected using the TOF camera in the head-mounted display device. Then, an industrial computer is used to downsample the TOF point cloud based on geometric features. Points with large curvature changes, such as wingtips, tail wingtips, and engine edges, that can effectively reflect the aircraft structure are selected, and the remaining points are filtered out for TOF point cloud downsampling.

[0050] A mesh is constructed using the downsampled TOF point cloud. The method is to first perform a two-dimensional projection on the downsampled TOF point cloud, and generate an initial triangulation using the Delaunay algorithm while maintaining the topological structure of the point cloud. Subsequently, the triangulation result is mapped back to a three-dimensional coordinate system so that the triangulated mesh can accurately cover the environmental surface captured by the head-mounted display device 1.

[0051] In step 9), the collision risk point set obtained in step 5) is converted to the TOF camera coordinate system using the internal and external parameters of the laser radar and RGB camera obtained in step 2) to obtain the correct display position of the warning area image in the head-mounted display device; then, the warning area image including the red pattern and the actual distance digital pattern obtained in step 7) is fitted into the grid generated in step 8) to achieve the effect that the user can directly see the warning area image overlaid on the real collision risk area in the augmented reality environment of the head-mounted display device. The method is as follows:

[0052] First, the collision risk point set in the radar coordinate system is converted to the RGB camera coordinate system based on the rotation matrix and displacement vector obtained by calibration in step 2). Then, the collision risk point set is converted from the RGB camera coordinate system to the TOF camera coordinate system using the built-in calibration relationship between the RGB camera and the TOF camera on the head-mounted display device.

[0053] Finally, the warning area image obtained in step 7) is fitted to the grid surface generated in step 8) according to the converted coordinate points. The method is to calculate the corresponding coordinates of the red pattern in the warning area image for each triangle on the grid, and use the texture mapping method to project the corrected warning area image onto the grid surface; during the fitting process, ensure that the transformation relationship of the triangle fragments is consistent with the grid topology, so that the warning area image accurately covers the collision risk area, so that the staff can directly see the effect of the warning area image covering the collision risk area in reality through the LCD display in the augmented reality environment of the head-mounted display device.

[0054] The aircraft dragging risk warning method based on a head-mounted display device and a lidar provided by the present invention designs a system that uses a head-mounted display device as a carrier, integrates multiple sensors such as a lidar, an RGB camera, a TOF camera, and integrates multiple technologies such as augmented reality, deep learning, point cloud processing, and distortion correction. It has stronger visibility, real-timeness, and accuracy, and can monitor and warn of various risks in complex environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of the aircraft dragging risk warning method based on a head-mounted display device and a laser radar provided by the present invention.

[0056] Figure 2Schematic diagram of the aircraft towing process risk warning system based on a head-mounted display device and a laser radar in the present invention. DETAILED DESCRIPTION

[0057] The aircraft dragging risk warning method based on a head-mounted display device and a laser radar provided by the present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0058] like Figure 1 、 Figure 2 As shown, the aircraft dragging risk warning method based on a head-mounted display device and a laser radar provided by the present invention includes the following steps performed in sequence:

[0059] 1) Establishing an aircraft towing process risk warning system based on a head-mounted display device and a laser radar; the aircraft towing process risk warning system includes a head-mounted display device 1 and a laser radar 2; wherein the head-mounted display device 1 includes a headband 3, an industrial computer, an LCD display screen 4, an RGB camera 5, and a depth camera 6; two LCD display screens 4 are respectively mounted on both sides of the front side of the headband 3; the RGB camera 5 and the depth camera 6 are mounted in the middle of the front side of the headband 3; the industrial computer is mounted on the headband 3 and is electrically connected to the LCD display screen 4, the RGB camera 5, and the depth camera 6, respectively, and is connected to the laser radar 2 mounted on the upper end of the headband 3 via an Ethernet interface;

[0060] The head-mounted display device 1 adopts Microsoft HoloLens2, which is a head-mounted device for augmented reality (AR) and mixed reality (MR); the laser radar 2 is a point cloud acquisition device, which adopts a 96-line Airy laser radar with a diameter of 60 mm and a weight of 230 g.

[0061] 2) Calibrate the RGB camera and lidar in the aircraft towing process risk warning system based on the head-mounted display device and lidar to obtain the intrinsic and extrinsic parameters of the RGB camera and the extrinsic parameters of the lidar;

[0062] The staff wears a head-mounted display device 1 equipped with a laser radar 2. First, a checkerboard calibration plate is used. Under the control of an industrial computer, the RGB camera 5 is used to collect images of the checkerboard calibration plate at different angles and distances through the principle of homography matrix mapping. Combined with the corner point positions in the image, the internal parameters of the RGB camera 5 including focal length, principal point position and distortion coefficient are calculated through a nonlinear optimization algorithm. Then, the RGB camera 5 and the laser radar 2 with calibrated internal parameters are used to collect multiple groups of images and point cloud data of the checkerboard calibration plate at different angles and distances. Then, according to the correspondence between the checkerboard corner points, the rotation matrix and translation vector of the two are calculated as external parameters, thereby obtaining the spatial position relationship between the RGB camera 5 and the laser radar 2.

[0063] 3) Collecting original target images, screening them, and annotating them. All annotated target images form a target image sample dataset and divide it into a training set and a test set in proportion. Build an original two-dimensional target detection model and train and evaluate it using the training set and test set respectively to obtain the final two-dimensional target detection model.

[0064] Use any camera to capture a sufficient number of original target images at different angles and depths to ensure comprehensiveness and diversity of the images. The targets include aircraft wings, tail fins, and engines. Then, filter the original target images to retain high-quality images. Then, use the labellmg.exe annotation method to manually mark the target positions in the filtered target images using square annotation boxes to obtain labeled target images. All labeled target images constitute a target image sample dataset, which is then divided into a training set and a test set in an 8:2 ratio.

[0065] In the industrial computer, the deep learning YOLOv9 algorithm model is used to build the original two-dimensional target detection model, and then the above training set is input into the original two-dimensional target detection model for training. By iteratively optimizing the loss function, the model performance is continuously adjusted. After that, the test set is input into the trained two-dimensional target detection model. The accuracy of the two-dimensional target detection model is evaluated by calculating the detection accuracy and average precision. Finally, the two-dimensional target detection model is optimized according to the evaluation results, and the two-dimensional target detection model with the best detection performance is selected as the final two-dimensional target detection model.

[0066] 4) Using the calibrated RGB camera in the head-mounted display device to continuously capture a series of real-time images of the target to be detected and input them into the final two-dimensional target detection model obtained in step 3), the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; using the calibrated lidar to capture the point cloud of the target to be detected in real time, combining the RGB camera intrinsic parameters and the RGB camera and lidar extrinsic parameters obtained by calibration in step 2), pre-processing the target point cloud corresponding to the target image at the same time, including depth screening, camera field of view (FOV) screening, and ground point cloud removal, then using the above-mentioned two-dimensional detection frame of the target to be detected combined with the intrinsic parameters of the calibrated RGB camera to calculate the target depth, construct a three-dimensional point cloud viewing cone based on the target depth, and then cluster and segment the pre-processed target point cloud based on the three-dimensional point cloud viewing cone to determine the target point cloud and obstacle point cloud, thereby completing three-dimensional target detection;

[0067] A series of real-time images of the target to be detected are continuously captured using the calibrated RGB camera 5 in the head-mounted display device and input into the final two-dimensional target detection model obtained in step 3), and the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; the laser radar 2 calibrated in step 2) is used to capture the point cloud of the target to be detected in real time and input into the industrial computer, a depth threshold of the target region of interest is set, and point clouds exceeding the depth threshold range are eliminated to obtain a depth-filtered target point cloud; combined with the internal parameters of the RGB camera 5 obtained in step 2), the depth-filtered target point cloud is projected onto the RGB camera image plane, and valid pixels within the field of view of the RGB camera 5 are extracted to obtain a target point cloud within the field of view of the RGB camera 5; finally, the RANSAC algorithm is used to fit the ground point cloud and eliminate it, thereby achieving preprocessing of the target point cloud to be detected;

[0068] Then use the intrinsic parameters of RGB camera 5 obtained in step 2):

[0069]

[0070] Where dx is the pixel size of RGB camera 5, which can be obtained from the factory parameters of RGB camera 5. Let f / dx be f x , calculate the camera focal length f by the following formula:

[0071] f=f x ·dx

[0072] Then, the horizontal pixel length of the two-dimensional detection frame of the target to be detected obtained in step 4) is recorded as the pixel width w, and the image width W of the target to be detected is expressed as follows:

[0073] W=w·dx

[0074] The actual length of the target to be detected is measured using a laser rangefinder and used as its prior length L. Based on the camera imaging principle and the relationship between the camera focal length f, image width W, and pixel size dx, the target depth calculation formula is obtained:

[0075]

[0076] Using the intrinsic and extrinsic parameters of the RGB camera 5 obtained in step 2), the above-mentioned two-dimensional detection frame of the target to be detected and the preprocessed point cloud of the target to be detected are converted to the camera coordinate system, and the three-dimensional point cloud cone containing the target to be detected is selected from the two-dimensional detection frame of the target to be detected. Then, according to the above-mentioned target depth d and the prior three-dimensional size of the target to be detected, a depth threshold is set, and the range of the three-dimensional point cloud cone is narrowed according to the set depth threshold. Within this range, the point cloud in the narrowed three-dimensional point cloud cone is clustered and segmented using the DBSCAN clustering algorithm, and the target point cloud is determined in the cluster cluster according to the prior length L of the target to be detected. Other clusters outside the target point cloud are regarded as obstacle point clouds, thereby completing three-dimensional target detection.

[0077] 5) Find the closest point pair between the target point cloud and the obstacle point cloud identified in step 4). The distance between the point pairs is the actual distance between the aircraft surface and the obstacle. Then, the target point cloud is magnified along the normal direction. Within the collision detection area, the bounding box detection method is first used to perform a rough collision detection on the target point cloud and the obstacle point cloud. If the bounding box is judged to have collided, the voxel detection method is then used to perform a precise collision detection. In this way, the target collision area point cloud, collision risk point set, and area are determined.

[0078] A KD tree-accelerated nearest neighbor search method is used to find the nearest neighbor point in the obstacle point cloud for each point in the target point cloud and calculate the Euclidean distance between the two points. Finally, the minimum value among all point pairs is selected as the actual distance between the aircraft surface and the obstacle.

[0079] The bounding box detection method uses a minimum cube to enclose a point cloud, thereby analyzing and processing the point cloud. Before performing collision detection, to ensure that no actual collision has occurred when the bounding box collision is detected, the target point cloud needs to be enlarged along the normal direction to create a certain safety distance. Then, by calculating the maximum and minimum values ​​of the target point cloud and the obstacle point cloud on the x, y, and z axes, the AABB bounding box of the corresponding point cloud is obtained. Then, in the camera coordinate system, it is determined whether the projections of the target point cloud AABB bounding box and the obstacle point cloud AABB bounding box on the x, y, and z axes intersect. As long as the projections on at least one axis do not intersect, it is judged that there is no collision risk. Otherwise, it is judged that there may be a collision risk and more detailed voxel detection is required.

[0080] Point cloud voxels divide the point cloud into voxel grids of equal size, and then perform collision detection by determining whether the voxel grids of different objects intersect. The detection method is similar to the bounding box detection method, but the detection accuracy can be controlled by defining different voxel sizes to provide accurate collision detection results.

[0081] In order to exclude the situation where the bounding boxes collide but the point clouds do not, the target point cloud and the obstacle point cloud are divided into voxel grids of equal size. In the camera coordinate system, if the projections of the target point cloud voxel grid and the obstacle point cloud voxel grid on the x, y, and z axes intersect, a collision is determined. Otherwise, no collision occurs, and the target collision area point cloud is determined.

[0082] In order to obtain the collision risk area, the overlapping area of ​​the target point cloud voxel grid and the obstacle voxel grid can be calculated through voxel indexing. The method is to first extract the voxel index sets corresponding to the target point cloud and the obstacle point cloud, and use set operations to calculate the intersection of the two to determine the voxel index of the collision risk area; then, in the target point cloud to be detected, the points whose indexes belong to the intersection part are screened out as the collision risk point set, and this point set is the collision risk area.

[0083] 6) Using the target point cloud obtained in step 4), calculate the principal curvature, Gaussian curvature, and mean curvature of the surface where the collision risk area is located, thereby determining the surface type of the surface where the collision risk area is located. Then, based on its geometric characteristics, divide the surface where the collision risk area is located into five different planes;

[0084] The surface morphology is determined by the maximum and minimum principal curvatures k1 and k2 of the point cloud. Based on the target point cloud obtained in step 4), the neighborhood point set of one point is taken. Its normal vector is obtained by calculating the covariance matrix and finding its eigenvalues. Then, the curvature tensor is constructed in the tangent plane of the normal vector and its eigenvalues ​​are solved to finally obtain the maximum and minimum principal curvatures k1 and k2.

[0085] Then, the Gaussian curvature K and mean curvature H are calculated based on the maximum and minimum principal curvatures k1 and k2. The Gaussian curvature reflects the overall curvature of the surface at that point. If K = 0, it indicates that the area is a cylinder or plane. If K > 0, it indicates that the area is a sphere. The calculation formula is as follows:

[0086] K=k1k2

[0087] The mean curvature H reflects the local concave-convex characteristics of the surface at that point. If H < 0, it indicates that the area is convex, and if H > 0, it indicates that the area is concave. The calculation formula is as follows

[0088]

[0089] Using curvature analysis, the surface where the collision risk area is located can be divided into five types: when k1=k2=0, the surface is a plane; when one of the principal curvatures is zero and the other is non-zero, the surface is a cylindrical surface, where H<0 is a convex cylindrical surface and H>0 is a concave cylindrical surface; when both principal curvatures are non-zero and K>0, the area is a spherical surface, where H<0 is a convex spherical surface and H>0 is a concave spherical surface.

[0090] 7) Calculate the convex hull of the collision risk point set obtained in step 5). Then, project the convex hull onto a normalized plane based on the lidar extrinsic parameters obtained by calibration in step 2) to obtain the approximate shape of the collision risk area. Based on the collision risk area surfaces of different surface types obtained in step 6), perform corresponding partitioning, making each partition as close to a small plane as possible. Perform perspective transformation correction on each partition separately, and then stitch the corrected partitions together to ultimately generate a complete warning area image.

[0091] The collision risk point set obtained in step 5) is processed using the convex hull method. The RGB of the convex hull is set to (255, 0, 0), i.e., red. The extrinsic parameters of LiDAR 2 obtained in step 2) are used to project the convex hull onto a normalized plane. The actual distance between the aircraft surface and the obstacle obtained in step 5) is projected as a pattern onto the normalized plane where the convex hull is located.

[0092] Subsequently, the pattern contained in the above-mentioned normalized plane is partitioned, and the partitioning method is designed according to the surface type obtained in step 6); no partitioning is required for the plane area; for the vertical cylindrical area, the collision risk area is divided into multiple longitudinal rectangles in the vertical direction, and its surface topological relationship is kept unchanged during the final mapping; for the horizontal cylindrical area, the risk area is divided into multiple horizontal strips in the horizontal direction, so that the expanded projection image maintains the original surface ratio; for the spherical area, since a single expansion method is difficult to ensure low distortion, a quadrilateral segmentation method is used to divide the spherical area into multiple regular quadrilateral areas; after partitioning, each partition is subjected to perspective transformation correction, and finally spliced ​​back to the surface through mapping, thereby obtaining a complete warning area image, which can accurately adapt to different surface shapes.

[0093] 8) Using the TOF camera in the head-mounted display device to collect the TOF point cloud, the TOF point cloud is then downsampled based on geometric features and meshed to ensure that the projected image matches the geometry of the real environment;

[0094] The TOF camera 6 of the head-mounted display device 1 can provide depth information, which can be converted into a point cloud. The TOF point cloud is collected by the TOF camera 6 in the head-mounted display device 1. Then, the TOF point cloud is downsampled based on geometric features using an industrial computer. Points with large curvature changes, including wingtips, tail wingtips, and engine edges that can effectively reflect the aircraft structure, are selected. The remaining points are filtered out to perform TOF point cloud downsampling.

[0095] A mesh is constructed using the downsampled TOF point cloud. The method is to first perform a two-dimensional projection on the downsampled TOF point cloud, and generate an initial triangulation using the Delaunay algorithm while maintaining the topological structure of the point cloud. Subsequently, the triangulation result is mapped back to a three-dimensional coordinate system so that the triangulated mesh can accurately cover the environmental surface captured by the head-mounted display device 1. Throughout the entire process, the mesh is prevented from being excessively distorted by limiting the aspect ratio of the triangles and optimizing the boundary processing to obtain a mesh structure that can correctly express the geometric form of the environment.

[0096] 9) Using the internal and external parameters of the lidar and RGB camera obtained in step 2), the collision risk point set obtained in step 5) is converted to the TOF camera coordinate system to obtain the correct display position of the warning area image in the head-mounted display device; then, the warning area image containing the red pattern and the actual distance digital pattern obtained in step 7) is fitted to the grid generated in step 8) to achieve the effect that the operator can directly see the warning area image overlaid on the actual collision risk area in the augmented reality environment of the head-mounted display device;

[0097] Since the lidar 2 and RGB camera 5 have been calibrated in step 2), and the RGB camera 5 and TOF camera 6 have been calibrated by the manufacturer, the conversion relationship provided by the system can be directly applied. First, the collision risk point set in the lidar coordinate system is converted to the RGB camera coordinate system based on the rotation matrix and displacement vector obtained in step 2). Then, the collision risk point set is converted from the RGB camera coordinate system to the TOF camera coordinate system using the built-in calibration relationship between the RGB camera 5 and TOF camera 6 on the head-mounted display device 1.

[0098] Finally, the warning area image obtained in step 7) is fitted to the grid surface generated in step 8) according to the converted coordinate points. The method is to calculate the corresponding coordinates of the red pattern in the warning area image for each triangle on the grid, and use the texture mapping method to project the corrected warning area image onto the grid surface; during the fitting process, ensure that the transformation relationship of the triangle fragments is consistent with the grid topology, so that the warning area image accurately covers the collision risk area, so that the staff can directly see the effect of the warning area image covering the collision risk area in reality through the LCD display 4 in the augmented reality environment of the head-mounted display device 1.

Claims

1. An aircraft dragging risk warning method based on a head-mounted display device and a laser radar, characterized by: The aircraft dragging risk warning method based on a head-mounted display device and a laser radar comprises the following steps performed in sequence: 1) Establish an aircraft towing process risk warning system based on head-mounted display devices and lidar; 2) Calibrate the RGB camera and lidar in the aircraft towing process risk warning system based on the head-mounted display device and lidar to obtain the intrinsic and extrinsic parameters of the RGB camera and the extrinsic parameters of the lidar; 3) Collecting original target images, screening them, and annotating them. All annotated target images form a target image sample dataset and divide it into a training set and a test set in proportion. Build an original two-dimensional target detection model and train and evaluate it using the training set and test set respectively to obtain the final two-dimensional target detection model. 4) Using the calibrated RGB camera in the head-mounted display device to continuously capture a series of real-time images of the target to be detected and input them into the final two-dimensional target detection model obtained in step 3), the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; using the calibrated lidar to capture the point cloud of the target to be detected in real time, combining the RGB camera intrinsic parameters and the RGB camera and lidar extrinsic parameters obtained by calibration in step 2), pre-processing the target point cloud corresponding to the target image at the same time, including depth screening, camera field of view screening, and ground point cloud removal, then using the above-mentioned two-dimensional detection frame of the target to be detected combined with the intrinsic parameters of the calibrated RGB camera to calculate the target depth, construct a three-dimensional point cloud viewing cone based on the target depth, and then cluster and segment the pre-processed target point cloud based on the three-dimensional point cloud viewing cone to determine the target point cloud and obstacle point cloud, thereby completing three-dimensional target detection; 5) Find the closest point pair between the target point cloud and the obstacle point cloud identified in step 4). The distance between the point pairs is the actual distance between the aircraft surface and the obstacle. Then, the target point cloud is magnified along the normal direction. Within the collision detection area, the bounding box detection method is first used to perform a rough collision detection on the target point cloud and the obstacle point cloud. If the bounding box is judged to have collided, the voxel detection method is then used to perform a precise collision detection. In this way, the target collision area point cloud, collision risk point set, and area are determined. 6) Using the target point cloud obtained in step 4), calculate the principal curvature, Gaussian curvature, and mean curvature of the surface where the collision risk area is located, thereby determining the surface type of the surface where the collision risk area is located. Then, based on its geometric characteristics, divide the surface where the collision risk area is located into five different planes; 7) Calculate the convex hull of the collision risk point set obtained in step 5). Then, project the convex hull onto a normalized plane based on the lidar extrinsic parameters obtained by calibration in step 2) to obtain the approximate shape of the collision risk area. Based on the collision risk area surfaces of different surface types obtained in step 6), perform corresponding partitioning, making each partition as close to a small plane as possible. Perform perspective transformation correction on each partition separately, and then stitch the corrected partitions together to ultimately generate a complete warning area image. 8) Using the TOF camera in the head-mounted display device to collect the TOF point cloud, then downsampling the TOF point cloud based on geometric features and generating a mesh to ensure that the projected image can match the geometry of the real environment; 9) Using the internal and external parameters of the lidar and RGB camera obtained in step 2), the collision risk point set obtained in step 5) is converted to the TOF camera coordinate system to obtain the correct display position of the warning area image in the head-mounted display device; then, the warning area image containing the red pattern and the actual distance digital pattern obtained in step 7) is fitted into the grid generated in step 8) to achieve the effect that the staff can directly see the warning area image overlaid on the real collision risk area in the augmented reality environment of the head-mounted display device.

2. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 1), the aircraft dragging process risk warning system based on a head-mounted display device and a laser radar comprises a head-mounted display device (1) and a laser radar (2); wherein the head-mounted display device (1) comprises a headband (3), an industrial computer, an LCD display screen (4), an RGB camera (5) and a depth camera (6); two LCD display screens (4) are respectively mounted on both sides of the front side of the headband (3); the RGB camera (5) and the depth camera (6) are mounted in the middle of the front side of the headband (3); the industrial computer is mounted on the headband (3) and is electrically connected to the LCD display screen (4), the RGB camera (5) and the depth camera (6) respectively, and is connected to the laser radar (2) mounted on the upper end of the headband (3) via an Ethernet interface; The head-mounted display device (1) adopts Microsoft HoloLens2, which is a head-mounted device for augmented reality and mixed reality; the laser radar (2) is a point cloud acquisition device, which adopts a 96-line Airy laser radar with a diameter of 60 mm and a weight of 230 g.

3. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 2), the method for calibrating the RGB camera and the lidar in the aircraft towing process risk warning system based on the head-mounted display device and the lidar to obtain the intrinsic and extrinsic parameters of the RGB camera and the extrinsic parameters of the lidar is: The staff wears a head-mounted display device (1) equipped with a laser radar (2), first uses a checkerboard calibration plate, and under the control of an industrial computer, uses an RGB camera (5) to collect images of the checkerboard calibration plate at different angles and distances using the principle of homography matrix mapping, and combines the corner point positions in the image to calculate the internal parameters of the RGB camera (5) including the focal length, the principal point position, and the distortion coefficient through a nonlinear optimization algorithm; Then, the RGB camera (5) and the laser radar (2) with calibrated internal parameters are used to collect multiple sets of images and point cloud data of the checkerboard calibration plate at different angles and distances; then, the rotation matrix and translation vector of the two are calculated as external parameters based on the corresponding relationship between the checkerboard corner points, thereby obtaining the spatial position relationship between the RGB camera (5) and the laser radar (2).

4. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 3), the original target images are collected and screened and annotated, and a target image sample data set is formed by all the annotated target images and divided into a training set and a test set in proportion; an original two-dimensional target detection model is built and trained and evaluated using the training set and the test set respectively, and the method for obtaining the final two-dimensional target detection model is: Use any camera to capture original target images at different angles and depths, including aircraft wings, tail fins, and engines. Then, filter the original target images to retain high-quality images. Then, use the labellmg.exe annotation method to manually mark the target locations in square annotation boxes on the filtered target images to obtain labeled target images. A target image sample dataset is constructed from all labeled target images, which is then divided into a training set and a test set in an 8:2 ratio. In the industrial computer, the deep learning YOLOv9 algorithm model is used to build the original two-dimensional target detection model, and then the above training set is input into the original two-dimensional target detection model for training. By iteratively optimizing the loss function, the model performance is continuously adjusted. After that, the test set is input into the trained two-dimensional target detection model. The accuracy of the two-dimensional target detection model is evaluated by calculating the detection accuracy and average precision. Finally, the two-dimensional target detection model is optimized according to the evaluation results, and the two-dimensional target detection model with the best detection performance is selected as the final two-dimensional target detection model.

5. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 4), the calibrated RGB camera in the head-mounted display device is used to continuously capture a series of real-time images of the target to be detected and input them into the final two-dimensional target detection model obtained in step 3), and the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; the calibrated lidar is used to capture the point cloud of the target to be detected in real time, and the RGB camera intrinsic parameters and the RGB camera and lidar extrinsic parameters obtained by calibration in step 2) are combined to perform preprocessing including depth screening, camera field of view screening, and ground point cloud removal on the point cloud of the target to be detected at the same time corresponding to the target image to be detected. Then, the target depth is calculated using the above-mentioned two-dimensional detection frame of the target to be detected in combination with the intrinsic parameters of the calibrated RGB camera, and a three-dimensional point cloud viewing cone is constructed based on the target depth. Then, the preprocessed point cloud of the target to be detected is clustered and segmented based on the three-dimensional point cloud viewing cone to determine the target point cloud and the obstacle point cloud. The method for completing three-dimensional target detection is as follows: A series of real-time images of the target to be detected are continuously collected using the calibrated RGB camera (5) in the head-mounted display device and input into the final two-dimensional target detection model obtained in step 3), and the final two-dimensional target detection model outputs a two-dimensional detection frame of the target to be detected; the laser radar (2) calibrated in step 2) is used to collect the point cloud of the target to be detected in real time and input into the industrial computer, and a depth threshold of the target area of ​​interest is set, and the point cloud exceeding the depth threshold range is eliminated to obtain the target point cloud after depth screening; combined with the internal parameters of the RGB camera (5) obtained in step 2), the target point cloud after depth screening is projected onto the image plane of the RGB camera, and the effective pixel points within the field of view of the RGB camera (5) are extracted to obtain the target point cloud within the field of view of the RGB camera (5); finally, the RANSAC algorithm is used to fit the ground point cloud and eliminate it, thereby realizing the preprocessing of the target point cloud to be detected; Then use the internal parameters of the RGB camera (5) obtained in step 2): Where dx is the pixel size of the RGB camera (5), which is obtained from the factory parameters of the RGB camera (5). Let f / dx be f x , calculate the camera focal length f by the following formula: f=f x ·dx Then, the horizontal pixel length of the two-dimensional detection frame of the target to be detected obtained in step 4) is recorded as the pixel width w, and the image width W of the target to be detected is expressed as follows: W=w·dx The actual length of the target to be detected is measured using a laser rangefinder and used as its prior length L. Based on the camera imaging principle and the relationship between the camera focal length f, image width W, and pixel size dx, the target depth calculation formula is obtained: By using the intrinsic and extrinsic parameters of the RGB camera (5) obtained in step 2), the above-mentioned two-dimensional detection frame of the target to be detected and the preprocessed point cloud of the target to be detected are converted into the camera coordinate system, and the three-dimensional point cloud cone containing the target to be detected is selected from the two-dimensional detection frame of the target to be detected. Then, according to the above-mentioned target depth d and the prior three-dimensional size of the target to be detected, a depth threshold is set, and the range of the three-dimensional point cloud cone is narrowed according to the set depth threshold. Within this range, the point cloud in the narrowed three-dimensional point cloud cone is clustered and segmented by the DBSCAN clustering algorithm, and the target point cloud is determined in the cluster cluster according to the prior length L of the target to be detected. The other clusters outside the target point cloud are regarded as obstacle point clouds, thereby completing the three-dimensional target detection.

6. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 5), the closest point pair between the target point cloud and the obstacle point cloud identified in step 4) is found. The distance between the point pairs is the actual distance between the aircraft surface and the obstacle. Then, the target point cloud is magnified along the normal direction. Within the collision detection area, a bounding box detection method is first used to perform a rough collision detection on the target point cloud and the obstacle point cloud. If it is determined that a collision has occurred within the bounding box, a voxel detection method is then used to perform a precise collision detection. The method for determining the target collision area point cloud, the collision risk point set, and the area is as follows: A KD tree-accelerated nearest neighbor search method is used to find the nearest neighbor point in the obstacle point cloud for each point in the target point cloud and calculate the Euclidean distance between the two points. Finally, the minimum value among all point pairs is selected as the actual distance between the aircraft surface and the obstacle. Before performing collision detection, in order to ensure that no actual collision has occurred when the bounding box collision is detected, the target point cloud needs to be enlarged along the normal direction to create a certain safety distance; then, the AABB bounding box of the corresponding point cloud is obtained by calculating the maximum and minimum values ​​of the target point cloud and the obstacle point cloud on the x, y, and z axes. After that, in the camera coordinate system, it is determined whether the projections of the target point cloud AABB bounding box and the obstacle point cloud AABB bounding box on the x, y, and z axes intersect. As long as the projections on at least one axis do not intersect, it is determined that there is no collision risk. Otherwise, it is determined that there may be a collision risk and more refined voxel detection is required; In order to exclude the situation where the bounding boxes collide but the point clouds do not, the target point cloud and the obstacle point cloud are divided into voxel grids of equal size. In the camera coordinate system, if the projections of the target point cloud voxel grid and the obstacle point cloud voxel grid on the x, y, and z axes intersect, a collision is determined. Otherwise, no collision occurs, and the target collision area point cloud is determined. In order to obtain the collision risk area, the overlapping area of ​​the target point cloud voxel grid and the obstacle voxel grid is calculated by voxel indexing. The method is to first extract the voxel index sets corresponding to the target point cloud and the obstacle point cloud, and use set operations to calculate the intersection of the two to determine the voxel index of the collision risk area; then, in the target point cloud to be detected, the points whose indexes belong to the intersection part are screened out as the collision risk point set, and this point set is the collision risk area.

7. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 6), the target point cloud obtained in step 4) is used to calculate the principal curvature, Gaussian curvature, and mean curvature of the surface where the collision risk area is located, thereby determining the surface type of the surface where the collision risk area is located. Then, based on its geometric characteristics, the method for dividing the surface where the collision risk area is located into five different planes is as follows: The surface morphology is determined by the maximum and minimum principal curvatures k1 and k2 of the point cloud. Based on the target point cloud obtained in step 4), the neighborhood point set of one point is taken. Its normal vector is obtained by calculating the covariance matrix and finding its eigenvalues. Then, the curvature tensor is constructed in the tangent plane of the normal vector and its eigenvalues ​​are solved to finally obtain the maximum and minimum principal curvatures k1 and k2. Then, the Gaussian curvature K and mean curvature H are calculated based on the maximum and minimum principal curvatures k1 and k2. The Gaussian curvature reflects the overall curvature of the surface at that point. If K = 0, it indicates that the area is a cylinder or plane. If K > 0, it indicates that the area is a sphere. The calculation formula is as follows: K=k1k2 The mean curvature H reflects the local concave-convex characteristics of the surface at that point. If H < 0, it indicates that the area is convex; if H > 0, it indicates that the area is concave. The calculation formula is as follows Using curvature analysis, the surface where the collision risk area is located is divided into five types: when k1=k2=0, the surface is a plane; when one of the principal curvatures is zero and the other is non-zero, the surface is a cylindrical surface, where H<0 is a convex cylindrical surface and H>0 is a concave cylindrical surface; when both principal curvatures are non-zero and K>0, the area is a spherical surface, where H<0 is a convex spherical surface and H>0 is a concave spherical surface.

8. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 7), the convex hull is calculated for the collision risk point set obtained in step 5), and then the convex hull is projected onto a normalized plane based on the laser radar extrinsic parameters obtained by calibration in step 2) to obtain the approximate shape of the collision risk area. The surfaces of the collision risk areas of different surface types obtained in step 6) are partitioned accordingly, so that each partition is as close to a small plane as possible. Perspective transformation correction is performed on each partition separately, and then the corrected partitions are spliced ​​to finally generate a complete warning area image. The method is as follows: The collision risk point set obtained in step 5) is processed using the convex hull method, and the RGB of the convex hull is set to (255, 0, 0), i.e., red. The convex hull is projected onto a normalized plane using the external parameters of the laser radar (2) obtained in step 2), and the actual distance between the aircraft surface and the obstacle obtained in step 5) is projected in the form of a pattern onto the normalized plane where the convex hull is located. Subsequently, the pattern contained in the above-mentioned normalized plane is partitioned, and the partitioning method is designed according to the surface type obtained in step 6); no partitioning is required for the plane area; for the vertical cylindrical area, the collision risk area is divided into multiple longitudinal rectangles in the vertical direction, and its surface topological relationship is kept unchanged during the final mapping; for the horizontal cylindrical area, the risk area is divided into multiple horizontal strips in the horizontal direction, so that the expanded projection image maintains the original surface ratio; for the spherical area, since a single expansion method is difficult to ensure low distortion, a quadrilateral segmentation method is used to divide the spherical area into multiple regular quadrilateral areas; after partitioning, each partition is subjected to perspective transformation correction, and finally spliced ​​back to the surface through mapping, thereby obtaining a complete warning area image, which can accurately adapt to different surface shapes.

9. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 8), the TOF point cloud is collected by using the TOF camera in the head-mounted display device, and then the TOF point cloud is downsampled based on geometric features and a grid is generated to ensure that the projected image can match the geometry of the real environment: A TOF point cloud is collected using a TOF camera (6) in a head-mounted display device (1), and then the TOF point cloud is downsampled based on geometric features using an industrial computer to screen out points with large curvature changes, including wing tips, tail wing tips, engine edges, and other points that can effectively reflect the aircraft structure, and the remaining points are filtered out to perform TOF point cloud downsampling; The mesh is constructed using the downsampled TOF point cloud. The method is to first perform a two-dimensional projection on the downsampled TOF point cloud, and then use the Delaunay algorithm to generate the initial triangulation while keeping the topological structure of the point cloud stable. Subsequently, the segmentation results are mapped back to a three-dimensional coordinate system so that the triangular mesh can accurately cover the surface of the environment captured by the head-mounted display device (1).

10. The aircraft dragging risk warning method based on a head-mounted display device and a laser radar according to claim 1, characterized in that: In step 9), the collision risk point set obtained in step 5) is converted to the TOF camera coordinate system using the internal and external parameters of the laser radar and RGB camera obtained in step 2) to obtain the correct display position of the warning area image in the head-mounted display device; then, the warning area image including the red pattern and the actual distance digital pattern obtained in step 7) is fitted into the grid generated in step 8) to achieve the effect that the user can directly see the warning area image overlaid on the real collision risk area in the augmented reality environment of the head-mounted display device. The method is as follows: First, the collision risk point set in the radar coordinate system is converted to the RGB camera coordinate system according to the rotation matrix and displacement vector obtained by calibration in step 2), and then the collision risk point set is converted from the RGB camera coordinate system to the TOF camera coordinate system using the built-in calibration relationship between the RGB camera (5) and the TOF camera (6) on the head-mounted display device (1); Finally, the warning area image obtained in step 7) is fitted to the grid surface generated in step 8) according to the converted coordinate points. The method is to calculate the corresponding coordinates of the red pattern in the warning area image for each triangle on the grid, and use the texture mapping method to project the corrected warning area image onto the grid surface; during the fitting process, it is ensured that the transformation relationship of the triangle fragments is consistent with the grid topology, so that the warning area image accurately covers the collision risk area, so that the staff can directly see the effect of the warning area image covering the collision risk area in reality through the LCD display (4) in the augmented reality environment of the head-mounted display device (1).

Images