Aircraft ice accretion dynamic monitoring method and system based on three-dimensional visual mapping

Abstract

The invention discloses an aircraft icing dynamic monitoring method and system based on three-dimensional visual mapping. The aircraft icing dynamic monitoring is carried out in two stages of global positioning mapping and local updating mapping. The global positioning mapping stage comprises the following steps: identifying icing points of a to-be-deiced airplane, reconstructing the identified icing points into an icing three-dimensional model, and fusing the icing three-dimensional model into an icing-free airplane three-dimensional model; outputting the fused aircraft icing three-dimensional model to a platform visual interface to realize visualization of aircraft icing global positioning mapping; the local updating and mapping stage comprises the following steps of: mapping ice accumulation thickness information in a local ice accumulation image into an airplane ice accumulation three-dimensional model; and adjusting and updating the Gaussian number and the Gaussian distribution density of the three-dimensional model of the ice accretion of the airplane to realize dynamic updating of the ice accretion ablation condition of each part of the airplane. According to the invention, the whole deicing process of the whole machine can be visually observed, and the deicing efficiency and precision can be improved, so that the deicing operation can be completed more efficiently and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional reconstruction and visual positioning, and particularly relates to an aircraft icing dynamic monitoring method and system based on three-dimensional visualization mapping. BACKGROUND

[0002] Aircraft deicing is one of the important work of winter airport flight service guarantee, and in the traditional aircraft deicing method, the aircraft icing situation is mainly judged by the human eye, which cannot accurately and quickly master the overall information of aircraft icing and cannot achieve precise deicing. Researchers use three-dimensional reconstruction technology to obtain the three-dimensional model of the aircraft, providing global visualization information of the aircraft skin surface for deicing workers. At the same time, in order to master the distribution and melting of the icing in the deicing work in real time, it is necessary to map the two-dimensional icing detection image to the three-dimensional model of the aircraft, so as to realize the visualization dynamic monitoring of the whole process of aircraft deicing from the beginning to the end.

[0003] At present, 2D-3D matching technology plays an important role in the application fields of automatic driving, medical imaging and augmented reality. Traditional 2D-3D matching technology mainly includes methods based on geometric matching, multi-sensor matching and local map matching. However, these methods are prone to mismatching due to insufficient depth estimation, high hardware cost and serious cumulative error, and the traditional methods are limited to indoor small scene applications and are not suitable for large scene applications such as aircraft icing three-dimensional visualization. Therefore, it is of great significance to study a method that can fuse two-dimensional icing detection image and three-dimensional aircraft model in the aircraft deicing scene and realize the visualization of the whole process of aircraft deicing, in order to improve the precision and speed of aircraft deicing work.

[0004] In the process of aircraft deicing, accurately mastering the volume of global icing before deicing and the real-time distribution of local residual icing during deicing operation can effectively improve the deicing effect of the aircraft and effectively control the loss cost of deicing liquid. Since aircraft icing three-dimensional visualization mapping involves outdoor large scene and the texture features of the aircraft surface are not obvious, and in order to map the local icing image to the three-dimensional model of the aircraft, the aircraft icing dynamic monitoring method based on three-dimensional visualization mapping adopts a global-local two-stage mapping architecture, uses the camera pose of the global image set calculated in the global mapping stage to perform global image-local image matching, and preliminarily judges the pose information of the local image, which lays a good foundation for accurately determining the mapping position of the local image in the three-dimensional model of the aircraft.

[0005] Through the above analysis, the problems and defects of the prior art are: (1) The prior art is prone to 2D-3D mismatching in three-dimensional mapping, and is not suitable for large scene applications such as aircraft icing three-dimensional visualization mapping.

[0006] (2) The prior art cannot accurately determine the camera pose of the local icing image, cannot accurately map the local icing image to the three-dimensional model of the aircraft, and cannot realize dynamic monitoring of aircraft icing in the deicing process. The matching of 2D pixels and 3D space points is realized through geometric constraints combined with cameras and radars, but such methods face problems such as high cost of hardware devices, complex calibration and fusion between sensors, and difficulty in device deployment. SUMMARY

[0007] In view of the above problems in the prior art, the aircraft icing dynamic monitoring method and system based on three-dimensional visualization mapping provided by the present application solve the problems of the prior art that false matching is prone to occur, accurate visualization of the deicing process of the aircraft cannot be realized, and the distribution and melting of the icing in the deicing process cannot be grasped in real time.

[0008] In order to achieve the above-mentioned purpose of the application, the technical scheme adopted by the present application is as follows: an aircraft icing dynamic monitoring method based on three-dimensional visualization mapping, which realizes dynamic monitoring of aircraft icing through two stages of global positioning mapping and local updating mapping; The global positioning mapping stage includes: obtaining a global icing image data set of the aircraft to be deiced; obtaining global aircraft icing point clouds and non-icing aircraft model point clouds from the global icing image data set of the aircraft to be deiced; performing point cloud registration on the global aircraft icing point clouds and the non-icing aircraft model point clouds; identifying icing points of the aircraft to be deiced according to the point cloud registration result, reconstructing the identified icing points into an icing three-dimensional model, fusing the obtained icing three-dimensional model into a non-icing aircraft three-dimensional model, and obtaining an aircraft icing three-dimensional model; outputting the fused aircraft icing three-dimensional model to a platform visualization interface to realize visualization of global positioning mapping of aircraft icing; The local updating mapping stage includes: obtaining local icing image data of each part of the aircraft to be deiced at each time period in the deicing process; calculating the camera pose and the intrinsic matrix corresponding to the local icing image according to the local icing image data; mapping the icing thickness information in the local icing image to the aircraft icing three-dimensional model according to the camera pose and the intrinsic matrix of the local icing image; adjusting and updating the number of Gaussians and the Gaussian distribution density of the aircraft icing three-dimensional model according to the icing thickness information of each pixel block mapped to the corresponding position of the aircraft icing three-dimensional model, and realizing dynamic updating of the ablation of icing of each part of the aircraft.

[0009] The present application also provides an aircraft icing dynamic monitoring system based on three-dimensional visualization mapping, comprising: An image acquisition module is configured to acquire a global ice accumulation image data set of the aircraft to be deiced and local ice accumulation image data of each part of the aircraft to be deiced at each time period during the deicing process. An image processing module is configured to perform point cloud reconstruction, point cloud registration, ice accumulation point identification, and calculation of camera pose and intrinsic matrix corresponding to the local ice accumulation image according to the global ice accumulation image data set and the local ice accumulation image data. A three-dimensional model processing module is configured to reconstruct the identified ice accumulation points into an ice accumulation three-dimensional model, fuse the obtained ice accumulation three-dimensional model into a three-dimensional model of the aircraft without ice accumulation, map ice thickness information in the local ice accumulation image into the aircraft ice accumulation three-dimensional model according to the camera pose and intrinsic matrix of the local ice accumulation image, and adjust and update the number and distribution density of Gaussians of the aircraft ice accumulation three-dimensional model. A visualization module is configured to show the ice ablation process of the aircraft ice accumulation three-dimensional model during deicing.

[0010] The present application has the following advantages: The present application maps the collected aircraft ice accumulation image into the three-dimensional model of the aircraft by combining point cloud processing, pose estimation, and image mapping, and realizes positioning and total ice accumulation estimation of global ice accumulation of the aircraft before deicing and real-time updating of each local ice accumulation during deicing. The global ice accumulation positioning and mapping can help the deicing staff to comprehensively master the ice distribution range and total ice accumulation of each part of the aircraft, so as to reasonably prepare the required deicing liquid and avoid resource waste caused by using excessive deicing liquid. The real-time mapping of local ice ablation can realize dynamic updating of the ice ablation of each part of the aircraft. With the aid of the aircraft ice accumulation three-dimensional visualization monitoring platform, the staff can intuitively observe the whole process of the aircraft deicing. According to the residual ice accumulation of each part, the deicing liquid is sprayed in the area with more residual ice accumulation, so as to improve the efficiency and accuracy of deicing and complete the deicing operation more efficiently and accurately. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A flow chart of an aircraft ice accumulation dynamic monitoring method based on three-dimensional visualization mapping provided by the embodiment. DETAILED DESCRIPTION

[0012] The specific embodiments of the present application are described below to facilitate understanding of the present application by those skilled in the art, but it should be clear that the present application is not limited to the scope of the specific embodiments. It is obvious to those skilled in the art that all the inventions utilizing the concept of the present application are within the scope of the present application as defined and limited by the appended claims.

[0013] As Figure 1As shown, in one embodiment of the application, the aircraft ice accretion dynamic monitoring method based on three-dimensional visualization mapping is composed of two stages of global positioning mapping and local updating mapping. This embodiment takes Airbus A320 aircraft as an example for explanation and description.

[0014] The global positioning mapping stage includes: A1, obtaining a global ice accretion image data set of the aircraft to be deiced.

[0015] When the aircraft to be deiced arrives at the designated parking position, a global ice accretion image is taken around the Airbus A320 aircraft using the Hikvision robot MV-CH050-10UC industrial camera, and the image data set is transmitted to the image processing module as input for reconstructing the global aircraft ice accretion point cloud.

[0016] A2, obtaining a global aircraft ice accretion point cloud and a non-ice accretion aircraft model point cloud from the global ice accretion image data set of the aircraft to be deiced.

[0017] The collected global images are reconstructed into three-dimensional point clouds using the colmap tool At the same time, during the point cloud reconstruction process, the SFIT features calculated in each global image need to be saved for image feature matching in local ice mapping. Then the aircraft three-dimensional model of Airbus A320 aircraft is called from the aircraft three-dimensional model database to the aircraft ice three-dimensional mapping visualization platform, and the point cloud is read from the aircraft three-dimensional model . Downsampling and denoising are performed on the point cloud

[0018] A3, point cloud registration of the global aircraft ice accretion point cloud and the non-ice accretion aircraft model point cloud.

[0019] The process of point cloud registration of the global aircraft ice accretion point cloud and the non-ice accretion aircraft model point cloud includes three stages of coarse registration, scale calibration and fine registration in turn.

[0020] In the coarse registration stage, the main purpose is to solve the problem of unknown initial pose, provide an initial transformation matrix for fine registration, and avoid falling into local optimum in fine registration. The coarse registration is specifically to align the principal directions of the global aircraft ice accretion point cloud and the non-ice accretion aircraft model point cloud by using principal component analysis method, and the specific calculation process is as follows: The center of mass of the global aircraft ice accretion point cloud and the center of mass of the non-ice accretion aircraft model point cloud are calculated, and the expression is as follows:

[0021]

[0022] Wherein, the center of mass of the global aircraft ice accretion point cloud is represented by denotes the total number of global aircraft icing point clouds, denotes the first point in the global aircraft icing point cloud; denotes the center of the aircraft model point cloud without icing, is the total number of the aircraft model point cloud without icing, denotes the first point in the aircraft model point cloud without icing; The covariance matrix of the global aircraft icing point cloud and the covariance matrix of the aircraft model point cloud without icing are calculated, and the expression is as follows:

[0023]

[0024] wherein, denotes the covariance matrix of the global aircraft icing point cloud, denotes the covariance matrix of the aircraft model point cloud without icing; The covariance matrix of the global aircraft icing point cloud and the covariance matrix of the aircraft model point cloud without icing are decomposed to obtain the eigenvector matrix and the eigenvector matrix ; The best rotation matrix R between and is calculated; the translation vector between is calculated. t ; R and t are applied to the global aircraft icing point cloud and the aircraft model point cloud without icing to align the principal directions of the two groups of point clouds.

[0025] After the coarse registration is completed, the source point cloud needs to be scaled, and in this embodiment, the diagonal length of the point cloud bounding box is used for correction, so that the size of the source point cloud is consistent with the target point cloud. In this embodiment, the actual size of the bounding box of the aircraft model point cloud without icing of Airbus A320 is 37.57m x 34.10m x 10.01m, and the diagonal length of the bounding box is about 51.716m. The diagonal length of the bounding box of the global icing point cloud is calculated, and the ratio of the two is the scale calibration factor. The scale calibration error is not more than 1mm, because this process will directly affect the calculation of the icing thickness and the total volume of icing. The specific operation is as follows: first, the bounding boxes of the source point cloud and the target point cloud are obtained, and then the diagonal lengths of the bounding boxes of the two point clouds are calculated and respectively, and then the scale scaling factor is calculated as follows:​​

[0026] The scale matrix is defined as:

[0027] In order to better achieve the goal of scale, it is necessary to move the ice accretion point cloud to the origin (0, 0, 0) for scale, and then move back to the starting position after scaling, so it is necessary to obtain the center point of the ice accretion point cloud Then define two translation matrices And :

[0028] Wherein, the translation matrix makes the ice accretion point cloud move to the origin, makes the ice accretion point cloud move back to the starting position from the origin.

[0029] The final scale transformation matrix is:

[0030] Using the scale transformation matrix to operate the ice accretion point cloud, the purpose of point cloud scale calibration is achieved.

[0031] Then, the ice accretion point cloud after scale correction is refined by using the iterative closest point algorithm (ICP), so that the distance between the corresponding points of the two groups of point clouds is further reduced. Based on the rotation matrix R and the translation vector t calculated in the coarse registration, the following steps are taken: (1) For each point in the transformed ice accretion point cloud , find the closest point in the ice-free aircraft point cloud to form a corresponding point pair ; (2) Based on the current corresponding point pair, solve the rotation matrix and the translation vector that minimize the error function ; (3) Update the position of the ice accretion point cloud using the newly calculated transformation matrix .

[0032] Repeat the above three steps until one of the following conditions is met, then stop iteration: ① the difference between the error functions of the previous two iterations is less than the threshold; ② the change of the transformation matrix between the previous two iterations is less than the threshold; ③ the maximum number of iterations is reached.

[0033] After stopping iteration, output the final ice accretion point cloud and the ice-free point cloud registration result .

[0034] A4, identify ice accretion points of the aircraft to be deiced according to the point cloud registration result, reconstruct the identified ice accretion points into an ice accretion three-dimensional model, and fuse into the three-dimensional model of the aircraft without ice accretion to obtain the three-dimensional model of the aircraft ice accretion.

[0035] The specific method for identifying ice accretion points of the aircraft to be deiced according to the point cloud registration result is: Calculate the distance between any point in the point cloud registration result and the corresponding point in the point cloud of the aircraft without ice accretion ; Calculate the average value and standard deviation of the distance between all points in the point cloud registration result and the corresponding points in the point cloud of the aircraft without ice accretion, and the expression is:

[0036]

[0037] wherein, is the average value, is the total number of the point cloud of the aircraft without ice accretion; represents the standard deviation; Define the distance threshold , and the expression is:

[0038] wherein, represents the sensitivity coefficient; When is greater than the distance threshold , then the point is identified as a preliminary ice accretion point.

[0039] However, the above method of identifying ice accretion points by simply comparing the threshold value has some errors, so in this embodiment, based on the characteristics that the ice accretion area of the aircraft is larger than the skin ice-free area, the curvature gradient changes greatly and the normal vector divergence is high, a correction method is proposed to reduce the error of ice accretion identification.

[0040] After identifying the preliminary ice accretion points, the error points in the preliminary ice accretion points are removed by the ice accretion correction method, and the specific method is: Use KD-Tree to find the l nearest neighbor points of the preliminary ice accretion points to obtain a neighborhood point set; Fit a quadratic surface to the neighborhood points in the neighborhood point set, and the expression is:

[0041] wherein, , , , , , are constant coefficients, ( , , ) represents the coordinates of the neighborhood points; Calculate the average curvature of the preliminary ice accumulation point and its neighborhood points , whose expression is: ; Take the preliminary ice accumulation point as the origin, and the neighborhood local normal vector as the Z-axis to establish a local ice accumulation point coordinate system. Assume that in this coordinate system, the average curvature of the ice accumulation point is a continuous function, and define its gradient vector as: ; Here, the curvature gradient is calculated using the difference approximation method, and the calculation formula for difference approximation is: ; Similarly, calculate according to the above formula to obtain the point cloud curvature gradient vector of the preliminary ice accumulation point neighborhood; Calculate the unit vector corresponding to the curvature gradient vector of each point in the preliminary ice accumulation point neighborhood. If the curvature gradient of the neighborhood point is 0, ignore the point; Calculate the local curvature gradient direction consistency, whose expression is:

[0042] wherein, represents the local curvature gradient direction consistency of the preliminary ice accumulation point, is the unit vector corresponding to the curvature gradient vector of the preliminary ice accumulation point, is the unit vector corresponding to the curvature gradient vector of the neighborhood point of the preliminary ice accumulation point; Obtain the normal vector of the preliminary ice accumulation point and the normal vectors of its neighborhood points. Calculate the cosine value between the normal vector of each neighborhood point of the preliminary ice accumulation point and the normal vector of the preliminary ice accumulation point with the normal vector of the preliminary ice accumulation point as the reference; Take the variance of the calculated cosine value as an index to measure the divergence degree of the normal vector of the ice accumulation point, whose expression is:

[0043] wherein, represents the divergence degree of the normal vector, represents the angle between the normal vector of the preliminary ice accumulation point and the normal vector of the neighborhood point of the preliminary ice accumulation point, represents the cosine function, ​the average value of the cosine of the angle between the normal vector of the initial icing point and the normal vector of its neighborhood points; The initial icing point with the local curvature gradient direction consistency greater than 0.9 and the normal vector divergence degree less than 0.1 is removed as an error point.

[0044] The icing point after the error point is removed is reconstructed into an icing three-dimensional model by using a three-dimensional Gaussian sputtering technology, and the obtained icing three-dimensional model is fused into the icing-free aircraft three-dimensional model. When the obtained icing three-dimensional model is fused into the icing-free aircraft three-dimensional model, the icing three-dimensional model is aligned with the skin surface of the icing-free aircraft three-dimensional model by minimizing a regularization term, and the specific method is as follows: The Gaussian density function of any spatial position point of the aircraft icing skin surface is calculated, and the expression is as follows:

[0045] wherein, The Gaussian density of any spatial position point of the aircraft icing skin surface is represented by The Gaussian ellipsoid representing the icing, , , The covariance of Gauss, the opacity of Gauss and the mean of Gauss are represented by The inverse of the covariance matrix of Gauss is represented by The superscript T in the above formula represents the transpose of the matrix; The exponential function with the natural logarithm as the base is represented by The ideal Gaussian density function is calculated, and the expression is as follows:

[0046] wherein, The ideal Gaussian density of any spatial position point of the aircraft icing skin surface is represented by The tangential scaling factor is represented by The tangential coordinate of any spatial position point of the aircraft icing skin surface is represented by The density loss is defined, and the expression is as follows:

[0047] wherein, The density loss value is represented by The distance between the center point of the Gaussian ellipsoid and the point is calculated, and the expression is as follows:

[0048] wherein, The distance between the center point of the Gaussian ellipsoid and the point the distance between the two, is the Gaussian ellipsoid representing the ice accretion in the ice accretion three-dimensional model, is the Gaussian ellipsoid is the mean value, is the normal vector at the point on the aircraft skin surface, represents the Euclidean norm; calculating the parallel relationship between the normal vector of the Gaussian and the normal vector of the aircraft skin surface at the point , which is expressed as:

[0049] wherein, represents the parallel relationship between the normal vector of the Gaussian and the normal vector of the aircraft skin surface at the point , represents the normal vector of the Gaussian ellipsoid , represents the norm; Finally, in order to optimize the ice accretion on all points of the aircraft icing surface, so that it can be aligned with the skin surface, a regularization term is defined, and the calculation formula is:

[0050] wherein, represents the regularization term, represents the set of all points on the skin surface of the ice-free aircraft three-dimensional model.

[0051] A5, output the fused aircraft ice accretion three-dimensional model to the platform visualization interface, realize the visualization of the global positioning mapping of the aircraft ice accretion.

[0052] By minimizing the regularization term, the ice accretion three-dimensional model is aligned with the aircraft skin surface. In the process of adjusting the Gaussian distribution, the thickness of the ice accretion at each place is not changed, only the Gaussian distribution of the ice accretion model at the fitting place of the aircraft model is adjusted, and the ice accretion reconstruction accuracy is 1mm. The fused aircraft ice accretion three-dimensional model is output to the visualization interface of the system platform, and the total volume of the aircraft ice accretion is calculated by using the range and thickness of the ice accretion, and is displayed in the visualization interface, which is fed back to the staff to assist in completing the deicing preparation work.

[0053] The local update mapping stage includes: B1, obtaining the local ice accretion image data of each part of the aircraft to be deiced at each time period in the deicing process.

[0054] In the aircraft deicing process, the related ice detection technology is used to shoot the local ice image of the aircraft at each angle of the Airbus A320 aircraft, wherein the resolution and shooting distance of the local ice image and the global ice image are consistent, and each pixel in the local ice image contains the thickness information of the aircraft ice.

[0055] B2, calculating the camera pose and the intrinsic matrix corresponding to the local ice image according to the local ice image data.

[0056] The specific method is: The SIFT algorithm is used to perform feature matching on the obtained local ice image and the global ice image data set, so as to obtain the number of matching points and the coordinates of the matching points corresponding to the matching points; According to the consistency of the number and coordinates of the matching points, the best matching image of the local ice image is found in the global ice image data set; The 2D-2D matching point relationship of the local ice image and the best matching image of the local ice image and the 2D-3D matching point relationship of the best matching image of the local ice image and the three-dimensional model of the aircraft ice are combined to establish the 2D-3D matching point relationship of the local ice image and the three-dimensional model of the aircraft ice; The camera intrinsic and pose parameters corresponding to the best matching image of the local ice image are used as the initial camera pose and the initial intrinsic of the local ice image, and then the Bundle Adjustment (BA) algorithm is used to jointly optimize the camera pose and the intrinsic, when the objective function converges, the iteration is stopped, and the optimal camera pose and the intrinsic are obtained, and the optimal camera pose and the intrinsic are defined as the optimal camera parameters.

[0057] The objective function is specifically:

[0058]

[0059]

[0060] Among them, is the objective function, represents the initial intrinsic, represents the initial camera pose, represents the re-projection error, represents any point in the three-dimensional model of the aircraft ice, represents the point in the plane by projecting to the plane by the initial camera pose and the initial intrinsic, is a projection function.

[0061] B3, mapping the ice thickness information in the local ice accretion image to the aircraft ice accretion three-dimensional model according to the camera pose and the intrinsic matrix of the local ice accretion image; Specifically, In the local ice accretion image, each pixel needs to save depth information and ice thickness information. In order to reduce noise and improve statistical stability, each 64 pixels (8x8 pixels) in the image is grouped into a pixel block, and the average ice thickness of each pixel block is calculated, and the expression is:

[0062] Among them, represents the average ice thickness of each pixel block, represents the ice thickness at the pixel, represents the pixel index; The average ice thickness of each pixel block is mapped to the Gaussian distribution area in the global aircraft ice accretion three-dimensional model using the optimal camera parameters, and the specific is: The center coordinates of each pixel block are projected into the camera coordinate system to obtain the three-dimensional position coordinates of each pixel block in the camera coordinate, and the expression is:

[0063] Among them, represents the three-dimensional position coordinates of the pixel block in the camera coordinate, represents the inverse matrix of the intrinsic matrix, is the center coordinates of the pixel block, is the depth information; The three-dimensional position coordinates of each pixel block in the camera coordinate are converted into the world coordinate of the aircraft ice accretion three-dimensional model, and the expression is:

[0064] Among them, is the position of the ice point of the local ice accretion image mapped to the Gaussian distribution area in the aircraft ice accretion three-dimensional model.

[0065] B4, adjusting and updating the Gaussian number and Gaussian distribution density of the aircraft ice accretion three-dimensional model according to the ice thickness information of each pixel block mapped to the corresponding position of the aircraft ice accretion three-dimensional model.

[0066] When the ice thickness changes, the number of Gaussian ellipsoids is adjusted according to the change amplitude of the ice thickness. Specifically, when the ice thickness in a certain pixel block changes more than 1mm, the following operations are performed: Remove the Gaussian ellipsoid: remove the Gaussian ellipsoid whose center point is higher than the current ice thickness based on the center point of the Gaussian ellipsoid, to ensure that the Gaussian ellipsoid reflects the real-time ice accretion.

[0067] Splitting Gaussian ellipsoid: for the Gaussian ellipsoid whose center point just falls on the boundary of the ice thickness, split it into two, keep the sub-Gaussian whose center point is below the ice thickness, and remove the other sub-Gaussian.

[0068] Processing Gaussian ellipsoid at the boundary of the pixel block: if the center point of the Gaussian ellipsoid is located at the boundary of two pixel blocks, and the ice thickness of the adjacent two pixel blocks differs by more than 1 mm, it is attributed to the pixel block with higher ice thickness.

[0069] In addition, in the initial local ice mapping process, due to the error of global ice positioning detection, the ice thickness value may abnormally increase. In order to correct this problem, the application adopts the method of cloning Gaussian ellipsoid and increasing its corresponding ice thickness information, so as to more accurately represent the ice distribution. It is worth noting that the three-dimensional model of the aircraft in the application is based on mesh reconstruction, while the three-dimensional model of the ice is based on Gaussian ellipsoid reconstruction. When updating the ice ablation in real time, only the Gaussian ellipsoid used for ice reconstruction is edited and adjusted, therefore, the local ice update mapping process will not affect the overall structure of the three-dimensional model of the aircraft.

[0070] After adjusting the number of Gaussians and the density of Gaussian distribution, the density of Gaussian distribution is optimized again by using the method mentioned in the global positioning mapping stage, that is, by minimizing the regularization term, the algorithm of aligning the ice Gaussian distribution with the skin surface, further adjusting the density of Gaussian distribution. The purpose of Gaussian density optimization in this step is to make the adjusted ice surface more smooth, so as to improve the visualization effect of local ice mapping.

[0071] In the process of dynamically updating the ice ablation of each part of the aircraft, a multi-thread synchronous updating strategy is adopted, that is, multiple local aircraft ice images are processed at the same time, and the real-time ice situation of each part of the aircraft is updated synchronously. The deicing staff can observe the gradual ablation process of the ice of each part of the aircraft on the ice visualization interface of the system platform. At the same time, the ice thickness mapping accuracy based on Gaussian editing reaches 1 mm, realizing millimeter-level aircraft ice visualization.

[0072] In summary, the present application combines point cloud processing, pose estimation, image mapping and other technical means to map the collected aircraft icing images to the aircraft three-dimensional model, realizing the positioning and total amount estimation of global icing before deicing, and the real-time update of each local icing during deicing. Among them, the global icing positioning and mapping can help the deicing staff to fully grasp the icing distribution range and total icing amount of each part of the aircraft, so as to reasonably prepare the required deicing liquid and avoid resource waste caused by excessive use of deicing liquid; the real-time mapping of local icing ablation can realize the dynamic update of the ablation of each part of the aircraft. With the help of the aircraft icing three-dimensional visualization monitoring platform, the staff can intuitively observe the whole process of aircraft deicing. According to the residual icing condition of each part, the deicing liquid is sprayed in the area with more residual icing, improving the efficiency and accuracy of deicing, so as to more efficiently and accurately complete the deicing operation.

Claims

1. An aircraft ice accretion dynamic monitoring method based on three-dimensional visualization mapping, characterized in that, The dynamic monitoring of aircraft icing is carried out in two stages: global positioning mapping and local update mapping. The global positioning and mapping phase includes: Obtain the global icing image dataset of the aircraft to be de-iced; The global icing point cloud and the non-icing model point cloud of the aircraft are obtained from the global icing image dataset of the aircraft to be de-iced. Register the global aircraft icing point cloud with the non-icing aircraft model point cloud. Based on the point cloud registration results, the icing points of the aircraft to be de-iced are identified, the identified icing points are reconstructed into a 3D model of icing, and the obtained 3D model of icing is merged into the 3D model of the aircraft without icing to obtain the 3D model of the aircraft with icing. The fused 3D model of aircraft icing is output to the platform's visualization interface to visualize the global positioning and mapping of aircraft icing. The partial update mapping phase includes: Acquire local ice accumulation image data of different parts of the aircraft at different times during the de-icing process; Calculate the camera pose and intrinsic parameter matrix corresponding to the local icing image based on the local icing image data; Based on the camera pose and intrinsic parameter matrix of the local icing image, the icing thickness information in the local icing image is mapped to the aircraft icing 3D model; Based on the ice thickness information mapped to the corresponding position of the aircraft icing 3D model for each pixel block, the Gaussian quantity and Gaussian distribution density of the aircraft icing 3D model are adjusted and updated to achieve dynamic updates of the ice melting status of various parts of the aircraft.

2. The method of claim 1, wherein, The process of registering the global aircraft icing point cloud with the non-icing aircraft model point cloud includes three stages: coarse registration, scale calibration, and fine registration. Coarse registration involves aligning the principal directions of the global aircraft icing point cloud and the non-icing aircraft model point cloud using principal component analysis. Scale calibration involves adjusting the scale of the global aircraft icing point cloud to match that of the non-icing aircraft model point cloud. Fine registration involves iteratively optimizing the scale-calibrated global aircraft icing point cloud using the iterative nearest point algorithm to minimize the distance between corresponding points in the global aircraft icing point cloud and the non-icing aircraft model point cloud, thus obtaining the point cloud registration result.

3. The method of claim 2, wherein, The specific method for identifying the icing points of the aircraft to be de-iced based on the point cloud registration results is as follows: Computing a point cloud registration result for any point Distance between corresponding points in the ice-free aircraft model point cloud ; The average and standard deviation of the distances between all points in the point cloud registration result and their corresponding points in the point cloud of the non-icing aircraft model are calculated using the following expression: wherein, is the average value, is the total number of point clouds of the ice-free aircraft model; denotes the standard deviation; Defining a distance threshold whose expression is: wherein represents the sensitivity coefficient; When greater than a distance threshold then the point is identified as a preliminary ice accretion point.

4. The method according to claim 3, characterized in that, After identifying the initial icing points, error points in the initial icing points are eliminated using an icing correction method. The specific method is as follows: Find the initial icing point l Find the nearest neighbor points to obtain the neighborhood point set; Fitting a quadratic surface to the neighborhood points of the neighborhood point set, its expression is: in, , , , , , All are constant coefficients, , , () represents the coordinates of a neighboring point; Calculate the average curvature of the initial icing point and its neighborhood points. Its expression is: ； Calculate the point cloud curvature gradient vector in the neighborhood of the initial icing point. ; Calculate the unit vector corresponding to the curvature gradient vector of each point in the neighborhood of the initial icing point; The expression for calculating the consistency of the local curvature gradient direction is as follows: in, This indicates the consistency of the local curvature gradient direction at the initial icing point. This is the unit vector corresponding to the curvature gradient vector of the initial icing point. This is the unit vector corresponding to the curvature gradient vector of the neighborhood points of the initial icing point; Obtain the normal vector of the initial ice accumulation point and the normal vectors of its neighboring points. Using the normal vector of the initial ice accumulation point as a reference, calculate the cosine value of the angle between the normal vector of each neighboring point of the initial ice accumulation point and the normal vector of the initial ice accumulation point. The variance of the calculated cosine value is used as an indicator to measure the degree of divergence of the normal vector at the ice accumulation point, and its expression is: in, Indicates the degree of divergence of the normal vector. This represents the angle between the normal vector of the initial icing point and the normal vectors of its neighboring points. Represents the cosine function. This represents the mean value of the cosine of the angle between the normal vector of the initial ice accumulation point and the normal vectors of its neighboring points; Preliminary icing points with a local curvature gradient direction consistency greater than 0.9 and a normal vector divergence degree less than 0.1 are discarded as error points.

5. The method according to claim 4, characterized in that, When fusing the obtained icing-covered 3D model into the ic-free aircraft 3D model, the skin surfaces of the icing-covered 3D model and the ic-free aircraft 3D model are aligned by minimizing the regularization term. The specific method is as follows: Calculate any spatial location on the icy surface of an aircraft skin The Gaussian density of is expressed as: in, Represents any spatial location on the icy surface of the aircraft skin. Gaussian density, To represent the Gaussian ellipsoid of ice, , , Let represent the covariance, opacity, and mean of the Gaussian, respectively. This represents the inverse of the Gaussian covariance matrix; The superscript T in the matrix indicates the transpose of the matrix; This represents an exponential function with the base of the natural logarithm. The ideal Gaussian density is calculated using the following expression: in, Represents any spatial location on the icy surface of the aircraft skin. Ideal Gaussian density, Tangential scaling factor The tangential coordinates of any point in space on the icy surface of the aircraft skin; The density loss is defined as follows: in, Indicates the density loss value; Calculate the center point and point of the Gaussian ellipsoid The distance between them is expressed as: in, Represent the center point and point of the Gaussian ellipsoid The distance between them The Gaussian ellipsoid representing ice in the 3D model of ice accumulation. For Gaussian ellipsoid The mean, Points on the surface of the aircraft skin The normal vector at that point, Denotes the Euclidean norm; Calculate the normal vector of Gaussian at point [point]. The parallel relationship between the normal vectors at a given point is expressed as follows: in, This represents the normal vector of Gauss intersecting the plane skin surface at a point. The parallel relationship between the normal vectors at a given location. Represents the Gaussian ellipsoid The normal vector, Represents the norm; By minimizing the regularization term, the skin surfaces of the icing 3D model and the non-icing aircraft 3D model are aligned. The expression for the regularization term is: in, Represents the regularization term. This represents the set of all points on the skin surface of a 3D model of an icing-free aircraft.

6. The method according to claim 4, characterized in that, The specific method for calculating the camera pose and intrinsic parameter matrix corresponding to the local icing image based on local icing image data is as follows: The SIFT algorithm is used to perform feature matching between the acquired local icing images and the global icing image dataset to obtain the number of matching points and the coordinates of the matching points; Based on the number of matching points and the consistency of their coordinate positions, find the best matching image for the local icing image in the global icing image dataset; By combining the 2D-2D matching point relationship between local icing images and the best matching image of local icing images and the 2D-3D matching point relationship between the best matching image of local icing images and the 3D model of aircraft icing, a 2D-3D matching point relationship between local icing images and the 3D model of aircraft icing is established. The camera intrinsics and pose parameters corresponding to the best matching image of the local icing image are used as the initial camera pose and initial intrinsics of the local icing image. The camera pose and intrinsic parameters are iteratively optimized. When the objective function converges, the iteration is stopped, and the optimal camera pose and intrinsic parameters are obtained. The optimal camera pose and intrinsic parameters are defined as the optimal camera parameters.

7. The method according to claim 6, characterized in that, The objective function for iterative optimization of camera pose and intrinsic parameters is as follows: in, Let be the objective function. Indicates the initial intrinsic parameters. This indicates the initial camera pose. This indicates the reprojection error. This represents any point in the 3D model of aircraft icing. This indicates that the initial camera pose and initial intrinsic parameters will be used to... Points projected onto the plane This is the projection function.

8. The method according to claim 7, characterized in that, Based on the camera pose and intrinsic parameter matrix of the local icing image, the local icing image is mapped onto the aircraft 3D icing model, specifically as follows: The average ice thickness of each pixel block is calculated by grouping 64 pixels into a pixel block in the local icing image. The expression is as follows: in, This represents the average ice thickness of each pixel block. Indicates the first Ice thickness at each pixel Indicates the pixel index; The average icing thickness of each pixel block is mapped to a Gaussian distribution region in the global aircraft icing 3D model using optimal camera parameters, specifically: Projecting the center coordinates of each pixel block onto the camera coordinate system yields the 3D position coordinates of each pixel block in the camera coordinate system, expressed as: in, This represents the three-dimensional position coordinates of the pixel block in camera coordinates. The inverse matrix of the intrinsic parameter matrix. The center coordinates of the pixel block For depth information; The expression for transforming the 3D position coordinates of each pixel block in camera coordinates to the world coordinates of the aircraft icing 3D model is as follows: in, The inverse matrix representing the camera rotation matrix; This maps the icing points in the local icing image to the locations of Gaussian distribution regions in the 3D model of aircraft icing.

9. The method according to claim 8, characterized in that, The Gaussian quantity and Gaussian distribution density of the aircraft icing 3D model are adjusted and updated based on the icing thickness information mapped to the corresponding position of each pixel block in the model. Specifically: The number of Gaussian ellipsoids is adjusted based on the variation in ice thickness within each pixel block. When the ice thickness variation within a pixel block exceeds 1mm, the following operations are performed: Remove Gaussian ellipsoid: Using the center point of the Gaussian ellipsoid as a reference, remove the Gaussian ellipsoid whose center point is higher than the current pixel block ice thickness; Split Gaussian ellipsoid: Split the Gaussian ellipsoid whose center point falls on the boundary of the current pixel block ice thickness into two sub-Gaussian ellipsoids, retain the sub-Gaussian ellipsoid whose center point is lower than the current pixel block ice thickness, and remove the other sub-Gaussian ellipsoid; Processing Gaussian ellipsoids at pixel block boundaries: If the center point of a Gaussian ellipsoid is located at the boundary between two pixel blocks, and the difference in ice thickness between the two pixel blocks is greater than 1 mm, then the Gaussian ellipsoid is assigned to the pixel block with the higher ice thickness. As the ice thickness value increases, the Gaussian ellipsoid is cloned to increase the ice thickness. The specific method for updating and adjusting the Gaussian distribution density is to adjust the Gaussian distribution density by minimizing the regularization term on the ice surface after the adjustment quantity.

10. A system for implementing the aircraft icing dynamic monitoring method based on three-dimensional visualization mapping as described in any one of claims 1 to 9, characterized in that, include: The image acquisition module is used to acquire the global icing image dataset of the aircraft to be de-iced and the local icing image data of the aircraft at different times and locations during the de-icing process. The image processing module is used to perform point cloud reconstruction, point cloud registration, ice point recognition, and calculation of camera pose and intrinsic parameter matrix corresponding to local ice images based on global ice image dataset and local ice image data. The 3D model processing module is used to reconstruct the identified icing points into an icing 3D model, integrate the obtained icing 3D model into the ic-free aircraft 3D model, map the icing thickness information in the local icing image to the aircraft icing 3D model based on the camera pose and intrinsic parameter matrix of the local icing image, and adjust and update the Gaussian number and Gaussian distribution density of the aircraft icing 3D model. The visualization module is used to show the ice melting process of a 3D model of aircraft ice during de-icing.

Images