Method for detecting ice on the surface of an aircraft using light field speckle imaging, reconstruction system and device

Abstract

The application belongs to the technical field of active three-dimensional reconstruction and light field three-dimensional imaging, and discloses a method for detecting ice on the surface of an airplane, a reconstruction system and equipment based on light field speckle imaging. A light field speckle measurement system is built; equivalent multi-camera calibration is performed based on the light field camera in the built light field speckle measurement system; ice layer local topography recovery of the surface of an airplane is performed based on the calibrated light field speckle measurement system; two frames of local point clouds are acquired at two different positions, and key points are extracted according to the repeated area features of the two frames of point clouds. The method for reconstructing the three-dimensional topography of the ice layer on the surface of an airplane based on light field speckle imaging can simultaneously capture the spatial distribution and angle distribution information of light in the mode of a single camera, and the active three-dimensional reconstruction mode is adopted in combination with an infrared speckle projector to acquire the distribution model of the ice layer thickness of the whole airplane through point cloud splicing technology, so that the hardware equipment is simplified and the cost is saved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of active three-dimensional reconstruction and light field three-dimensional imaging, and particularly relates to a method for detecting ice on the surface of an airplane, a reconstruction system and equipment for light field speckle imaging. BACKGROUND

[0002] Airplane icing is an important cause of airplane accidents and is one of the six meteorological factors affecting airplane flight. Under icing conditions, the ice on the surface of an airplane can increase the mass of the airplane, affect the control performance of the airplane, and cause serious damage to the power performance of the airplane, and even affect flight safety. The ice on the surface of an airplane also has an important influence on many sensors. When the ice is on the surface of a sensor, it can affect the detection data of the sensor and have an adverse effect on the state detection of the engine and the judgment of the flight parameters by the pilot.

[0003] In traditional three-dimensional reconstruction technology, binocular stereo vision, motion recovery structure, and structured light three-dimensional reconstruction methods are often used. However, when the surface of an airplane is used as the object to be measured, it has the significant characteristics of large size and weak texture, and therefore, the traditional methods cannot be used for reconstruction. Therefore, it is of great significance to study a fast, accurate, and flexible three-dimensional reconstruction method for the ice layer on the surface of an airplane.

[0004] When there is ice on the surface of an airplane, detecting the distribution of the residual ice on the surface of the airplane is of great significance for improving the ice removal efficiency of the airplane. Since the surface of an airplane has no obvious texture characteristics and large-scale measurement needs to be achieved, the special internal structure of a light field camera can record the propagation path of light in a single camera mode, which enables the light field camera to be used for three-dimensional reconstruction. The speckle pattern has good autocorrelation, which also provides good conditions for stereo matching.

[0005] Through the above analysis, the problems and defects of the prior art are as follows:

[0006] (1) In the prior art, the detection accuracy of the distribution of the residual ice on the surface of an airplane is low, and the ice removal efficiency of the airplane is poor.

[0007] (2) The prior art cannot simultaneously capture the spatial distribution and angular distribution information of light in a single camera mode. The active three-dimensional reconstruction method combined with an infrared speckle projector and the point cloud splicing technology are used to obtain the distribution model of the ice layer thickness of the entire airplane, which requires a large number of hardware devices to complete the related work, and the cost is increased. SUMMARY

[0008] To overcome the problems in the related art, the present application provides a method for detecting ice on the surface of an airplane, a reconstruction system and equipment for light field speckle imaging.

[0009] The technical solution is as follows: the plane surface ice detection method based on light field speckle imaging of an airplane comprises the following steps:

[0010] S1, a light field speckle measurement system is built;

[0011] S2, based on the light field camera in the built light field speckle measurement system, equivalent multi-view camera calibration is performed;

[0012] S3, based on the calibrated light field speckle measurement system, the local topography of the ice layer on the surface of the airplane is recovered; two frames of local point clouds are acquired at two different positions, key points are extracted according to the repeated area features of the two frames of point clouds, a three-dimensional rigid body transformation matrix of the two groups of key points is calculated, an initial registration matrix of the latter frame of point clouds relative to the former frame of point clouds is obtained, the nearest points of the initial registration matrix are iterated constantly through the ICP point cloud splicing technology, and the iteration is terminated when the error is less than a given threshold value, so that the registration matrix is obtained;

[0013] S4, according to the point cloud splicing between frames, images of the whole airplane are acquired at different positions, the splicing method is used to splice the point clouds between adjacent frames in turn according to the shooting order, and a distribution model of the ice layer thickness of the whole airplane is recovered.

[0014] In step S1, the light field speckle measurement system comprises a light field camera, a speckle projector and a support; the light field camera is arranged on the left side of the support, the speckle projector is installed on the right side of the light field camera, and the relative positions are fixed and fixed on the support;

[0015] The angle between the light field camera and the speckle projector is adjusted so that the light field camera can completely shoot the image after speckle modulation, then the light field camera and the speckle projector are fixed, the relative positions between the light field camera and the speckle projector are kept unchanged, and the building of the light field speckle measurement system is completed.

[0016] In one embodiment, the microlens array inside the light field camera is equivalent to a multi-view camera, the camera coordinate system is established on the center point of the main lens, the internal parameters of the light field camera are calibrated by using a chessboard, and the rotation and translation relationship between the main lens and each sub-lens and the coordinates of the center point of each sub-lens in the camera coordinate system are established through geometric relationship;

[0017] The internal parameters comprise imaging main point coordinates, and the distance from the main lens to the microlens array and the distance from the microlens array to the sensor plane.

[0018] In step S2, the equivalent multi-view camera calibration specifically comprises: the internal parameters of the light field camera are obtained through a light field camera calibration method based on line features, and the internal parameters comprise {f x , f y , c u , c v , K1, K2}, fx , f y are pixel focal length in x, y direction respectively, c u , c v is the coordinate of the center point of the main lens on the sensor plane, K1, K2 are calibration parameters; the expression is:

[0019]

[0020]

[0021] wherein d is the distance from the main lens to the sensor plane, D is the distance from the sensor plane to the microlens array, f L is the focal length of the main lens, s x , s y is the size of the pixel;

[0022] After the calibration of the light field camera is completed and the calibration parameters are obtained, according to the similar triangles and the spatial position relationship, the position coordinates of the center points of each equivalent sub-camera in the camera coordinate system [L x , L y , L z ] and the position coordinates on the sensor plane [L u , L v ] are determined, which are expressed by parameters K1, K2 as:

[0023]

[0024] wherein (i u , i v ) is the center pixel coordinate of the microlens image, (c u , c v ) represents the coordinate point of the center point of the main lens on the pixel plane;

[0025] The microlens array is parallel to the main lens, and there is a translation relationship between the main lens coordinate system and each sub-camera coordinate system, the rotation matrix is a unit matrix, and the translation matrix is [L x , L y , L z ] T ; the equivalent multi-camera model of the light field camera is established, and the pose conversion relationship between each sub-camera and the main camera is obtained.

[0026] In step S3, the ice layer local topography recovery of the aircraft surface based on the calibrated light field speckle measurement system specifically includes:

[0027] The speckle pattern is projected on the surface of the aircraft, the texture features of the surface of the aircraft are enriched, the texture features are locally uniquely coded, then the sub-aperture images are extracted, the same-named pixel points are matched through the normalized cross-correlation algorithm of de-meaning, and the local point cloud of the ice layer on the surface of the aircraft is obtained by solving the over-determined equation through multi-frame triangulation principles in the multiple sub-aperture images.

[0028] In one embodiment, the sub-aperture image extraction adopts a multi-view speckle stereo matching method to obtain multiple parallax maps, corresponding points are matched through the gray similarity of image blocks, a window sub-region is demarcated around the center of the reference point, and then the center point with the maximum gray correlation in the same size sub-region is found, and the cost function of the corresponding point digital speckle correlation matching of the reference point is:

[0029]

[0030] Wherein: f(x i , y j ) is a to-be-matched point, is the gray mean value of the left image sub-region image, g(x′ i , y′ j ) is a corresponding point of the right image, is the gray mean value of the right image sub-region window, C zncc represents the cost calculated through the ZNCC algorithm, and m is the window size.

[0031] In the light field equivalent multi-view camera model, a space point will produce projection in multiple cameras, and in the case that the observation value is greater than 2, two-frame triangulation is extended to multi-frame triangulation, and the projection model is:

[0032]

[0033] Wherein: x is a homogeneous coordinate, P j is a projection matrix, represents a row vector of the projection matrix, (u j , v j ) is the jth projection, R j t j is a rotation matrix and a translation matrix t of the jth position; is the image observation point homogeneous coordinate of the jth camera; [x, y, z] is the three-dimensional coordinate of the previous frame data, [x, y, z, 1] is the homogeneous coordinate of the point cloud of the next frame, s represents a scale factor, and K represents a camera intrinsic matrix.

[0034] The least square solution of the over-determined equation is used to solve the multi-frame triangulation problem, and is expressed as:

[0035]

[0036] Wherein: the first row of the pose transformation matrix under the i-th view, the second row of the pose transformation matrix under the i-th view, the third row of the pose transformation matrix under the i-th view, the first row of the pose transformation matrix under the n-th view, the second row of the pose transformation matrix under the n-th view, the third row of the pose transformation matrix under the n-th view, P is the homogeneous coordinates of a three-dimensional point, x i , y i represent the pixel coordinates of the i-th frame;

[0037] Complete multi-frame triangulation to obtain local point cloud data of the measurement system at one pose.

[0038] In step S3, key points are extracted according to the repeated area features of the two frames of point clouds, and a three-dimensional rigid body transformation matrix of the two groups of key points is calculated, which specifically includes:

[0039] Local topography of the ice layer on the surface of the aircraft is obtained under different views, features are extracted through the overlapping area of the local topography data under two adjacent views, a group of homonymous points under different views is obtained, and a transformation matrix under two views is calculated. The transformation matrix under adjacent views is represented by a rotation matrix R and a translation matrix t, and the mathematical expression is:

[0040]

[0041] Where: [x, y, z] is the three-dimensional coordinates of the previous frame data, [x, y, z, 1] is the homogeneous coordinates of the next frame of point cloud, [R, t] is the rotation matrix and translation matrix t between [x, y, z] and [x, y, z, 1];

[0042] The least square solution of the transformation matrix [R, t] can be obtained by SVD decomposition, and the SVD decomposition is represented as:

[0043] W = UΛV T

[0044] Where W is the matrix to be decomposed, U and V are orthogonal matrices, and Λ is a diagonal matrix.

[0045] In the diagonal matrix obtained by decomposition, the diagonal elements are arranged from large to small, and the rotation matrix R is represented as:

[0046] R = UV T

[0047] The centroid coordinates of the two groups of points are p1 and p2, and the translation matrix t is obtained:

[0048] t = p1 - R p2

[0049] The pose conversion relationship under different view angle coordinate systems between adjacent frames is obtained, and the adjacent local point cloud data is classified into the same coordinate system to achieve point cloud splicing.

[0050] In step S4, the splicing method is used to sequentially splice the point clouds between adjacent frames in the shooting order, and the splicing method specifically comprises the following steps.

[0051] The conversion relationship between the adjacent frames of the local data is obtained, the world coordinate system is established on the center point of the main lens, and the local data is converted to the coordinate system of the first frame to achieve the purpose of global data fusion, and the pose conversion relationship between the first frame and the second frame is:

[0052]

[0053] In the formula, (x1, y1, z1) is the camera coordinates in the first frame camera coordinate system, (x2, y2, z2) is the camera coordinates in the second frame camera coordinate system, T ij (i = 1, 2, 3; j = 1, 2, 3) is a conversion matrix;

[0054] The pose conversion matrix is obtained from the obtained pose conversion relationship between the first frame and the second frame and the pose conversion relationship between the second frame and the third frame, and is represented as:

[0055]

[0056] Wherein: R1, T1 is the pose conversion matrix of the first frame and the second frame, R2, T2 is the conversion matrix of the second frame and the third frame; (x3, y3, z3) is the camera coordinates in the third frame camera coordinate system;

[0057] The obtained third frame local data is classified into the first frame coordinate system, the local data coordinate system is classified into the global coordinate system, all local data is sequentially classified into the first frame camera coordinate system through pose conversion and pose transmission, and the distribution model reconstruction of the ice layer thickness of the whole aircraft is obtained.

[0058] Another object of the application is to provide a light field speckle imaging based aircraft surface ice layer three-dimensional topography reconstruction system for implementing the light field speckle imaging based aircraft surface ice detection method, the light field speckle imaging based aircraft surface ice layer three-dimensional topography reconstruction system comprising:

[0059] The light field speckle measurement system comprises a light field camera, an infrared speckle projector and a support; wherein the light field camera and the infrared speckle projector are fixed on the support, the light field camera is located on the left side of the support, and the infrared speckle projector is located on the right side of the support.

[0060] The calibration module of the light field equivalent multi-camera is used for calibrating the distance from the micro-lens array inside the light field camera to the sensor plane and the distance from the micro-lens array to the main lens according to the light field camera calibration method based on line features, equivalent the micro-lens array inside the light field camera to a multi-camera, and determining the positional relationship between each sub-camera and the main lens through the similar triangle relationship according to geometric features, and adopting a rotation matrix R and a translation matrix t to represent the positional relationship;

[0061] The local topography recovery module of the ice layer on the aircraft surface is used for three-dimensional reconstruction in an active projection manner, projects a speckle pattern on the aircraft surface through an infrared speckle projector in the measurement system, images the same scene at different viewing angles through the multi-camera model equivalent to the light field camera, adopts a correlation matching algorithm to perform correlation matching on each sub-aperture image, obtains a plurality of image points corresponding to a spatial point, and solves the overdetermined equation through multi-frame triangulation to obtain the local point cloud data of the ice layer on the aircraft surface.

[0062] The registration matrix acquisition module according to the repeated region features is used for acquiring the local point cloud data through the light field equivalent multi-camera calibration module in different poses by the measurement system, acquiring a set of homonymous points of two frames of point clouds through the features between the overlapping regions of the adjacent two pieces of point clouds, and calculating the pose conversion relationship of the two frames of point clouds through the similar transformation relationship.

[0063] The distribution model reconstruction module of the ice layer thickness of the whole aircraft is used for acquiring the pose conversion relationship between each adjacent frame through the registration matrix acquisition module according to the repeated region features, setting a world coordinate system, and sequentially converting the local data to the same coordinate system to complete the distribution model reconstruction of the ice layer thickness of the whole aircraft.

[0064] Another object of the present application is to provide a computer device comprising a memory and a processor, the memory storing a computer program, the computer program being executed by the processor to enable the processor to execute the aircraft surface ice detection method of the light field speckle imaging.

[0065] In combination with all the above technical solutions, the present application has the following advantages and positive effects:

[0066] (1) The application provides a method for detecting ice on the surface of an airplane by light field speckle imaging, a light field speckle measurement system is built, a multi-camera equivalent calibration of a light field camera is performed, a local topography of an ice layer on the surface of an airplane is recovered, a registration matrix is obtained according to repeated area features, and a distribution model of the thickness of the ice layer on the whole airplane is reconstructed.

[0067] (2) The method for reconstructing the three-dimensional topography of an ice layer on the surface of an airplane based on light field speckle imaging can simultaneously capture the spatial distribution and angle distribution information of light in a single camera mode, adopts an active three-dimensional reconstruction mode in combination with an infrared speckle projector, and obtains the distribution model of the thickness of the ice layer on the whole airplane through point cloud splicing technology, so that the hardware equipment is simplified and the cost is saved. BRIEF DESCRIPTION OF DRAWINGS

[0068] The accompanying drawings, which are incorporated into and form part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the principles of the present disclosure;

[0069] Figure 1 is a flow chart of the method for reconstructing the three-dimensional topography of an ice layer on the surface of an airplane by light field speckle imaging provided by the embodiment of the application;

[0070] Figure 2 is a principle diagram of the method for reconstructing the three-dimensional topography of an ice layer on the surface of an airplane by light field speckle imaging provided by the embodiment of the application;

[0071] Figure 3 is a schematic diagram of light field imaging provided by the embodiment of the application;

[0072] Figure 4 is a schematic diagram of data acquisition of a measurement system provided by the embodiment of the application;

[0073] Figure 5 is a schematic diagram of the inter-frame pose conversion relationship provided by the embodiment of the application. DETAILED DESCRIPTION

[0074] In order to make the above objectives, characteristics and advantages of the present application more obvious and easy to understand, the specific embodiments of the present application are described in detail below with reference to the drawings. In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without departing from the concept of the present application, so the present application is not limited to the specific implementation disclosed below.

[0075] I. Explanation of the embodiment

[0076] The embodiment of the present application provides a three-dimensional topography reconstruction method of aircraft surface ice layer based on light field speckle imaging, which measures the topography of the aircraft surface ice layer by combining a light field camera with an infrared speckle projector, completes the reconstruction of the three-dimensional topography of the aircraft surface ice layer, and realizes the reconstruction of the distribution model of the ice layer thickness of the whole aircraft through a point cloud splicing method.

[0077] As shown in Figure 1 , the three-dimensional topography reconstruction method sequentially performs the following steps:

[0078] S101, building a light field speckle measurement system;

[0079] The measurement system includes a light field camera, a speckle projector and a bracket. The light field camera and the speckle projector are placed on the bracket at a certain angle, the angle between the light field camera and the speckle projector is adjusted so that the light field camera can completely capture the image after speckle modulation, then the light field camera and the speckle projector are fixed, and the relative position between them is kept unchanged, and the building of the light field speckle measurement system is completed;

[0080] S102, equivalent multi-camera calibration of the light field camera;

[0081] The microlens array inside the light field camera is equivalent to a multi-camera, the camera coordinate system is established on the center point of the main lens, the internal parameters of the light field camera are calibrated by a chessboard, including the imaging principal point coordinates, the distance from the main lens to the microlens array and the distance from the microlens array to the sensor plane, and the rotation and translation relationship between the main lens and each sub-lens and the coordinates of the center point of each sub-lens in the camera coordinate system are established through geometric relationship;

[0082] S103, local topography recovery of the aircraft surface ice layer;

[0083] The speckle pattern is projected on the aircraft surface by the measurement system, the texture features of the aircraft surface are enriched, and the aircraft surface is uniquely coded locally. Then the sub-aperture images are extracted, the same pixel points are matched by the de-meaning normalized cross-correlation algorithm, and the local point cloud of the aircraft surface ice layer is obtained by solving the overdetermined equation in multiple sub-aperture images using the multi-frame triangulation principle.

[0084] S104, obtaining a registration matrix according to the repeated region feature;

[0085] The measurement system obtains two frames of local point clouds at two different positions by the previous step, extracts key points according to the repeated region features of the two frames of point clouds, calculates a three-dimensional rigid transformation matrix of the two groups of key points, obtains an initial registration matrix of the latter frame of point clouds relative to the former frame of point clouds, and continuously iterates the initial registration matrix according to the nearest points through an ICP point cloud splicing technology, and the iteration is terminated when the error is less than a given threshold value, so as to obtain the registration matrix.

[0086] S105, reconstruction of the distribution model of the ice layer thickness of the whole aircraft;

[0087] The point cloud splicing between frames is completed by the previous step, images of the whole aircraft are collected at different positions, the splicing method of the previous step is used between adjacent frames of point clouds to sequentially splice according to the shooting order, and the distribution model of the ice layer thickness of the whole aircraft is restored.

[0088] In the embodiment of the application, the light field camera of the imaging system in step S101 is a German Raytrix R29 industrial light field camera, and the imaging system is built by establishing a world coordinate system at the center point of the main lens of the light field camera, placing the light field camera on the left side of the support, and placing the speckle projector at a certain angle on the right side of the light field camera, keeping the relative position fixed and fixed on the support.

[0089] In the embodiment of the application, the calibration of the light field equivalent multi-camera in step S102, due to the special internal structure of the light field camera, there is a microlens array in front of its sensor plane, each sub-lens in the microlens array can be equivalent to a sub-camera, so the light field camera can be equivalent to a multi-camera model, the camera coordinate system is established at the center point of the main lens, regarded as the main camera, the internal parameter model of the light field camera is calibrated through the light field camera calibration method based on line features, the relationship between the center coordinates of the sub-camera and the coordinates of the microlens center on the sensor plane can be obtained through similar triangle relationship, and then the coordinates of each sub-camera center in the camera coordinate system can be obtained. The orientation of each lens in the microlens array is the same and meets the coplanar characteristic, so there is a pure translation relationship between each sub-camera and the main camera, and the positional relationship between each sub-camera and the main camera can be established through the coordinates of the sub-camera center point in the camera coordinate system, that is, the calibration of the light field camera equivalent multi-camera is completed.

[0090] In the embodiment of the present application, the step S103 recovers the local morphology of the ice layer on the aircraft surface, and the infrared speckle projector in the light field speckle imaging system projects the infrared speckle to the local area of the aircraft surface. Since the light field camera is equivalent to a multi-view camera model in the step S102, the multi-view stereo matching is performed by using the ZNCC correlation matching algorithm, one point in space corresponds to multiple image points, and the coordinates of the three-dimensional points in space are solved by using the multi-frame triangulation method.

[0091] In the embodiment of the present application, the step S104 obtains the registration matrix according to the repeated area features. The local point cloud data of the ice layer on the aircraft surface is obtained in the step S103, a group of homonymous points in the two frames of point cloud data is found through the similar features of the point cloud overlap area, the initial conversion matrix of the two frames of point cloud data is obtained by calculating the similar transformation group between the homonymous points, the more accurate conversion matrix is obtained by continuously iterating the conversion matrix through the ICP point cloud splicing technology, and the point cloud splicing of the local data is realized by converting the two frames of point cloud data to the same coordinate system through the conversion matrix.

[0092] In the embodiment of the present application, the step S105 reconstructs the distribution model of the ice layer thickness of the whole aircraft. The interframe conversion matrix of the local point cloud is obtained in the step 4), the point cloud splicing is sequentially performed on each frame of local data through the conversion matrix in the step S104, and the distribution model of the ice layer thickness of the whole aircraft is reconstructed.

[0093] Figure 2 It is the principle of the light field speckle imaging method for reconstructing the three-dimensional morphology of the ice layer on the aircraft surface provided in the embodiment of the present application.

[0094] Embodiment 1

[0095] The light field speckle imaging method for reconstructing the three-dimensional morphology of the ice layer on the aircraft surface provided in the embodiment of the present application comprises the following steps:

[0096] Step 1, build a light field speckle measurement system: the system comprises a light field camera, a speckle projector and a support clamp, wherein the light field camera and the speckle projector are placed at a certain angle, the angle is adjusted to be substantially the same as the field of view range of the light field camera, the light field camera is located on the left side, the speckle projector is located on the right side, and the world coordinate system is established to be coincident with the main lens coordinate system of the light field camera. The speckle projector is used for projecting a random coded speckle pattern to code the space of the measured object and enrich the texture features for facilitating feature matching. The support clamp is used for fixing the positions of the light field camera and the speckle projector, and the relative positions are kept unchanged during the measurement.

[0097] Step 2, Calibration of light field equivalent multi-camera: Due to the internal structure of the light field camera, it places a microlens array in front of the sensor plane. Due to the action of the microlens array, a point in space will present multiple image points inside the light field camera, Figure 3 For the schematic diagram of light field imaging, each lens in the microlens array can be equivalent to a sub-camera, and the main lens of the light field camera is the main camera. The origin of the camera coordinate system is established at the center point of the main lens of the light field camera. Through the proposed light field camera calibration method based on line features, the internal parameters of the light field camera are obtained, including {f x , f y , c u , c v , K1, K2}, f x , f y are the pixel focal lengths in x and y directions, c u , c v are the coordinates of the center point of the main lens on the sensor plane, and K1 and K2 are calibration parameters. The expression is:

[0098]

[0099]

[0100] where d is the distance from the main lens to the sensor plane, D is the distance from the sensor plane to the microlens array, f L is the focal length of the main lens, and s x , s y are the pixel sizes.

[0101] After completing the calibration of the light field camera and obtaining the calibration parameters, according to the similar triangles and the spatial position relationship, the position coordinates of the center points of each equivalent sub-camera in the camera coordinate system [L x , L y , L z ] and on the sensor plane [L u , L v ] are determined. They are represented by parameters K1 and K2 as follows:

[0102]

[0103] where (i u , i v ) are the center pixel coordinates of the microlens image, and (c u , c v ) represent the coordinates of the center point of the main lens on the pixel plane.

[0104] The microlens array is parallel to the main lens, and there is a translation relationship between the main lens coordinate system and each sub-camera coordinate system. The rotation matrix is the unit matrix, and the translation matrix is [L x , Ly , L z ] T ; the equivalent multi-camera model of the light field camera is established, and the pose conversion relationship between each sub-camera and the main camera is obtained. x , L y , L z ] T is a 3x1 translation matrix.

[0105] Step 3, local topography recovery of the ice layer on the aircraft surface, Figure 4 is a schematic diagram of data acquisition of the measuring system in the application, as shown in the figure, Figure 4 the speckle pattern is projected onto the surface of the aircraft by the infrared speckle projector in the light field speckle measurement system, and the aircraft ice layer surface is spatially encoded. At this time, the local window of the aircraft surface has good autocorrelation, and the image of each viewing angle is the encoding of all light rays in the direction collected by its corresponding microlens, which is called a sub-aperture image. Each sub-aperture image of the equivalent multi-camera of the light field camera is extracted, a plurality of parallax maps are obtained by using the multi-view speckle stereo matching method, the corresponding points are matched by the gray similarity of the image blocks, a window sub-area is demarcated around the reference point in the left image, and the center point with the greatest gray correlation in the sub-area of the same size in the right image is found as the corresponding point of the reference point. The cost function of digital speckle correlation matching is:

[0106]

[0107] wherein: f(x i , y j ) is a to-be-matched point, is the gray mean value of the sub-area image of the left image, g(x′ i , y′ j ) is the corresponding point of the right image, is the gray mean value of the sub-area window of the right image, C zncc represents the cost calculated by the ZNCC algorithm, and m is the window size.

[0108] In the light field equivalent multi-camera model, a spatial point will produce a projection in multiple cameras. In the case where the observation value is greater than 2, the two-frame triangulation is extended to multi-frame triangulation, and the projection model is as follows:

[0109]

[0110] wherein: x is a homogeneous coordinate, P j is a projection matrix, represents a row vector of the projection matrix, (u j , v j ) is the jth projection, R j t jRotation matrix and translation matrix for the jth position; Homogeneous coordinates of image observation point of the jth camera, s represents a scale factor, and K represents a camera intrinsic parameter matrix;

[0111] The method of solving the least square solution by using an overdetermined equation can solve the multi-frame triangulation problem, which is represented as:

[0112]

[0113] Wherein: The second row of the pose conversion matrix under the ith view, The third row of the pose conversion matrix under the ith view, The second row of the pose conversion matrix under the nth view, The third row of the pose conversion matrix under the nth view, P is the homogeneous coordinates of a three-dimensional point, x i , y i Indicates the pixel coordinates of the ith frame;

[0114] The multi-frame triangulation is completed, and the local point cloud data of the measurement system under one pose is obtained.

[0115] Step 4, obtaining a registration matrix according to the repeated area features, Figure 5 The inter-frame pose conversion relationship in the application is shown in Figure Figure 5 The local topography of the ice layer on the surface of the aircraft under different views is obtained by step 3, and through the overlapping area of the local topography data under two adjacent views, features are extracted, a group of homonymous points under different views is obtained, and the conversion matrix under two views is calculated. The conversion matrix under adjacent views is represented by a rotation matrix R and a translation matrix t, and the mathematical expression is:

[0116]

[0117] Wherein: [x, y, z] is the three-dimensional coordinates of the previous frame data, [x, y, z, 1] is the homogeneous coordinates of the point cloud of the next frame, [R, t] is the rotation matrix and translation matrix t between [x, y, z] and [x, y, z, 1];

[0118] The least square solution of the conversion matrix [R, t] can be obtained by SVD decomposition, and the SVD decomposition is represented as:

[0119] W=UΛV T

[0120] Wherein, W is the matrix to be decomposed, U and V are orthogonal matrices, and Λ is a diagonal matrix;

[0121] In the diagonal matrix obtained by decomposition, the diagonal elements are arranged from large to small, and the rotation matrix R can be represented as:

[0122] R = UV T

[0123] Determine the centroid coordinates of the two sets of points as p1 and p2, and obtain the translation matrix t as follows:

[0124] t = p1 - Rp2

[0125] It can obtain the pose transformation relationship between adjacent frames under different viewpoint coordinate systems, and classify adjacent local point cloud data into the same coordinate system to achieve the purpose of point cloud stitching.

[0126] Step 5: Reconstruct the distribution model of the ice layer thickness of the entire aircraft. Based on the transformation relationship between adjacent frames of local data obtained in Step 4, establish the world coordinate system at the center point of the main shot in Step 2, and transform each local data to the coordinate system of the first frame to achieve global data fusion. The pose transformation relationship between the first and second frames is as follows:

[0127]

[0128] In the formula, (x1, y1, z1) are the camera coordinates in the first frame's camera coordinate system, (x2, y2, z2) are the camera coordinates in the second frame's camera coordinate system, and T... ij (i = 1, 2, 3; j = 1, 2, 3) is the transformation matrix;

[0129] Based on the pose transformation relationships obtained from the first and second frames, as well as between the second and third frames, the pose transformation matrix is ​​obtained and expressed as:

[0130]

[0131] Where R1 and T1 are the pose transformation matrices between the first and second frames, and R2 and T2 are the transformation matrices between the second and third frames; the obtained local data of the third frame is assigned to the coordinate system of the first frame, and the local data coordinate system is assigned to the global coordinate system. All local data are then assigned to the coordinate system of the first frame camera through pose transformation and pose transfer in sequence to obtain the distribution model reconstruction of the ice thickness of the entire aircraft.

[0132] Therefore, the local data of the third frame obtained in step 3 can be assigned to the coordinate system of the first frame, thereby achieving the purpose of assigning the local data coordinate system to the global coordinate system. In turn, all local data are assigned to the coordinate system of the first frame camera through pose transformation and pose transfer, so as to obtain the distribution model reconstruction of the ice thickness of the entire aircraft.

[0133] Example 2

[0134] This invention provides a system for reconstructing the three-dimensional topography of aircraft surface ice layers based on speckle imaging, comprising:

[0135] The light field speckle measurement system comprises a light field camera, an infrared speckle projector and a support, wherein the light field camera and the infrared speckle projector are fixed on the support, the light field camera is located on the left side of the support, and the infrared speckle projector is located on the right side of the support.

[0136] The calibration module of the light field equivalent multi-camera is used for calibrating the distance from the micro-lens array in the light field camera to the sensor plane and the distance from the micro-lens array to the main lens according to a line feature-based light field camera calibration method, equivalently taking the micro-lens array in the light field camera as a multi-camera, determining the positional relationship between each sub-camera and the main lens through a similar triangle relationship, and using a rotation matrix R and a translation matrix t.

[0137] The aircraft surface ice layer local topography recovery module is used for three-dimensional reconstruction in an active projection mode, the infrared speckle projector in the measurement system projects a speckle pattern to the aircraft surface, the multi-camera equivalent model of the light field camera images the same scene at different viewing angles, a correlation matching algorithm is used to perform correlation matching on each sub-aperture image, a plurality of image points corresponding to a spatial point are obtained, and a plurality of frames of triangulation are used to solve the overdetermined equation to obtain the local point cloud data of the aircraft surface ice layer.

[0138] The registration matrix acquisition module according to the repeated region features is used for acquiring the local point cloud data by using the principle of the calibration module of the light field equivalent multi-camera in the measurement system at different poses, acquiring a set of homonymous points of two frames of point cloud data by using the features between the overlapping regions of two adjacent pieces of point cloud data, and calculating the pose conversion relationship of the two frames of point cloud data, that is, the conversion matrix.

[0139] The distribution model reconstruction module of the ice layer thickness of the whole aircraft is used for acquiring the pose conversion relationship between adjacent frames by using the registration matrix acquisition module according to the repeated region features, setting a world coordinate system, and sequentially converting the local data to the same coordinate system to complete the distribution model reconstruction of the ice layer thickness of the whole aircraft.

[0140] Further, the line feature-based light field camera calibration method in the calibration module of the light field equivalent multi-camera is Geometric calibration of micro-lens-based light field cameras using line features published by Bok Y et al in IEEE Transactions on Pattern Analysis and Machine Intelligence in 2016.

[0141] Further, the related matching algorithm in the ice layer local topography recovery module of the aircraft surface is ZNCC (Zero-based Normalized Cross Correlation) which uses block matching to calculate the correlation between integer pixels.

[0142] Further, the homonym in the registration matrix acquisition module according to the repeated area feature is the representation of the same point in the space under different camera coordinate systems.

[0143] Further, the world coordinate system in the ice layer thickness distribution model reconstruction module of the whole aircraft is established under the main lens coordinate system of the light field camera for acquiring the first frame of local data, and the same coordinate system is used to convert all the local data of the frames to the world coordinate system through the conversion matrix.

[0144] In the above embodiments, the description of each embodiment focuses on different aspects, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0145] The information interaction, execution process and the like between the above devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and the technical effects brought by the method embodiments can be referred to the method embodiments part, which will not be repeated here.

[0146] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of software functional unit. In addition, the specific names of the functional units and modules are only for convenient distinction, and do not limit the protection scope of the present application. The specific working process of the units and modules in the system can be referred to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0147] II. Application Embodiments

[0148] The application embodiments of the present application provide a computer device, which comprises at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above method embodiments when executing the computer program.

[0149] The application further provides a computer readable storage medium storing a computer program, which, when executed by a processor, enables the steps in the above-mentioned various method embodiments.

[0150] The application further provides an information data processing terminal for providing a user input interface to implement the steps in the above-mentioned various method embodiments when executed on an electronic device, and the information data processing terminal is not limited to a mobile phone, a computer, or a switch.

[0151] The application further provides a server for providing a user input interface to implement the steps in the above-mentioned various method embodiments when executed on an electronic device.

[0152] The application provides a computer program product, which, when executed on an electronic device, enables the electronic device to perform the steps in the above-mentioned various method embodiments.

[0153] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the application can implement all or part of the processes in the above-mentioned embodiments by a computer program to instruct related hardware, and the computer program can be stored in a computer readable storage medium. The computer program, when executed by a processor, can implement the steps in the above-mentioned various method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, an executable file, or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunications signal, and a software distribution medium. For example, a U disk, a mobile hard disk, a magnetic disk, or an optical disk.

[0154] The above-mentioned is only a preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, and any modification, equivalent replacement, and improvement within the technical range disclosed by the application and within the spirit and principle of the application should be covered in the protection scope of the application.

Claims

1. A method for aircraft surface de-icing detection by light field speckle imaging, characterized in that, The method comprises: S1, building an optical field speckle measurement system; S2, based on the light field camera in the built optical field speckle measurement system, equivalent multi-camera calibration is carried out; S3, based on the calibrated optical field speckle measurement system, the local topography of the ice layer on the surface of the airplane is recovered; two frames of local point clouds are acquired at two different positions, key points are extracted according to the repeated area features of the two frames of point clouds, a three-dimensional rigid body transformation matrix of the two groups of key points is calculated, an initial registration matrix of the latter frame of point clouds relative to the former frame of point clouds is obtained, the nearest points of the initial registration matrix are iterated continuously through the ICP point cloud splicing technology, and the iteration is terminated when the error is less than a given threshold value, so that a registration matrix is obtained; S4, according to the point cloud splicing between frames, images of the whole airplane are acquired at different positions, the splicing method is used to splice the point clouds between adjacent frames in turn according to the shooting order, and a distribution model of the ice layer thickness of the whole airplane is recovered; In step S3, the local topography recovery of the ice layer on the surface of the airplane based on the calibrated optical field speckle measurement system specifically comprises: A speckle pattern is projected on the surface of the airplane, the texture features of the surface of the airplane are enriched, the texture features are locally uniquely coded, then sub-aperture images are extracted, the same-named pixel points are matched through a de-meaning normalized cross-correlation algorithm, and the local point cloud of the ice layer on the surface of the airplane is obtained by solving the overdetermined equation through multi-frame triangulation. The multi-view speckle stereo matching method is used to obtain a plurality of parallax maps, corresponding points are matched through the gray similarity of image blocks, a window sub-region is demarcated around the center point, a center point with the maximum gray correlation in a sub-region of the same size is found, and the cost function of the corresponding point digital speckle correlation matching of the reference point is: where f(x i ,y j ) is the point to be matched, is the left image sub-region image gray mean, g(x′ i ,y′ j ) is the corresponding point of the right image, is the right image sub-region window gray mean, C zncc represents the cost calculated by the ZNCC algorithm, and m is the window size. In the light field equivalent multi-camera model, a space point will produce projection in multiple cameras, and in the case that the observation value is greater than 2, the two-frame triangulation is extended to multi-frame triangulation, and the projection model is: wherein: x is a homogeneous coordinate, P j is a projection matrix, represents a row vector of the projection matrix, (u j ,v j ) is the jth projection, R j t j is a rotation matrix and a translation matrix t of the jth position; is the image observation point homogeneous coordinate of the jth camera; [x, y, z] is the three-dimensional coordinate of the previous frame data, [x, y, z, 1] is the homogeneous coordinate of the next frame point cloud, s represents a scale factor, and K represents a camera intrinsic matrix; The least square solution of the overdetermined equation can solve the multi-frame triangulation problem, which is expressed as: wherein: represents the first row of the pose transformation matrix at the i-th view, represents the second row of the pose transformation matrix at the i-th view, represents the third row of the pose transformation matrix at the i-th view, represents the first row of the pose transformation matrix at the n-th view, represents the second row of the pose transformation matrix at the n-th view, represents the third row of the pose transformation matrix at the n-th view, P is the homogeneous coordinate of a three-dimensional point, x i ,y i represents the pixel coordinate of the i-th frame; The multi-frame triangulation is completed, and the local point cloud data of the measurement system at one pose is obtained.

2. The method of claim 1, wherein, In step S1, the optical field speckle measurement system comprises a light field camera, a speckle projector and a support; the light field camera is arranged on the left side of the support, and the speckle projector is installed on the right side of the light field camera, and the relative positions are fixed and fixed on the support; The angle between the light field camera and the speckle projector is adjusted so that the light field camera can completely capture the image after speckle modulation, then the light field camera and the speckle projector are fixed, the relative positions between the light field camera and the speckle projector are kept unchanged, and the optical field speckle measurement system is built.

3. The method of claim 2, wherein, The microlens array inside the light field camera is equivalent to a multi-camera, the camera coordinate system is established on the center point of the main lens, the internal parameters of the light field camera are calibrated by a chessboard, and the rotation and translation relationship between the main lens and each sub-lens and the coordinates of the center point of each sub-lens in the camera coordinate system are established through geometric relationship; The internal parameters comprise main point coordinates, the distance from the main lens to the microlens array and the distance from the microlens array to the sensor plane.

4. The method of claim 1, wherein, In step S2, the performing equivalent multi-view camera calibration specifically comprises: obtaining internal parameters of the light field camera by a line feature based light field camera calibration method, including {f x ,f y ,c u ,c v ,K1,K2}f x ,f y respectively being pixel focal lengths in x and y directions, c u ,c v being a coordinate of a center point of a main lens on a sensor plane, and K1 and K2 being calibration parameters; and an expression being: Where d is the distance from the main lens to the sensor plane, D is the distance from the sensor plane to the microlens array, and f is the distance from the sensor plane to the microlens array. L Main lens focal length, s x ,s y Pixel size; After the calibration of the light field camera is completed to obtain the calibration parameters, according to similar triangles and spatial position relations, the position coordinates of the center points of each equivalent sub-camera in the camera coordinate system [L x ,L y ,L z ] and in the sensor plane [L u ,L v ] are determined, which are represented by parameters K1, K2: wherein: (i u , v ) is the center pixel coordinate of the microlens image, (c u , v ) represents the coordinate point of the center point of the main lens on the pixel plane; The microlens array is parallel to the main lens, and there is a translation relationship between the main lens coordinate system and each sub-camera coordinate system, the rotation matrix is a unit matrix, and the translation matrix is [L x , L y , L z ] T ; An equivalent multi-camera model of the light field camera is established, and the pose conversion relationship between each sub-camera and the main camera is obtained.

5. The method of claim 1, wherein, In step S3, key points are extracted according to the repeated area features of the two frames of point clouds, and a three-dimensional rigid body transformation matrix of the two groups of key points is calculated, which specifically includes: The local topography of the ice layer on the surface of the aircraft is obtained under different viewing angles, the features are extracted through the overlapping area of the local topography data under two adjacent viewing angles, a group of homonymous points under different viewing angles is obtained, a conversion matrix under two viewing angles is calculated, the conversion matrix under adjacent viewing angles is represented by a rotation matrix R and a translation matrix t, and the mathematical expression is: Wherein: [x, y, z] is the three-dimensional coordinates of the previous frame data, [x, y, z, 1] is the homogeneous coordinates of the next frame of point cloud, [R, t] is the rotation matrix and translation matrix t between [x, y, z] and [x, y, z, 1]; The least square solution of the conversion matrix [R, t] can be obtained by SVD decomposition, and the SVD decomposition is represented as: W = UΛV T Wherein, W is the matrix to be decomposed, U and V are orthogonal matrices, and A is a diagonal matrix; The diagonal elements in the diagonal matrix obtained by decomposition are arranged from large to small, and the rotation matrix R is represented as: R = UV T The decentered coordinates of the two groups of points are p1 and p2, and the translation matrix t is obtained: t = p1-Rp2 The pose conversion relationship under the coordinate system of different viewing angles between adjacent frames is obtained, and the adjacent local point cloud data is classified into the same coordinate system to achieve point cloud splicing.

6. The method of claim 1, wherein, In step S4, the adjacent frames of point cloud are spliced in sequence according to the shooting order by using the splicing method, which specifically includes: After obtaining the conversion relationship between the adjacent frames of local data, the world coordinate system is established on the center point of the main lens, and each local data is converted to the coordinate system of the first frame to achieve the purpose of global data fusion, and the pose conversion relationship between the first frame and the second frame is: where (x1, y1, z1) is the camera coordinate in the first frame camera coordinate system, (x2, y2, z2) is the camera coordinate in the second frame camera coordinate system, T ij (i = 1, 2, 3; j = 1, 2, 3) is a transformation matrix; The pose conversion matrix is obtained from the obtained pose conversion relationship between the first frame and the second frame and the second frame and the third frame, and is represented as: Wherein: R1, T1 is the pose conversion matrix of the first frame and the second frame, R2, T2 is the conversion matrix of the second frame and the third frame; (x3, y3, z3) is the camera coordinate in the third frame camera coordinate system; The third frame local data obtained is classified into the first frame coordinate system, the local data coordinate system is classified into the global coordinate system, all local data is sequentially classified into the first frame camera coordinate system through pose conversion and pose transmission, and the distribution model reconstruction of the ice layer thickness of the whole aircraft is obtained.

7. A light field speckle imaging based aircraft surface ice layer three-dimensional profile reconstruction system implementing the method of detecting aircraft surface ice according to any one of claims 1-6, characterized in that, The aircraft surface ice layer three-dimensional topography reconstruction system based on light field speckle imaging includes: A light field speckle measurement system includes a light field camera, an infrared speckle projector and a support; wherein the light field camera and the infrared speckle projector are placed on the support to be fixed, the light field camera is located on the left side of the support, and the infrared speckle projector is located on the right side of the support; A calibration module of the light field equivalent multi-camera is used to calibrate the distance from the internal microlens array of the light field camera to the sensor plane and the distance from the microlens array to the main lens according to the light field camera calibration method based on line features, equivalent the internal microlens array of the light field camera to a multi-camera, determine the positional relationship between each sub-camera and the main lens through similar triangle relationship according to geometric features, and use a rotation matrix R and a translation matrix t to represent. The aircraft surface ice layer local appearance recovery module is used for three-dimensional reconstruction in an active projection manner. An infrared speckle projector in a measurement system projects a speckle pattern to an aircraft surface. A multi-view model equivalent to a light field camera images the same scene at different viewing angles. A correlation matching algorithm is used to perform correlation matching on each sub-aperture image to obtain multiple image points corresponding to a spatial point. A multi-frame triangulation method is used to solve an overdetermined equation to obtain local point cloud data of the aircraft surface ice layer. The registration matrix acquisition module according to repeated region features is used for acquiring local point cloud data by using a calibration module principle of a light field equivalent multi-view camera in a measurement system at different poses. A set of homonymous points of two frames of point clouds is obtained by using features between overlapping regions of two adjacent pieces of point clouds. A pose conversion relationship of the two frames of point clouds is obtained by calculating a similarity transformation relationship. The distribution model reconstruction module of the ice layer thickness of the whole aircraft is used for acquiring a pose conversion relationship between each adjacent frame by the registration matrix acquisition module according to repeated region features. A world coordinate system is set. The local data is sequentially converted to the same coordinate system to complete the distribution model reconstruction of the ice layer thickness of the whole aircraft in the order of shooting.

8. A computer device, comprising: The computer device includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor executes the aircraft surface ice detection method of the light field speckle imaging according to any one of claims 1-6.

Images