Aircraft surface damage three-dimensional positioning method, device, equipment, medium and product

Abstract

The invention discloses an aircraft surface damage three-dimensional positioning method, device and equipment, a medium and a product, and relates to the technical field of damage detection. The method comprises the steps of determining a damage detection result according to a target image based on a damage detection model; determining a three-dimensional coordinate of the damage point in a camera coordinate system by adopting a binocular vision positioning algorithm based on a pixel coordinate of a center point of a damage area contained in a damage detection result; based on the relative pose relation between the binocular camera and the aircraft, according to the three-dimensional coordinates of the damage point in the camera coordinate system, determining the three-dimensional coordinates of the damage point in the aircraft body coordinate system; and mapping the three-dimensional coordinates of the damage point in the aircraft body coordinate system to a three-dimensional point cloud model of the aircraft, and carrying out visualization processing to obtain a visualization display result. The invention aims to accurately determine the position of the damage on the aircraft body and improve the aircraft maintenance efficiency.

Description

Technical Field

[0001] The present application relates to the field of damage detection technology, and in particular to a method, device, equipment, medium and product for three-dimensional positioning of aircraft surface damage. Background Art

[0002] The rapid and accurate detection and location of aircraft surface damage (such as scratches, dents, and corrosion) is a critical component of aviation maintenance and support. Currently, airlines primarily rely on manual visual inspections for aircraft surface damage detection, which is subject to significant human influence, prone to missed detections and false positives, and inefficient. Therefore, the development of automated methods for detecting aircraft surface damage is essential.

[0003] In recent years, researchers have applied computer vision and deep learning technologies to aircraft surface damage detection, developing automated inspection methods with high accuracy and speed. These methods can detect the presence and type of damage based on captured skin images, but they cannot pinpoint the specific location of the damage on the fuselage. Aircraft structural repair manuals specify different damage tolerances for the same type of damage in different locations on the fuselage, necessitating maintenance personnel to consider both the type and location of the damage to develop an appropriate repair plan.

[0004] Current research on damage localization uses structured light to perform rapid, high-precision global 3D reconstruction of aircraft structures, providing a methodological basis for detecting damage to aircraft skins. However, the calibration process is cumbersome. Other researchers have constructed an aircraft skin image scanning and localization system using a pan-tilt-mounted zoom camera and a laser rangefinder. This system can obtain high-quality, detailed images of the aircraft skin. Point cloud registration is then used to determine the relative pose of the system and the aircraft. Combined with the calibration results, this system achieves spatial localization of detailed images. However, this requires extensive equipment coordination and is not very convenient. Therefore, a direct, accurate, and efficient method is needed to achieve 3D localization of aircraft surface damage. Summary of the Invention

[0005] The purpose of this application is to provide a three-dimensional positioning method, device, equipment, medium and product for aircraft surface damage, which can accurately determine the location of the damage on the aircraft body and improve the efficiency of aircraft maintenance.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides a three-dimensional positioning method for aircraft surface damage, comprising:

[0008] Acquire a target image; the target image is an aircraft surface image acquired by a binocular camera;

[0009] Determining a damage detection result based on the target image based on a damage detection model; the damage detection model is trained using a deep learning method based on an aircraft surface damage dataset; the aircraft surface damage dataset includes: surface damage information and corresponding annotated damage detection results;

[0010] Using a binocular vision positioning algorithm, based on the pixel coordinates of the center point of the damage area included in the damage detection result, the three-dimensional coordinates of the damage point in the camera coordinate system are determined;

[0011] Based on the relative pose relationship between the binocular camera and the aircraft, the three-dimensional coordinates of the damage point in the aircraft body coordinate system are determined according to the three-dimensional coordinates of the damage point in the camera coordinate system. The relative pose relationship between the binocular camera and the aircraft is obtained by using a multi-sensor information fusion method based on the extended Kalman filter to determine the rotation matrix and translation vector between the camera coordinate system and the aircraft body coordinate system.

[0012] The three-dimensional coordinates of the damage point in the aircraft body coordinate system are mapped to the three-dimensional point cloud model of the aircraft, and visualization processing is performed to obtain a visualization display result.

[0013] In a second aspect, the present application provides a three-dimensional positioning device for aircraft surface damage, comprising:

[0014] A target image acquisition module is used to acquire a target image; the target image is an aircraft surface image acquired by a binocular camera;

[0015] a damage detection module, configured to determine damage detection results based on the target image based on a damage detection model; the damage detection model is trained using a deep learning method based on an aircraft surface damage dataset; the aircraft surface damage dataset includes surface damage information and corresponding annotated damage detection results;

[0016] A binocular vision positioning module is used to determine the three-dimensional coordinates of the damage point in the camera coordinate system based on the pixel coordinates of the center point of the damage area contained in the damage detection result using a binocular vision positioning algorithm;

[0017] The coordinate conversion module is used to determine the 3D coordinates of the damage point in the aircraft's coordinate system based on the relative pose relationship between the binocular camera and the aircraft, and the 3D coordinates of the damage point in the camera's coordinate system. The relative pose relationship between the binocular camera and the aircraft is obtained by using a multi-sensor information fusion method based on the extended Kalman filter to determine the rotation matrix and translation vector between the camera coordinate system and the aircraft's coordinate system.

[0018] The visualization module is used to map the three-dimensional coordinates of the damage point in the aircraft body coordinate system to the three-dimensional point cloud model of the aircraft, perform visualization processing, and obtain a visualization display result.

[0019] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-mentioned three-dimensional positioning method for aircraft surface damage.

[0020] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned three-dimensional positioning method for aircraft surface damage.

[0021] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the above-mentioned three-dimensional positioning method for aircraft surface damage.

[0022] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0023] The present application provides a method, device, equipment, medium and product for three-dimensional positioning of aircraft surface damage. Based on a damage detection model, the damage detection result is determined according to the target image; a binocular vision positioning algorithm is used to determine the three-dimensional coordinates of the damage point in the camera coordinate system based on the pixel coordinates of the center point of the damage area contained in the damage detection result; based on the relative posture relationship between the binocular camera and the aircraft, the three-dimensional coordinates of the damage point in the camera coordinate system are used to determine the three-dimensional coordinates of the damage point in the aircraft body coordinate system; the three-dimensional coordinates of the damage point in the aircraft body coordinate system are mapped to the three-dimensional point cloud model of the aircraft, and visualization processing is performed to obtain a visualization display result. The present application combines the damage detection model with the binocular vision positioning algorithm to realize damage point detection and positioning, and through mapping, realizes visualization display in the three-dimensional point cloud model of the aircraft to determine the location of the damage on the aircraft body, so that maintenance personnel can go directly to the damaged location for repair, thereby improving the efficiency of aircraft maintenance. Therefore, the present application can accurately determine the location of the damage on the aircraft body and improve the efficiency of aircraft maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0025] Figure 1This is a flow chart of the three-dimensional positioning method for aircraft surface damage;

[0026] Figure 2 A flowchart of the overall method is provided;

[0027] Figure 3 Schematic diagram of the left camera image of the aircraft surface taken by the binocular camera;

[0028] Figure 4 Schematic diagram of the right camera image of the aircraft surface taken by the binocular camera;

[0029] Figure 5 Schematic diagram of damage detection results of the left camera image;

[0030] Figure 6 Schematic diagram of relative posture measurement;

[0031] Figure 7 The figure shows the damage points on the 3D point cloud model of the aircraft.

[0032] Figure 8 This is a structural diagram of a three-dimensional positioning device for aircraft surface damage;

[0033] Figure 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0035] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0036] In an exemplary embodiment, Figure 1 As shown, a three-dimensional positioning method for aircraft surface damage is provided, comprising:

[0037] Step 100: Acquire a target image. The target image is an aircraft surface image acquired by a binocular camera.

[0038] Step 200: Determine damage detection results based on the target image based on the damage detection model. The damage detection model is trained using deep learning methods based on an aircraft surface damage dataset. The aircraft surface damage dataset includes surface damage information and corresponding annotated damage detection results.

[0039] Step 300: Using a binocular vision positioning algorithm, based on the pixel coordinates of the center point of the damage area contained in the damage detection result, determine the three-dimensional coordinates of the damage point in the camera coordinate system.

[0040] Among them, a binocular vision positioning algorithm is used to determine the three-dimensional coordinates of the damage point in the camera coordinate system based on the pixel coordinates of the center point of the damage area contained in the damage detection results. Specifically, it includes:

[0041] The MATLAB toolbox is used to calibrate the binocular camera and obtain the camera's intrinsic parameters.

[0042] The left and right images in the target image are corrected using the camera's intrinsic parameters to obtain a corrected image.

[0043] The SGBM algorithm is used to perform stereo matching on the corrected image, so that the pixels in the left and right images of the target image are paired to obtain the pairing result.

[0044] According to the pairing results and the pixel coordinates of the center point of the damage area contained in the damage detection results, the parallax of the damage point is calculated to obtain the three-dimensional coordinates of the damage point in the camera coordinate system.

[0045] The three-dimensional coordinates of the damage point in the camera coordinate system, the corresponding mathematical expression is:

[0046]

[0047]

[0048] Where Z is the coordinate of the damage point along the optical axis in the camera coordinate system; X is the coordinate of the damage point along the optical center line in the camera coordinate system; Y is the coordinate of the damage point in the direction perpendicular to the optical axis and the optical center line in the camera coordinate system; b is the camera baseline length of the binocular camera; f is the focal length of the binocular camera; d is the parallax of the damage point; x l is the horizontal coordinate of the damage point in the left camera coordinate system; l is the vertical coordinate of the damage point in the left camera coordinate system; (u l , v1) is the coordinate of the damage point in the pixel coordinate system; u0, d x , v0 and d y All are camera internal parameters.

[0049] Step 400: Based on the relative pose relationship between the binocular camera and the aircraft, the three-dimensional coordinates of the damage point in the aircraft coordinate system are determined using the three-dimensional coordinates of the damage point in the camera coordinate system. The relative pose relationship between the binocular camera and the aircraft is determined by determining the rotation matrix and translation vector between the camera coordinate system and the aircraft coordinate system using a multi-sensor information fusion method based on an extended Kalman filter.

[0050] The relative position relationship between the binocular camera and the aircraft is determined by the following method:

[0051] A multi-sensor information fusion method based on extended Kalman filter is used to determine the real-time position and attitude of the binocular camera when it acquires the aircraft surface image.

[0052] The invention relates to a method for determining a rotation matrix of a binocular camera relative to an aircraft based on a real-time posture; determining a translation vector of the binocular camera relative to the aircraft based on a real-time position; determining a rotation matrix and a translation vector of the binocular camera relative to a mounting device according to an installation position of the binocular camera; and mounting the binocular camera on the mounting device.

[0053] The relative position relationship between the binocular camera and the aircraft is determined according to the rotation matrix of the binocular camera relative to the aircraft, the translation vector of the binocular camera relative to the aircraft, and the rotation matrix and translation vector of the binocular camera relative to the mounting device.

[0054] The relative position relationship between the binocular camera and the aircraft is expressed as follows:

[0055]

[0056]

[0057] in, is the rotation matrix between the camera coordinate system and the aircraft body coordinate system; is the rotation matrix of the binocular camera relative to the aircraft; is the rotation matrix of the binocular camera relative to the mounting device; T c w is the translation vector between the camera coordinate system and the aircraft body coordinate system; T ca is the translation vector of the binocular camera relative to the mounting device; T a w is the translation vector of the binocular camera relative to the aircraft.

[0058] Step 500: Map the three-dimensional coordinates of the damage point in the aircraft body coordinate system to the three-dimensional point cloud model of the aircraft, perform visualization processing, and obtain a visualization display result.

[0059] The three-dimensional coordinates of the damage point in the aircraft body coordinate system are expressed as follows:

[0060]

[0061] in, is the rotation matrix between the camera coordinate system and the aircraft body coordinate system; T c wis the translation vector between the camera coordinate system and the aircraft body coordinate system; Z is the coordinate of the damage point along the optical axis in the camera coordinate system; X is the coordinate of the damage point along the direction of the optical center in the camera coordinate system; Y is the coordinate of the damage point in the direction perpendicular to the optical axis and the optical center in the camera coordinate system; X w Y is the horizontal coordinate of the damage point in the three-dimensional coordinate system of the aircraft body; w Z is the vertical coordinate of the damage point in the three-dimensional coordinate system of the aircraft body; w It is the vertical coordinate of the damage point in the three-dimensional coordinate system of the aircraft body.

[0062] This application uses a binocular camera mounted on an unmanned vehicle to obtain an image of the aircraft surface; the image is input into a damage detection model based on deep learning to obtain the type, shape, size and pixel coordinates of the center point of the damage area; the three-dimensional coordinates of the damage point in the camera coordinate system are obtained based on the pixel coordinates of the center point of the damage area using the binocular vision positioning principle; the relative position and posture relationship between the camera and the aircraft under test is calculated using the position and posture information provided by the sensor mounted on the unmanned vehicle; the three-dimensional coordinates of the damage point in the aircraft body coordinate system are calculated based on the three-dimensional coordinates of the damage point in the camera coordinate system, combined with the relative position and posture relationship between the camera and the aircraft under test; the damage point is mapped to the three-dimensional point cloud model of the aircraft to achieve three-dimensional positioning and visualization of damage on the aircraft surface. This application can determine the location of the damage on the aircraft body, allowing maintenance personnel to go directly to the damaged location for repair, thereby improving the efficiency of aircraft maintenance.

[0063] Specifically, the implementation steps are as follows:

[0064] Acquire the target image; the target image is an image of the aircraft surface taken by an unmanned vehicle equipped with a binocular camera.

[0065] Specifically, the target image is acquired by using a mobile platform such as an unmanned vehicle or a drone equipped with a binocular camera to capture the surface image of the aircraft, including the left camera image and the right camera image.

[0066] The target image is fed into a deep learning-based damage detection model to obtain damage detection results, including the damage type, shape, size, and pixel coordinates of the damaged area's center. The deep learning-based damage detection model is trained using a dataset of aircraft surface damage. The YOLOv11 model is used for deep learning-based damage detection.

[0067] Specifically, the left camera image is input into a deep learning-based damage detection model to obtain the damage category, shape, size, and pixel coordinates of the center point of the damaged area; the deep learning-based damage detection model uses YOLO11m-seg as a pre-training model and then trains using an aircraft surface damage dataset; the aircraft surface damage dataset includes pictures of common aircraft surface damage such as scratches, dents, paint peeling, and corrosion, and the labelme tool is used to annotate the pictures.

[0068] Using binocular vision positioning, the three-dimensional coordinates of the damage point in the camera coordinate system are calculated based on the target image and the pixel coordinates of the damage point in the target image. In other words, the three-dimensional coordinates of the damage point in the camera coordinate system can be determined based on the pixel coordinates of the center point of the damaged area included in the damage detection results.

[0069] The left camera image and the right camera image are input into the binocular vision positioning algorithm to calculate the coordinates of the damage point in the camera coordinate system; the camera coordinate system is the left camera coordinate system; the binocular vision positioning algorithm includes: correcting the image according to the camera internal parameters to eliminate the influence of camera distortion; using the SGBM algorithm for stereo matching to pair the pixels in the left and right images; calculating the disparity of the center point of the damage area in the left and right images; and calculating the coordinates of the damage point in the camera coordinate system based on the camera focal length, baseline length, disparity, and pixel coordinates of the damage point.

[0070] According to the relative posture relationship between the camera and the aircraft under test, the three-dimensional coordinates of the damage point in the aircraft body coordinate system are calculated; the relative posture relationship between the camera and the aircraft under test refers to the rotation matrix and translation vector between the camera coordinate system and the aircraft body coordinate system.

[0071] The measurement of relative posture relationship includes: setting the aircraft body coordinate system and measuring the aircraft heading angle. Since the aircraft is usually parked on a flat horizontal ground, the aircraft's pitch angle and roll angle are considered to be zero.

[0072] The measurement of relative pose is completed using a multi-sensor information fusion method based on extended Kalman filtering. The multi-sensor information includes data collected by the unmanned vehicle's inertial measurement unit, odometer, and RTK (Real-Time Kinematic) receiver.

[0073] The system collects attitude quaternions output by the IMU (Inertial Measurement Unit), displacements recorded by the odometry, and latitude, longitude, and altitude obtained by the RTK receiver. The extended Kalman filter is used to fuse these sensor data to obtain the real-time position and attitude of the UAV, thereby calculating the relative position and attitude between the UAV and the aircraft being measured. The position and angle of the camera on the UAV are used to determine the position and attitude of the camera relative to the UAV, and ultimately the position and attitude of the camera relative to the aircraft being measured.

[0074] The 3D coordinates of the damage points in the aircraft's body coordinate system are mapped to the aircraft's 3D point cloud model, enabling 3D location and visualization of damage on the aircraft's surface. Specifically, points of a different color than the aircraft's body point cloud are added to the 3D point cloud model to represent the damage, enabling visualization of the damage.

[0075] The training process of the deep learning-based damage detection model includes:

[0076] Obtain an aircraft surface damage dataset, which includes images of common aircraft surface damage, such as scratches, dents, paint peeling, and corrosion. Use the labelme tool to annotate the images, and use YOLO11m-seg as the pre-trained model.

[0077] The binocular vision positioning method is based on the binocular vision imaging principle and uses the camera focal length, baseline length, parallax, and pixel coordinates of the damage point in the left camera image coordinate system to calculate the three-dimensional coordinates of the damage point in the left camera coordinate system.

[0078] Relative pose measurement uses an extended Kalman filter to fuse data collected by the unmanned vehicle's inertial measurement unit, odometer, and RTK (Real-Time Kinematic) receiver to calculate the position and attitude of the unmanned vehicle relative to the measured aircraft. The position and attitude of the camera relative to the measured aircraft are then calculated based on the camera's installation position and angle on the unmanned vehicle.

[0079] The damage points are mapped to the aircraft's 3D point cloud model based on the 3D coordinates of the damage points in the aircraft's body coordinate system. In the 3D point cloud model of the aircraft being tested, points of a different color from the body point cloud are added to represent the damage.

[0080] In practical applications, the methods mentioned in this application, such as Figure 2 As shown, the following steps are included:

[0081] S1: Collect images of aircraft surface damage, construct an aircraft surface damage dataset, and train a damage detection model based on YOLOv11.

[0082] S1.1: Use a handheld camera or a camera mounted on a drone or unmanned vehicle to capture images of aircraft surface damage.

[0083] S1.2: Use the labelme tool to annotate the images, construct an aircraft surface damage dataset, and convert it into the YOLO dataset format.

[0084] S1.3: Use the aircraft surface damage dataset constructed in step S1.2 to train a YOLOv11-based damage detection model. Use yolo11m-seg.pt provided by the ultralytics official website as a pre-training model to improve the model convergence difficulties and poor learning results caused by the small size of the training dataset.

[0085] S2: Take an image of the aircraft surface and input it into the damage detection model to obtain the pixel coordinates of the center point of the damaged area.

[0086] S2.1: Use the unmanned vehicle equipped with binocular cameras to circle around the aircraft under test and take pictures of the aircraft surface, including left camera pictures and right camera pictures, such as Figure 3 and Figure 4 shown.

[0087] S2.2: Input the left camera image taken in step S2.1 into the damage detection model based on YOLOv11 trained in S1.3 for detection. The detection results are as follows: Figure 5 As shown in , the type, size, shape and location of the damage in the image can be obtained, and each damage will be framed by a detection box, as shown in Figure 5 The blue portion in the upper right corner represents the label name for the damage, indicating paint chipping. The pixel coordinates of the center point of the detection frame are used as the pixel coordinates of the damage point.

[0088] S3: Use binocular vision positioning algorithm to calculate the coordinates of the damage point in the camera coordinate system.

[0089] The binocular vision positioning algorithm includes three steps: image correction, stereo matching, and disparity calculation.

[0090] S3.1: Use the Stereo Camera Calibrator toolbox in MATLAB to calibrate the binocular camera and obtain the camera's intrinsic parameters.

[0091] S3.2: Use the camera internal parameters to correct the left and right images to eliminate the influence of camera distortion.

[0092] S3.3: Use the SGBM algorithm to perform stereo matching on the images and pair the pixels in the left and right images.

[0093] S3.4: Calculate the parallax of the damage point in step S2.2 to obtain the coordinates (X, Y, Z) of the damage point in the camera coordinate system. The calculation formula is as follows:

[0094]

[0095] In step S2.1, the coordinates of the damage point in the pixel coordinate system (u l , v1), according to the pinhole camera imaging principle, (x l ,y1) and (u l , v1) satisfy the following conversion relationship:

[0096]

[0097] u0,d x , v0 and d y All are camera internal parameters. x and d y Respectively represent the physical size of a single pixel in the x and y directions of the imaging plane, v0 and u0 represent the coordinates of the camera optical center in the pixel coordinate system.

[0098] S4: Calculate the relative pose relationship between the camera and the aircraft under test.

[0099] The relative position relationship between the camera and the aircraft under test is calculated as follows: Figure 6 As shown, the camera coordinate system O c X c Y c Z c The optical center of the left camera is the origin, and the optical axis of the left camera is Z c Axis, the line between the two optical centers is X c Axis, Y c The axis is vertically downward, the aircraft body coordinate system O w X w Y w Z w The nose is the origin and the vertical axis of the body is Y w Axis, X w Axis along the wing direction, Z w The axis is vertically upward, Y w The angle between the axis and the N axis of the ENU (Northeast Sky) coordinate system is θ, that is, the heading angle of the aircraft is θ.

[0100] S4.1: Use the extended Kalman filter to fuse the attitude quaternion output by the IMU on the UAV, the displacement recorded by the odometer, and the longitude, latitude and altitude obtained by the RTK receiver to obtain the real-time position and attitude of the UAV when capturing the aircraft surface image.

[0101] S4.2: Convert the real-time posture (quaternion) of the unmanned vehicle obtained in step S4.1 into Euler angles to obtain the pitch angle α of the unmanned vehicle a , roll angle β a and heading angle γ a , thus obtaining the rotation matrix of the unmanned vehicle relative to the measured aircraft The calculation formula is:

[0102]

[0103] S4.3: According to the conversion relationship between the WSG84 coordinate system and the ENU coordinate system, the real-time position of the unmanned vehicle (latitude, longitude, altitude) obtained in step S4.1 is converted to the ENU coordinate system, and then converted to the Y coordinate system of the aircraft body. w The angle between the axis and the N axis of the ENU coordinate system is calculated to obtain the coordinates of the unmanned vehicle in the aircraft body coordinate system (x a ,y a ,z a ), then the translation vector T of the unmanned vehicle relative to the measured aircraft a w =[x a ,y a ,z a ] T .

[0104] S4.4: Obtain the rotation matrix of the camera relative to the unmanned vehicle based on the camera's installation position on the unmanned vehicle and the translation vector T c a .

[0105] S4.5: Calculate the camera's position relative to the aircraft under test ( and the translation vector T c w ), the formula is as follows:

[0106]

[0107]

[0108] S5: Calculate the coordinates of the damage point in the aircraft body coordinate system and map them to the aircraft 3D point cloud model for visualization.

[0109] S5.1: Based on the coordinates (X, Y, Z) of the damage point in the camera coordinate system calculated in step S2.2 and the relative position between the camera and the aircraft under test obtained in step S4.5, calculate the coordinates (X, Y, Z) of the damage point in the aircraft body coordinate system. w ,Y w ,Z w ), the formula is as follows:

[0110]

[0111] S5.2: Based on the coordinates of the damage point calculated in step S5.1 in the aircraft body coordinate system, map them to the aircraft 3D point cloud model to visualize the damage location. The effect is as follows: Figure 7 As shown, the green points represent the damage points.

[0112] In an exemplary embodiment, Figure 8 As shown, a three-dimensional positioning device for aircraft surface damage is provided, comprising:

[0113] The target image acquisition module is used to acquire the target image, which is the aircraft surface image acquired by the binocular camera.

[0114] The damage detection module is used to determine damage detection results based on the target image based on a damage detection model. The damage detection model is trained using a deep learning method based on an aircraft surface damage dataset. The aircraft surface damage dataset includes: surface damage information and corresponding annotated damage detection results.

[0115] The binocular vision positioning module is used to determine the three-dimensional coordinates of the damage point in the camera coordinate system based on the pixel coordinates of the center point of the damage area contained in the damage detection results using the binocular vision positioning algorithm.

[0116] The coordinate conversion module is used to determine the three-dimensional coordinates of the damage point in the aircraft body coordinate system based on the relative posture relationship between the binocular camera and the aircraft and the three-dimensional coordinates of the damage point in the camera coordinate system. The relative posture relationship between the binocular camera and the aircraft is obtained by using a multi-sensor information fusion method based on the extended Kalman filter to determine the rotation matrix and translation vector between the camera coordinate system and the aircraft body coordinate system.

[0117] The visualization module is used to map the three-dimensional coordinates of the damage point in the aircraft body coordinate system to the three-dimensional point cloud model of the aircraft, perform visualization processing, and obtain a visualization display result.

[0118] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Figure 9As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store three-dimensional positioning data of aircraft surface damage. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a three-dimensional positioning method for aircraft surface damage is implemented.

[0119] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0120] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0121] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0122] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0123] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0124] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0125] The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may include, but are not limited to, general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic units, data processing logic units based on quantum computing, and the like.

[0126] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0127] This document uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A three-dimensional positioning method for aircraft surface damage, characterized in that: include: Acquire the target image; The target image is an aircraft surface graphic acquired by a binocular camera; determining a damage detection result according to the target image based on a damage detection model; The damage detection model is based on an aircraft surface damage dataset and is trained using a deep learning method. The aircraft surface damage dataset includes: surface damage information and corresponding annotated damage detection results; Using a binocular vision positioning algorithm, based on the pixel coordinates of the center point of the damage area included in the damage detection result, the three-dimensional coordinates of the damage point in the camera coordinate system are determined; Based on the relative pose relationship between the binocular camera and the aircraft, the three-dimensional coordinates of the damage point in the aircraft body coordinate system are determined according to the three-dimensional coordinates of the damage point in the camera coordinate system. The relative pose relationship between the binocular camera and the aircraft is obtained by using a multi-sensor information fusion method based on the extended Kalman filter to determine the rotation matrix and translation vector between the camera coordinate system and the aircraft body coordinate system. The three-dimensional coordinates of the damage point in the aircraft body coordinate system are mapped to the three-dimensional point cloud model of the aircraft, and visualization processing is performed to obtain a visualization display result.

2. The three-dimensional positioning method for aircraft surface damage according to claim 1, characterized in that: A binocular vision positioning algorithm is used to determine the three-dimensional coordinates of the damage point in the camera coordinate system based on the pixel coordinates of the center point of the damage area contained in the damage detection result, specifically including: Use MATLAB toolbox to calibrate the binocular camera and obtain the camera's intrinsic parameters; Correcting the left and right images in the target image using the camera intrinsic parameters to obtain a corrected image; Using the SGBM algorithm to perform stereo matching on the corrected image, so that the pixels in the left and right images of the target image are paired to obtain a pairing result; The disparity of the damage point is calculated based on the pairing result and the pixel coordinates of the center point of the damage area included in the damage detection result to obtain the three-dimensional coordinates of the damage point in the camera coordinate system.

3. The three-dimensional positioning method for aircraft surface damage according to claim 2, characterized in that: The three-dimensional coordinates of the damage point in the camera coordinate system, the corresponding mathematical expression is: Where Z is the coordinate of the damage point along the optical axis in the camera coordinate system; X is the coordinate of the damage point along the optical center line in the camera coordinate system; Y is the coordinate of the damage point in the direction perpendicular to the optical axis and the optical center line in the camera coordinate system; b is the camera baseline length of the binocular camera; f is the focal length of the binocular camera; d is the parallax of the damage point; x l is the horizontal coordinate of the damage point in the left camera coordinate system; l is the vertical coordinate of the damage point in the left camera coordinate system; (u l , v1) is the coordinate of the damage point in the pixel coordinate system; u0, d x , v0 and d y All are camera internal parameters.

4. The three-dimensional positioning method for aircraft surface damage according to claim 1, characterized in that: The relative position relationship between the binocular camera and the aircraft is determined by the following methods: A multi-sensor information fusion method based on extended Kalman filtering is used to determine the real-time position and attitude of the binocular camera when it acquires the image of the aircraft surface; Determine the rotation matrix of the binocular camera relative to the aircraft based on the real-time attitude; Determine the translation vector of the binocular camera relative to the aircraft based on the real-time position; Determining a rotation matrix and a translation vector of the binocular camera relative to a mounting device according to an installation position of the binocular camera; the binocular camera is mounted on the mounting device; The relative position relationship between the binocular camera and the aircraft is determined according to the rotation matrix of the binocular camera relative to the aircraft, the translation vector of the binocular camera relative to the aircraft, and the rotation matrix and translation vector of the binocular camera relative to the mounting device.

5. The three-dimensional positioning method for aircraft surface damage according to claim 4, characterized in that: The relative position relationship between the binocular camera and the aircraft is expressed as follows: in, is the rotation matrix between the camera coordinate system and the aircraft body coordinate system; is the rotation matrix of the binocular camera relative to the aircraft; is the rotation matrix of the binocular camera relative to the mounting device; is the translation vector between the camera coordinate system and the aircraft body coordinate system; is the translation vector of the binocular camera relative to the mounting device; is the translation vector of the binocular camera relative to the aircraft.

6. The three-dimensional positioning method for aircraft surface damage according to claim 1, characterized in that: The three-dimensional coordinates of the damage point in the aircraft body coordinate system are expressed as follows: in, is the rotation matrix between the camera coordinate system and the aircraft body coordinate system; is the translation vector between the camera coordinate system and the aircraft body coordinate system; Z is the coordinate of the damage point along the optical axis in the camera coordinate system; X is the coordinate of the damage point along the direction of the optical center in the camera coordinate system; Y is the coordinate of the damage point in the direction perpendicular to the optical axis and the optical center in the camera coordinate system; X w Y is the horizontal coordinate of the damage point in the three-dimensional coordinate system of the aircraft body; w Z is the vertical coordinate of the damage point in the three-dimensional coordinate system of the aircraft body; w It is the vertical coordinate of the damage point in the three-dimensional coordinate system of the aircraft body.

7. A three-dimensional positioning device for aircraft surface damage, characterized in that: include: A target image acquisition module is used to acquire a target image; The target image is an aircraft surface graphic acquired by a binocular camera; a damage detection module, configured to determine a damage detection result according to the target image based on a damage detection model; The damage detection model is based on an aircraft surface damage dataset and is trained using a deep learning method. The aircraft surface damage dataset includes: surface damage information and corresponding annotated damage detection results; A binocular vision positioning module is used to determine the three-dimensional coordinates of the damage point in the camera coordinate system based on the pixel coordinates of the center point of the damage area contained in the damage detection result using a binocular vision positioning algorithm; The coordinate conversion module is used to determine the 3D coordinates of the damage point in the aircraft's coordinate system based on the relative pose relationship between the binocular camera and the aircraft, and the 3D coordinates of the damage point in the camera's coordinate system. The relative pose relationship between the binocular camera and the aircraft is obtained by using a multi-sensor information fusion method based on the extended Kalman filter to determine the rotation matrix and translation vector between the camera coordinate system and the aircraft's coordinate system. The visualization module is used to map the three-dimensional coordinates of the damage point in the aircraft body coordinate system to the three-dimensional point cloud model of the aircraft, perform visualization processing, and obtain a visualization display result.

8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the three-dimensional positioning method for aircraft surface damage according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the three-dimensional positioning method for aircraft surface damage according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the three-dimensional positioning method for aircraft surface damage according to any one of claims 1 to 6 is implemented.

Images