Device Detection Method and Device Detection System

Abstract

The present invention relates to a device detection method and a device detection system. The device detection method includes the following steps: a detection task setting step; a detection path setting step; a navigation and positioning step, enabling the carrying platform to move along the detection path and determining the pose of the carrying platform in real time; a detection and shooting step, when the carrying platform moves along the detection path, the detection and shooting device shoots a detection image; and a detection and analysis step, analyzing the state of the object to be detected based on the pose of the carrying platform and the detection image. In the navigation and positioning step, the positioning and shooting device shoots an image of the environment where the object to be detected is located as a positioning image, and determines the pose of the carrying platform in real time based on the positioning image and the measurement value of the positioning measurement device. According to the device detection method and the device detection system of the present invention, real-time pose estimation is performed based on environmental information and positioning measurement data, avoiding the problem of affecting positioning due to insufficient texture of the object to be detected itself, and capable of improving the positioning accuracy.

Description

Technical Field

[0001] The present invention relates to a device detection method and a device detection system, and more particularly, to a device detection method and a device detection system for detecting a device using a mobile device (such as a drone or a robot). Background Art

[0002] The content of this section only provides background information related to the present invention, which may not constitute the prior art.

[0003] In the production, manufacturing and use of many industrial devices, various inspections and maintenance need to be carried out on the devices. With the development of technology, the device detection technology based on drones or robots has been widely applied in various industrial fields. For example, in the field of aircraft, the device detection technology based on drones or robots has been widely applied in the digital manufacturing, pre-flight inspection, maintenance and repair of aircraft.

[0004] In the device detection technology based on drones or robots, the accuracy of the navigation and positioning of the drones or robots is crucial for the accuracy of the device detection performed. The higher the accuracy of the navigation and positioning, the higher the accuracy of the device detection performed. Common positioning technologies include a positioning scheme based on the Global Positioning System (GPS), a positioning scheme based on Time Difference of Arrival (TDOA), and a positioning scheme based on Simultaneous Localization and Mapping (SLAM).

[0005] When using a drone or a detection robot to perform detection outdoors without occlusion, the drone or the robot can achieve accurate navigation and positioning by means of GPS. However, when performing detection indoors or when occluded, the application of the Global Positioning System is limited, and accurate navigation and positioning cannot be achieved only by means of GPS. And the positioning scheme based on TDOA involves wireless positioning technology, requires the establishment of a ground base station, and the application is easily restricted by ground facilities, and there are error problems caused by multipath effects in the positioning scheme based on TDOA.

[0006] SLAM-based positioning solutions can be used to provide positioning in the above application scenarios where GPS applications are restricted. Currently, they are widely used in fields such as robotics, drones, autonomous driving, AR, and VR. Relying on sensors, they can achieve functions such as autonomous positioning, mapping, and path planning of machines. According to the different sensors used, SLAM-based positioning solutions include laser-based SLAM solutions and vision-based SLAM solutions. Laser-based SLAM solutions perform positioning and mapping based on the point cloud information returned by lidar, with relatively high positioning accuracy. However, they need to perform a large number of data alignment operations, resulting in a large amount of computational processing, high power consumption, and are not suitable for use in unknown application scenarios. Vision-based SLAM solutions perform positioning and mapping based on the image information returned by cameras, and have received extensive attention due to advantages such as large amount of information, wide application range, low cost, and high feature discrimination. However, vision-based SLAM solutions have the problem of cumulative error and are easily affected by image quality, and cannot handle dark areas (textureless areas) well.

[0007] Therefore, there is still a need to improve existing positioning technologies to improve the navigation positioning accuracy of drones or inspection robots during the inspection process, thereby improving the accuracy of equipment inspection. Summary of the Invention

[0008] An object of the present invention is to provide a device inspection method to improve positioning accuracy and inspection accuracy. Another object of the present invention is to provide a device inspection system to improve the positioning accuracy of the platform carried by the device inspection system during the device inspection process, thereby improving the accuracy of device inspection.

[0009] One aspect of the present invention is to provide a device detection method, including the following steps: a detection task setting step, in which information of an object to be detected is input and a detection task is set; a detection path setting step, in which a detection path is set according to the detection task; a navigation and positioning step, in which a carrying platform is started to move the carrying platform along the detection path and the pose of the carrying platform is determined in real time, wherein the carrying platform is provided with a positioning sensing device and a detection imaging device, the positioning sensing device includes a positioning imaging device and a positioning measurement device, the positioning imaging device is configured to capture a positioning image during the movement of the carrying platform, and the positioning measurement device is configured to measure the movement of the carrying platform during the movement of the carrying platform; a detection imaging step, in which the detection imaging device captures a detection image of the object to be detected when the carrying platform moves along the detection path; and a detection analysis step, in which the state of the object to be detected is analyzed based on the pose of the carrying platform and the detection image. In the navigation and positioning step, the positioning imaging device captures an image of the environment where the object to be detected is located as the positioning image, and the pose of the carrying platform is determined in real time based on the positioning image and the measurement value of the positioning measurement device.

[0010] The navigation and positioning step includes: processing the positioning image and extracting image features; processing the measurement value to obtain the movement data of the carrying platform; performing a fusion process on the image features and the movement data to obtain a first pose estimate of the carrying platform; and optimizing the first pose estimate to obtain a second pose estimate.

[0011] The navigation and positioning step further includes calculating the positioning confidence in real time and associating the positioning confidence with the second pose estimate and the detection image to form detection image position metadata.

[0012] In the navigation and positioning step, feature points of the positioning image are extracted and tracked, and the image quality confidence, feature point distribution compactness, tracking confidence, and reprojection error of the positioning image are calculated. The positioning confidence is calculated based on the image quality confidence, feature point distribution compactness, tracking confidence, and reprojection error.

[0013] The detection analysis step includes: obtaining the detection image position metadata, stitching the obtained detection images, and calibrating the obtained detection image position metadata based on the design data of the object to be detected; and analyzing the state of the object to be detected based on the calibrated detection image position metadata.

[0014] Preferably, the detection analysis step is performed after the carrying platform returns.

[0015] The detection path setting steps include: constructing a virtual detection profile of the object to be detected according to the design parameters of the object to be detected, dividing the virtual detection profile into multiple sub-detection regions, and setting the planned detection paths of the multiple sub-detection regions; and setting the detection path as the planned detection path of at least one of the multiple sub-detection regions according to the detection task.

[0016] In one embodiment, the measurement value is the measurement value of one or more inertial measurement units.

[0017] In one embodiment, the object to be detected is an aircraft, and the positioning image is an image of the hangar where the aircraft is located.

[0018] Another aspect of the present invention is to provide a device detection system, including: a carrying platform, the carrying platform being adapted to move along the detection path of the object to be detected; a positioning sensing device, the positioning sensing device being arranged on the carrying platform, the positioning sensing device including a positioning photographing device and a positioning measuring device, the positioning photographing device being configured to photograph a positioning image during the movement of the carrying platform, and the positioning measuring device being configured to measure the movement of the carrying platform during the movement of the carrying platform; a detection photographing device, the detection photographing device being arranged on the carrying platform, the detection photographing device being configured to photograph a detection image of the object to be detected when the carrying platform moves along the detection path; and a calculation and analysis unit, the calculation and analysis unit being configured to: process the sensing results of the positioning sensing device and the detection image to analyze the object to be detected. The positioning photographing device is configured to: during the movement of the carrying platform along the detection path, the positioning photographing device photographs an image of the environment where the object to be detected is located as the positioning image, and the calculation and analysis unit determines the pose of the carrying platform in real time based on the positioning image and the measurement value of the positioning measuring device.

[0019] The calculation and analysis unit includes a positioning unit, and the positioning unit includes: a front-end processing unit, the front-end processing module being configured to process the positioning image to extract image features, process the measurement value to obtain the movement data of the carrying platform, and perform a fusion process on the image features and the movement data to obtain a first pose estimate of the carrying platform; a back-end processing unit, the back-end processing unit being configured to optimize the first pose estimate to obtain a second pose estimate; and a positioning credibility calculation unit, the positioning credibility calculation unit being configured to calculate the positioning credibility of the second pose estimate.

[0020] The calculation and analysis unit further includes a detection unit, and the detection unit is configured to analyze the state of the detection object according to the pose of the carrying platform and the detection image. The detection unit includes: an image stitching unit configured to obtain the detection image and the corresponding second pose estimation, the detection image and the second pose estimation form detection image position metadata, perform image stitching on the obtained detection images, and calibrate the detection image position metadata based on the design data of the object to be detected; and an analysis unit configured to analyze the state of the object to be detected based on the calibrated detection image position metadata.

[0021] Preferably, the detection unit analyzes the state of the detection object after the carrying platform returns.

[0022] The device detection system further includes a ground control device, and the ground control device can be operated to input the information of the object to be detected. The calculation and analysis unit constructs a virtual detection profile of the object to be detected according to the input information of the object to be detected, divides the virtual detection profile into multiple sub-detection regions, and sets the planned detection paths of the multiple sub-detection regions. The detection path is the planned detection path of at least one of the multiple sub-detection regions.

[0023] In one embodiment, the positioning and measuring device is one or more inertial measurement units.

[0024] In one embodiment, the object to be detected is an aircraft, and the positioning image is an image of the hangar where the aircraft is located.

[0025] The present invention provides an improved device detection method and a device detection system. The device detection method and the device detection system according to the present invention construct a virtual detection profile of the detection object based on the design parameters of the detection object, divide the detection area of the detection object into multiple sub-detection regions, and respectively plan the detection paths of the sub-detection regions, which can achieve more reasonable path planning and make the detection process safer. Moreover, the device detection method and the device detection system according to the present invention perform data fusion based on the information of the environment where the detection object is located and in combination with positioning measurement data (such as IMU data), and perform loop detection and correction to minimize pose drift, can achieve accurate real-time navigation and positioning without relying on the information of the detection object, and can avoid the influence caused by insufficient texture of the detection object itself. In addition, the device detection method and the device detection system according to the present invention also calculate the positioning confidence in real time, associate the pose estimation with the positioning confidence and the detection image to form detection image position metadata, perform image stitching on the detection image, further calibrate the detection image position metadata, and perform analysis based on the calibrated detection image position metadata, which can obtain more accurate analysis results. Description of the Drawings

[0026] Embodiments of the present invention will be described below by way of example only with reference to the accompanying drawings. In the drawings, the same features or components are denoted by the same reference numerals, and the drawings are not necessarily drawn to scale, and in the drawings:

[0027] Figure 1 A schematic block diagram of a device detection system according to an embodiment of the present invention is shown;

[0028] Figure 2 An example of a detection object detected using the device detection system according to the present invention is shown;

[0029] Figure 3 A flowchart of a device detection method of the device detection system according to the present invention is shown;

[0030] Figure 4 Shows the detection using the device detection method according to the present invention Figure 2 A schematic diagram of a detection path of the detection object shown; and

[0031] Figure 5 A flowchart of a navigation and positioning process of the device detection method according to the present invention is shown. Detailed Embodiments

[0032] The following description is essentially exemplary only and is not intended to limit the present invention, its applications and uses. It should be understood that in all these drawings, similar reference numerals indicate the same or similar parts and features. Each drawing only schematically shows the concept and principle of the embodiment of the present invention, and does not necessarily show the specific dimensions and their ratios of each embodiment of the present invention. Specific parts in a specific drawing may be exaggerated to illustrate relevant details or structures of the embodiment of the present invention.

[0033] Based on the consideration of the above-mentioned existing positioning technologies, the inventors found that in a vision-based SLAM solution, especially for indoor application scenarios, the image information of the environment where the object to be detected is located is often richer than the image information of the detection object. If the environmental information is used as the sensing output of the positioning sensing device, for example, taking an image of the environment where the object to be detected is located and performing subsequent processing based on this to achieve navigation and positioning, the problem of insufficient image texture of the object to be detected can be overcome. In addition, in a vision-based SLAM solution, loop detection is used to eliminate cumulative errors, and by performing multiple small loop detections to achieve large loop detection, loop detection can be optimized, which is beneficial to further correct pose estimation. In addition, by stitching the captured detection images, the positioning accuracy can be further improved, thereby providing a more reliable basis for subsequent detection and analysis.

[0034] The device detection system according to the present invention will be described below in conjunction with the accompanying drawings.

[0035] Figure 1 A schematic block diagram of a device detection system 1 according to the present invention is shown. As Figure 1 shown, the device detection system 1 includes a ground control device 10, a carrying platform 20, a positioning and sensing device 30, a detection and imaging device 40, a calculation and analysis unit 50, and a memory 60.

[0036] The ground control device 10 is a control device of the device detection system 1 and can be a handheld controller operated by an operator. The ground control device 10 is configured to start or stop the movement of the carrying platform 20 to start or stop the detection of the detection object. The detection object can be various industrial devices, such as airplanes, vehicles, electric towers, etc., or can be certain specific spaces or facilities, such as workshops or factories. The ground control device 10 is provided with a user input interface, which may include a detection task setting interface 11, a start detection interface 13, and a stop detection interface 15. The operator can input information of the object to be detected through the detection task setting interface 11 of the ground control device 10, including the type of the object to be detected, the location where the object to be detected is located, the detection alignment point of the object to be detected, the detection items of the object to be detected, etc., to set the detection task. The detection alignment point of the object to be detected can be a point on the detection path of the object to be detected that is aligned with a certain part of the object to be detected. For example, when the object to be detected is an airplane, the detection alignment point can be a point on the detection path of the airplane that is aligned with the nose. The detection path of the object to be detected can be a path located outside the outer contour of the object to be detected that surrounds the object to be detected, or can be a path located inside the object to be detected. Then, the operator starts the movement of the carrying platform 20 by operating the start detection interface 13, so that the carrying platform 20 moves along the detection path to start the detection of the object to be detected. During the process of the carrying platform 20 moving to perform the detection, the operator can terminate the detection at any time by operating the stop detection interface 15 of the ground control device 10.

[0037] The carrying platform 20 is a mobile device capable of moving along the detection path to perform detection. For example, it can be a drone or a robot for detection. The carrying platform 20 can carry multiple devices of the device detection system 1. For example, the positioning and sensing device 30, the detection and imaging device 40, the calculation and analysis unit 50, and the memory 60 can all be provided on the carrying platform 20. The carrying platform 20 moves according to the command of the ground control device 10 to move to the detection position to perform detection or return from the detection position.

[0038] The positioning sensing device 30 is disposed on the carrying platform 20 and is configured to capture positioning images during the movement of the carrying platform 20 and measure the movement of the carrying platform 20 to determine the position and attitude of the carrying platform 20. The positioning sensing device 30 includes a positioning imaging device 31 and a positioning measurement device 33. The positioning imaging device 31 can be mounted on the carrying platform 20 such that the relative pose between the positioning imaging device 31 and the carrying platform 20 is adjustable. For example, the positioning imaging device 31 can be mounted on the carrying platform 20 via a pan-tilt head with controllable movement and can move relative to the carrying platform 20, such as horizontally and vertically pitching relative to the carrying platform 20, to adjust the shooting azimuth and pitch angle. The positioning imaging device 31 is configured to capture images of the environment where the object to be detected is located in real time during the movement of the carrying platform 20 to determine the position and attitude of the carrying platform 20. Preferably, the positioning imaging device 31 employs a high-speed binocular camera. The positioning imaging device 31 is configured such that its optical axis forms an angle with the detection surface of the object to be detected (i.e., does not face the detection surface of the object to be detected directly, that is, is not perpendicular to the detection surface of the object to be detected) to align with the environment where the object to be detected is located and capture images of the environment where the object to be detected is located as positioning images. For example, in one application example, when the object to be detected is an aircraft parked in a hangar, the positioning imaging device 31 is configured such that its optical axis forms an angle with the surface of the aircraft, for example, an angle of approximately 60 degrees, to align with the ceiling of the hangar, or the peripheral wall or other parts of the hangar. Thus, during the movement of the carrying platform 20 following the positioning imaging device 31, images of the hangar are captured as positioning images, and corresponding processing and calculations are performed to determine the pose of the carrying platform 20 and the devices mounted thereon in real time. Through the above configuration, during the movement of the carrying platform 20, accurate positioning of the carrying platform 20 and the devices mounted thereon (such as the positioning imaging device 31 and the detection imaging device 40) is achieved based on the environmental information of the object to be detected, without relying on the features of the object to be detected, thereby avoiding the problem of affecting the positioning accuracy due to insufficient texture of the object to be detected.

[0039] The positioning and measuring device 33 is also disposed on the carrying platform 20, and is used to sense the six-degree-of-freedom motion of the carrying platform 20 in real time, so as to determine the pose of the carrying platform 20 and the devices carried thereon (for example, the positioning and photographing device 31, the detection and photographing device 40). The pose of the carrying platform 20 refers to the position and attitude of the carrying platform 20 in a three-axis coordinate system in space. The relative positioning between the carrying platform 20 and the devices carried thereon (for example, the positioning and photographing device 31, the detection and photographing device 40) is known. Therefore, after obtaining the pose of the carrying platform 20, the pose of the devices carried on the carrying platform 20 can be obtained according to the relative positioning between the carrying platform and them, and vice versa. In this embodiment, the positioning and measuring device 33 includes one or more inertial measurement units (IMUs). The inertial measurement unit can provide six-degree-of-freedom measurements, and can include three single-axis accelerometers and three single-axis gyroscopes, which are used to detect the acceleration and angular velocity of the carrying platform 20 in three-dimensional space, so as to solve the six-degree-of-freedom motion of the carrying platform 20 in space, that is, the movement along the three rectangular coordinate axes of the three-axis coordinate system in space and the rotation around these three axes, for determining the attitude of the carrying platform 20. In other examples, the inertial measurement unit of the positioning and measuring device 33 can provide nine-degree-of-freedom measurements. For example, in addition to including accelerometers and gyroscopes, it can also include three single-axis magnetic sensors to provide heading information.

[0040] The detection and photographing device 40 can be mounted on the carrying platform 20 via a gimbal with controllable movement, for example, and can move relative to the carrying platform 20, such as horizontal movement and vertical pitching movement relative to the carrying platform 20, to adjust the photographing azimuth and pitch angle. The detection and photographing device 40 is configured to take images of the object to be detected as detection images at a predetermined time interval when the carrying platform 20 moves along the detection path of the object to be detected, and store the detection images in the memory 60 for detecting and analyzing the detection object.

[0041] The calculation and analysis unit 50 is configured to communicate with the ground control device 10, the positioning and sensing device 30, the detection and imaging device 40, and the memory 60 in a wired or wireless manner, and perform various processes and calculations. For example, the calculation and analysis unit 50 can obtain the set detection task from the ground control device 10, and obtain the corresponding design parameters (such as a digital design model) and the detection safety distance from the memory 60 according to the set detection task, and construct a virtual detection profile of the object to be detected based on the obtained design parameters, divide the constructed virtual detection profile into multiple sub-detection regions, and set the planned detection paths of the respective sub-detection regions according to the detection safety distance. The calculation and analysis unit 50 is further configured to receive and process the sensing data of the positioning and sensing device 30 (including the positioning image and the measurement values of the positioning measurement device) to calculate the pose of the carrying platform 20, receive and process the detection image captured by the detection and imaging device 40 to perform the detection and analysis of the object to be detected, etc. In the present embodiment, the calculation and analysis unit 50 is provided on the carrying platform 20. Alternatively, in other embodiments according to the present invention, the calculation and analysis unit 50 may also be provided on the ground control device 10.

[0042] The calculation and analysis unit 50 mainly includes a detection path planning unit 51, a positioning unit 53, and a detection unit 55. The detection path planning unit 51 is configured to construct a virtual detection profile of the object to be detected using the design data of the object to be detected according to the information of the object to be detected, divide the virtual detection profile into multiple sub-detection regions, set the planned detection paths of each sub-detection region according to the detection safety distance, and set the detection path as the planned detection path of at least one of the multiple sub-detection regions of the virtual detection profile according to the set detection task.

[0043] The positioning unit 53 is configured to process the sensing data of the positioning sensing device 30 to determine the pose of the carrying platform 20. The positioning unit 53 includes a front-end processing unit 531, a back-end processing unit 532, and a positioning confidence calculation unit 533. The front-end processing unit 531 includes a visual odometry unit 5311 and a loop detection unit 5312. The visual odometry unit 5311 is configured to receive and process the continuous frame positioning images captured by the positioning imaging device 31 and the sensing signals of the positioning measurement device 33, and process the received positioning images and sensing signals to calculate a first pose estimate of the carrying platform 20. The loop detection unit 5312 is configured to perform loop detection on the positioning images captured by the positioning imaging device 31, and establish a constraint relationship between the current frame and the previous frames by comparing the similarity of the positioning images captured by the positioning imaging device 31, so as to eliminate the cumulative error. The back-end processing unit 532 is configured to receive the first pose estimate calculated by the visual odometry unit 5311 and the detection result of the loop detection unit 5312, optimize the first pose estimate to obtain a second pose estimate, and use this as the real-time pose of the carrying platform. The positioning confidence calculation unit 533 is configured to calculate the positioning confidence of the positioning unit 53, and store the calculated positioning confidence and the second pose estimate in the memory 60.

[0044] The detection unit 55 is configured to process the detection images captured by the detection imaging device 40 according to the real-time pose calculated by the positioning unit 53 to analyze the state of the detection object. Preferably, the detection unit 55 is configured to process and analyze the detection images after the carrying platform 20 moves around the detection object along the detection path to complete the capture of the detection object and returns, rather than performing real-time analysis and processing during the movement of the carrying platform 20, so as to reduce the real-time calculation amount of the calculation and analysis unit 50 and reduce the configuration requirements of the calculation and analysis unit 50. The detection unit 55 includes an image stitching unit 551 and an analysis unit 552. The image stitching unit 551 is configured to read from the memory the second pose estimate calculated by the positioning unit 53, the corresponding positioning confidence, and the detection images, form detection image position metadata, and perform image stitching on the read detection images to form a model contour of the object to be detected, so as to calibrate the read detection image position metadata, and update the detection image position metadata with the calibrated detection image position metadata to further improve the position accuracy of the detection images. The analysis unit 552 is configured to process and analyze the detection images based on the calibrated detection image position metadata to determine the state of the detection object.

[0045] The memory 60 is configured to store various information, data, algorithms, etc., including: design parameters corresponding to various types of detection objects, detection alignment points, detection safety distances, processing algorithms, state judgment criteria, sensing data of each sensor, intermediate data of each calculation process, etc. The design parameters of each detection object include the design parameters of each part of the detection object, the assembly parameters between each part, and the overall assembly parameters of the detection object, etc. The design parameters of the detection object can be based on the digital mock-up model (DMU) data of the detection object. The detection alignment point is the starting point of the detection path of the object to be detected. After the carrying platform 20 moves to this detection alignment point, the detection starts to be executed. The detection safety distance is the minimum distance that the carrying platform 20 should maintain from the detection object during movement. During the process of the carrying platform 20 moving along the detection path of the detection object for detection, the movement of the carrying platform 20 can be corrected according to this detection safety distance to make it follow the detection path of the detection object. The state judgment criterion is the criterion for judging the state of the detection object. For example, this criterion can be the defect judgment criterion corresponding to each detection item in the maintenance manual of the detection object. The sensing data of the sensor includes the positioning image captured by the positioning shooting device 31, the measurement result of the positioning measurement device 33, and the detection image captured by the detection shooting device 40. During the process of using the equipment detection system 1 to detect the detection object, the information, algorithms, etc. stored in the memory 60 are accessed and called by the calculation and analysis unit 50 in a wired or wireless manner to perform corresponding processing.

[0046] The schematic block diagram of the equipment detection system 1 has been introduced above. According to the equipment detection system 1 of the present invention, based on the design parameters of the object to be detected, a virtual detection contour of the object to be detected is constructed, and a detection path is set to perform detection based on the digital model of the detection object. Moreover, the image of the environment where the object to be detected is located is used as the positioning image, and data fusion is performed on the positioning image and the measurement value of the positioning detection device to achieve positioning. Loop detection and correction are performed on each part of the object to be detected, which can eliminate cumulative errors, minimize pose drift, optimize pose estimation, achieve simultaneous localization and mapping based on the environment, and improve the positioning accuracy. In addition, after the detection shooting device 40 completes the detection shooting of the object to be detected and the carrying platform 20 returns, the equipment detection system 1 performs image stitching on the detection image to calibrate the position information of the detection image, further improving the position accuracy of the detection image, thereby providing a more reliable basis for subsequent detection analysis and enabling more accurate detection analysis.

[0047] Next, with reference to the accompanying drawings, taking an airplane as an example of the detection object, the detection method for detecting an airplane using the equipment detection system 1 according to the present invention will be described.

[0048] Figure 2The schematic diagram of the aircraft M to be detected is shown, and the outline of the aircraft M is shown. The aircraft M mainly includes a nose M1, a left wing M2, a right wing M3, a fuselage M4, a tail M5, a left horizontal tail M6, a right horizontal tail M7, and a vertical tail M8. The equipment detection system 1 can fly around the aircraft M, as Figure 2 shown by the arrowed line in

[0049] Figure 3 to detect the aircraft M.

[0050] As Figure 3 shown, first, in step S1, the detection task is set. The user inputs the information of the object to be detected in the detection task setting interface 11 of the ground control device 10, including the type of the object to be detected, the location of the object to be detected, the detection alignment point of the object to be detected, the detection area, etc., to set the detection task to be performed. In this example, the object to be detected is the Figure 2 aircraft M shown in

[0051] which is parked in the hangar, the detection alignment point is the nose M1 of the aircraft M, and the detection area is the nose M1 of the aircraft M. Figure 2 Next, according to the detection task set in step S1, in step S2, the detection path of the object to be detected (i.e., the aircraft M) is set. In step S21, a virtual detection profile is constructed and the detection area is divided. The detection path planning unit 51 reads the design parameters corresponding to the aircraft M and the corresponding detection safety distance from the memory 60, and constructs the virtual detection profile of the aircraft M based on the read design parameters. For example, the DMU data of the aircraft M can be read from the memory 60, and the virtual detection profile of the aircraft M is constructed based on the read DMU data. The established virtual detection profile of the aircraft M is divided into multiple sub-detection areas, and then according to the read detection safety distance corresponding to the aircraft M, the planned detection paths of each detection area are generated. For example, based on the virtual detection profile of the aircraft M, the profile detection area of the aircraft M is divided into Figure 4As shown by the dashed line in []. In step S22, based on the set detection task, the detection path that the carrying platform 20 needs to follow when performing detection is set as the planned detection path of at least one of the multiple sub-detection areas of the contour detection area of the aircraft M. For example, when it is necessary to inspect the nose M1 of the aircraft M, the detection path of the carrying platform 20 is set as the planned detection path P1.

[0052] After the detection path is set, in step S3, the user operates the start detection interface 13 of the ground control device 10 to start the movement of the carrying platform 20, so that the carrying platform 20 moves along the detection path to perform detection. For example, when preparing to perform the inspection of the nose M1 of the aircraft M, the movement of the carrying platform 20 is started. When moving near the nose M1, for the purpose of obstacle avoidance, the positioning and photographing device 31 takes a photo of the nose. Then, the carrying platform 20 moves to the detection alignment point and moves along the planned detection path P1 to perform the detection of the nose M1.

[0053] Next, in step S4, during the process that the carrying platform 20 moves along the detection path to perform detection, the positioning unit 53 performs navigation and positioning. Figure 5 Shows the detailed process of step S4.

[0054] As Figure 5 shown, in step S41, the positioning unit 53 obtains the sensing result from the positioning sensing device 30. During the process that the carrying platform 20 moves along the detection path, the positioning and photographing device 31 of the positioning sensing device 30 is set so that its optical axis forms an angle with the surface of the detection area of the aircraft M. For example, the optical axis of the positioning and photographing device 31 forms a 60-degree angle with the surface of the detection area of the aircraft M, so that the optical axis of the positioning and photographing device 31 is aligned with the ceiling or the peripheral wall of the hangar where the aircraft M is located, thereby taking an image of the hangar where the aircraft M is located (for example, an image of the ceiling or the peripheral wall of the hangar), and using this image as the positioning image. And, during this process, the positioning measurement device 33 of the positioning sensing device 30 senses the six-degree-of-freedom movement of the carrying platform 20 in real time. In this example, the positioning measurement device 33 includes one or more IMUs, and the IMUs are used to measure the six-degree-of-freedom movement of the carrying platform 20 in three-dimensional space. The positioning unit 53 obtains the positioning image from the positioning and photographing device 31 and obtains the positioning measurement data (measurement data of the IMUs) from the positioning measurement device 33.

[0055] Then, in step S42, front-end processing is performed to establish the association between the obtained positioning image and the positioning measurement data (that is, the measurement data of the IMUs), including the visual odometry performed in step S421 and the loop detection performed in step S422. In this example, the visual odometry in step S421 uses the feature point method to calculate the three-dimensional movement trajectory of the carrying platform 20 between consecutive frames of the positioning image, and obtains the first pose estimate.

[0056] In step S421, the positioning measurement data is processed to obtain the motion data of the carrying platform 20, and the positioning image is processed to extract image features. Specifically, in step S4211, the obtained positioning measurement data (the measurement data of the IMU) is processed. For example, data integration is performed and the error is calculated.

[0057] Meanwhile, in step S4212, the visual odometry unit 5311 extracts the feature points of the positioning image and calculates the image quality credibility of the positioning image. The corner points of the positioning image can be selected as the key points, and the corresponding feature descriptors are calculated. According to the ratio of the area occupied by the feature points to the total area of the positioning image, the image quality credibility of the positioning image is calculated. Then, in step S4213, the distribution of the feature points is statistically analyzed. According to the calculated feature descriptors and combined with the processing of the positioning measurement data in step S4211, the distribution of the feature points of the positioning image is statistically analyzed, and the compactness of the distribution of the feature points of the positioning image is calculated. The compactness of the distribution of the feature points can be calculated based on the distance between adjacent feature points. Next, in step S4214, feature point matching is performed. According to the calculated feature descriptors, key frames are determined, and the tracking credibility is calculated based on the number of feature points tracked in the positioning images of the front and rear frames. For the key frames of the positioning image, at least one of the following conditions needs to be satisfied: 1) The change in the attitude angle between consecutive frame positioning images exceeds a predetermined angle change threshold. For example, in one example, the predetermined angle change threshold is 5 degrees; 2) The change in the spatial displacement in consecutive frame positioning images exceeds a predetermined displacement threshold. For example, in one example, the predetermined displacement threshold is 0.1 m; 3) The number of tracking feature points in consecutive frame positioning images is lower than the feature point threshold.

[0058] Next, in step S4215, the image features of the extracted positioning image and the processed data of the positioning measurement data are fused, the reprojection error of the positioning image is optimized, and the first pose estimate of the carrying platform 20 is obtained. This first pose estimate includes the position (X, Y, Z coordinates in space) and attitude (attitude quaternion Qx, Qy, Qz, Qw) of the carrying platform 20.

[0059] In addition, in step S422, the loop detection unit 5312 performs loop detection. Based on the distribution of feature points of the positioning image counted in step 4213, the same feature points that are passed through multiple times during the movement of the carrying platform 20 are determined, and the co-visible key frames of the current frame of the positioning image are identified. The current positioning image is compared with the co-visible key frames, and co-visible feature points are found in the co-visible key frames of the current frame. When the number of matching co-visible feature points exceeds the threshold, a closed loop is formed, the motion matrix is calculated according to the pose of the positioning image, geometric constraints are used for matching, and triangulation is performed, thereby performing loop detection. When new closed-loop information appears, the previous information used for environmental mapping is corrected with the new closed-loop information. In this example, when performing loop detection, small closed-loop detection is first performed, and then large closed-loop detection is performed, which can improve the mapping reliability.

[0060] Then, in step S43, the back-end processing unit 532 performs back-end optimization on the first pose estimate, calibrates the drift, eliminates the influence of the cumulative error of the positioning image, and obtains the second pose estimate.

[0061] Next, in step S44, the positioning credibility calculation unit 533 performs positioning credibility calculation. According to the various parameters determined in the previous steps, such as the error of the positioning measurement data, the compactness of the feature point distribution, the tracking credibility, the quality credibility, etc., corresponding weights are respectively assigned, and the positioning credibility of this pose estimate is calculated. Therefore, the obtained second pose estimate contains not only the pose information but also the positioning credibility information of this pose estimate.

[0062] Return to Figure 3 , in step S5, the second pose estimate containing the positioning credibility obtained in step S4 is determined as the current pose of the carrying platform, thereby obtaining the real-time pose of the carrying platform 20.

[0063] During the movement of the carrying platform 20 along the set detection path (for example, the planned detection path P1), the detection imaging device 40 is activated and an image of the aircraft M is taken as the detection image. And in step S7, the image of the aircraft M taken by the detection imaging device 40 is stored in the memory 60 as the detection image. In step S8, the real-time pose of the carrying platform 20 obtained in step S5 is associated with the detection image obtained in step S7 to form detection image position metadata, which is stored in the memory 60.

[0064] Then, in step S9, it is determined whether the shooting of the set detection task has been completed. If the carrying platform 20 has not traversed the entire set detection path, the carrying platform 20 continues to move along the detection path, and the processing in the above steps S4 to S8 is repeatedly executed. If the carrying platform 20 has traversed the entire set detection path and the shooting of the set detection task has been completed, the carrying platform 20 returns and the detection shooting ends.

[0065] Next, in step S11, the image stitching unit 551 of the detection unit 55 reads the detection image position metadata from the memory 60, stitches the read detection images, and calibrates the read detection image position metadata based on the design data of the detection object (for example, the geometric features of the digital model). For example, according to the read detection image position metadata, the detection images of the aircraft M are stitched to construct the contour of the aircraft M, and calibration is performed based on the geometric features of the digital model of the aircraft M, thereby calibrating the read detection image position metadata.

[0066] In step S12, the image stitching unit 551 updates the read detection image position metadata using the calibrated detection image position metadata.

[0067] Finally, in step S13, the analysis unit 552 of the detection unit 55 performs detection analysis based on the calibrated detection image position metadata to determine the state of the detection object. Thus, the detection of the detection object is completed.

[0068] The device detection system 1 and the device detection method according to the present invention construct a virtual detection profile of the detection object based on the design parameters of the detection object, divide the detection area of the detection object into multiple sub-detection areas, and respectively plan the detection paths of each sub-detection area to perform detection based on the digital model of the detection object, which can achieve more reasonable path planning and make the detection process safer. Moreover, the device detection system 1 and the device detection method according to the present invention achieve navigation and positioning based on the information of the environment where the detection object is located and in combination with position measurement data (for example, the measurement data of the IMU), without relying on the information of the object to be detected (for example, the paint on the surface of the detection object), which can avoid the problem that the positioning is affected due to insufficient texture of the detection object itself. And by performing loop detection and correction during the positioning process, the cumulative error can be eliminated, the pose drift can be minimized, and simultaneous localization and mapping based on the environment can be achieved, enabling accurate real-time positioning, with the position error of the positioning being less than 5 cm and the angle error being less than 1 degree, achieving satisfactory positioning accuracy. In addition, the device detection system 1 and the device detection method according to the present invention also perform image stitching on the captured detection images, further calibrate the position metadata of the detection images, further improve the positioning accuracy, and based on the calibrated position metadata of the detection images for analysis, more accurate analysis results can be obtained.

[0069] The device detection system 1 according to the present invention has been introduced above in conjunction with the accompanying drawings, and the device detection method according to the present invention has been introduced by taking an aircraft as an example of the detection object. However, the above example should not be regarded as a limitation to the device detection system according to the present invention. The device detection system according to the present invention can also be applied to the detection of other devices.

[0070] Here, the exemplary embodiments of the present invention have been described in detail, but it should be understood that the present invention is not limited to the specific embodiments described and illustrated above in detail. Without departing from the gist and scope of the present invention, those skilled in the art can make various modifications and variations to the present invention. All such modifications and variations fall within the scope of the present invention. Moreover, all the components described herein can be replaced by other technically equivalent components.

Claims

1. An equipment detection method, including the following steps: A detection task setting step, in which information of an object to be detected is input and a detection task is set; A detection path setting step, in which a detection path is set according to the detection task; A navigation and positioning step, in which a carrying platform is started to make the carrying platform move along the detection path and determine the pose of the carrying platform in real time. Wherein, the carrying platform is provided with a positioning sensing device and a detection imaging device. The positioning sensing device includes a positioning imaging device and a positioning measurement device. The positioning imaging device is configured to capture positioning images during the movement of the carrying platform, and the positioning measurement device is configured to measure the movement of the carrying platform during the movement of the carrying platform; A detection imaging step, in which the detection imaging device captures detection images of the object to be detected when the carrying platform moves along the detection path; and A detection analysis step, in which the state of the object to be detected is analyzed based on the pose of the carrying platform and the detection images, characterized in that, in the navigation and positioning step, the positioning imaging device captures an image of the environment where the object to be detected is located as the positioning image, and the pose of the carrying platform is determined in real time based on the positioning image and the measurement value of the positioning measurement device, wherein, the navigation and positioning step includes: Processing the positioning image and extracting image features; Processing the measurement value to obtain the movement data of the carrying platform; Performing fusion processing on the image features and the movement data to obtain a first pose estimate of the carrying platform; and Optimizing the first pose estimate to obtain a second pose estimate, wherein, the navigation and positioning step further includes calculating the positioning credibility in real time and associating the positioning credibility with the second pose estimate and the detection images to form detection image position metadata, wherein, the detection analysis step includes: Obtaining the detection image position metadata, splicing the obtained detection images, and calibrating the obtained detection image position metadata based on the design data of the object to be detected; and Analyzing the state of the object to be detected based on the calibrated detection image position metadata, wherein, the detection path setting step includes: Constructing a virtual detection contour of the object to be detected according to the design parameters of the object to be detected, dividing the virtual detection contour into a plurality of sub-detection regions, and setting the planned detection paths of the plurality of sub-detection regions; and Setting the detection path as the planned detection path of at least one of the plurality of sub-detection regions according to the detection task.

2. The equipment detection method according to claim 1, wherein, in the navigation and positioning step, feature points of the positioning image are extracted and tracked, and the image quality credibility, feature point distribution compactness, tracking credibility, and reprojection error of the positioning image are calculated, Among them, the positioning confidence is calculated based on the image quality confidence, the compactness of feature point distribution, the tracking confidence, and the reprojection error.

3. The device detection method according to claim 1, wherein, the detection and analysis step is executed after the return of the carrying platform.

4. The device detection method according to any one of claims 1-3, wherein, the measured value is the measured value of one or more inertial measurement units.

5. The device detection method according to any one of claims 1-3, wherein, the object to be detected is an aircraft, and the positioning image is an image of the hangar where the aircraft is located.

6. A device detection system, comprising: a carrying platform adapted to move along a detection path of an object to be detected; a positioning sensing device provided on the carrying platform, the positioning sensing device including a positioning photographing device and a positioning measuring device, the positioning photographing device being configured to photograph a positioning image during the movement of the carrying platform, and the positioning measuring device being configured to measure the movement of the carrying platform during the movement of the carrying platform; a detection photographing device provided on the carrying platform, the detection photographing device being configured to photograph a detection image of the object to be detected when the carrying platform moves along the detection path; and a calculation and analysis unit configured to: process the sensing results of the positioning sensing device and the detection image to analyze the object to be detected, characterized in that the positioning photographing device is configured to: during the movement of the carrying platform along the detection path, the positioning photographing device photographs an image of the environment where the object to be detected is located as the positioning image, and the calculation and analysis unit determines the pose of the carrying platform in real time based on the positioning image and the measured value of the positioning measuring device, wherein the calculation and analysis unit includes a positioning unit, and the positioning unit includes: a front-end processing unit configured to process the positioning image to extract image features, process the measured value to obtain the movement data of the carrying platform, and perform a fusion process on the image features and the movement data to obtain a first pose estimate of the carrying platform; a back-end processing unit configured to optimize the first pose estimate to obtain a second pose estimate; and a positioning confidence calculation unit configured to calculate the positioning confidence of the second pose estimate, wherein the calculation and analysis unit further includes a detection unit configured to analyze the state of the object to be detected according to the pose of the carrying platform and the detection image, and the detection unit includes: An image stitching unit, configured to obtain the detected image and the corresponding second pose estimation, the detected image and the second pose estimation forming detected image position metadata, perform image stitching on the obtained detected image, and calibrate the detected image position metadata based on the design data of the object to be detected; and An analysis unit, configured to analyze the state of the object to be detected based on the calibrated detected image position metadata, wherein the device detection system further includes a ground control device that can be operated to input information about the object to be detected; wherein the calculation and analysis unit constructs a virtual detection profile of the object to be detected using the design parameters of the object to be detected according to the input information about the object to be detected, divides the virtual detection profile into a plurality of sub-detection regions, and sets a planned detection path for the plurality of sub-detection regions, wherein the detection path is the planned detection path for at least one of the plurality of sub-detection regions.

7. The device detection system according to claim 6, wherein, the detection unit analyzes the state of the object to be detected after the carrying platform returns.

8. The device detection system according to claim 6 or 7, wherein, the positioning and measuring device is one or more inertial measurement units.

9. The device detection system according to claim 6 or 7, wherein, the object to be detected is an aircraft, and the positioning image is an image of the hangar where the aircraft is located.

Images