An unmanned aerial vehicle automatic inspection navigation system based on machine vision and adaptive flight control

Abstract

The application provides an unmanned aerial vehicle automatic inspection navigation system based on machine vision and adaptive flight control, which comprises a machine vision perception module, an adaptive flight control module and a communication feedback module; the machine vision perception module is used for collecting environmental information in real time and constructing a three-dimensional environmental model of a flight area; the adaptive flight control module is used for dynamically planning a flight path of a subsequent unmanned aerial vehicle in combination with the three-dimensional environmental model of the flight area of the unmanned aerial vehicle; and the communication feedback module is used for realizing bidirectional communication between the unmanned aerial vehicle and a ground station; through construction of a high-precision three-dimensional environmental model, in combination with adaptive path optimization and dynamic obstacle avoidance, the unmanned aerial vehicle can realize accurate inspection, efficient navigation and stable flight in a complex environment, and the intelligence, safety and reliability of the inspection task are improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) inspection system technology, and in particular to an automatic inspection and navigation system for UAVs based on machine vision and adaptive flight control. Background Technology

[0002] With the increasing demand for applications such as power line inspection, industrial testing, and infrastructure monitoring, drone technology has been widely used in the field of automated inspection. Compared with traditional manual inspection methods, drones have the advantages of high efficiency, low cost, and wide coverage, enabling them to perform inspection tasks in large-scale, high-precision, and complex environments. However, current drone inspection systems still face many challenges in practical applications. First, the inspection accuracy and navigation stability in complex environments are key factors affecting the efficiency of drone inspections. In scenarios such as power line inspection and oil and gas pipeline inspection, drones need to accurately locate and avoid obstacles in complex structures and dynamic environments, while traditional GPS navigation methods are prone to failure under severe obstruction or signal interference. Second, existing inspection systems still have limitations in dynamic environmental perception and path optimization, making it difficult to make real-time path corrections based on environmental changes and obstacle adjustments, affecting the safety and quality of inspection completion.

[0003] A review of publicly available technical solutions reveals that CN116126027A proposes a UAV inspection system for power distribution rooms based on machine vision and multi-sensor fusion. This system includes a micro-UAV equipped with machine vision; the micro-UAV takes off and lands on a helipad equipped with a wireless charging module and a power management module; the inspection system also includes edge devices located on the helipad, and positioning identifiers affixed to the areas to be inspected, instruments, and indicator lights, which can be visually recognized by the micro-UAV; these positioning identifiers guide the micro-UAV to the inspection target; the edge devices include edge computing devices and edge control devices, which will... The IMU attitude trajectory data transmitted back by the UAV during inspection is fused with the coordinate data of the UAV's visual positioning to form fused positioning data. When the information used by the UAV for visual positioning is lost, the edge device actively intervenes and obtains the UAV coordinates through the fused positioning data. This solution can achieve three-dimensional, blind-spot-free coverage inspection of the power distribution room. This solution mainly relies on the positioning identification code of visual recognition to guide the UAV inspection. When the identification code is blocked or affected by changes in ambient light, the positioning accuracy may decrease, and it will be unable to perceive the inspection objects and obstacles in the dynamic environment in real time, thus affecting the reliability of the UAV inspection task. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of current systems by proposing an automatic inspection and navigation system for unmanned aerial vehicles (UAVs) based on machine vision and adaptive flight control.

[0005] The present invention adopts the following technical solution:

[0006] An automatic inspection and navigation system for unmanned aerial vehicles (UAVs) based on machine vision and adaptive flight control is disclosed. The system includes a machine vision perception module, an adaptive flight control module, and a communication feedback module. The machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area. The adaptive flight control module is used to dynamically plan the flight path of the UAV in the subsequent flight area based on the three-dimensional environmental model of the UAV flight area. The communication feedback module is used to realize two-way communication between the UAV and the ground station.

[0007] The machine vision perception module includes an environment perception unit, a target recognition unit, and a visual positioning unit; the environment perception unit is used to collect environmental information; the target recognition unit is used to identify inspection objects and environmental obstacles by combining environmental information; and the visual positioning unit is used to construct a three-dimensional environmental model of the flight area by combining the recognition results of the target recognition unit and GPS data.

[0008] Furthermore, the environmental perception unit includes a stereo camera and a lidar; the stereo camera is used to acquire continuous multi-frame image information during the flight of the UAV; the lidar is used to scan the surrounding environment of the UAV during flight and acquire spatial relative position and distance information between the UAV and objects in the surrounding environment.

[0009] Furthermore, the target recognition unit identifies inspection objects and environmental obstacles in the following ways:

[0010] S11: Acquire image information from multiple consecutive frames in the environmental information;

[0011] S12: Use the YOLO model to perform target detection on multiple consecutive frames of image information, and identify the inspection objects and environmental obstacles in the multiple consecutive frames of image information;

[0012] S13: Use the Mask-R-CNN model to segment the identified inspection objects and environmental obstacles in the image and extract the edge morphology information of the inspection objects and environmental obstacles.

[0013] Furthermore, the visual positioning unit constructs a three-dimensional environmental model of the flight area in the following manner:

[0014] S21: Acquire GPS positioning data of the drone during its flight to determine the drone's real-time location information;

[0015] S22: By fusing the information acquired by the lidar with the identification content of the target recognition unit through spatial data, the relative position and distance information of the UAV to the inspection object and environmental obstacles during flight is obtained;

[0016] S23: Construct a three-dimensional environment model of the UAV flight area based on the information obtained in step S22. The three-dimensional environment model includes the coordinate information of the UAV, the inspection object, and environmental obstacles.

[0017] S24: During the drone's flight, the update frequency of the 3D environment model is adjusted in accordance with dynamic environmental changes. Further, in step S24, the update frequency of the 3D environment model is adjusted specifically in the following ways:

[0018] S241: Set a continuous incremental acquisition period, and acquire the incremental environmental change within each incremental acquisition period; the incremental environmental change is specifically represented as:

[0019] ΔM object =|M new -M old |;

[0020] Where, ΔM object M represents the incremental environmental change within a certain incremental data collection period. new M represents the area of ​​the inspected object and environmental obstacles in the image acquired by the UAV at the end time of this incremental acquisition cycle. old The area of ​​the inspected object and environmental obstacles in the image acquired by the UAV at the start time of this incremental acquisition cycle; M new and M old All are obtained through further calculations based on the edge morphology information of the inspection objects and environmental obstacles identified by the target recognition unit;

[0021] S242: Calculate the update frequency of the 3D environment model after each incremental acquisition cycle by combining the incremental environmental changes within each incremental acquisition cycle.

[0022] f=α·[1-exp(-β·ΔM object )];

[0023] Where f is the update frequency of the 3D environment model after a certain incremental acquisition cycle; α is the preset maximum update frequency; β is the incremental sensitivity adjustment coefficient, which is used to adjust the influence of the incremental change in the environment on the update frequency, and is set through pre-experimentation.

[0024] Furthermore, the adaptive flight control module performs dynamic planning of the UAV's subsequent flight path in the following manner:

[0025] S31: Define the initial flight path of the UAV according to mission requirements;

[0026] S32: After each update of the 3D environment model, obtain the edge coordinate information of the inspected objects and environmental obstacles in the 3D environment model;

[0027] S33: After each update of the 3D environment model, the inspection path points related to the inspection objects in the current 3D environment model are corrected.

[0028] The beneficial effects achieved by this invention are as follows:

[0029] This invention constructs a dynamic 3D environment model by collecting environmental information in real time and combining it with target recognition, lidar ranging, and GPS positioning. This provides precise environmental perception and path planning support for UAV inspection missions. During flight, the inspection path is adaptively adjusted based on the boundary information of the inspection object and the relative position of environmental obstacles, ensuring that the UAV can accurately cover the target and dynamically avoid obstacles. In addition, by incrementally controlling the update frequency of the 3D environment model through environmental changes, computational resources are optimized, improving the real-time performance and stability of the inspection mission. Attached Figure Description

[0030] The invention will be further understood from the following description taken in conjunction with the accompanying drawings. The components in the drawings are not necessarily drawn to scale, but rather the emphasis is on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0031] Figure 1 This is a schematic diagram of the overall modules of the present invention.

[0032] Figure 2 This is a schematic diagram of the working process of the target recognition unit of the present invention.

[0033] Figure 3 This is a schematic diagram of the working process of the visual positioning unit of the present invention.

[0034] Figure 4 This is a schematic diagram of the workflow of the adaptive flight control module of the present invention. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to its embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Other systems, methods, and / or features of this embodiment will become apparent to those skilled in the art after reviewing the following detailed description. It is intended that all such additional systems, methods, features, and advantages are included within this specification, are included within the scope of the present invention, and are protected by the appended claims. Further features of the disclosed embodiments are described in the following detailed description, and these features will be apparent from the following detailed description.

[0036] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or component referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0037] Example 1:

[0038] like Figure 1 As shown, this embodiment provides an automatic inspection and navigation system for unmanned aerial vehicles (UAVs) based on machine vision and adaptive flight control. The system includes a machine vision perception module, an adaptive flight control module, and a communication feedback module. The machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area. The adaptive flight control module is used to dynamically plan the subsequent flight path of the UAV based on the three-dimensional environmental model of the UAV's flight area. The communication feedback module is used to realize two-way communication between the UAV and the ground station.

[0039] The machine vision perception module includes an environment perception unit, a target recognition unit, and a visual positioning unit; the environment perception unit is used to collect environmental information; the target recognition unit is used to identify inspection objects and environmental obstacles by combining the environmental information; the visual positioning unit is used to construct a three-dimensional environmental model of the flight area by combining the recognition results of the target recognition unit and GPS data.

[0040] Furthermore, the environmental perception unit includes a stereo camera and a lidar; the stereo camera is used to acquire continuous multi-frame image information during the flight of the UAV; the lidar is used to scan the surrounding environment of the UAV during its flight and acquire spatial relative position and distance information between the UAV and objects in the surrounding environment.

[0041] Furthermore, such as Figure 2 As shown, the target recognition unit identifies inspection objects and environmental obstacles in the following ways:

[0042] S11: Acquire image information from multiple consecutive frames in the environmental information;

[0043] S12: Use the YOLO model to perform target detection on multiple consecutive frames of image information, and identify the inspection objects and environmental obstacles in the multiple consecutive frames of image information;

[0044] S13: Use the Mask-R-CNN model to segment the identified inspection objects and environmental obstacles in the image and extract the edge morphology information of the inspection objects and environmental obstacles;

[0045] Furthermore, such as Figure 3 As shown, the visual positioning unit constructs a three-dimensional environmental model of the flight area in the following manner:

[0046] S21: Acquire GPS positioning data of the drone during its flight to determine the drone's real-time location information;

[0047] S22: By fusing the information acquired by the lidar with the identification content of the target recognition unit through spatial data, the relative position and distance information of the UAV to the inspection object and environmental obstacles during flight is obtained;

[0048] S23: Construct a three-dimensional environment model of the UAV flight area based on the information obtained in step S22. The three-dimensional environment model includes the coordinate information of the UAV, the inspection object, and environmental obstacles.

[0049] S24: During the flight of the UAV, adjust the update frequency of the 3D environment model in combination with dynamic environmental changes;

[0050] Furthermore, in step S24, the update frequency of the 3D environment model is adjusted in the following ways:

[0051] S241: Set a continuous incremental acquisition period, and acquire the incremental environmental change within each incremental acquisition period; the incremental environmental change is specifically represented as:

[0052] ΔM object =|M new -M old |;

[0053] Where, ΔM object M represents the incremental environmental change within a certain incremental data collection period. new M represents the area of ​​the inspected object and environmental obstacles in the image acquired by the UAV at the end time of this incremental acquisition cycle. old The area of ​​the inspected object and environmental obstacles in the image acquired by the UAV at the start time of this incremental acquisition cycle; M new and M old All are obtained through further calculations based on the edge morphology information of the inspection objects and environmental obstacles identified by the target recognition unit;

[0054] S242: Calculate the update frequency of the 3D environment model after each incremental acquisition cycle by combining the incremental environmental changes within each incremental acquisition cycle.

[0055] f=α·[1-exp(-β·ΔM object )];

[0056] Where f is the update frequency of the 3D environment model after a certain incremental acquisition cycle; α is the preset maximum update frequency; β is the incremental sensitivity adjustment coefficient, which is used to adjust the influence of the change in the incremental environment on the update frequency, and is set through pre-experimentation.

[0057] Furthermore, the code implementing the function of adjusting the update frequency of the 3D environment model is as follows:

[0058]

[0059]

[0060]

[0061] This solution collects environmental information during UAV flight and combines this information to build a 3D environmental model that includes the inspection objects and environmental obstacles. This provides accurate environmental perception and navigation basis for the dynamic adjustment of the UAV's flight path. Furthermore, it adjusts the update frequency of the 3D environmental model based on dynamic environmental changes, thereby optimizing computing resources while ensuring the accuracy of environmental information and improving the real-time performance and stability of UAV inspection.

[0062] Example 2:

[0063] This embodiment should be understood to include at least all the features of any of the foregoing embodiments, and to further improve upon them;

[0064] This embodiment provides an automatic inspection and navigation system for unmanned aerial vehicles (UAVs) based on machine vision and adaptive flight control. The system includes a machine vision perception module, an adaptive flight control module, and a communication feedback module. The machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area. The adaptive flight control module is used to dynamically plan the subsequent flight path of the UAV based on the three-dimensional environmental model of the UAV's flight area. The communication feedback module is used to realize two-way communication between the UAV and the ground station.

[0065] Furthermore, such as Figure 4 As shown, the adaptive flight control module performs dynamic planning of the UAV's subsequent flight path in the following manner:

[0066] S31: Define the initial flight path of the UAV according to the mission requirements; the initial flight path of the UAV is represented as:

[0067]

[0068] Where P0 is the initial flight path of the UAV, argmin() is the minimum objective function; n is the number of inspection objects obtained from the task requirements, P i,j G represents the coordinates of the j-th inspection path point set by the UAV for the u-th inspection object; i d(P) represents the coordinates of the object being inspected. i,j G i ) is P i,j With G i Distance between; S(P) i,j [] represents the smoothness adjustment coefficient for the coordinates of the j-th inspection path point corresponding to the i-th inspection object, and μ is the path smoothness weight, which is set according to pre-experimentation; m i Let be the total number of all inspection path points corresponding to the i-th inspection object, satisfying:

[0069] m i =m base +ρ·I target (G i );

[0070] Where, m base I is the preset minimum total number of inspection points. target (G i ) represents the target importance coefficient of the i-th inspection object, which is set according to task requirements; ρ represents the weight of the number of path points, which is set according to pre-experimentation.

[0071] S32: After each update of the 3D environment model, obtain the edge coordinate information of the inspected objects and environmental obstacles in the 3D environment model;

[0072] S33: After each update of the 3D environment model, correct the inspection path points related to the inspection objects in the current 3D environment model:

[0073]

[0074] Where t represents the current 3D environment model update time, P new (i, j, t) are the coordinates of the corrected inspection path points, P old (i, j, t) are the coordinates of the inspection path points before the correction; This is the direction vector pointing to the adjustment point of the inspected object; γ1 is the direction vector pointing away from the obstacle point; γ2 is the step size for adjusting the observation of the inspection object, set according to pre-experimentation; where:

[0075]

[0076] Among them, B1(G i ,t) represents the coordinates of the adjustment point for the inspection object, which are obtained in the following way:

[0077] S3311: Obtain the set of edge coordinates of the inspected object;

[0078] S3312: Obtain the set of edge coordinates of the inspected object and point P old (i, j, t) represents the nearest boundary point of the inspected object;

[0079] S3313: Based on the boundary point of the inspection object, after applying a preset observation safety distance along the normal direction of the plane of the inspection object where the point is located, the obtained coordinate position is the coordinate of the adjustment point of the inspection object.

[0080]

[0081] Among them, B2(G i ,t) represents the coordinates of the point far from the obstacle, which are obtained in the following way:

[0082] S3321: Obtain the set of coordinates of environmental obstacles edges;

[0083] S3322: Obtain the set of environmental obstacle edge coordinates and point P old (i, j, t) represents the nearest boundary point of the environmental obstacle;

[0084] S3323: Based on the boundary point of the environmental obstacle, after applying a preset obstacle avoidance safety distance along the normal direction of the plane of the obstacle where the point is located, the obtained coordinate position is the coordinate of the point far away from the obstacle.

[0085] γ2=d(P old (i, j, t), B2(G) i ,t))·[1+ln(1+γ3·d(P old (i, j, t), B2(G) i ,t)))];

[0086] Wherein d(P old (i, j, t), B2(G) i ,t)) is P old (i, j, t) and B2(G) i The distance between t) and γ3 is the obstacle avoidance correction amplitude coefficient, which is set through pre-experimentation;

[0087] This solution dynamically adjusts the drone's inspection path points, combining boundary information of the inspection object and environmental obstacles. After updating the 3D environment model, the flight path is dynamically optimized and corrected to ensure inspection accuracy. This allows the drone's inspection path to more accurately cover the inspection object while maintaining a reasonable observation safety distance, thus improving inspection quality. At the same time, by dynamically adjusting the obstacle avoidance step size, the solution ensures that the drone enhances obstacle avoidance when approaching obstacles and reduces unnecessary deviations when moving away from obstacles, thereby improving flight safety and stability.

[0088] The content disclosed above is only a preferred and feasible embodiment of the present invention, and is not intended to limit the scope of protection of the present invention. Therefore, all equivalent technical changes made based on the content of the present invention specification and drawings are included within the scope of protection of the present invention. Furthermore, the elements therein can be updated as technology develops.

Claims

1. An automatic inspection and navigation system for unmanned aerial vehicles (UAVs) based on machine vision and adaptive flight control, characterized in that, The system includes a machine vision perception module, an adaptive flight control module, and a communication feedback module. The machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area. The adaptive flight control module is used to dynamically plan the flight path of the UAV based on the three-dimensional environmental model of the UAV's flight area. The communication feedback module is used to realize two-way communication between the UAV and the ground station. The machine vision perception module includes an environment perception unit, a target recognition unit, and a visual positioning unit; the environment perception unit is used to collect environmental information; the target recognition unit is used to identify inspection objects and environmental obstacles by combining the environmental information; the visual positioning unit is used to construct a three-dimensional environmental model of the flight area by combining the recognition results of the target recognition unit and GPS data. The visual positioning unit constructs a three-dimensional environmental model of the flight area in the following manner: S21: Acquire GPS positioning data of the drone during its flight to determine the drone's real-time location information; S22: By fusing the information acquired by the lidar with the identification content of the target recognition unit through spatial data, the relative position and distance information of the UAV to the inspection object and environmental obstacles during flight is obtained; S23: Construct a three-dimensional environment model of the UAV flight area based on the information obtained in step S22. The three-dimensional environment model includes the coordinate information of the UAV, the inspection object, and environmental obstacles. S24: During the flight of the UAV, adjust the update frequency of the 3D environment model in combination with dynamic environmental changes; In step S24, the update frequency of the 3D environment model is adjusted in the following ways: S241: Set a continuous incremental acquisition period, and acquire the incremental environmental change within each incremental acquisition period; the incremental environmental change is specifically represented as: ； in, This represents the incremental change in the environment within a specific data collection period. The area of ​​the inspected objects and environmental obstacles in the UAV-acquired images at the end of the incremental acquisition cycle; The area of ​​the inspected object and environmental obstacles in the image acquired by the UAV at the start time of this incremental acquisition cycle; and All are obtained through further calculations based on the edge morphology information of the inspection objects and environmental obstacles identified by the target recognition unit; S242: Calculate the update frequency of the 3D environment model after each incremental acquisition cycle by combining the incremental environmental changes within each incremental acquisition cycle. ； in, The update frequency of the 3D environment model after a certain incremental acquisition cycle; This is the preset maximum update frequency; This is the incremental sensitivity adjustment coefficient, used to adjust the impact of environmental change increments on the update frequency, and is set through pre-experimentation. The adaptive flight control module performs dynamic planning of the UAV's subsequent flight path in the following manner: S31: Define the initial flight path of the UAV according to the mission requirements; the initial flight path of the UAV is represented as: ； in, This is the initial flight path of the drone. The minimum objective function is the optimization function; The number of inspection objects obtained from the task requirements. For drones as the first The first inspection target is set Coordinates of each inspection path point; The coordinates of the object being inspected; for and Distance between; For the first The inspection target corresponds to the first Smoothness adjustment coefficient for the coordinates of each inspection path point. The path smoothness weights are set based on prior experiments. For the first The total number of all inspection path points corresponding to a given inspection object satisfies: ； in, This is the preset minimum total number of inspection points. For the first The target importance coefficient for each inspection object is set according to task requirements; The weights for the number of path points are set based on pre-experimentation. S32: After each update of the 3D environment model, obtain the edge coordinate information of the inspected objects and environmental obstacles in the 3D environment model; S33: After each update of the 3D environment model, the inspection path points related to the inspection objects in the current 3D environment model are corrected; ； in, This represents the current update time of the 3D environment model. The coordinates of the inspection path points are as follows. The coordinates of the inspection path points before correction; This is the direction vector pointing to the adjustment point of the inspected object; It is a direction vector pointing away from the obstacle point; Adjust the step size for the inspection target based on the pre-test settings; To adjust the step size for obstacle avoidance; where: ； in, Adjust the coordinates of the points for the inspection targets.

2. The UAV automatic inspection and navigation system based on machine vision and adaptive flight control according to claim 1, characterized in that, The environmental perception unit includes a stereo camera and a lidar; the stereo camera is used to acquire continuous multi-frame image information during the flight of the UAV; the lidar is used to scan the surrounding environment of the UAV during its flight and acquire spatial relative position and distance information between the UAV and objects in the surrounding environment.

3. The UAV automatic inspection and navigation system based on machine vision and adaptive flight control according to claim 1, characterized in that, The target recognition unit identifies inspection objects and environmental obstacles in the following ways: S11: Acquire image information from multiple consecutive frames in the environmental information; S12: Use the YOLO model to perform target detection on multiple consecutive frames of image information, and identify the inspection objects and environmental obstacles in the multiple consecutive frames of image information; S13: Use the Mask-R-CNN model to segment the identified inspection objects and environmental obstacles in the image and extract the edge morphology information of the inspection objects and environmental obstacles.

Images