Unmanned aerial vehicle automatic inspection navigation system based on machine vision and adaptive flight control

Abstract

The invention provides an unmanned aerial vehicle automatic inspection navigation system based on machine vision and adaptive flight control. The system comprises a machine vision sensing module, an adaptive flight control module and a communication feedback module, the machine vision perception module is used for collecting environment information in real time and constructing a three-dimensional environment model of a flight area; the adaptive flight control module is used for dynamically planning a subsequent flight path of the unmanned aerial vehicle in combination with the three-dimensional environment model of the flight area of the unmanned aerial vehicle; the communication feedback module is used for realizing two-way communication between the unmanned aerial vehicle and the ground station; by constructing the high-precision three-dimensional environment model and combining adaptive path optimization and dynamic obstacle avoidance, it is ensured that the unmanned aerial vehicle achieves precise inspection, efficient navigation and stable flight in a complex environment, and the intelligence, safety and reliability of an inspection task are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned aerial vehicle inspection systems, and in particular to an unmanned aerial vehicle automatic inspection navigation system based on machine vision and adaptive flight control. Background Art

[0002] With the growing demand for applications such as power inspection, industrial inspection and infrastructure monitoring, drone technology has been widely used in the field of automatic inspection. Compared with traditional manual inspection methods, drones have the advantages of high efficiency, low cost and wide coverage, and can perform inspection tasks in large-scale, high-precision and complex environments. However, the current drone inspection system still faces many challenges in practical applications. First, inspection accuracy and navigation stability in complex environments are key factors affecting the efficiency of drone inspections. In scenarios such as power inspections and oil and gas pipeline inspections, drones need to accurately locate and avoid obstacles in complex structures and dynamic environments, while traditional GPS navigation methods are prone to failure in the case of severe occlusion or signal interference. Secondly, the existing inspection system still has limitations in dynamic environment perception and path optimization. It is difficult to make real-time path corrections based on environmental changes, obstacle adjustments and other factors, which affects the safety of inspections and the quality of task completion.

[0003] After consulting the relevant published technical solutions, the technology with publication number CN116126027A proposes a distribution station drone inspection system based on machine vision and multi-sensor fusion, including a micro drone with machine vision; the micro drone takes off and lands at a helipad equipped with a wireless charging module and a power management module; the inspection system also includes edge equipment arranged on the helipad, and positioning identification codes that can be visually identified by the micro drone and are attached to the areas to be inspected, instruments and indicator lights; the positioning identification code is used to guide the micro drone to the inspection target; the edge equipment includes an edge computing device and an edge control device, which will The IMU attitude trajectory data sent back during drone inspection and the coordinate data of the drone's visual positioning are integrated to form fused positioning data; when the information used for visual positioning of the drone is lost, the edge device actively intervenes to obtain the drone's coordinates through fused positioning data; this solution can achieve three-dimensional, no-dead-angle coverage inspection of distribution stations; this solution mainly relies on visually recognized positioning identification codes to guide drone inspections. When the identification codes are blocked or affected by changes in ambient light, the positioning accuracy may be reduced, and it is impossible to perceive the inspection objects and obstacles in a dynamic environment in real time, thereby affecting the reliability of the drone inspection mission. Summary of the invention

[0004] The purpose of the present invention is to propose an automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control in response to the current deficiencies.

[0005] The present invention adopts the following technical solution:

[0006] An automatic inspection and navigation system for unmanned aerial vehicles (UAVs) based on machine vision and adaptive flight control, the system comprising a machine vision perception module, an adaptive flight control module and a communication feedback module; the machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area; the adaptive flight control module is used to dynamically plan the flight path of subsequent UAVs in combination with the three-dimensional environmental model of the UAV flight area; the communication feedback module is used to realize two-way communication between the UAV and a ground station.

[0007] The machine vision perception module includes an environment perception unit, a target recognition unit and a visual positioning unit; the environment perception unit is used to collect environmental information; the target recognition unit is used to identify inspection objects and environmental obstacles in combination with environmental information; the visual positioning unit is used to construct a three-dimensional environmental model of the flight area in combination with the recognition results of the target recognition unit and GPS data.

[0008] Furthermore, the environmental perception unit includes a stereo camera and a laser radar; the stereo camera is used to obtain continuous multi-frame image information during the flight of the drone; the laser radar is used to scan the surrounding environment during the flight of the drone and obtain the spatial relative position and distance information between the drone and objects in the surrounding environment.

[0009] Furthermore, the target recognition unit recognizes the inspection object and environmental obstacles in the following manner:

[0010] S11: Acquire continuous multiple frames of image information in the environmental information;

[0011] S12: Use the YOLO model to perform target detection on continuous multi-frame image information and identify inspection objects and environmental obstacles in continuous multi-frame image information;

[0012] S13: Use the Mask-R-CNN model to segment the identified inspection objects and environmental obstacles in the image and extract the edge morphological information of the inspection objects and environmental obstacles.

[0013] Furthermore, the visual positioning unit constructs a three-dimensional environment model of the flight area in the following manner:

[0014] S21: Acquire the GPS positioning data of the drone during the flight of the drone to determine the real-time location information of the drone;

[0015] S22: The information obtained by the laser radar and the identification content of the target identification unit are fused through spatial data to obtain the relative position and distance information between the UAV and the inspection object and environmental obstacles during the flight;

[0016] S23: constructing a three-dimensional environment model of the UAV flight area in combination with the information obtained in step S22, wherein the three-dimensional environment model includes coordinate information of the UAV, the inspection object, and environmental obstacles;

[0017] S24: During the flight of the drone, the update frequency of the three-dimensional environment model is adjusted in combination with dynamic environmental changes. Further, in the step S24, the update frequency of the three-dimensional environment model is adjusted in the following manner:

[0018] S241: Setting a continuous incremental collection cycle, and obtaining an environmental change increment in each incremental collection cycle; the environmental change increment is specifically expressed as:

[0019] ΔM object =|M new -M old |;

[0020] Among them, ΔM object Indicates the incremental change of the environment within a certain incremental collection cycle, M new M is the area of ​​the inspection object and environmental obstacles in the image collected by the UAV at the end time of the incremental collection cycle; old is the area of ​​the inspection object and environmental obstacles in the image collected by the UAV at the start time of the incremental collection cycle; new and M old The edge morphological information of the inspection object and environmental obstacles identified by the target recognition unit is further calculated and obtained;

[0021] S242: Calculate the update frequency of the three-dimensional environment model after each incremental collection cycle based on the incremental environment change in each incremental collection cycle:

[0022] f = α·[1-exp(-β·ΔM object )];

[0023] Among them, f is the update frequency of the three-dimensional environment model after a certain incremental acquisition cycle; α is the preset maximum update frequency; β is the incremental sensitivity adjustment coefficient, which is used to adjust the impact of the change in the incremental environmental change on the update frequency and is set through pre-experimental settings.

[0024] Furthermore, the adaptive flight control module completes the dynamic planning of the subsequent flight path of the UAV by:

[0025] S31: Define the initial flight path of the UAV according to the mission requirements;

[0026] S32: after each update of the three-dimensional environment model, obtaining edge coordinate information of the inspection object and environmental obstacles in the three-dimensional environment model;

[0027] S33: After each update of the three-dimensional environment model, the inspection path points related to the inspection object in the current three-dimensional environment model are corrected.

[0028] The beneficial effects achieved by the present invention are:

[0029] The present invention collects environmental information in real time, combines target recognition, lidar ranging and GPS positioning, and constructs a dynamic three-dimensional environmental model to provide accurate environmental perception and path planning support for UAV inspection tasks. During the flight, the inspection path is adaptively adjusted based on the boundary information of the inspection object and the relative position of environmental obstacles to ensure that the UAV can accurately cover the target and dynamically avoid obstacles. In addition, the update frequency of the three-dimensional environmental model is controlled by incremental environmental changes, which optimizes computing resources and improves the real-time and stability of the inspection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The present invention can be further understood from the following description in conjunction with the accompanying drawings. The components in the figures are not necessarily drawn to scale, but the emphasis is placed on illustrating the principles of the embodiments. In different views, the same reference numerals designate corresponding parts.

[0031] Figure 1 It is a schematic diagram of the overall module of the present invention.

[0032] Figure 2 Schematic diagram of the work flow of the target identification unit of the present invention.

[0033] Figure 3 Schematic diagram of the working process of the visual positioning unit of the present invention.

[0034] Figure 4 The figure is a schematic diagram of the working process of the adaptive flight control module of the present invention. DETAILED DESCRIPTION

[0035] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with its embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention; for those skilled in the art, other systems, methods and / or features of the present embodiment will become apparent after reviewing the following detailed description; it is intended that all such additional systems, methods, features and advantages are included in this specification; included within the scope of the present invention and protected by the appended claims; additional features of the disclosed embodiments are described in the following detailed description, and these features will be apparent from the following detailed description.

[0036] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right" and the like indicate directions or positional relationships based on the directions or positional relationships shown in the drawings, it is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or component referred to must have a specific direction, be constructed and operated in a specific direction. Therefore, the terms describing the positional relationship in the drawings are only used for illustrative purposes and cannot be understood as limitations on this patent. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0037] Embodiment 1:

[0038] like Figure 1 As shown, this embodiment provides an automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control, the system comprising a machine vision perception module, an adaptive flight control module and a communication feedback module; the machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area; the adaptive flight control module is used to dynamically plan the flight path of the subsequent unmanned aerial vehicles in combination with the three-dimensional environmental model of the unmanned aerial vehicle flight area; the communication feedback module is used to realize two-way communication between the unmanned aerial vehicle and the ground station;

[0039] The machine vision perception module includes an environment perception unit, a target recognition unit and a visual positioning unit; the environment perception unit is used to collect environmental information; the target recognition unit is used to identify inspection objects and environmental obstacles in combination with environmental information; the visual positioning unit is used to build a three-dimensional environmental model of the flight area in combination with the recognition result of the target recognition unit and GPS data;

[0040] Furthermore, the environment perception unit includes a stereo camera and a laser radar; the stereo camera is used to obtain continuous multi-frame image information during the flight of the drone; the laser radar is used to scan the surrounding environment during the flight of the drone and obtain the spatial relative position and distance information between the drone and objects in the surrounding environment;

[0041] Further, such as Figure 2 As shown, the target recognition unit recognizes the inspection object and environmental obstacles in the following manner:

[0042] S11: Acquire continuous multiple frames of image information in the environmental information;

[0043] S12: Use the YOLO model to perform target detection on continuous multi-frame image information and identify inspection objects and environmental obstacles in continuous multi-frame image information;

[0044] S13: Use the Mask-R-CNN model to segment the identified inspection objects and environmental obstacles in the image and extract the edge morphological information of the inspection objects and environmental obstacles;

[0045] Further, such as Figure 3 As shown, the visual positioning unit constructs a three-dimensional environment model of the flight area in the following manner:

[0046] S21: Acquire the GPS positioning data of the drone during the flight of the drone to determine the real-time location information of the drone;

[0047] S22: The information obtained by the laser radar and the identification content of the target identification unit are fused through spatial data to obtain the relative position and distance information between the UAV and the inspection object and environmental obstacles during the flight;

[0048] S23: constructing a three-dimensional environment model of the UAV flight area in combination with the information obtained in step S22, wherein the three-dimensional environment model includes coordinate information of the UAV, the inspection object, and environmental obstacles;

[0049] S24: During the flight of the UAV, the update frequency of the three-dimensional environment model is adjusted in combination with dynamic environmental changes;

[0050] Furthermore, in step S24, the update frequency of the three-dimensional environment model is adjusted in the following manner:

[0051] S241: Setting a continuous incremental collection cycle, and obtaining an environmental change increment in each incremental collection cycle; the environmental change increment is specifically expressed as:

[0052] ΔM object =|M new -M old |;

[0053] Among them, ΔM object Indicates the incremental change of the environment within a certain incremental collection cycle, M new M is the area of ​​the inspection object and environmental obstacles in the image collected by the UAV at the end time of the incremental collection cycle; old is the area of ​​the inspection object and environmental obstacles in the image collected by the UAV at the start time of the incremental collection cycle; new and M old The edge morphological information of the inspection object and environmental obstacles identified by the target recognition unit is further calculated and obtained;

[0054] S242: Calculate the update frequency of the three-dimensional environment model after each incremental collection cycle based on the incremental environment change in each incremental collection cycle:

[0055] f = α·[1-exp(-β·ΔM object )];

[0056] Among them, f is the update frequency of the three-dimensional environment model after a certain incremental acquisition cycle; α is the preset maximum update frequency; β is the incremental sensitivity adjustment coefficient, which is used to adjust the impact of the change of the environmental change increment on the update frequency and is set by pre-experimentation;

[0057] Furthermore, the partial function implementation code for adjusting the update frequency of the three-dimensional environment model is as follows:

[0058]

[0059]

[0060]

[0061] This solution collects environmental information during the flight of the UAV and uses the environmental information to establish a three-dimensional environmental model that includes inspection objects and environmental obstacles, thereby providing accurate environmental perception and navigation basis for the dynamic adjustment of the subsequent UAV flight path; and, in combination with the dynamic environmental change increments, the update frequency of the three-dimensional environmental model is adjusted, thereby optimizing computing resources while ensuring the accuracy of environmental information, and improving the real-time and stability of UAV inspections.

[0062] Embodiment 2:

[0063] This embodiment should be understood to include at least all the features of any of the above embodiments, and further improve upon them;

[0064] This embodiment provides an automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control, the system comprising a machine vision perception module, an adaptive flight control module and a communication feedback module; the machine vision perception module is used to collect environmental information in real time and construct a three-dimensional environmental model of the flight area; the adaptive flight control module is used to dynamically plan the flight path of the subsequent unmanned aerial vehicles in combination with the three-dimensional environmental model of the unmanned aerial vehicle flight area; the communication feedback module is used to realize two-way communication between the unmanned aerial vehicle and the ground station;

[0065] Further, such as Figure 4 As shown, the adaptive flight control module completes the dynamic planning of the subsequent flight path of the UAV in the following manner:

[0066] S31: Define the initial flight path of the UAV according to the mission requirements; the initial flight path of the UAV is expressed as:

[0067]

[0068] Among them, P 0 is the initial flight path of the UAV, argmin() is the minimum target optimization function; n is the number of inspection objects obtained from the task requirements, P i,j G is the coordinate of the jth inspection path point set by the UAV for the uth inspection object; i is the coordinate of the inspection object; d(P i,j ,G i ) is P i,j With G i Distance between; S(P i,j ] is the smoothness adjustment coefficient of the coordinates of the jth inspection path point corresponding to the ith inspection object, μ is the path smoothness weight, which is set according to the pre-experimental setting; m i is the total number of all inspection path points corresponding to the i-th inspection object, satisfying:

[0069] m i =m base +ρ·I target (G i );

[0070] Among them, m base is the preset minimum total number of inspection points, I target (G i ) is the target importance coefficient of the i-th inspection object, which is set according to the task requirements; ρ is the weight of the number of path points, which is set according to the pre-experimental setting;

[0071] S32: after each update of the three-dimensional environment model, obtaining edge coordinate information of the inspection object and environmental obstacles in the three-dimensional environment model;

[0072] S33: After each update of the three-dimensional environment model, the inspection path points related to the inspection object in the current three-dimensional environment model are corrected:

[0073]

[0074] Among them, t represents the current 3D environment model update time, P new (i, j, t) is the coordinate of the corrected inspection path point, P old (i, j, t) is the coordinates of the inspection path point before correction; The direction vector of the adjustment point pointing to the inspection object; is the direction vector pointing away from the obstacle; 1 Adjust the step size for inspection object observation according to the pre-experimental setting; γ 2 Adjust the step size for obstacle avoidance; where:

[0075]

[0076] Among them, B1 (G i , t) is the coordinate of the inspection object adjustment point, which can be obtained by the following method:

[0077] S3311: Obtaining the edge coordinate set of the inspection object;

[0078] S3312: Obtain the edge coordinate set of the inspection object and point P old (i, j, t) is the nearest inspection object boundary point;

[0079] S3313: After applying a preset observation safety distance along the normal direction of the plane of the inspection object where the point is located on the basis of the boundary point of the inspection object, the coordinate position obtained is the coordinate of the inspection object adjustment point;

[0080]

[0081] Among them, B 2 (G i , t) is the coordinate of the point far away from the obstacle, which can be obtained by the following method:

[0082] S3321: Obtaining the edge coordinate set of environmental obstacles;

[0083] S3322: Obtain the coordinate set of the edge of the environmental obstacle and point P old (i, j, t) is the nearest environmental obstacle boundary point;

[0084] S3323: After applying a preset obstacle avoidance safety distance along the normal direction of the obstacle plane where the point is located based on the boundary point of the environmental obstacle, the coordinate position obtained is the coordinate of the point away from the obstacle;

[0085] γ 2 =d(P old (i,j,t), B 2 (G i , t))·[1+ln(1+γ 3 ·d(P old (i,j,t), B 2 (G i , t)))];

[0086] Among them, d(P old (i,j,t), B 2 (G i , t)) is P old (i, j, t) and B 2 (G i , t), γ 3 To correct the amplitude coefficient for obstacle avoidance, it is set by pre-experimentation;

[0087] This solution dynamically adjusts the UAV inspection path points, combines the boundary information of the inspection object and environmental obstacles, and dynamically optimizes and corrects the flight path after the three-dimensional environmental model is updated, thereby ensuring the inspection accuracy, making the UAV inspection path cover the inspection object more accurately, and maintaining a reasonable observation safety distance, thereby improving the inspection quality. At the same time, through the dynamic adjustment of the obstacle avoidance adjustment step length, it is ensured that the UAV enhances its obstacle avoidance when approaching obstacles and reduces unnecessary deviations when far away from obstacles, thereby improving flight safety and stability.

[0088] The contents disclosed above are only preferred feasible embodiments of the present invention, and do not limit the protection scope of the present invention. Therefore, all equivalent technical changes made using the contents of the present invention specification and drawings are included in the protection scope of the present invention. In addition, the elements therein can be updated as technology develops.

Claims

1. An automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control, characterized in that: The system includes a machine vision perception module, an adaptive flight control module and a communication feedback module; the machine vision perception module is used to collect environmental information in real time and build a three-dimensional environmental model of the flight area; the adaptive flight control module is used to dynamically plan the flight path of the subsequent UAV in combination with the three-dimensional environmental model of the UAV flight area; the communication feedback module is used to realize two-way communication between the UAV and the ground station; The machine vision perception module includes an environment perception unit, a target recognition unit and a visual positioning unit; the environment perception unit is used to collect environmental information; the target recognition unit is used to identify inspection objects and environmental obstacles in combination with environmental information; the visual positioning unit is used to construct a three-dimensional environmental model of the flight area in combination with the recognition results of the target recognition unit and GPS data.

2. According to claim 1, the automatic inspection and navigation system of unmanned aerial vehicle based on machine vision and adaptive flight control is characterized in that: The environmental perception unit includes a stereo camera and a laser radar; the stereo camera is used to obtain continuous multi-frame image information during the flight of the drone; the laser radar is used to scan the surrounding environment during the flight of the drone and obtain the spatial relative position and distance information between the drone and objects in the surrounding environment.

3. The automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control according to claim 1 is characterized in that: The target recognition unit recognizes the inspection object and environmental obstacles in the following ways: S11: Acquire continuous multiple frames of image information in the environmental information; S12: Use the YOLO model to perform target detection on continuous multi-frame image information and identify inspection objects and environmental obstacles in continuous multi-frame image information; S13: Use the Mask-R-CNN model to segment the identified inspection objects and environmental obstacles in the image and extract the edge morphological information of the inspection objects and environmental obstacles.

4. The automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control according to claim 1 is characterized in that: The visual positioning unit constructs a three-dimensional environment model of the flight area in the following manner: S21: Acquire the GPS positioning data of the drone during the flight of the drone to determine the real-time location information of the drone; S22: The information obtained by the laser radar and the identification content of the target identification unit are fused through spatial data to obtain the relative position and distance information between the UAV and the inspection object and environmental obstacles during the flight; S23: constructing a three-dimensional environment model of the UAV flight area in combination with the information obtained in step S22, wherein the three-dimensional environment model includes coordinate information of the UAV, the inspection object, and the environmental obstacles; S24: During the flight of the UAV, the update frequency of the three-dimensional environment model is adjusted in combination with dynamic environmental changes.

5. The automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control according to claim 4 is characterized in that: In step S24, the update frequency of the three-dimensional environment model is adjusted specifically in the following manner: S241: Setting a continuous incremental collection cycle, and obtaining an environmental change increment in each incremental collection cycle; the environmental change increment is specifically expressed as: ΔM object =|M new -M old |; Among them, ΔM object Indicates the incremental change of the environment within a certain incremental collection cycle, M new is the area of ​​the inspection object and environmental obstacles in the image collected by the UAV at the end time of the incremental collection cycle; M old is the area of ​​the inspection object and environmental obstacles in the image collected by the UAV at the start time of the incremental collection cycle; new and M old The edge morphological information of the inspection object and environmental obstacles identified by the target recognition unit is further calculated and obtained; S242: Calculate the update frequency of the three-dimensional environment model after each incremental collection cycle based on the incremental environment change in each incremental collection cycle: f=α·[1-exp(-β·ΔM object )]; Among them, f is the update frequency of the three-dimensional environment model after a certain incremental acquisition cycle; α is the preset maximum update frequency; β is the incremental sensitivity adjustment coefficient, which is used to adjust the impact of the change in the incremental environmental change on the update frequency and is set through pre-experimental settings.

6. The automatic inspection and navigation system for unmanned aerial vehicles based on machine vision and adaptive flight control according to claim 5 is characterized in that: The adaptive flight control module completes the dynamic planning of the subsequent flight path of the UAV in the following ways: S31: Define the initial flight path of the UAV according to the mission requirements; S32: after each update of the three-dimensional environment model, obtaining edge coordinate information of the inspection object and environmental obstacles in the three-dimensional environment model; S33: After each update of the three-dimensional environment model, the inspection path points related to the inspection object in the current three-dimensional environment model are corrected.

Images