Unmanned aerial vehicle subway inspection method and equipment based on visual guidance and storage medium

Abstract

The invention discloses an unmanned aerial vehicle subway inspection method and device based on visual guidance and a readable storage medium, and the method comprises the steps: carrying out the image collection of a track through a binocular camera, and carrying out the data point marking of the collected image; dividing the data points into left track key points and right track key points, and matching the track key points of the left image and the right image; after a track key point matching result of the left and right images is obtained, depth information of track key points on the images is calculated according to the focal length of the camera and parallax of the left and right images; and calculating a middle point of the matched key points of the tracks on the two sides to extract a reference path for unmanned aerial vehicle inspection. According to the method, full-automatic inspection of the unmanned aerial vehicle can be realized only by using a binocular camera and a sensor, the inspection cost is greatly reduced, a subway tunnel model does not need to be established in advance, the inspection path does not need to be marked, the inspection path can be automatically identified and tracked on line, and the inspection process is more intelligent and portable.

Description

[0001] UAV subway inspection method, equipment and storage medium based on vision guidance Technical Field

[0002] The present invention relates to a method, equipment and storage medium for subway inspection by an unmanned aerial vehicle based on vision guidance. Background Art

[0003] As a new type of intelligent inspection tool, subway inspection drones have many advantages that traditional manual inspection methods cannot match. Drones can quickly and comprehensively fly and inspect every corner of the tunnel, including some places that are difficult to reach manually (such as the top of the tunnel or narrow passages). It can quickly cover the entire tunnel section, saving a lot of time compared to traditional manual inspection methods. Subway inspections often involve dangerous environments, such as water accumulation, landslides, and poor air circulation in the tunnel. Traditional inspectors need to enter these dangerous areas and may face health and safety risks. Drones can be operated remotely to avoid manual entry into complex or dangerous environments and ensure personnel safety. Drones can also upload the collected data to the cloud platform or control center in real time, so that operators can obtain inspection results and analyze them in the first time and make decisions quickly.

[0004] Among the existing technical solutions, some solutions use ground track robots for inspection, control the robot to travel on the track, and detect the surrounding environment through the sensors on its load. Although this method does not require odometers and reference path information, it lacks flexibility during inspections. When obstacles appear on the track, the robot cannot inspect, and the detection of the top of the tunnel is not clear enough. Some solutions use lidar to build maps in advance, then mark the reference path, and control the drone to fly along the preset path during the actual inspection. This method requires marking the reference path in the point cloud offline in advance, and then during the actual inspection, it is necessary to use relocation or scene positioning algorithms to determine the position of the drone in the previously collected point cloud. The accuracy requirements for the algorithm are high, and there will be problems of accumulated errors during the positioning process. There is also a method of positioning the drone by deploying multiple UWB base stations, but if the tunnel distance is long, this method is costly to implement and difficult to deploy.

[0005] The patent with publication number CN115951704A discloses a method and equipment for subway inspection using drones based on BIM models (building digital models). The method for subway inspection using drones based on BIM models includes: obtaining mapping data of railway sections in various places, and generating subway inspection BIM models based on the mapping data; obtaining the size information of the inspection drone, and generating inspection path information corresponding to the railway sections in various places based on the size information and the subway inspection BIM model, and sending the inspection plan information to the inspection control module of the inspection drone to control the drone to inspect according to the preset path. The proposal requires evenly setting monitoring points at various locations in the subway tunnel, and mapping data through total stations and monitoring positioning, and mapping data based on the BIM model generated based on the design drawings. The modeling process is cumbersome, the equipment required is expensive, and it is difficult to implement. In addition, in the actual application process, it is also necessary to align the BIM model coordinate system with the coordinate system of the drone inspection, which increases the difficulty of development.

[0006] The patent with publication number CN109343548A discloses an inspection system for a subway tunnel inspection robot. It includes: a robot body, a control unit and a detection unit, the detection unit is arranged on the robot body; the control unit is used to send control instructions to the robot body and the detection unit respectively; the robot body is used to move along the track in the subway tunnel according to the control instructions; the detection unit is used to detect the environment in the subway tunnel when the robot body moves along the track in the subway tunnel according to the control instructions, and send the data obtained by the detection to the control unit, thereby realizing the automated inspection of the subway tunnel. This proposal controls the robot to travel on the track for inspection. The advantage is that no additional odometer and environmental information are required to control the robot to patrol in the center of the subway tunnel, but it also limits the flexibility of the robot during inspection. Since it is traveling on the track, if there are obstacles on the track, it may cause danger and inspection failure, and the on-track driving method is far away from the top of the tunnel, making it difficult to detect data from the top of the tunnel.

[0007] The patent with publication number CN117518158A discloses a tunnel full-section inspection equipment and method based on unmanned aerial vehicle geological radar. The patent installs millimeter wave micro-communication base stations at equal intervals in advance in the subway tunnel. When the unmanned aerial vehicle conducts inspections in the subway tunnel, multiple millimeter wave micro-communication base stations respectively locate the millimeter wave radar module and receive the communication signal transmitted by the unmanned aerial vehicle flight control module through the wireless communication module. Multiple millimeter wave micro-communication base stations are respectively connected to the tunnel scanning control center for communication. When the GPS signal is not good, the current position of the unmanned aerial vehicle can be accurately obtained, and then the unmanned aerial vehicle can be controlled to conduct inspections along the preset path. Although this method can accurately obtain the position of the unmanned aerial vehicle, it is necessary to deploy multiple millimeter wave micro-communication base stations in the subway tunnel. If the tunnel is long and there is no good deployment environment, the cost and implementation difficulty of this method will increase significantly. In addition, it is difficult to maintain in the later stage. If one of the millimeter wave micro-communication base stations fails, the unmanned aerial vehicle cannot be located near the location, resulting in failure of the entire system. Summary of the invention

[0008] The main purpose of the present invention is to provide a vision-guided UAV subway inspection method, equipment and storage medium, aiming to solve the above-mentioned technical problems.

[0009] To achieve the above objectives, the present invention provides a UAV subway inspection method based on vision guidance.

[0010] The visually guided UAV subway inspection method comprises the following steps: The track is imaged using a binocular camera, and data points are marked on the imaged; Divide the data points into left track key points and right track key points, and match the track key points of the left and right images; After obtaining the trajectory key point matching results of the left and right images, the depth information of the trajectory key points on the image is calculated based on the focal length of the camera and the parallax of the left and right images; The midpoint of the matching key points on both sides of the track is calculated to extract the reference path for drone inspection.

[0011] In one embodiment, after the step of collecting images of the track by a binocular camera and marking data points on the collected images, the visually guided UAV subway inspection method further includes: The prior information of the track size is used to remove the falsely detected key points and supplement the missed key points, so that the number of key points on the left track is equal to the number of key points on the right track.

[0012] In one embodiment, the steps of eliminating falsely detected key points and supplementing missed key points by using prior information of track size are specifically as follows: Whether the key points are misdetected or missed is determined by judging whether the distance between the adjacent key points exceeds the maximum value of the sleeper distance or is less than the minimum value of the sleeper distance.

[0013] In one embodiment, after obtaining the trajectory key point matching results of the left and right images, and calculating the depth information of the trajectory key points on the image according to the focal length of the camera and the parallax of the left and right images, the visually guided UAV subway inspection method further includes: It is converted into real 3D space through the camera's internal and external parameter matrix.

[0014] In one embodiment, after the step of calculating the middle point of the matching key points of the two side tracks to extract the reference path for the drone inspection, the visually guided drone subway inspection method further includes: The expected position of the drone on the reference path at the current moment is obtained, and the drone is instructed to inspect along the expected position on the reference path.

[0015] In one embodiment, the step of obtaining the expected position of the drone on the reference path at the current moment and causing the drone to inspect along the reference path at the expected position includes: Project the UAV onto the orbital plane, draw a circle with the UAV projection position as the center and the preview distance as the radius, and use the intersection of the circle and the reference path as the optimal expected position in the horizontal direction, and let the UAV patrol along the reference path along the optimal expected position.

[0016] In addition, to achieve the above-mentioned objectives, the present invention also provides a vision-guided drone subway inspection method, which includes: a memory, a processor, and a vision-guided drone subway inspection program stored in the memory and executable on the processor. When the vision-guided drone subway inspection program is executed by the processor, the steps of the vision-guided drone subway inspection method as described above are implemented.

[0017] In addition, to achieve the above-mentioned objectives, the present invention also provides a computer-readable storage medium, on which a vision-guided drone subway inspection program is stored. When the vision-guided drone subway inspection program is executed by a processor, the steps of the vision-guided drone subway inspection method as described above are implemented.

[0018] The beneficial effects that can be achieved by the present invention are as follows: a vision-guided drone subway inspection method proposed in an embodiment of the present invention collects images of tracks through a binocular camera and marks data points of the collected images; the data points are divided into left track key points and right track key points, and the track key points of the left and right images are matched; after the track key point matching results of the left and right images are obtained, the depth information of the track key points on the image is calculated according to the focal length of the camera and the parallax of the left and right images; the middle point of the matching key points of the tracks on both sides is calculated to extract the reference path for the drone inspection.

[0019] This application only requires a binocular camera as one sensor to achieve fully automated inspections by drones, greatly reducing the cost of inspections. There is no need to build a subway tunnel model in advance and mark the inspection path. The inspection path can be automatically identified and tracked online, making the inspection process more intelligent and lightweight. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the structure of a device in a hardware operating environment involved in an embodiment of the present invention; Figure 2 It is a flow chart of the method for subway inspection using a UAV based on vision guidance of the present invention; Figure 3 A schematic diagram of marking track marking points according to an embodiment of the present invention; Figure 4 This is a schematic diagram of misdetection of track marking points according to an embodiment of the present invention; Figure 5 Schematic diagram of missed detection of track marking points according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the preview range of the UAV according to an embodiment of the present invention; Figure 7 Schematic diagram of a reference path for tracking a drone according to an embodiment of the present invention.

[0021] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0022] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0023] like Figure 1 As shown, Figure 1 It is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiment of the present invention.

[0024] The terminal of the embodiment of the present invention can be a PC, or it can be a smart phone, a tablet computer, an e-book reader, an MP3 (Moving Picture Experts Group Audio Layer III) player, an MP4 (Moving Picture Experts Group Audio Layer IV) player, a portable computer, or other portable terminal devices with display function.

[0025] like Figure 1 As shown, the terminal may include: a processor 1001, such as a CPU, a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0026] Optionally, the terminal may also include a camera, an RF (Radio Frequency) circuit, a sensor, an audio circuit, a WiFi module, and the like. Among them, sensors include light sensors, motion sensors, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display screen according to the brightness of the ambient light, and the proximity sensor may turn off the display screen and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile terminal (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; of course, the mobile terminal can also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., which will not be repeated here.

[0027] Those skilled in the art will understand that Figure 1 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0028] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a vision-guided drone subway inspection program.

[0029] exist Figure 1 In the terminal shown, the network interface 1004 is mainly used to connect to the backend server and communicate data with the backend server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the visual guidance-based drone subway inspection program stored in the memory 1005 and perform the following operations: The track is imaged using a binocular camera, and data points are marked on the imaged; Divide the data points into left track key points and right track key points, and match the track key points of the left and right images; After obtaining the trajectory key point matching results of the left and right images, the depth information of the trajectory key points on the image is calculated based on the focal length of the camera and the parallax of the left and right images; The midpoint of the matching key points on both sides of the track is calculated to extract the reference path for drone inspection.

[0030] Further, the processor 1001 may call the visually guided UAV subway inspection program stored in the memory 1005, and further perform the following operations: The prior information of the track size is used to remove the falsely detected key points and supplement the missed key points, so that the number of key points on the left track is equal to the number of key points on the right track.

[0031] Further, the processor 1001 may call the visually guided UAV subway inspection program stored in the memory 1005, and further perform the following operations: Whether the key points are misdetected or missed is determined by judging whether the distance between the adjacent key points exceeds the maximum value of the sleeper distance or is less than the minimum value of the sleeper distance.

[0032] Further, the processor 1001 may call the visually guided UAV subway inspection program stored in the memory 1005, and further perform the following operations: It is converted into real 3D space through the camera's internal and external parameter matrix.

[0033] Further, the processor 1001 may call the visually guided UAV subway inspection program stored in the memory 1005, and further perform the following operations: The expected position of the drone on the reference path at the current moment is obtained, and the drone is instructed to inspect along the expected position on the reference path.

[0034] Further, the processor 1001 may call the visually guided UAV subway inspection program stored in the memory 1005, and further perform the following operations: Project the UAV onto the orbital plane, draw a circle with the UAV projection position as the center and the preview distance as the radius, and use the intersection of the circle and the reference path as the optimal expected position in the horizontal direction, and let the UAV patrol along the reference path along the optimal expected position.

[0035] The specific embodiments of the data storage device of the present invention are basically the same as the embodiments of the visual-guided UAV subway inspection method described below, and will not be described in detail here.

[0036] Reference Figure 2 The first embodiment of the present invention provides a UAV subway inspection method based on visual guidance, and the UAV subway inspection method based on visual guidance includes: The track is imaged using a binocular camera, and data points are marked on the imaged; Divide the data points into left track key points and right track key points, and match the track key points of the left and right images; After obtaining the trajectory key point matching results of the left and right images, the depth information of the trajectory key points on the image is calculated based on the focal length of the camera and the parallax of the left and right images; The midpoint of the matching key points on both sides of the track is calculated to extract the reference path for drone inspection.

[0037] Please refer to Figure 3 First, the key point detection model in deep learning is used to detect the track key points of the collected binocular images. When annotating the data, the track key points are defined as the intersection of the rails and the sleepers, and are divided into two categories: left track key points and right track key points. The left and right directions are based on the shooting direction. The red marked points indicate the left track key points, and the green marked points indicate the right track key points. Optionally, the key point detection model can adopt a spatial embedding model. The image data is input, and three feature maps are output through the backbone network, namely the confidence feature map, the offset feature map (x and y), and the embedding vector branch. The position index of the point on the feature map with a confidence greater than a certain threshold indicates that there is a lane line key point at the corresponding position of the image. Combined with the offset in the x and y directions, the accurate position of the lane line point can be obtained. Then the Euclidean distance of the embedding feature vectors of different lane line points is calculated for clustering, and a complete lane line instance can be obtained. The trajectory key point detection result of the key point detection model is also similar to Figure 3 shown.

[0038] After obtaining the track key points of the left and right images respectively, the track key points of the left and right images are matched. The feature descriptor of the pixel position can be extracted, and then the feature similarity of the track key points can be calculated for matching. Other methods can also be used for matching. For example, the SIFT feature matching method performs SIFT feature detection on the two images respectively, extracts the key points and their feature descriptors, and the descriptor of each key point is a 128-dimensional vector, which represents the gradient distribution near the point. Then the Euclidean distance between the feature points is calculated, and for each descriptor in the first image, the closest descriptor in the second image is found for matching. Other commonly used feature matching methods include surf feature matching and orb feature matching methods.

[0039] After obtaining the trajectory key point matching results of the left and right images, the depth information of the trajectory key points on the image is calculated according to the focal length of the camera and the parallax of the left and right images, and then converted into the real 3D space through the camera's internal and external parameter matrix.

[0040] Since the track spacing and the spacing between the front and rear sleepers can be measured in advance, the detection results of the model can be corrected using this prior information.

[0041] It can be mainly divided into the following categories: (1) Track key point classification error; (2) False detection of track key points; (3) Missing inspection of key track points; For track key point classification errors, such as Figure 4 As shown in Figure 2. The red keypoints in the right track are incorrectly detected.

[0042] At the same time, if Figure 5 As shown in the figure, the change of track slope can be judged to locate the key points of incorrect classification. If the slope of adjacent track points changes, they will be changed to the correct category. For the misdetection and missed detection of track key points, it can be judged by judging whether the spacing between the adjacent key points exceeds the maximum sleeper spacing or is less than the minimum sleeper spacing threshold. Eliminate the misdetected points and fill in the missed points.

[0043] After correcting the detection results of the model according to the prior information of the track, the matching relationship of the key points of the left and right tracks is obtained through the Hungarian matching method, and then the middle point of the matching key points of the tracks on both sides is calculated to extract the reference path for UAV inspection.

[0044] At the same time, the complete track points can be back-projected into the image and added to the image feature points detected by the visual positioning algorithm to jointly calculate the position information of the drone. The position of the drone can be obtained by SLAM positioning algorithms based on feature points such as orbslam.

[0045] In addition to obtaining the current position of the drone, it is also necessary to know the expected position of the drone at the current moment before it can be passed to the controller to control the drone to patrol along the reference path.

[0046] Since the UAV has a forward flight speed, if the point closest to the reference path is directly used as the expected position of the UAV at the current moment, it may cause the UAV's flight trajectory to oscillate. In fact, the UAV should track a certain point on the reference path ahead.

[0047] Therefore, this application tracks the expected path by previewing, and calculates the preview distance based on the take-off altitude of the drone, the current flight speed, and the curvature of the track. Since the camera is installed at a fixed angle downward, the take-off altitude of the drone determines the upper and lower limits of the preview distance. like Figure 7 As shown in the figure, the faster the UAV flies, the longer the preview distance should be, otherwise it will cause the UAV flight trajectory to oscillate; if the road curvature radius is larger, it means that the track is more curved. At this time, the preview distance should be reduced to enable the UAV to better track the reference path for inspection. The calculated preview distance should not exceed its upper and lower limits.

[0048] Then ignore the height of the drone, project the drone onto the orbital plane, draw a circle with the drone's projection position as the center and the preview distance as the radius, and solve its intersection with the reference path as the optimal expected position in the horizontal direction. Use the drone's current flight altitude as the vertical position coordinate, and send the expected position information to the controller.

[0049] In summary, the present application can realize the fully automated inspection of UAVs in subway tunnels by relying solely on binocular camera sensors, which greatly saves costs. The key points of the tracks on both sides are detected through the key point detection model in deep learning, and the coordinates of the track key points in the real world are restored by matching them in the left and right images. The algorithm detection results are post-processed using the prior information of the track size, the erroneous key points are eliminated, and the missed key points are supplemented, which can improve the accuracy of the algorithm model detection and obtain a more accurate reference path. The optimized 3D track key points are back-projected into the image, added to the image feature points detected by the visual positioning algorithm, and the current position of the UAV is calculated together. Due to the increase in the number of accurately matched feature points, the positioning accuracy of the UAV can be improved.

[0050] In order to ensure that the drone accurately and smoothly tracks the expected path, the expected position of the drone inspection is calculated by pre-aiming. The pre-aiming distance is calculated according to the current flight speed, flight altitude and track curvature information of the drone. The optimal expected position is selected on the reference path, which can make the drone track the reference path more smoothly and accurately for inspection. If an obstacle appears within the field of view, the obstacle avoidance algorithm is called to plan the path to bypass the obstacle. When there is no obstacle within the field of view, the reference path is re-tracked to ensure the safety of the drone inspection.

[0051] In another embodiment, the image track key points can be projected into 3D space through an RGBD camera, or a radar plus camera method can be used. Because the depth information of the image track key points needs to be known, the depth camera and radar in the RGBD camera can both measure their depth.

[0052] For the positioning of drones, other algorithms can be used for positioning. This proposal uses a binocular camera to run an algorithm to locate the drone. It can also be positioned through an RGBD camera, radar or other sensors.

[0053] The present invention proposes a UAV subway inspection method based on visual guidance. This method only needs a binocular camera sensor to realize fully automated inspection of the UAV, greatly reducing the inspection cost, and does not require the establishment of a subway tunnel model in advance to mark the inspection path. The inspection path can be automatically identified and tracked online, making the inspection process more intelligent and convenient. By controlling the UAV to inspect above the track, the environmental information in the subway tunnel can also be fully obtained.

[0054] The detection results of track key points are corrected by using the prior information of tracks and sleepers, which improves the accuracy of track detection results. At the same time, the corrected track key points are back-projected onto the image and added to the image feature points detected by the visual positioning algorithm to jointly locate the UAV, which increases the number of feature points and improves the accuracy and stability of UAV positioning.

[0055] The reference path is tracked by previewing, and the preview distance is calculated according to the flight speed and track curvature of the drone. The upper and lower boundaries of the preview distance are calculated according to the take-off height of the drone and the camera installation angle. The preview distance is constrained, and the optimal expected position is found on the reference path according to the preview distance and transmitted to the controller. This enables the drone to smoothly and accurately track the reference path for inspection.

[0056] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, on which a vision-guided UAV subway inspection program is stored. When the vision-guided UAV subway inspection program is executed by a processor, the following operations are implemented: The track is imaged using a binocular camera, and data points are marked on the imaged; Divide the data points into left track key points and right track key points, and match the track key points of the left and right images; After obtaining the trajectory key point matching results of the left and right images, the depth information of the trajectory key points on the image is calculated based on the focal length of the camera and the parallax of the left and right images; The midpoint of the matching key points on both sides of the track is calculated to extract the reference path for drone inspection.

[0057] Furthermore, when the visually guided UAV subway inspection program is executed by the processor, the following operations are also implemented: The prior information of the track size is used to remove the falsely detected key points and supplement the missed key points, so that the number of key points on the left track is equal to the number of key points on the right track.

[0058] Furthermore, when the visually guided UAV subway inspection program is executed by the processor, the following operations are also implemented: Whether the key points are misdetected or missed is determined by judging whether the distance between the adjacent key points exceeds the maximum value of the sleeper distance or is less than the minimum value of the sleeper distance.

[0059] Furthermore, when the visually guided UAV subway inspection program is executed by the processor, the following operations are also implemented: It is converted into real 3D space through the camera's internal and external parameter matrix.

[0060] Furthermore, when the visually guided UAV subway inspection program is executed by the processor, the following operations are also implemented: The expected position of the drone on the reference path at the current moment is obtained, and the drone is instructed to inspect along the expected position on the reference path.

[0061] Furthermore, when the visually guided UAV subway inspection program is executed by the processor, the following operations are also implemented: Project the UAV onto the orbital plane, draw a circle with the UAV projection position as the center and the preview distance as the radius, and use the intersection of the circle and the reference path as the optimal expected position in the horizontal direction, and let the UAV patrol along the reference path along the optimal expected position.

[0062] The specific embodiments of the computer-readable storage medium of the present invention are basically the same as the embodiments of the above-mentioned vision-guided UAV subway inspection method, and will not be described in detail here.

[0063] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0064] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0065] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0066] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A UAV subway inspection method based on visual guidance, characterized in that: The visually guided UAV subway inspection method comprises the following steps: The track is imaged using a binocular camera, and data points are marked on the imaged; Divide the data points into left track key points and right track key points, and match the track key points of the left and right images; After obtaining the trajectory key point matching results of the left and right images, the depth information of the trajectory key points on the image is calculated based on the focal length of the camera and the parallax of the left and right images; The midpoint of the matching key points on both sides of the track is calculated to extract the reference path for drone inspection.

2. The visually guided UAV subway inspection method according to claim 1, characterized in that: After the step of collecting images of the track by using a binocular camera and marking data points on the collected images, the visually guided UAV subway inspection method further includes: The prior information of the track size is used to remove the falsely detected key points and supplement the missed key points, so that the number of key points on the left track is equal to the number of key points on the right track.

3. The visually guided UAV subway inspection method according to claim 2, characterized in that: The specific steps of using the prior information of track size to eliminate falsely detected key points and supplement missed key points are as follows: Whether the key points are misdetected or missed is determined by judging whether the distance between the adjacent key points exceeds the maximum value of the sleeper distance or is less than the minimum value of the sleeper distance.

4. The visually guided UAV subway inspection method according to claim 2, characterized in that: After obtaining the tracking key point matching results of the left and right images, and calculating the depth information of the tracking key points on the image according to the focal length of the camera and the parallax of the left and right images, the visually guided UAV subway inspection method further includes: It is converted into real 3D space through the camera's internal and external parameter matrix.

5. The visually guided UAV subway inspection method according to claim 1 is characterized in that: After the step of calculating the middle point of the matching key points of the tracks on both sides to extract the reference path for the drone inspection, the drone subway inspection method based on visual guidance also includes: The expected position of the drone on the reference path at the current moment is obtained, and the drone is instructed to inspect along the expected position on the reference path.

6. The visually guided UAV subway inspection method according to claim 5 is characterized in that: The step of obtaining the expected position of the drone on the reference path at the current moment and making the drone inspect along the expected position on the reference path comprises: Project the UAV onto the orbital plane, draw a circle with the UAV projection position as the center and the preview distance as the radius, and use the intersection of the circle and the reference path as the optimal expected position in the horizontal direction. Allow the UAV to patrol along the reference path along the optimal expected position.

7. A UAV subway inspection device based on visual guidance, characterized in that: The vision-guided UAV subway inspection device includes: a memory, a processor, and a vision-guided UAV subway inspection program stored in the memory and executable on the processor. When the vision-guided UAV subway inspection program is executed by the processor, the steps of the vision-guided UAV subway inspection method according to any one of claims 1 to 6 are implemented.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a vision-guided UAV subway inspection program, which, when executed by a processor, implements the steps of the vision-guided UAV subway inspection method according to any one of claims 1 to 6.

Images