Unmanned aerial vehicle highway inspection and warning device and method

Abstract

The invention discloses an unmanned aerial vehicle highway inspection and warning device, which can quickly warn drivers of vehicles behind when an unmanned aerial vehicle recognizes abnormal road conditions and abnormal traffic conditions, so as to avoid or reduce traffic accidents. The system comprises an inspection unmanned aerial vehicle which mounts a visual detection module and a projection module; the auxiliary unmanned aerial vehicle is used for mounting the projection screen; the inspection unmanned aerial vehicle and the auxiliary unmanned aerial vehicle are matched one by one to construct an inspection system, and the inspection system has an inspection working mode and a warning mode; in the inspection working mode, the inspection unmanned aerial vehicle inspects along a highway, the visual inspection module shoots video image data of a front road surface, automatically identifies the road surface condition and assists the unmanned aerial vehicle to follow behind the inspection unmanned aerial vehicle; in the inspection working mode, when the inspection unmanned aerial vehicle detects that abnormal conditions such as dangerous road conditions or traffic accidents exist on the road surface, the system is switched into the warning mode.

Description

Technical Field

[0001] The present invention relates to the technical field of highway inspection, and specifically to an unmanned aerial vehicle highway inspection and warning device. The present invention also provides an unmanned aerial vehicle highway inspection and warning method. Background Art

[0002] With the rapid development of the transportation industry, the number, mileage and traffic volume of highways are increasing, and the difficulty of highway inspection and maintenance is increasing. Conventional highway inspection methods such as vehicle inspection and foot inspection occupy a lot of human resources, have a long inspection cycle, and also pose safety risks.

[0003] The rapid progress of drone technology in recent years and its widespread popularity in various industries have gradually attracted attention for its application in the field of highways. Drone inspections do not occupy road resources and are not restricted by traffic conditions. They can inspect designated sections efficiently and flexibly. At present, conventional drone inspections require professional pilots to operate, and automated and intelligent inspections need to be further improved. Drone inspections can only detect road problems and traffic accidents. How to remind and warn vehicles and drivers on the road in real time with inspection results also needs to be strengthened. Especially when emergencies such as geological disasters occur in mountainous areas, early warnings as soon as possible can avoid or reduce the occurrence of major accidents. Summary of the invention

[0004] In response to the above problems, the present invention provides a drone highway inspection and warning device, which can quickly warn the drivers of vehicles behind when the drone identifies abnormal road conditions and abnormal traffic conditions, thereby avoiding or reducing the occurrence of traffic accidents.

[0005] A drone highway inspection and warning device, characterized in that it comprises:

[0006] Inspection drones, which are equipped with visual inspection modules and projection modules;

[0007] and an auxiliary drone, which carries a projection screen;

[0008] The inspection drones and auxiliary drones are matched one by one to form an inspection system, and the inspection system has an inspection working mode and an alarm mode;

[0009] In the inspection working mode, the inspection drone inspects along the highway. Its visual detection module captures video image data of the road ahead and automatically identifies the road conditions. The auxiliary drone follows the inspection drone.

[0010] In the inspection working mode, when the inspection drone detects abnormal situations such as dangerous road conditions or traffic accidents on the road, the system switches to warning mode. The inspection drone sends its coordinate position to the auxiliary drone. The auxiliary drone opens the projection screen behind the inspection drone and sends its position to the inspection drone. The inspection drone flies behind the auxiliary drone and detects the appearance of the projection screen. Its projection module then projects a warning image onto the projection screen to alert the drivers of the vehicles behind to the accident ahead and avoid danger.

[0011] It is further characterized by:

[0012] The inspection drone uses a rotor drone that supports hovering as a flight platform, and the inspection drone is equipped with a visual module, a projection module, a positioning and attitude module, and a wireless communication module;

[0013] The visual module is a camera with a zoom function installed on a three-axis motion gimbal, and its field of view supports: short focus can cover the width of the road to facilitate automatic recognition of road conditions, and long focus supports covering the range of the auxiliary drone projection screen to facilitate detection of the auxiliary drone projection screen and positioning to achieve the transformation of the projection screen;

[0014] The visual module outputs the current camera gimbal posture, namely the pitch angle, yaw angle, and roll angle;

[0015] The visual module includes a recognition processing unit, which processes the video images taken by the camera in real time and automatically detects and recognizes abnormal conditions such as dangerous road conditions or traffic accidents on the road surface;

[0016] The recognition processing unit can automatically detect the road and lane lines ahead based on the video images captured by the camera, and automatically plan the inspection path according to the outermost lane line;

[0017] The projection module is a projection device fixedly installed on the inspection drone;

[0018] The projection module further includes a warning image generation unit, which sets projection transformation parameters according to the position and posture relationship between the inspection drone and the projection screen of the auxiliary drone obtained by the inspection drone, generates an image after projection transformation, and projects it onto the projection screen of the auxiliary drone by the projector, ensuring that the image on the projection screen is not deformed;

[0019] The auxiliary drone uses a rotor drone that supports hovering as a flight platform. The auxiliary drone is equipped with a projection screen module, a flight path calculation module, a positioning and attitude module, and a wireless communication module. The auxiliary drone has two working modes: a following mode and a warning mode.

[0020] The projection screen module adopts a screen, a fog screen or a water screen; in the following mode of the auxiliary drone, the projection screen module is in a retracted or closed state; when in the warning mode, the auxiliary drone lowers the screen or opens the fog screen / water screen after reaching the calculated warning position;

[0021] The flight path calculation module is used to calculate the flight path of the following inspection drone in the follow mode and calculate the appropriate projection position in real time; in the warning mode, it calculates the road direction and the tilt direction of the projection screen at the projection position;

[0022] The positioning and attitude determination modules of the inspection UAV and the auxiliary UAV obtain the position information (x, y, z coordinates) and attitude information (pitch, roll, yaw angle) of the corresponding UAV, and output it to the other UAV of the paired inspection system. The positioning and attitude determination module can be implemented by using the existing GPS or Beidou navigation system, gyroscope, and accelerometer, which is a mature technology and will not be described in detail;

[0023] The wireless communication module is used to communicate with another drone of the paired inspection system to mutually transmit position and posture information, which is achieved using a wireless WIFI network.

[0024] The drone highway inspection and warning method comprises the following steps:

[0025] Step 1: Initialize the drone’s location information and camera field of view;

[0026] Step 2: The inspection drone inspects along the road;

[0027] Step 3: Assist the drone to follow;

[0028] Step 4: Assist the drone to calculate the rear warning point relative to the current position;

[0029] Step 5: The inspection drone determines whether an abnormality is detected;

[0030] Step 6: Assist the drone to place the projection screen at the warning position;

[0031] Step 7: The inspection drone reaches the projection location;

[0032] Step 8: Calculate the corrected projection image and issue a projection warning;

[0033] If the inspection drone does not detect an abnormality, after step five, the device switches to step two to perform the inspection working mode; if the inspection drone detects an abnormality, after step five, the device sequentially enters step six, step seven, and step eight until the road abnormality is resolved and enters the inspection working mode again.

[0034] After adopting the present invention, it uses a drone as a platform and carries a visual module to automatically identify and report highway road conditions and traffic conditions, thereby reducing manual operations and improving work efficiency; when the drone identifies abnormal road conditions and traffic conditions, it uses a drone projection display method to warn the drivers of the vehicles behind in real time, thereby avoiding or reducing the occurrence of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a schematic diagram of the inspection working mode of the present invention;

[0036] Figure 2 It is a schematic diagram of the warning working mode of the present invention;

[0037] Figure 3 It is a schematic diagram of the inspection working mode and coordinate definition diagram applicable to the present invention;

[0038] Figure 4 It is a warning mode schematic diagram and coordinate posture definition diagram applicable to the present invention;

[0039] Figure 5 Schematic diagram of the coordinate relationship of the XY plane (i.e., top view) applicable to the screen projection of the present invention. DETAILED DESCRIPTION

[0040] A drone highway inspection and warning device, Figure 1-Figure 4 , which includes inspection drones and auxiliary drones;

[0041] The inspection drone uses a hovering rotor drone as a flight platform, and the inspection drone is equipped with a visual module, a projection module, a positioning and attitude module, and a wireless communication module;

[0042] The auxiliary drone uses a rotor drone that supports hovering as a flight platform. The auxiliary drone is equipped with a projection screen module, a flight path calculation module, a positioning and attitude module, and a wireless communication module. The auxiliary drone working modes are divided into two types: a follow mode and a warning mode, and an auxiliary drone is equipped with a projection screen;

[0043] The inspection drones and auxiliary drones are matched one by one to form an inspection system, and the inspection system has an inspection working mode and an alarm mode;

[0044] In the inspection working mode, the inspection drone inspects along the highway. Its visual detection module captures video image data of the road ahead and automatically identifies the road conditions. The auxiliary drone follows the inspection drone.

[0045] In the inspection working mode, when the inspection drone detects abnormal situations such as dangerous road conditions or traffic accidents on the road, the system switches to warning mode. The inspection drone sends its coordinate position to the auxiliary drone. The auxiliary drone opens the projection screen behind the inspection drone and sends its position to the inspection drone. The inspection drone flies behind the auxiliary drone and detects the appearance of the projection screen. Its projection module then projects a warning image onto the projection screen to alert the drivers of the vehicles behind to the accident ahead and avoid danger.

[0046] In specific implementation: the visual module of the inspection drone is a camera with zoom function installed on a three-axis motion gimbal, and its field of view supports: short focus can cover the width of the road to facilitate automatic recognition of road conditions, and long focus supports covering the range of the auxiliary drone projection screen to facilitate detection of the auxiliary drone projection screen and positioning to achieve the transformation of the projection screen; in a specific embodiment, the camera includes two types: visible light camera and infrared camera, which support obtaining good quality video images both during the day and at night;

[0047] The visual module outputs the current camera gimbal posture (angles of three-axis rotation), namely, pitch angle, yaw angle, and roll angle;

[0048] The visual module includes a recognition processing unit, which processes the video images taken by the camera in real time, and automatically detects and recognizes abnormal conditions such as dangerous road conditions or traffic accidents on the road surface; in a specific embodiment, the recognition processing unit of the visual module is implemented using a deep learning CNN detection model; the recognition processing unit automatically detects the road ahead and lane lines based on the video images taken by the camera, and automatically plans the inspection path according to traveling along the outermost lane line;

[0049] The projection module is a projection device fixedly installed on the inspection drone; the projection module is specifically a laser projector to ensure that the brightness can also be clear in the daytime;

[0050] The projection module further includes a warning image generation unit, which sets projection transformation parameters according to the position and posture relationship between the inspection drone and the projection screen of the auxiliary drone obtained by the inspection drone, generates an image after projection transformation, and projects it onto the projection screen of the auxiliary drone by the projector, ensuring that the image on the projection screen is not deformed;

[0051] The projection screen module of the auxiliary drone is realized by an electric rolling curtain. In order to reduce the impact of strong winds outdoors and beside the road, a counterweight is installed at the lower end of the curtain. When the weather conditions are good, the projection screen module is mounted with a fog curtain or a water curtain device to form a fog curtain or a water curtain to project the display image. In the following mode of the auxiliary drone, the projection screen is in a retracted or closed state. When in the warning mode, the auxiliary drone lowers the curtain or opens the fog curtain / water curtain after reaching the calculated warning position. The plane of the projection screen is as perpendicular to the road direction as possible or tilted slightly forward, so that the driver can see the warning picture on the projection screen.

[0052] The flight path calculation module is used to calculate the flight path of the following inspection drone in the follow mode and calculate the appropriate projection position in real time; in the warning mode, it calculates the road direction and the tilt direction of the projection screen at the projection position;

[0053] The positioning and attitude determination modules of the inspection UAV and the auxiliary UAV obtain the position information (x, y, z coordinates) and attitude information (pitch, roll, yaw angle) of the corresponding UAV, and output it to the other UAV of the paired inspection system. The positioning and attitude determination module can be implemented by using the existing GPS or Beidou navigation system, gyroscope, and accelerometer, which is a mature technology and will not be described in detail;

[0054] The wireless communication module is used to communicate with another drone of the paired inspection system to mutually transmit position and posture information, which is achieved using a wireless WIFI network.

[0055] The drone highway inspection and warning method comprises the following steps:

[0056] Step 1: Initialize the drone’s location information and camera field of view;

[0057] Start the inspection drone and auxiliary drone, and let the inspection drone rise to a suitable inspection height. The current position of the inspection drone is used as the coordinate origin of the inspection. Figure 3 As shown, the coordinate axis X direction corresponds to the GPS coordinate longitude direction, the Y direction corresponds to the latitude direction, and the Z direction corresponds to the height direction; record the latitude and longitude values ​​output by the positioning and attitude determination module of the current position of the inspection drone

[0058] Step 2: The inspection drone inspects along the road;

[0059] The inspection drone flies along the outermost lane line of the road. Its visual module captures video image data of the road ahead, and the recognition processing unit automatically identifies the road conditions and transmits its location information to the auxiliary drone in real time.

[0060] Among them, the inspection route of the inspection drone along the road can be set in advance by the operator according to the road coordinates of the GIS map; the inspection drone's recognition processing unit can also be used to process the video image taken by its visual module camera, automatically detect the road and lane lines ahead, and automatically plan the inspection route along the outermost lane line. This type of technology is relatively mature in the field of autonomous driving and will not be described in detail.

[0061] The automatic detection and identification of abnormal situations such as dangerous road conditions or traffic accidents on the road are preferably implemented using a deep learning target detection model, such as the YOLO series models, the Swin Transformer model, etc.

[0062] During the inspection flight of the inspection drone, its positioning and attitude module obtains its current latitude and longitude at each moment. And calculate the coordinate position in the coordinate system of the origin established in step 1 initialization based on the longitude and latitude, and send its coordinate position to the auxiliary drone in real time. The coordinate position calculation method is as follows:

[0063]

[0064] in t i-1 The location coordinates at the moment, that is, the location coordinates recorded last time, The latitude and longitude values ​​given by the current GPS module, t i-1 The latitude and longitude values ​​at the time, R is the radius of the earth, usually 6371000 meters;

[0065] Step 3: Assist the drone to follow;

[0066] The auxiliary drone receives the coordinate position of each moment sent by the inspection drone in real time through the wireless communication module. According to these coordinate positions, delayed flight to each point is realized to achieve follow-up flight; the delay time length is set according to the flight speed, for example, 2 seconds, to avoid collision between the auxiliary drone and the inspection drone; at the same time, the auxiliary drone records each coordinate position point during the flight to form a completed flight path;

[0067] Step 4: Assist the drone to calculate the rear warning point relative to the current position;

[0068] The auxiliary drone calculates the appropriate warning point on the previous path relative to the current position based on the completed flight path recorded in step 3. The design principle is to avoid warnings at large bends in the road so that drivers can see the warning picture in advance on relatively straight road sections, leaving enough time for other appropriate actions such as pulling over or slowing down. The calculation method is as follows:

[0069] According to a set of coordinate point data recorded in step 3 above Use the least squares method to fit the cubic curve: y = a3x 3 +a2x 2 +a1x+a0, the cubic curve is The curvature value at is:

[0070]

[0071] Search the curvature of the curve in reverse order When the curvature of a road section is less than the set threshold and the distance of the road section is long enough, k(x i )<Kt, i∈(j,c) and 0<j<c<t i ,

[0072] and Where Kt is the curvature threshold, Dt is the distance threshold, then (x j ,y j ) is the appropriate warning point location.

[0073] Step 5: The inspection drone determines whether an abnormality is detected;

[0074] If the inspection drone vision module does not detect any abnormality on the road, go to step 2; if it detects abnormality on the road, go to step 6 and enter the warning mode;

[0075] Step 6: Assist the drone to place the projection screen at the warning position;

[0076] The auxiliary drone flies to the warning position (x j ,y j ), hover and put down the projection screen, and then obtain the current position and posture information from its positioning and posture module, and calculate the three-dimensional coordinate position (x mj ,y mj , z mj ) and attitude angle The wireless transmission module is used to send the data to the inspection drone. Since the drone is small, the deviation between the position of its GPS module and the center position of the projection screen in the X-axis and Y-axis directions is not large and can usually be ignored. If you want to get the precise relationship between the two, you can paste the target point at the center of the projection screen before the system works, and use the WGS-84 coordinate system and the local coordinate system of the fuselage to convert and calibrate. This method belongs to the existing technology and will not be described in detail. Z value coordinate z mj The simple calculation method is as follows: mj = -MH / 2, where MH is the height of the projection screen. The UAV can usually maintain level flight during flight. When hovering stably, its pitch and ball angles are usually small and can be ignored. Only the yaw angle needs to be considered, that is, Figure 4 The angle of rotation of the coordinate system around the Z axis

[0077] Step 7: The inspection drone reaches the projection location;

[0078] The inspection drone receives the warning position sent by the auxiliary drone, flies around the auxiliary drone to the appropriate projection position behind the auxiliary drone facing the projection screen. In order to reduce the obstruction of the driver's vision, the inspection drone is farther away from the road than the auxiliary drone and the projection screen. Figure 4 As shown, the projection module mounted on the inspection drone will have an angle θ relative to the normal direction of the projection screen. Figure 5 As shown in the figure, in the XY plane, z pj =z mj , you can get the location of the inspection drone (x pj ,y pj , z pj ), where D is the set distance value, which can be selected according to the flight safety of the drone and the projection range of the projection equipment, for example, 4 meters; the yaw angle of the inspection drone It should be noted that according to Figure 5 For easy understanding here The absolute value is used, which can be negative when applied, that is, counterclockwise rotation around the Z axis is a positive angle, and clockwise rotation is a negative angle;

[0079] Step 8: Calculate the corrected projection image and issue a projection warning;

[0080] The inspection drone projection module corrects the projected image according to its position and attitude relationship with the projection screen mounted on the auxiliary drone, and then projects it onto the projection screen after correction. The trapezoidal correction method can be used for correction. Since the inclination angle of the optical axis of the projection device in the yaw direction is left-leaning θ, a trapezoid with a longer left side than the right side will appear after projection. If the optical axis of the projection device passes through the center of the projection screen, the ratio of the left and right sides of the projection area after the left-leaning θθ projection to the side of the projection at the center position is: Where α is half of the horizontal field of view of the projection device. When the projector is projecting forward, the original screen size is W×H, and the four vertices of its projection range are (0,0), (W,0), (0,H), (W,H); corresponding to the left tilt θ, the four vertices become (0,-H*(r l -1) / 2)、(W,H*(1-r r ) / 2)、(0,H*(r l +1) / 2)、(W,H*(1+r r ) / 2), the corresponding correction transformation matrix can be calculated according to the coordinates of the four vertices in the original projection range and the four vertices after the change. When correcting, multiplying the transformation matrix can get the image to be projected. The method of calculating the transformation matrix can be realized by using the perspective transformation function getPerspectiveTransform in OpenCV.

[0081] This technology uses drones as a platform and is equipped with a visual module to automatically identify and report highway road and traffic conditions, reducing manual operations and improving work efficiency. When the drone identifies abnormal road and traffic conditions, it uses drone projection display to warn drivers of vehicles behind in real time, avoiding or reducing the occurrence of traffic accidents.

[0082] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0083] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A drone highway inspection and warning device, characterized in that: It includes: Inspection drones, which are equipped with visual inspection modules and projection modules; and an auxiliary drone, which carries a projection screen; The inspection drones and auxiliary drones are matched one by one to form an inspection system, and the inspection system has an inspection working mode and an alarm mode; In the inspection working mode, the inspection drone inspects along the highway. Its visual detection module captures video image data of the road ahead and automatically identifies the road conditions. The auxiliary drone follows the inspection drone. In the inspection working mode, when the inspection drone detects abnormal situations such as dangerous road conditions or traffic accidents on the road, the system switches to warning mode. The inspection drone sends its coordinate position to the auxiliary drone. The auxiliary drone opens the projection screen behind the inspection drone and sends its position to the inspection drone. The inspection drone flies behind the auxiliary drone and detects the appearance of the projection screen. Its projection module then projects a warning image onto the projection screen to alert the drivers of the vehicles behind to the accident ahead and avoid danger.

2. The unmanned aerial vehicle highway inspection and warning device according to claim 1 is characterized by: The inspection UAV adopts a rotor UAV that supports hovering as a flight platform, and the inspection UAV is equipped with a vision module, a projection module, a positioning and attitude module, and a wireless communication module.

3. The unmanned aerial vehicle highway inspection and warning device according to claim 2 is characterized by: The visual module is a camera with zoom function installed on a three-axis motion gimbal, and its field of view supports: The short focus can cover the width of the road so as to capture the road conditions for automatic identification, and the long focus can cover the range of the auxiliary drone projection screen so as to detect the auxiliary drone projection screen and locate it to achieve the change of the projection screen; The visual module outputs the current camera gimbal posture, namely the pitch angle, yaw angle, and roll angle; The visual module includes a recognition processing unit, which processes the video images taken by the camera in real time and automatically detects and recognizes abnormal situations such as dangerous road conditions or traffic accidents on the road.

4. The unmanned aerial vehicle highway inspection and warning device according to claim 3 is characterized by: The recognition processing unit can automatically detect the road ahead and lane lines based on the video images captured by the camera, and automatically plan an inspection path by traveling along the outermost lane line.

5. The unmanned aerial vehicle highway inspection and warning device according to claim 2 is characterized by: The projection module is a projection device fixedly installed on the inspection drone; The projection module also includes a warning image generation unit, which sets projection transformation parameters according to the position and posture relationship between the inspection drone and the projection screen of the auxiliary drone obtained by the inspection drone, and generates an image after projection transformation, which is projected by the projector onto the projection screen of the auxiliary drone.

6. The unmanned aerial vehicle highway inspection and warning device according to claim 2 is characterized by: The auxiliary UAV adopts a rotor UAV that supports hovering as a flight platform. The auxiliary UAV is equipped with a projection screen module, a flight path calculation module, a positioning and attitude determination module, and a wireless communication module. The working modes of the auxiliary UAV are divided into two types: a following mode and a warning mode.

7. The unmanned aerial vehicle highway inspection and warning device according to claim 6 is characterized by: The projection screen module adopts a curtain, a fog curtain or a water curtain; in the following mode of the auxiliary drone, the projection screen module is in a retracted or closed state, and when in the warning mode, the auxiliary drone lowers the curtain or opens the fog curtain / water curtain after reaching the calculated warning position.

8. The unmanned aerial vehicle highway inspection and warning device according to claim 6 is characterized by: The flight path calculation module is used to calculate the flight path of the following inspection drone in the following mode and calculate the appropriate projection position in real time; in the warning mode, it calculates the road direction and the tilt direction of the projection screen at the projection position.

9. The unmanned aerial vehicle highway inspection and warning device according to claim 6, characterized in that: The wireless communication module is used to communicate with another drone of the paired inspection system to mutually transmit position and posture information, which is achieved using a wireless WIFI network.

10. A drone highway inspection and warning method, which uses a drone highway inspection and warning device as claimed in any one of claims 1 to 9, characterized in that: It includes the following steps: Step 1: Initialize the drone’s location information and camera field of view; Step 2: The inspection drone inspects along the road; Step 3: Assist the drone to follow; Step 4: Assist the drone to calculate the rear warning point relative to the current position; Step 5: The inspection drone determines whether an abnormality is detected; Step 6: Assist the drone to place the projection screen at the warning position; Step 7: The inspection drone reaches the projection location; Step 8: Calculate the corrected projection image and issue a projection warning; If the inspection drone does not detect an abnormality, after step five, the device switches to step two to perform the inspection working mode; if the inspection drone detects an abnormality, after step five, the device sequentially enters step six, step seven, and step eight until the road abnormality is resolved and enters the inspection working mode again.

Images