Unmanned aerial vehicle refined station inspection path planning method based on deep learning

Abstract

The invention discloses an unmanned aerial vehicle refined station inspection path planning method based on deep learning. The method comprises the following steps: S100, initial modeling of an unattended station; and S200, performing intelligent route planning based on the obtained model. According to the method, the work of manually drawing the route is greatly reduced, and the method is more suitable for fine inspection in various complex environments.

Description

Technical Field

[0001] The present disclosure belongs to the technical fields of computer vision and image processing, and particularly relates to a method for refined inspection path planning of an unmanned aerial vehicle (UAV) for a station yard based on deep learning. Background Art

[0002] In recent years, with the continuous growth of domestic demand for oil and gas, the scale of oil and gas exploration and production has been expanding. With the continuous progress of automation technology, the level of oil and gas production has been greatly improved. Especially with the popularization and application of Internet of Things (IoT) and communication technologies in oil and gas field management, unmanned station yards are becoming the mainstream mode of oil and gas field operation management. With the wide application of technologies such as industrial Internet, IoT, 5G, and big data, however, some problems still need to be observed externally to ensure the normal operation of automation equipment.

[0003] Currently, for unmanned station yards and wells, each oil and gas field unit still retains a small number of inspection personnel to conduct on-site inspections. However, unmanned station yards are usually distributed in the wild, in complex and remote areas such as mountains, hills, and deserts. Manual inspection is difficult, with high labor intensity, low efficiency, and high inspection costs.

[0004] Therefore, UAV inspection has become an ideal solution. However, the current technical solutions are difficult to achieve the purpose of refined inspection. In existing similar technical solutions, before inspection, based on the coordinate information of the inspection station yard and well, the UAV inspection points are pre-entered, and a series of parameters such as height, direction, pan-tilt angle, zoom ratio, etc. are set for the inspection point to conduct the inspection. In this solution, the camera of the UAV is difficult to focus on refined detail targets, such as valve instruments, etc. Since each inspection point needs to be manually entered, there is a high probability that the flight path will be incorrect due to insufficient work experience of the personnel planning the inspection path, and in severe cases, there may even be an unreasonable flight path collision between inspection points and on-site equipment. Summary of the Invention

[0005] To solve the above problems, the present disclosure provides a method for refined inspection path planning of an unmanned aerial vehicle for a station yard based on deep learning, which includes the following steps:

[0006] S100: Initial modeling of the unmanned station yard;

[0007] S200: Intelligent flight path planning based on the obtained model;

[0008] The S200 further includes the following steps:

[0009] S201: Collecting laser point clouds and oblique photography images of the target area to be inspected;

[0010] S202: Establish the correspondence between image pixels and point clouds;

[0011] S203: Identify the target to be inspected from the oblique photography images, extract the central pixel coordinates of the target, and convert them into spatial coordinates;

[0012] S204: Remove duplicates from the spatial coordinates of the center points of the above inspection targets, then calculate the normal vector of this point based on the 3D model, and thus obtain the waypoints of the UAV flight path. Connect all the inspection waypoints to create a flight path;

[0013] S205: Based on the natural language large model algorithm, modify the key parameters of the waypoints based on natural language, and adjust and update the flight path information.

[0014] In addition, the present invention also discloses a refined UAV field inspection path planning device based on deep learning, including:

[0015] A device for initial modeling of unmanned stations;

[0016] A device for intelligent flight path planning based on the obtained model;

[0017] The device for intelligent flight path planning based on the obtained model further includes:

[0018] A device for collecting laser point clouds and oblique photography images of the target area to be inspected;

[0019] A device for establishing the correspondence between image pixels and point clouds;

[0020] A device for identifying the target to be inspected from the images and converting it into spatial coordinates;

[0021] A device for removing duplicates from the spatial coordinates of the center points of the above inspection targets, then calculating the normal vector of this point based on the 3D model, and thus obtaining the waypoints of the UAV flight path. Connect all the inspection waypoints to create a flight path;

[0022] A device for modifying the key parameters of the waypoints based on natural language using the natural language large model algorithm, and adjusting and updating the flight path information.

[0023] In addition, the present invention also discloses a computer storage medium, wherein the storage medium includes computer instructions, and when it runs on a computer, it causes the computer to execute the above method.

[0024] In addition, the present invention also discloses an electronic device, wherein the electronic device includes:

[0025] A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein,

[0026] When the processor executes the program, the method described above is implemented.

[0027] This method applies natural language processing (NLP) and computer vision (CV) technologies, which can simplify the route planning method. The automatic generation of the entire task can be completed through simple configuration work, greatly reducing the work of manually drawing routes, and is more suitable for fine inspection in various complex environments. Brief Description of the Drawings

[0028] Figure 1 It is a flowchart of a method for fine inspection path planning of an unmanned aerial vehicle (UAV) for a station yard based on deep learning provided in an embodiment of the present disclosure;

[0029] Figure 2 It is a distribution diagram of photos corresponding to target points provided in an embodiment of the present disclosure;

[0030] Figure 3 It is a schematic diagram of automatically planning the position of waypoints based on a three-dimensional space algorithm provided in an embodiment of the present disclosure;

[0031] Figure 4 It is a schematic diagram of the intelligent route of the UAV provided in an embodiment of the present disclosure. Detailed Embodiments

[0032] In one embodiment, as Figure 1 shown, the present disclosure provides a method for fine inspection path planning of an unmanned aerial vehicle (UAV) for a station yard based on deep learning, which includes the following steps:

[0033] S100: Initial modeling of an unmanned station yard;

[0034] S200: Intelligent route planning based on the obtained model;

[0035] The S200 further includes the following steps:

[0036] S201: Collect the laser point cloud and oblique photography pictures of the target area to be inspected;

[0037] S202: Establish the correspondence between picture pixels and the point cloud;

[0038] S203: Identify the target to be inspected from the oblique photography pictures, extract the central pixel coordinates of the target, and convert them into spatial coordinates;

[0039] S204: Remove duplicates from the spatial coordinates of the center points of the above-mentioned inspection targets, then calculate the normal vector of the point based on the three-dimensional model, and further obtain the waypoints of the UAV route. Connect all the inspection waypoints to create a route;

[0040] S205: Modify the key waypoint parameters based on natural language through the natural language large model algorithm, and adjust and update the route information.

[0041] For this embodiment, the method can adaptively plan a refined inspection route for the oil and gas field station scenario, and can optimize the route in subsequent line inspection tasks, improve the route planning efficiency and avoid collision safety accidents. The algorithm has good computing performance and stronger environmental adaptability.

[0042] The method includes: establishing a UAV inspection system, selecting a multi-rotor UAV to perform flight tasks, and the UAV has an algorithm module; collecting the coordinate positions of the oil and gas field stations, collecting lidar and oblique photography data of the stations to be inspected and performing modeling processing; importing the modeling data into the UAV inspection system to establish a mapping relationship between the three-dimensional spatial point cloud and photos of the stations. Based on the pre-trained image recognition algorithm, automatically identify the inspection targets and confirm the targets; based on the pre-trained route planning algorithm, set the parameters of each waypoint and automatically connect them to form a route path, and avoid the possibility of collision of the model; input the route adjustment description text information, and adjust and update the route information through the natural language large model algorithm; after confirming the route information, the UAV uploads the route, completes the route flight and synchronously stores it in the cloud server. After multiple inspection tasks, the route stored in the cloud will be automatically updated to the optimal route for the current inspection target.

[0043] Among them, in step S203, identify the target to be inspected from the photographic image, extract the central pixel coordinates of the target, and convert them into spatial coordinates, marked as P i . In step S204, calculate the waypoint information, collect all the spatial coordinates of the center points of the above inspection targets, and perform deduplication processing (for example, calculate the distance between two coordinates. If it is less than the threshold, for example, 50 meters, then remove one coordinate point), and record the processed coordinate points as {P1, P2,..., P N}}, then calculate the normal vector of this point based on the three-dimensional model, extend each point a certain distance (for example, 20 meters) to obtain the waypoints of the UAV route, recorded as {P ′ 1, P ′ 2,..., P ′ N}, at the same time, reverse the direction of the normal to calculate the photographing direction of the waypoint (the algorithm is to resolve the spatial vector into Euler angles) and create a route.

[0044] In another embodiment, the S100 further includes the following steps:

[0045] S101: Establish a UAV inspection system;

[0046] S102: Obtain radar data and oblique photography pictures by using the method of lidar fusion with oblique photography, and perform point cloud modeling on the area to be inspected in the unmanned aerial vehicle (UAV) - manned station yard;

[0047] S103: Import the point cloud modeling data into the UAV inspection system, obtain the coordinate values of each three - dimensional space point in the relative coordinate system of the area to be inspected in the station yard, and establish the mapping relationship between the three - dimensional space points and each picture;

[0048] S104: First, train the target recognition algorithm model to identify common targets to be inspected, and then train the natural language processing model to improve the flight path instructions.

[0049] For this embodiment, for the refined inspection of the station yard, the method of lidar fusion with oblique photography is adopted to obtain radar data and original photos, and point cloud modeling is performed on the inspection area of the UAV - manned station yard. During the modeling process, ensure that the inspection targets have sufficient point cloud density: the point cloud density in the high - density area of the inspection points is not less than 500 pts / m 2 . The point cloud density in the low - density area of the flight environment is not less than 80 pts / m 2 .

[0050] Basic data preparation: Point cloud data processing. After the modeling is completed, obtain the coordinate (x, y, z) values of each three - dimensional space point in the relative coordinate system of the inspection area in the station yard. Establish the mapping relationship between the three - dimensional space points and each picture, that is, each pixel point on the picture can correspond to a unique spatial coordinate point, as Figure 2 shown.

[0051] Image data processing. Pre - process the collected images, including cropping, scaling, denoising, etc., to improve the efficiency and accuracy of subsequent processing.

[0052] Model training: Use two types of deep - learning algorithm models.

[0053] First, it is necessary to select a target recognition algorithm model, such as a convolutional neural network (e.g., YOLO series models, or Faster R - CNN models). Train the model with the goal of being able to identify common targets to be inspected in the imported photos, such as: valves, instruments, signs, switches, etc.

[0054] Secondly, use natural language to reduce the complexity of UAV flight path creation and improve the flight path instructions. Therefore, first select a set of natural language processing (NLP) systems, such as selecting the open - source BLOOM model. The training steps are as follows:

[0055] 1) Training data preparation. It is necessary to prepare a file (such as in JSON or CSV format), and the data format can be dialogue data, instruction data, etc., for example:

[0056] {

[0057] "instruction": "Increase the zoom factor",

[0058] "input": "Take a larger photo of the instrument at the first inspection point",

[0059] "output": "Call adjust_zoom(1, 1.5)"

[0060] }

[0061] The above data indicates that if natural language similar to \"Adjust the zoom factor\" or \"Take a larger photo of the instrument at the first inspection point\" is detected, the call the back-end algorithm adjust_zoom to adjust the zoom.

[0062] 2) Adjust the model parameters and train the model as follows:

[0063] from transformers import Trainer, TrainingArguments

[0064] # Define the training parameters

[0065] training_args = TrainingArguments(

[0066] output_dir = ". / results",

[0067] per_device_train_batch_size = 4,

[0068] num_train_epochs = 3,

[0069]

[0070] if method_name == "adjust_zoom":

[0071] adjust_zoom(*params)

[0072] elif method_name == "adjust_height":

[0073] adjust_height(*params)

[0074] else:

[0075] print("Unknown method")

[0076] # Example background method

[0077] def adjust_zoom(point_id, zoom_level):

[0078] print(f"Adjust the zoom factor of inspection point {point_id} to {zoom_level}")

[0079] def adjust_height(point_id, height):

[0080] print(f"Adjust the height of inspection point {point_id} to {height}")

[0081] # Execute method call

[0082] execute_method_call(method_call)

[0083] Evaluate the accuracy rate of the test results. If it meets the usage requirements (for example, greater than 90%), stop the training; otherwise, continue iterative training or increase the training parameters.

[0084] In another embodiment, S203 further includes automatically identifying the target to be inspected based on a pre-trained image recognition algorithm and confirming the target.

[0085] For this embodiment, after importing the point cloud model results and oblique photography pictures into the UAV route planning platform, call the model data, select the target points according to the requirements for automatic route planning, and perform optimization and adjustment by setting corresponding parameters and rules.

[0086] Target point recognition and selection: After importing the point cloud model and oblique photography pictures into the UAV route planning platform, click on the target points that need to be inspected with refined photography. The system automatically recognizes and marks the targets to be inspected based on the visual algorithm model, such as items like valves, instruments, signs, switches, etc.

[0087] Convert the pixel coordinates of the image target into spatial point cloud coordinates, and perform deduplication to obtain the sequence of target points to be inspected {P1, P2,..., P n}.

[0088] In another embodiment, S204 further includes setting various parameters for each waypoint based on a pre-trained route planning algorithm, automatically connecting to form a flight path, and avoiding the possibility of model collision.

[0089] For this embodiment, waypoint information calculation: After the user confirms that the target recognition of each inspection point is correct, the system calculates an appropriate shooting direction according to the three-dimensional space algorithm. This algorithm calculates the normal through the point cloud data near the target point P i and then extends a certain safety distance d along the normal direction and then checks whether the minimum distance d between the current position and the target point cloud safe is greater than or equal to d i If not, continue to extend a certain distance Δd (for example, 1 meter) outward and iterate until d safe is satisfied i ≥d safe .

[0090] According to the flight altitude, camera parameters, the proportion of the inspection target in the frame, etc., the waypoint information is automatically determined, as Figure 3 shown

[0091] Path planning and optimization: Connect the paths of all inspection waypoints, calculate the shortest distance between the connections of each waypoint, and ensure that the distance between the connection path of two waypoints and the closest point of the point cloud model meets the minimum safety distance requirement. The goal of path planning is to minimize the total flight distance D, that is:

[0092]

[0093] where (x i , y i , z i ) are the coordinates of the i-th waypoint

[0094] In another embodiment, step S200 further includes:

[0095] Adaptive modification of the default inspection model to obtain the optimal flight path

[0096] For this embodiment, adaptive model optimization: The system algorithm model records the input of the user's required parameters and adaptively modifies the default inspection model. Compare various algorithm paths and recommend the most suitable flight path. The goal of optimization is the shortest path, and the constraint condition is that the distance between each waypoint and the shooting target is greater than the preset safety distance (for example, d safe = 20 meters), as follows:

[0097]

[0098] where d i is the distance between the i-th waypoint and the closest point of the point cloud model, and d safe is the minimum safety distance

[0099] In another embodiment, the waypoint key parameters include adjusting the zoom ratio and adjusting the waypoint height.

[0100] For this embodiment, the model parameters are adjusted as follows: If the preset model parameters are not appropriate, the algorithm model parameters preset by the system are adjusted through natural language prompt words. For example:

[0101] 1) Adjusting the zoom ratio: For example, "Make the instrument in the first inspection point look larger", and the system will adjust the zoom ratio zoom of the corresponding waypoint;

[0102] 2) Adjusting the waypoint height: For example, "Raise all waypoints a little", and the system will adjust the height height of all waypoints and recalculate the azimuth angle parameters and zoom parameters of the pan-tilt camera.

[0103] Through the above algorithm, a set of route files for each target point in the field station is formed, as Figure 4 shown, a route file with 21 waypoints is generated for 21 equipment inspection target points in a certain field station.

[0104] In another embodiment, the waypoint information includes: the longitude, latitude and altitude of the UAV, the attitude angles of the UAV pan-tilt camera, and the zoom ratio.

[0105] For this embodiment, the waypoint information includes:

[0106] 1) The longitude lon, latitude lat and altitude height of the UAV.

[0107] 2) The attitude angles (roll, pitch, yaw) of the UAV pan-tilt camera.

[0108] 3) The zoom ratio: zoom.

[0109] In another embodiment, a device for fine-tuning the inspection path of a UAV for a field station based on deep learning includes:

[0110] A device for initial modeling of an unmanned field station;

[0111] A device for intelligent route planning based on the obtained model;

[0112] The device for intelligent route planning based on the obtained model further includes:

[0113] A device for collecting laser point clouds and oblique photography pictures of the target area to be inspected;

[0114] A device for establishing the correspondence between picture pixels and point clouds;

[0115] A device for identifying a target to be inspected from an oblique photography image, extracting the central pixel coordinates of the target, and converting them into spatial coordinates;

[0116] A device for de-duplicating the spatial coordinates of the center point of the above-mentioned inspection target, then calculating the normal vector of this point based on a 3D model, and further obtaining the waypoints of the UAV flight path, and connecting all the inspection waypoints to create a flight path;

[0117] A device for modifying the key parameters of the waypoints based on natural language through a natural language large model algorithm, and adjusting and updating the flight path information.

[0118] For this embodiment, the system can automatically generate and optimize the inspection flight path to ensure the efficiency and safety of the inspection process.

[0119] In addition, the present invention also discloses a computer storage medium, wherein the storage medium includes computer instructions, which when run on a computer, cause the computer to execute any of the methods described above.

[0120] In addition, the present invention also discloses an electronic device, wherein the electronic device includes:

[0121] A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein,

[0122] When the processor executes the program, it implements any of the methods described above.

[0123] Although the embodiments of the present invention have been described above in conjunction with the accompanying drawings, the present invention is not limited to the above specific embodiments and application fields. The above specific embodiments are merely illustrative and guiding, rather than restrictive. Those of ordinary skill in the art can also make many forms under the inspiration of this specification and without departing from the scope protected by the claims of the present invention, and these all belong to the scope of protection of the present invention.

Claims

1. A method for fine - grained inspection path planning of an unmanned aerial vehicle (UAV) for a station yard based on deep learning, which comprises the following steps: S100: Initial modeling of an unmanned station yard; S200: Intelligent route planning based on the obtained model; The S200 further comprises the following steps: S201: Collect laser point clouds and oblique photography pictures of the target area to be inspected; S202: Establish the correspondence between picture pixels and point clouds; S203: Identify the targets to be inspected from the oblique photography pictures, extract the central pixel coordinates of the targets, and convert them into spatial coordinates; S204: Remove duplicates from the spatial coordinates of the central points of the above - mentioned inspection targets, then calculate the normal vector of this point based on the 3D model, and further obtain the waypoints of the UAV route. Connect all the inspection waypoints to create a route; S205: Based on the natural language large - model algorithm, modify the key parameters of the waypoints based on natural language, and adjust and update the route information.

2. According to the method described in claim 1, the S100 further comprises the following steps, preferably: S101: Establish a UAV inspection system; S102: Use the method of fusing lidar with oblique photography to obtain radar data and oblique photography pictures, and perform point cloud modeling on the area to be inspected in the UAV - manned station yard; S103: Import the point cloud modeling data into the UAV inspection system, obtain the coordinate values of each three - dimensional space point in the relative coordinate system of the area to be inspected in the station yard, and establish the mapping relationship between the three - dimensional space points and each picture; S104: First train the target recognition algorithm model to recognize common targets to be inspected, and then train the natural language processing model to improve the route instructions.

3. According to the method described in claim 1, the S203 further comprises automatically identifying the targets to be inspected based on a pre - trained image recognition algorithm and confirming the targets.

4. According to the method described in claim 1, the S204 further comprises setting the parameters of each waypoint based on a pre - trained route planning algorithm, automatically connecting to form a route path, and avoiding the possibility of collision of the model.

5. According to the method described in claim 1, step S200 further comprises: Adaptive modification of the default inspection model to obtain the optimal route path.

6. According to the method described in claim 1, the key parameters of the waypoints include adjusting the zoom ratio and adjusting the altitude of the waypoint.

7. The method according to claim 1, wherein the waypoint information comprises: The longitude, latitude and altitude of the UAV, the attitude angles of the UAV gimbal camera, and the zoom ratio.

8. A device for fine - grained inspection path planning of an unmanned aerial vehicle (UAV) for a station yard based on deep learning, comprising: A device for initial modeling of an unmanned station yard; A device for intelligent route planning based on the obtained model; The device for intelligent route planning based on the obtained model further comprises: a device for collecting laser point clouds and oblique photography pictures of the target area to be inspected; A device for establishing the correspondence between picture pixels and point clouds; A device for identifying the targets to be inspected from the oblique photography pictures, extracting the central pixel coordinates of the targets, and converting them into spatial coordinates; A device for removing duplicates from the spatial coordinates of the center point of the above-mentioned inspection target, then calculating the normal vector of this point based on the 3D model, and further obtaining the waypoints of the UAV flight path, and connecting all the inspection waypoints to create a flight path; A device for modifying the key parameters of waypoints based on natural language and adjusting and updating flight path information through natural language large model algorithms.

9. A computer storage medium, wherein, The storage medium includes computer instructions, which when running on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

10. An electronic device, wherein, The electronic device includes: A memory, a processor, and a computer program stored on the memory and executable on the processor, wherein When the processor executes the program, it implements the method according to any one of claims 1 to 7.

Images