Ground-air cooperative inspection method and system for oil depot inspection

Abstract

The invention provides a ground-air cooperative inspection method and system for oil depot inspection, and the method comprises the steps: determining a corresponding inspection position based on the coordinates of a selected inspection point in a pre-constructed three-dimensional model of an oil depot; determining an unmanned aerial vehicle task set and a ground task set based on the inspection position corresponding to each coordinate; constructing a ground inspection route based on the ground task set and the parking position of the inspection robot; constructing a flight inspection route based on the ground inspection route and the unmanned aerial vehicle task set, wherein the flight inspection route comprises a ground inspection route and a target inspection route; carrying out image recognition based on an image returned by the unmanned aerial vehicle in the flight process of the ground patrol route, and judging whether the corresponding sub-route is blocked or not; and if the sub-route is blocked, marking the sub-route, and updating the ground inspection route, so that the inspection robot performs inspection based on the updated ground inspection route.

Description

Technical Field

[0001] The present invention relates to the field of robot inspection technology, and in particular to a ground-to-air collaborative inspection method and system for oil depot inspection. Background Art

[0002] In the energy supply system, oil depots are the core of storage and allocation of petroleum and its products. Their safe and stable operation is related to the continuity of energy supply, the normal operation of social economy, and the safety of people’s lives and property. Inspection of oil depots is an important line of defense to ensure all of this, and its importance cannot be ignored.

[0003] From a safety perspective, the oil stored in the oil depot has dangerous characteristics such as being flammable, explosive, and volatile. Once a leak, fire, or explosion occurs, the consequences will be disastrous. Inspection work is like a "physical examination" for the oil depot. Through a detailed inspection of equipment and facilities such as oil tanks, pipelines, valves, and pumps, potential hidden dangers can be discovered in a timely manner. For example, check the corrosion of the oil tank. If rust or cracks are found on the tank body, timely repair can prevent oil leakage from causing a fire; check whether the pipeline connection is well sealed to prevent oil and gas from volatilizing and gathering to form an explosive mixed gas. Only through regular inspections can safety hazards be nipped in the bud and a solid safety barrier can be built for the oil depot.

[0004] In terms of equipment maintenance, inspection is the key to ensuring the normal operation of oil depot equipment. Various equipment in the oil depot is in operation for a long time, which is prone to problems such as wear and aging. Inspection personnel can judge whether the equipment is in normal working condition by observing the operating parameters, sound, temperature and other indicators of the equipment. For example, monitoring the vibration frequency and temperature of the pump. If any abnormality is found, timely inspection and maintenance can avoid downtime accidents caused by equipment failure, ensure the normal receipt and storage of oil products, and improve the operating efficiency of the oil depot.

[0005] In the original oil depot inspection, manual inspection is usually adopted to reach each inspection position, and then the data of the corresponding position is collected through equipment to complete the inspection work; with the development of technology, in the current oil depot inspection, inspection robots are usually used to carry data collection equipment to complete the inspection work. However, the inspection robot is limited by the length of its own mechanical arm, and the inspection range is limited. In the inspection work, it is difficult to pre-judge the road conditions, and it is easy to be affected by the road conditions during driving, causing the inspection to be interrupted. Summary of the invention

[0006] In view of this, an embodiment of the present invention provides a ground-to-air collaborative inspection method and system for oil depot inspection to eliminate or improve one or more defects existing in the prior art.

[0007] One aspect of the present invention provides a ground-air collaborative inspection method for oil depot inspection, the method comprising the following steps: Obtain the selected inspection points, construct an inspection task including multiple inspection points, and determine the inspection positions corresponding to the coordinates in the three-dimensional model based on the coordinates of the selected inspection points in the pre-constructed three-dimensional model of the oil depot; Determine the UAV task set and the ground task set based on the inspection positions corresponding to each coordinate; Constructing a ground inspection route based on the ground task set and the parking position of the inspection robot, wherein the ground inspection route includes multiple sub-routes; Based on the ground inspection route and the drone task set, a flight inspection route is constructed, wherein the flight inspection route includes a ground inspection route and a target inspection route, wherein the ground inspection route is an aerial projection route of the ground inspection route, and the drone flies along the ground inspection route to collect images of the ground; Based on the images sent back by the drone during the ground inspection route, image recognition is performed to determine whether the corresponding sub-route is blocked; If there is a blockage in a sub-route, the sub-route is marked and the ground inspection route is updated so that the inspection robot performs inspection based on the updated ground inspection route.

[0008] By adopting the above scheme, on the one hand, the oil tanks of the oil depot equipment are inspected through two devices. During one inspection process, the drone can be used to conduct the inspection first. During the inspection process, the drone can inspect the ground inspection route of the inspection robot in the air. If there are obstacles on the route, the ground inspection route will be marked and updated in advance to bypass the sub-route with obstacles. In the subsequent inspection process of the inspection robot, the inspection interruption caused by the obstacles can be avoided.

[0009] In some embodiments of the present invention, in the step of constructing a ground inspection route based on a ground task set and the parking positions of the inspection robot, the inspection positions in the ground task set belonging to the same oil tank are constructed into a ground inspection group, and the parking position of the inspection robot corresponding to each inspection position in the sub-route is determined based on the inspection positions in the ground task set, the parking positions in the same ground inspection group are merged to construct a parking area, and the starting point of the inspection robot is used as the starting point of the ground inspection route. Based on the parking area corresponding to each ground inspection group, the distance from the starting point to each parking area is calculated to determine the inspection order of each ground inspection group; the ground inspection route is determined based on the inspection order.

[0010] In some embodiments of the present invention, in the step of determining the ground inspection route based on the inspection sequence, the sub-route that the inspection robot passes from the starting point to the first docking area, the sub-route that the inspection robot passes from one docking area to another docking area, and the sub-route that the inspection robot passes from the last docking area back to the starting point are sequentially combined as the ground inspection route.

[0011] In some embodiments of the present invention, a flight inspection route is constructed based on the ground inspection route and the drone task set, the flight inspection route includes a ground inspection route and a target inspection route, the ground inspection route is an aerial projection route of the ground inspection route, and the drone flies along the ground inspection route. The step of collecting images of the ground includes: The inspection locations belonging to the same oil tank in the UAV mission set are constructed into an aerial inspection team; For the ground inspection team and the aerial inspection team corresponding to the same oil tank in the ground task set and the drone task set, when the drone flies to the aerial inspection team along the ground inspection route, the drone conducts an inspection at the inspection position of the aerial inspection team; For the aerial inspection team in the drone task set that does not correspond to any oil tank in the ground task set, the distance between the corresponding oil tank of the aerial inspection team and the corresponding oil tank of each ground inspection team in the ground task set is calculated, and the transfer tanks are screened based on the distance value. When the drone flies to the ground inspection team corresponding to the transfer tank, it flies to the corresponding oil tank of the aerial inspection team and inspects the aerial inspection team of the oil tank. After completing the inspection, it returns to the corresponding transfer tank.

[0012] In some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the drone during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, a pre-trained obstacle recognition model is used for recognition, and the obstacle recognition model includes a backbone network, an obstacle detection branch, a width calculation branch and an output fusion layer.

[0013] In some embodiments of the present invention, the backbone network includes a basic convolution block, a downsampling module and a multi-scale feature extraction module arranged in sequence, the basic convolution block is composed of a convolution layer, a batch normalization layer and an activation function; the downsampling module realizes feature map size reduction through a pooling layer or a stride convolution; the multi-scale feature extraction module adopts a cross-level connection structure in FPN, and outputs a multi-level feature map for subsequent branch processing; The obstacle detection branch inputs a multi-level feature map, and the obstacle detection branch includes a feature enhancement layer composed of 1×1 convolution and 3×3 convolution, an upsampling layer and a Sigmoid activation layer, and outputs an obstacle binary segmentation map through the Sigmoid activation layer for marking obstacles; The width calculation branch inputs a multi-level feature map, and the width calculation branch includes an attention layer, a variable convolution layer, and a fully connected layer, and outputs the width of the obstacle through the fully connected layer; The output fusion layer receives the obstacle coordinates of the obstacle binary segmentation map from the detection branch and the obstacle width from the width branch. The output fusion layer includes a concat function layer, a channel attention layer, a spatial attention layer and a fully connected layer, and outputs the structured detection results of the obstacle position and obstacle width through the fully connected layer.

[0014] In some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the UAV during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, when the UAV flies along the ground inspection route and collects images of the ground, the images are transmitted back to identify obstacles within the sub-route in the transmitted images. If obstacles exist, the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route are further identified. Based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, it is determined whether the current sub-route is blocked.

[0015] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route; and whether there is a blockage is determined based on the drivable path width.

[0016] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, a blocking determination value is calculated based on the width of the drivable path and the width of the inspection robot, and the blocking determination value determines whether there is a blockage.

[0017] In some embodiments of the present invention, in the step of calculating the drivable path width based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated using the following formula: ; in, represents the width of the drivable path, D represents the width of the sub-route, r represents the length of the obstacle in the width direction of the sub-route, Indicates the middle position of the sub-route in width direction. Indicates the middle position of the obstacle in the width direction of the sub-route. Indicates the offset of the middle position of the obstacle in the width direction relative to the middle position of the sub-route in the width direction.

[0018] In some embodiments of the present invention, in the step of determining whether there is a blockage based on the width of the drivable path, the blockage determination value is calculated based on the following formula; ; in, Indicates the width of the inspection robot. Indicates the blocking judgment value, Indicates the preset calculation parameters.

[0019] In some embodiments of the present invention, if a sub-route is blocked, the sub-route is marked, and in the step of updating the ground inspection route, the sub-route is deleted from the feasible route, and the exit position of the previous ground inspection team and the entry position of the next ground inspection team are used as the starting point and the end point, and the path of the starting point and the end point is recalculated using the shortest path calculation method.

[0020] In some embodiments of the present invention, in the step of the inspection robot performing inspection based on the updated ground inspection route, the inspection robot collects thermal imaging images and RGB images during the inspection process; For the thermal imaging image, the thermal imaging image is matched with the pre-set standard image of the inspection point to obtain a contrast matrix, and the contrast matrix is ​​input into a pre-trained first neural network model. The first neural network model is used to determine whether a first abnormal state occurs. For the inspection data collected by the inspection robot or the UAV; for the RGB image, it is input into a pre-trained second neural network model. The second neural network model is used to determine whether a second abnormal state occurs.

[0021] The second aspect of the present invention also provides a ground-to-air collaborative inspection system for oil depot inspections, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0022] The third aspect of the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned traffic congestion processing method for a high-speed data center network.

[0023] Additional advantages, purposes, and features of the present invention will be described in part in the following description, and will become apparent to those skilled in the art after studying the following, or may be learned from the practice of the present invention. The purposes and other advantages of the present invention may be specifically pointed out and obtained in the specification and the accompanying drawings.

[0024] Those skilled in the art will appreciate that the objectives and advantages that can be achieved with the present invention are not limited to the above specific description, and the above and other objectives that can be achieved by the present invention will be more clearly understood from the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0026] Figure 1 This is a schematic diagram of the architecture of an implementation method of the ground-air collaborative inspection method for oil depot inspection in this solution; Figure 2 This is a schematic diagram of the architecture of another implementation method of the ground-air collaborative inspection method for oil depot inspection in this solution; Figure 3 Schematic diagram of the implementation steps of step S400 in this solution; Figure 4 A top-down view of a sub-route when calculating the width of the drivable path; Figure 5 This is a schematic diagram of the structure of the electronic equipment of this solution. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0028] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, only structures and / or processing steps closely related to the solutions according to the present invention are shown in the accompanying drawings, while other details that are not closely related to the present invention are omitted.

[0029] like Figure 1 As shown, the present invention proposes a ground-air collaborative inspection method for oil depot inspection, and the steps of the method include: Step S100, obtaining the selected inspection points, constructing an inspection task including multiple inspection points, and determining the inspection positions corresponding to the coordinates in the three-dimensional model based on the coordinates of the selected inspection points in the pre-constructed three-dimensional model of the oil depot; In the specific implementation process, the oil depot is scanned in advance using scanning equipment to scan the oil depot equipment therein, record the location of the oil depot equipment, and build a three-dimensional model of the oil depot; Specifically, during the scanning process of the oil depot, a drone equipped with a laser radar (LiDAR) was used to cruise at a speed of 5m / s to generate a centimeter-level accurate 3D point cloud map (error ≤ 5cm), marking complex areas such as the top of the oil tank and the intersection of pipelines; In the process of 3D modeling, the improved Dubins curve algorithm is applied to smooth the path, eliminate the jagged flight trajectory, and improve the consistency of map construction.

[0030] Specifically, the staff selects inspection points in the system's three-dimensional model, and the system constructs the multiple inspection points selected by the staff into inspection tasks. The system determines the corresponding inspection positions as the tank top, the upper area of ​​the tank body, the lower area of ​​the tank body, the tank bottom or the pipeline, etc. based on the coordinates of the inspection points selected by the staff in the system's three-dimensional model.

[0031] Step S200, determining a UAV task set and a ground task set based on the inspection positions corresponding to the coordinates; Specifically, the inspection positions on the tank top and the upper area of ​​the tank body are constructed as a drone task set; the lower area of ​​the tank body, the tank bottom or the pipeline are constructed as a ground task set.

[0032] Specifically, the upper area of ​​the can body and the lower area of ​​the can body are divided by a preset height threshold.

[0033] Through the above solution, staff can select coordinates by clicking in the three-dimensional map of the system to determine whether to join the drone mission set or the ground mission set.

[0034] Step S300, constructing a ground inspection route based on the ground task set and the parking position of the inspection robot, wherein the ground inspection route includes a plurality of sub-routes; In the specific implementation process, the inspection position is pre-set with a parking position for the inspection robot and an aerial parking position for the UAV.

[0035] Specifically, the parking position of the inspection robot is the ground area, and the aerial parking position of the drone is the aerial area.

[0036] Specifically, the sub-route is a straight driving section, and there is no intersection where the vehicle can turn in the section.

[0037] Step S400, constructing a flight inspection route based on the ground inspection route and the drone task set, wherein the flight inspection route includes a ground inspection route and a target inspection route, wherein the ground inspection route is an aerial projection route of the ground inspection route, and the drone flies along the ground inspection route to collect images of the ground; Through the above scheme, after the UAV reaches the location of the inspected oil tank along the ground inspection route or the target inspection route, the fast random tree algorithm or the A-star algorithm is used to plan a specific flight path.

[0038] Step S500, performing image recognition based on the images transmitted back by the drone during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked; Step S600: If a sub-route is blocked, the sub-route is marked and the ground inspection route is updated so that the inspection robot performs inspection based on the updated ground inspection route.

[0039] By adopting the above scheme, on the one hand, the oil tanks of the oil depot equipment are inspected through two devices. During one inspection process, the drone can be used to conduct the inspection first. During the inspection process, the drone can inspect the ground inspection route of the inspection robot in the air. If there are obstacles on the route, the ground inspection route will be marked and updated in advance to bypass the sub-route with obstacles. In the subsequent inspection process of the inspection robot, the inspection interruption caused by the obstacles can be avoided.

[0040] like Figure 2 As shown, in some embodiments of the present invention, the step of constructing a ground inspection route based on a ground task set and a parking position of an inspection robot includes: Step S310, constructing the inspection positions belonging to the same oil tank in the ground task set into a ground inspection team; Step S320, determining the parking position of the inspection robot corresponding to each inspection position in the sub-route based on the inspection positions in the ground task set, merging the parking positions in the same ground inspection team, and constructing a parking area; Step S330, taking the starting point of the inspection robot as the starting point of the ground inspection route, calculating the distance from the starting point to each docking area based on the docking area corresponding to each ground inspection team, and determining the inspection order of each ground inspection team; Step S340, determining a ground inspection route based on the inspection sequence.

[0041] In some embodiments of the present invention, in the step of determining the ground inspection route based on the inspection sequence, the sub-route that the inspection robot passes from the starting point to the first docking area, the sub-route that the inspection robot passes from one docking area to another docking area, and the sub-route that the inspection robot passes from the last docking area back to the starting point are sequentially combined as the ground inspection route.

[0042] Specifically, a shortest path algorithm is used to calculate a path from a starting point to a first docking area, from one docking area to another docking area, and from the last docking area back to the starting point.

[0043] Through the above scheme, the inspection order of each ground inspection team (i.e., oil tank) is determined by calculating the distance from the starting point to each docking area. After the inspection order is determined, the inspection path between the inspection teams is planned by the shortest path algorithm to ensure the efficiency of the inspection.

[0044] In some embodiments of the present invention, a flight inspection route is constructed based on the ground inspection route and the drone task set, the flight inspection route includes a ground inspection route and a target inspection route, the ground inspection route is an aerial projection route of the ground inspection route, and the drone flies along the ground inspection route. The step of collecting images of the ground includes: like Figure 3 As shown, in some embodiments of the present invention, the ground inspection route is an aerial vertical projection route of the ground inspection route.

[0045] Step S410, constructing the inspection positions belonging to the same oil tank in the drone task set into an aerial inspection team; Step S420, for the ground inspection team and the aerial inspection team corresponding to the same oil tank in the ground task set and the drone task set, when the drone flies to the aerial inspection team along the ground inspection route, the drone inspects the inspection position of the aerial inspection team; Step S430, for the aerial inspection team in the UAV task set that does not correspond to any oil tank in the ground task set, the distance between the corresponding oil tank of the aerial inspection team and the corresponding oil tank of each ground inspection team in the ground task set is calculated, and the transfer tank is selected based on the distance value. When the UAV flies to the ground inspection team corresponding to the transfer tank, it flies to the corresponding oil tank of the aerial inspection team and inspects the aerial inspection team of the oil tank. After completing the inspection, it returns to the corresponding transfer tank.

[0046] Step S440, after returning to the transfer oil tank, continue to fly along the ground inspection route.

[0047] In some embodiments of the present invention, when the drone flies to the ground inspection team corresponding to the diverted oil tank, in the step of flying to the corresponding oil tank of the aerial inspection team, a fast random tree algorithm or an A-star algorithm is used to plan a specific flight path.

[0048] By adopting the above scheme, the drone can complete its own inspection task at the same time during the inspection process. On the one hand, when the drone flies to its own aerial inspection team during the inspection process, it can complete the inspection of its own aerial inspection team. On the other hand, it can determine the diversion position through the diversion tank. When the drone flies close to the oil tank of its own aerial inspection team, it flies to the oil tank of its own aerial inspection team to complete the inspection task, thereby ensuring the efficient completion of the aerial inspection task.

[0049] In some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the drone during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, a pre-trained obstacle recognition model is used for recognition, and the obstacle recognition model includes a backbone network, an obstacle detection branch, a width calculation branch and an output fusion layer.

[0050] In some embodiments of the present invention, the backbone network includes a basic convolution block, a downsampling module and a multi-scale feature extraction module arranged in sequence, wherein the basic convolution block is composed of a convolution layer (Conv), a batch normalization layer (BN) and an activation function (such as ReLU); the downsampling module achieves feature map size reduction through a pooling layer (MaxPooling) or a strided convolution (Strided Conv); the multi-scale feature extraction module adopts a cross-level connection structure in a FPN (feature pyramid network) and outputs a multi-level feature map for subsequent branch processing; Specifically, the backbone network adopts a hierarchical structure such as CSPDarknet, and realizes multi-level perception of obstacles from contour to details through deep and shallow feature fusion (such as CSP module segmentation feature map for divide-and-conquer processing). For example, the shallow layer captures edge textures, and the deep layer extracts semantic information to improve adaptability to complex scenes.

[0051] The obstacle detection branch inputs a multi-level feature map, and the obstacle detection branch includes a feature enhancement layer composed of 1×1 convolution and 3×3 convolution, an upsampling layer and a Sigmoid activation layer, and outputs an obstacle binary segmentation map through the Sigmoid activation layer for marking obstacles; The width calculation branch inputs a multi-level feature map, and the width calculation branch includes an attention layer, a variable convolution layer, and a fully connected layer, and outputs the width of the obstacle through the fully connected layer; Specifically, the width calculation branch introduces deformable convolution in the regression layer, dynamically adjusts the convolution kernel sampling points, accurately captures the geometric features of the obstacle edge, and reduces the width calculation error to within ±3cm.

[0052] The output fusion layer receives the obstacle coordinates of the obstacle binary segmentation map from the detection branch and the obstacle width from the width branch. The output fusion layer includes a concat function layer, a channel attention layer, a spatial attention layer and a fully connected layer, and outputs the structured detection results of the obstacle position and obstacle width through the fully connected layer.

[0053] The output fusion layer performs Kalman filtering tracking on the detection results of consecutive frames, and removes instantaneous outliers based on the width change trend, improving the output stability by 60%. It also designs confidence weights for the output of each branch (such as detection box score + width regression variance), and dynamically adjusts the fusion ratio, reducing the false alarm rate by 22% in occlusion scenarios. Through learnable fusion parameters (such as the gating mechanism), it achieves global optimization from feature extraction to final output, avoiding suboptimal solutions introduced by manual rules.

[0054] like Figure 4 As shown, in some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the UAV during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, when the UAV flies along the ground inspection route and collects images of the ground, the images are transmitted back to identify obstacles within the sub-route in the transmitted images. If obstacles exist, the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route are further identified, and based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, it is determined whether the current sub-route is blocked.

[0055] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route; and whether there is a blockage is determined based on the drivable path width.

[0056] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, a blocking determination value is calculated based on the width of the drivable path and the width of the inspection robot, and the blocking determination value determines whether there is a blockage.

[0057] In some embodiments of the present invention, in the step of calculating the drivable path width based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated using the following formula: ; in, represents the width of the drivable path, D represents the width of the sub-route, r represents the length of the obstacle in the width direction of the sub-route, Indicates the middle position of the sub-route in width direction. Indicates the middle position of the obstacle in the width direction of the sub-route. Indicates the offset of the middle position of the obstacle in the width direction relative to the middle position of the sub-route in the width direction.

[0058] In some embodiments of the present invention, in the step of determining whether there is a blockage based on the width of the drivable path, the blockage determination value is calculated based on the following formula; ; in, Indicates the width of the inspection robot. Indicates the blocking judgment value, Indicates the preset calculation parameters.

[0059] With the above scheme, the UAV patrols the ground inspection route along the ground inspection route, and uses real-time feedback image recognition to identify whether there are obstacles on the road, such as puddles and piled items. If there are obstacles, the obstacle recognition model is used to identify the position and width of the obstacle, and then the drivable width is calculated through the width of the sub-route using the above formula. The drivable width is used to determine whether the remaining width allows the inspection robot to pass normally. It can accurately determine whether there is congestion in the sub-route. If there is congestion, the route is replanned to avoid obstacles blocking normal inspections.

[0060] In some embodiments of the present invention, if a sub-route is blocked, the sub-route is marked, and in the step of updating the ground inspection route, the sub-route is deleted from the feasible route, and the exit position of the previous ground inspection team and the entry position of the next ground inspection team are used as the starting point and the end point, and the path of the starting point and the end point is recalculated using the shortest path calculation method, and the recalculated path consists of at least one sub-route.

[0061] In some embodiments of the present invention, in the step of the inspection robot performing inspection based on the updated ground inspection route, the inspection robot collects thermal imaging images and RGB images during the inspection process; For the thermal imaging image, the thermal imaging image is matched with the pre-set standard image of the inspection point to obtain a contrast matrix, and the contrast matrix is ​​input into a pre-trained first neural network model. The first neural network model is used to determine whether a first abnormal state occurs. For the inspection data collected by the inspection robot or the UAV; for the RGB image, it is input into a pre-trained second neural network model. The second neural network model is used to determine whether a second abnormal state occurs.

[0062] In the specific implementation process, in the step of matching the thermal imaging image with the pre-set standard image of the inspection point to obtain a comparison matrix, the pixel values ​​at the corresponding positions of the thermal imaging image and the standard image are subtracted as the matrix values ​​at the same position in the comparison matrix to obtain the comparison matrix.

[0063] Specifically, the first abnormal state mentioned above is used to determine the difference between the heat distribution in the image and the standard image. If there is a large difference, it means that the first abnormal state has occurred; the second abnormal state mentioned above is used to determine whether corrosion or leakage occurs at the corresponding position. If excessive corrosion or leakage occurs, it is determined that the second abnormal state has occurred.

[0064] The first neural network model and the second neural network model are pre-trained deep learning models, and the deep learning model can be a multi-layer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer model or a graph neural network (GNN).

[0065] On the one hand, this solution uses two types of equipment to inspect the oil tanks of the oil depot equipment. During one inspection, the drone can be the first to conduct the inspection. During the inspection process, the drone can inspect the ground inspection route of the inspection robot in the air. If there are obstacles on the route, the ground inspection route will be marked in advance and updated to bypass the sub-route with obstacles. In the subsequent inspection process of the inspection robot, inspection interruptions caused by obstacles can be avoided.

[0066] During the inspection process of the drone, for inspection points at higher positions, such as the tank top and higher tank body positions, the drone flies to the height above the oil tank or the tank body to collect images, and uses image analysis to determine whether the corrosion situation and temperature distribution are normal. If abnormal, timely maintenance can be carried out. During the inspection process of higher positions, the problem of inspection range limitation caused by the height of the inspection robot's mechanical arm can be solved.

[0067] An embodiment of the present application also provides a ground-to-air collaborative inspection system for oil depot inspections, the system comprising a computer device, the computer device comprising a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0068] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned method for handling traffic congestion for a high-speed data center network.

[0069] like Figure 5 As shown, an embodiment of the present application also provides an electronic device, the device comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the above-mentioned load-balanced power grid computing power network resource scheduling method is implemented.

[0070] The electronic device may include a processor 1201 and a memory 1202 storing computer program instructions.

[0071] Specifically, the processor 1201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0072] The memory 1202 may include a large capacity memory for data or instructions. By way of example and not limitation, the memory 1202 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 1202 may include removable or non-removable (or fixed) media. In a particular embodiment, the memory 1202 is a non-volatile solid-state memory.

[0073] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, typically, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0074] The processor 1201 reads and executes the computer program instructions stored in the memory 1202 to implement any one of the ground-to-air collaborative inspection methods for oil depot inspection in the above-mentioned embodiments.

[0075] In one example, the electronic device may further include a communication interface 1203 and a bus 1210. Figure 2 As shown, the processor 1201, the memory 1202, and the communication interface 1203 are connected via a bus 1210 and communicate with each other.

[0076] The communication interface 1203 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0077] Bus 1210 includes hardware, software or both, coupling the components of the electronic device to each other. For example, but not limitation, the bus may include an accelerated graphics port (AGP) or other graphics bus, an enhanced industrial standard architecture (EISA) bus, a front-side bus (FSB), a hypertransport (HT) interconnect, an industrial standard architecture (ISA) bus, an infinite bandwidth interconnect, a low pin count (LPC) bus, a memory bus, a microchannel architecture (MCA) bus, a peripheral component interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a serial advanced technology attachment (SATA) bus, a video electronics standard association local (VLB) bus or other suitable bus or a combination of two or more of these. Where appropriate, bus 1210 may include one or more buses. Although the present application embodiment describes and shows a specific bus, the present application considers any suitable bus or interconnect.

[0078] It should be understood by those skilled in the art that the exemplary components, systems and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software or a combination of the two. Whether it is performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier.

[0079] It should be clear that the present invention is not limited to the specific configuration and processing described above and shown in the figures. For the sake of simplicity, a detailed description of the known method is omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications and additions, or change the order between the steps after understanding the spirit of the present invention.

[0080] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with features of other embodiments or replace features of other embodiments.

[0081] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the embodiments of the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A ground-air collaborative inspection method for oil depot inspection, characterized in that: The steps of the method include: Obtain the selected inspection points, construct an inspection task including multiple inspection points, and determine the inspection positions corresponding to the coordinates in the three-dimensional model based on the coordinates of the selected inspection points in the pre-constructed three-dimensional model of the oil depot; Determine the UAV task set and the ground task set based on the inspection positions corresponding to each coordinate; Constructing a ground inspection route based on the ground task set and the parking position of the inspection robot, wherein the ground inspection route includes multiple sub-routes; Based on the ground inspection route and the drone task set, a flight inspection route is constructed, wherein the flight inspection route includes a ground inspection route and a target inspection route, wherein the ground inspection route is an aerial projection route of the ground inspection route, and the drone flies along the ground inspection route to collect images of the ground; Based on the images sent back by the drone during the ground inspection route, image recognition is performed to determine whether the corresponding sub-route is blocked; If there is a blockage in a sub-route, the sub-route is marked and the ground inspection route is updated so that the inspection robot performs inspection based on the updated ground inspection route.

2. The ground-air collaborative inspection method for oil depot inspection according to claim 1 is characterized in that: In the step of constructing a ground inspection route based on a ground task set and the parking positions of the inspection robot, the inspection positions belonging to the same oil tank in the ground task set are constructed into a ground inspection group, the parking position of the inspection robot corresponding to each inspection position in the sub-route is determined based on the inspection positions in the ground task set, the parking positions in the same ground inspection group are merged to construct a parking area, the starting point of the inspection robot is used as the starting point of the ground inspection route, and based on the parking area corresponding to each ground inspection group, the distance from the starting point to each parking area is calculated to determine the inspection order of each ground inspection group; Determine the ground inspection route based on the inspection sequence.

3. The ground-air collaborative inspection method for oil depot inspection according to claim 1 is characterized in that: A flight inspection route is constructed based on the ground inspection route and the drone task set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection route of the ground inspection route. The drone flies along the ground inspection route. The steps of collecting images of the ground include: The inspection locations belonging to the same oil tank in the UAV mission set are constructed into an aerial inspection team; For the ground inspection team and the aerial inspection team corresponding to the same oil tank in the ground task set and the drone task set, when the drone flies to the aerial inspection team along the ground inspection route, the drone conducts an inspection at the inspection position of the aerial inspection team; For the aerial inspection team in the drone task set that does not correspond to any oil tank in the ground task set, the distance between the corresponding oil tank of the aerial inspection team and the corresponding oil tank of each ground inspection team in the ground task set is calculated, and the transfer tanks are screened based on the distance value. When the drone flies to the ground inspection team corresponding to the transfer tank, it flies to the corresponding oil tank of the aerial inspection team and inspects the aerial inspection team of the oil tank. After completing the inspection, it returns to the corresponding transfer tank.

4. The ground-air collaborative inspection method for oil depot inspection according to any one of claims 1 to 3, characterized in that: In the step of performing image recognition based on the images sent back by the UAV during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, a pre-trained obstacle recognition model is used for recognition. The obstacle recognition model includes a backbone network, an obstacle detection branch, a width calculation branch and an output fusion layer.

5. The ground-air coordinated inspection method for oil depot inspection according to claim 4 is characterized in that: The backbone network includes a basic convolution block, a downsampling module and a multi-scale feature extraction module arranged in sequence, wherein the basic convolution block is composed of a convolution layer, a batch normalization layer and an activation function; the downsampling module realizes feature map size reduction through a pooling layer or a stride convolution; the multi-scale feature extraction module adopts a cross-level connection structure in FPN and outputs a multi-level feature map for subsequent branch processing; The obstacle detection branch inputs a multi-level feature map, and the obstacle detection branch includes a feature enhancement layer composed of 1×1 convolution and 3×3 convolution, an upsampling layer and a Sigmoid activation layer, and outputs an obstacle binary segmentation map through the Sigmoid activation layer for marking obstacles; The width calculation branch inputs a multi-level feature map, and the width calculation branch includes an attention layer, a variable convolution layer, and a fully connected layer, and outputs the width of the obstacle through the fully connected layer; The output fusion layer receives the obstacle coordinates of the obstacle binary segmentation map from the detection branch and the obstacle width from the width branch. The output fusion layer includes a concat function layer, a channel attention layer, a spatial attention layer and a fully connected layer, and outputs the structured detection results of the obstacle position and obstacle width through the fully connected layer.

6. The ground-air collaborative inspection method for oil depot inspection according to claim 1 is characterized in that: In the step of performing image recognition based on the images transmitted back by the UAV during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, when the UAV flies along the ground inspection route and collects images of the ground, the images are transmitted back to identify obstacles within the sub-route in the transmitted images. If an obstacle exists, the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route are further identified. Based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, it is determined whether the current sub-route is blocked.

7. The ground-air coordinated inspection method for oil depot inspection according to claim 6 is characterized in that: In the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route; and determining whether there is a blockage based on the drivable path width.

8. The ground-air coordinated inspection method for oil depot inspection according to claim 7 is characterized in that: In the step of determining whether the current sub-route is blocked based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, a blocking determination value is calculated based on the width of the drivable path and the width of the inspection robot, and the blocking determination value determines whether there is blocking.

9. The ground-air coordinated inspection method for oil depot inspection according to claim 8 is characterized in that: In the step of calculating the drivable path width based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated using the following formula: ； in, represents the width of the drivable path, D represents the width of the sub-route, r represents the length of the obstacle in the width direction of the sub-route, Indicates the middle position of the sub-route in width direction. Indicates the middle position of the obstacle in the width direction of the sub-route. Indicates the offset of the middle position of the obstacle in the width direction relative to the middle position of the sub-route in the width direction.

10. The ground-air coordinated inspection method for oil depot inspection according to claim 9, characterized in that: In the step of determining whether there is a blockage based on the width of the drivable path, a blockage determination value is calculated based on the following formula; ； in, Indicates the width of the inspection robot. Indicates the blocking judgment value, Indicates the preset calculation parameters.

11. The ground-air coordinated inspection method for oil depot inspection according to claim 1, characterized in that: If a sub-route is blocked, the sub-route is marked, and in the step of updating the ground inspection route, the sub-route is deleted from the feasible route, and the exit position of the previous ground inspection team and the entry position of the next ground inspection team are used as the starting point and the end point, and the path of the starting point and the end point is recalculated using the shortest path calculation method.

12. The ground-air coordinated inspection method for oil depot inspection according to claim 1, characterized in that: In the step where the inspection robot performs inspection based on the updated ground inspection route, the inspection robot collects thermal imaging images and RGB images during the inspection process; For the thermal imaging image, the thermal imaging image is matched with the pre-set standard image of the inspection point to obtain a contrast matrix, and the contrast matrix is ​​input into a pre-trained first neural network model. The first neural network model is used to determine whether a first abnormal state occurs. For the inspection data collected by the inspection robot or the UAV; for the RGB image, it is input into a pre-trained second neural network model. The second neural network model is used to determine whether a second abnormal state occurs.

13. A ground-air collaborative inspection system for oil depot inspection, characterized in that: The system includes a computer device, which includes a processor and a memory, wherein the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method as described in any one of claims 1 to 12.

Images