A ground-air collaborative inspection method and system for oil depot inspection

Abstract

The present invention provides a ground-to-air collaborative inspection method and system for oil depot inspection, the method comprising the following steps: determining corresponding inspection positions based on the coordinates of selected inspection points in a pre-constructed three-dimensional model of the oil depot; determining a drone task set and a ground task set based on the inspection positions corresponding to each coordinate; constructing a ground inspection route based on the ground task set and the parking position of an inspection robot; constructing a flight inspection route based on the ground inspection route and the drone task set, the flight inspection route including a ground inspection route and a target inspection route; performing image recognition based on images transmitted back by the drone during flight along the ground inspection route to determine whether a corresponding sub-route is blocked; if a sub-route is blocked, marking the sub-route and updating the ground inspection route so that the inspection robot performs inspection based on the updated ground inspection route.

Description

Technical Field

[0001] The present invention relates to the field of robot inspection technology, and in particular to a ground-air collaborative inspection method and system for oil depot inspection. Background Art

[0002] In the energy supply system, oil depots are the core of storage and distribution of petroleum and its products. Their safe and stable operation is related to the continuity of energy supply, the normal operation of social economy, and the safety of people's lives and property. Inspection of oil depots is an important line of defense to ensure all of this and is of great importance that cannot be ignored.

[0003] From a safety perspective, the oil stored in oil depots is flammable, explosive, and volatile. The consequences of a leak, fire, or explosion are disastrous. Inspections are like a physical examination of an oil depot. Through meticulous inspections of equipment and facilities like tanks, pipelines, valves, and pumps, potential hazards can be identified promptly. For example, tank corrosion inspections can be used to promptly repair any rust or cracks found, preventing leaks and fires. Pipeline joints can also be properly sealed to prevent the evaporation and accumulation of oil and gas, which could potentially form explosive mixtures. Only through regular inspections can safety hazards be nipped in the bud and a strong safety barrier established for oil depots.

[0004] When it comes to equipment maintenance, inspections are crucial for ensuring the proper functioning of oil depot equipment. The various equipment within an oil depot is in constant operation, making it susceptible to wear and aging. By observing equipment operating parameters, sound levels, temperature, and other indicators, inspectors can determine whether the equipment is functioning properly. For example, by monitoring the vibration frequency and temperature of a pump, and detecting any anomalies, prompt inspection and maintenance can prevent downtime caused by equipment failure, ensure the normal receipt, delivery, and storage of oil products, and improve oil depot operational efficiency.

[0005] In the original oil depot inspection, manual inspection is usually adopted to reach each inspection location, and then the data of the corresponding location is collected by equipment to complete the inspection work. With the development of technology, in the current oil depot inspection, inspection robots are usually used to carry data collection equipment to carry out inspections and complete the inspection work. However, the inspection robot is limited by the length of its own mechanical arm, and the inspection range is limited. In addition, it is difficult to pre-judge the road conditions during the inspection work, and it is easy to be affected by the road conditions during driving, causing the inspection to be interrupted. Summary of the Invention

[0006] In view of this, an embodiment of the present invention provides a ground-air collaborative inspection method and system for oil depot inspection to eliminate or improve one or more defects in the prior art.

[0007] One aspect of the present invention provides a ground-air collaborative inspection method for oil depot inspection, the method comprising the following steps:

[0008] Obtain the selected inspection points, construct an inspection task including multiple inspection points, and determine the inspection positions corresponding to the coordinates in the pre-built 3D model of the oil depot based on the coordinates of the selected inspection points;

[0009] Determine the UAV task set and the ground task set based on the inspection positions corresponding to each coordinate;

[0010] Constructing a ground inspection route based on the ground task set and the parking position of the inspection robot, wherein the ground inspection route includes multiple sub-routes;

[0011] A flight inspection route is constructed based on the ground inspection route and the drone task set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection of the ground inspection route. The drone flies along the ground inspection route to collect images of the ground.

[0012] Image recognition is performed based on the images sent back by the drone during the ground inspection route to determine whether the corresponding sub-route is blocked;

[0013] If a sub-route is blocked, the sub-route is marked and the ground inspection route is updated so that the inspection robot performs inspection based on the updated ground inspection route.

[0014] The above scheme adopts, on the one hand, the scheme uses two devices to inspect the oil tanks of the oil depot equipment, and can let the drone conduct the inspection first during one inspection process. During the inspection process, the drone can inspect the ground inspection route of the inspection robot in the air. If there are obstacles on the route, the ground inspection route will be marked in advance and updated to bypass the sub-route with obstacles. In the subsequent inspection process of the inspection robot, the inspection interruption caused by obstacles is avoided.

[0015] In some embodiments of the present invention, in the step of constructing a ground inspection route based on a ground task set and the stopping positions of the inspection robot, the inspection positions belonging to the same oil tank in the ground task set are constructed into a ground inspection group, and the stopping positions of the inspection robot corresponding to each inspection position in the sub-route are determined based on the inspection positions in the ground task set, and the stopping positions in the same ground inspection group are merged to construct a stopping area. The starting point of the inspection robot is used as the starting point of the ground inspection route, and based on the stopping area corresponding to each ground inspection group, the distance from the starting point to each stopping area is calculated to determine the inspection order of each ground inspection group; and the ground inspection route is determined based on the inspection order.

[0016] In some embodiments of the present invention, in the step of determining the ground inspection route based on the inspection sequence, the sub-route that the inspection robot passes from the starting point to the first docking area, the sub-route that the inspection robot passes from one docking area to another docking area, and the sub-route that the inspection robot passes from the last docking area back to the starting point are sequentially combined as the ground inspection route.

[0017] In some embodiments of the present invention, a flight inspection route is constructed based on the ground inspection route and the drone mission set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection route of the ground inspection route. The drone flies along the ground inspection route, and the step of capturing images of the ground includes:

[0018] The inspection locations belonging to the same oil tank in the UAV mission set are constructed into an aerial inspection team;

[0019] For the ground inspection team and the aerial inspection team corresponding to the same oil tank in the ground task set and the drone task set, when the drone flies along the ground inspection route to the aerial inspection team, it inspects the inspection position of the aerial inspection team;

[0020] For the aerial inspection team in the drone task set that does not correspond to any oil tank in the ground task set, the distance between the corresponding oil tank of the aerial inspection team and the corresponding oil tank of each ground inspection team in the ground task set is calculated, and the diverted oil tanks are screened based on the distance value. When the drone flies to the ground inspection team corresponding to the diverted oil tank, it flies to the corresponding oil tank of the aerial inspection team and inspects the aerial inspection team of the oil tank. After completing the inspection, it returns to the corresponding diverted oil tank.

[0021] In some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the drone during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, a pre-trained obstacle recognition model is used for recognition. The obstacle recognition model includes a backbone network, an obstacle detection branch, a width calculation branch and an output fusion layer.

[0022] In some embodiments of the present invention, the backbone network includes a basic convolution block, a downsampling module, and a multi-scale feature extraction module arranged in sequence, wherein the basic convolution block is composed of a convolution layer, a batch normalization layer, and an activation function; the downsampling module reduces the size of the feature map through a pooling layer or a strided convolution; the multi-scale feature extraction module adopts the cross-level connection structure in FPN and outputs a multi-level feature map for subsequent branch processing;

[0023] The obstacle detection branch inputs a multi-level feature map. The obstacle detection branch includes a feature enhancement layer composed of 1×1 convolution and 3×3 convolution, an upsampling layer, and a Sigmoid activation layer. The Sigmoid activation layer outputs an obstacle binary segmentation map for marking obstacles.

[0024] The width calculation branch inputs a multi-level feature map, and the width calculation branch includes an attention layer, a variable convolution layer, and a fully connected layer, and outputs the width of the obstacle through the fully connected layer;

[0025] The output fusion layer receives the obstacle coordinates of the obstacle binary segmentation map from the detection branch and the obstacle width from the width branch. The output fusion layer includes a concat function layer, a channel attention layer, a spatial attention layer and a fully connected layer, and outputs the structured detection results of the obstacle position and obstacle width through the fully connected layer.

[0026] In some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the UAV during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, when the UAV flies along the ground inspection route and collects images of the ground, the images are transmitted back to identify obstacles within the sub-route in the transmitted images. If an obstacle exists, the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route are further identified. Based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, it is determined whether the current sub-route is blocked.

[0027] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route; and whether there is a blockage is determined based on the drivable path width.

[0028] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, a blocking determination value is calculated based on the width of the drivable path and the width of the inspection robot, and the blocking determination value determines whether there is blocking.

[0029] In some embodiments of the present invention, in the step of calculating the drivable path width based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated using the following formula:

[0030] ;

[0031] in, represents the width of the drivable path, D represents the width of the sub-route, and r represents the length of the obstacle in the width direction of the sub-route. Indicates the middle position of the sub-route in the width direction. Indicates the middle position of the obstacle in the width direction of the sub-route. Indicates the offset of the middle position of the obstacle in the width direction relative to the middle position of the sub-route in the width direction.

[0032] In some embodiments of the present invention, in the step of determining whether there is a blockage based on the width of the drivable path, the blockage determination value is calculated based on the following formula:

[0033] ;

[0034] in, Indicates the width of the inspection robot. Indicates the blocking judgment value, Indicates the preset calculation parameters.

[0035] In some embodiments of the present invention, if a sub-route is blocked, the sub-route is marked, and in the step of updating the ground inspection route, the sub-route is deleted from the feasible route, and the exit position of the previous ground inspection team and the entry position of the next ground inspection team are used as the starting point and the end point, and the path of the starting point and the end point is recalculated using the shortest path calculation method.

[0036] In some embodiments of the present invention, in the step of the inspection robot performing an inspection based on the updated ground inspection route, the inspection robot collects thermal imaging images and RGB images during the inspection process;

[0037] For the thermal imaging image, the thermal imaging image is matched with the pre-set standard image of the inspection point to obtain a comparison matrix, and the comparison matrix is input into a pre-trained first neural network model. The first neural network model is used to determine whether a first abnormal state occurs. For the inspection data collected by the inspection robot or drone; for the RGB image, the comparison matrix is input into a pre-trained second neural network model. The second neural network model is used to determine whether a second abnormal state occurs.

[0038] The second aspect of the present invention also provides a ground-to-air collaborative inspection system for oil depot inspections, the system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0039] The third aspect of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps implemented by the aforementioned method for handling traffic congestion for a high-speed data center network.

[0040] Additional advantages, objects, and features of the present invention will be described in part in the following description and will become apparent to those skilled in the art after studying the following or may be learned by practice of the present invention. The objects and other advantages of the present invention may be particularly pointed out and attained in the description and drawings.

[0041] Those skilled in the art will understand that the purposes and advantages that can be achieved by the present invention are not limited to the above specific descriptions, and the above and other purposes that can be achieved by the present invention will be more clearly understood based on the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The drawings described herein are used to provide a further understanding of the present invention, constitute a part of this application, and do not constitute a limitation of the present invention.

[0043] Figure 1 This is a schematic diagram of the architecture of one implementation method of the ground-air collaborative inspection method for oil depot inspections in this solution;

[0044] Figure 2 This is a schematic diagram of the architecture of another implementation method of the ground-air collaborative inspection method for oil depot inspections in this solution;

[0045] Figure 3 Schematic diagram of the implementation steps of step S400 in this solution;

[0046] Figure 4 A top-down view of a sub-route when calculating the drivable path width;

[0047] Figure 5 This is a schematic diagram of the structure of the electronic equipment of this solution. DETAILED DESCRIPTION

[0048] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0049] It should also be noted that, in order to avoid obscuring the present invention due to unnecessary details, the accompanying drawings only show structures and / or processing steps closely related to the solutions according to the present invention, while other details that are not closely related to the present invention are omitted.

[0050] like Figure 1 As shown, the present invention proposes a ground-air collaborative inspection method for oil depot inspection, the method comprising the following steps:

[0051] Step S100: obtaining selected inspection points, constructing an inspection task including multiple inspection points, and determining inspection positions corresponding to the coordinates in the pre-constructed three-dimensional model of the oil depot based on the coordinates of the selected inspection points;

[0052] During the specific implementation process, the oil depot is scanned in advance using scanning equipment to scan the oil depot equipment, record the location of the oil depot equipment, and build a three-dimensional model of the oil depot;

[0053] Specifically, during the oil depot scanning process, a drone equipped with a laser radar (LiDAR) cruises at a speed of 5m / s to generate a centimeter-level accurate 3D point cloud map (error ≤ 5cm), marking complex areas such as the top of the oil tank and pipeline intersections;

[0054] In the process of 3D modeling, the improved Dubins curve algorithm is applied to smooth the path, eliminate jagged flight trajectories, and improve the consistency of mapping.

[0055] Specifically, the staff selects inspection points in the system's three-dimensional model, and the system constructs the multiple inspection points selected by the staff into inspection tasks. Based on the coordinates of the inspection points selected by the staff in the system's three-dimensional model, the system determines the corresponding inspection positions as the tank top, the upper area of the tank body, the lower area of the tank body, the tank bottom or the pipeline, etc.

[0056] Step S200, determining a UAV task set and a ground task set based on the inspection positions corresponding to the respective coordinates;

[0057] Specifically, the inspection locations of the tank top and upper area of the tank body are constructed as a drone task set; the lower area of the tank body, the tank bottom or the pipeline are constructed as a ground task set.

[0058] Specifically, the upper area of the can body and the lower area of the can body are divided by a preset height threshold.

[0059] Through the above solution, staff can select coordinates by clicking in the system's three-dimensional map to determine whether to join the drone mission set or the ground mission set.

[0060] Step S300: constructing a ground inspection route based on the ground task set and the parking position of the inspection robot, wherein the ground inspection route includes multiple sub-routes;

[0061] During the specific implementation process, the inspection position is pre-set with a parking position for the inspection robot and an air parking position for the UAV.

[0062] Specifically, the parking position of the inspection robot is the ground area, and the aerial parking position of the drone is the air area.

[0063] Specifically, the sub-route is a straight driving section, and there is no turning intersection in the section.

[0064] Step S400: constructing a flight inspection route based on the ground inspection route and the drone mission set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection of the ground inspection route. The drone flies along the ground inspection route to capture images of the ground.

[0065] Through the above scheme, after the UAV reaches the location of the inspected oil tank along the ground inspection route or the target inspection route, the fast random tree algorithm or the A-star algorithm is used to plan the specific flight path.

[0066] Step S500 , performing image recognition based on the images transmitted by the drone during its flight along the ground inspection route to determine whether the corresponding sub-route is blocked;

[0067] Step S600: If a sub-route is blocked, the sub-route is marked and the ground inspection route is updated so that the inspection robot performs inspection based on the updated ground inspection route.

[0068] The above scheme adopts, on the one hand, the scheme uses two devices to inspect the oil tanks of the oil depot equipment, and can let the drone conduct the inspection first during one inspection process. During the inspection process, the drone can inspect the ground inspection route of the inspection robot in the air. If there are obstacles on the route, the ground inspection route will be marked in advance and updated to bypass the sub-route with obstacles. In the subsequent inspection process of the inspection robot, the inspection interruption caused by obstacles is avoided.

[0069] like Figure 2 As shown, in some embodiments of the present invention, the step of constructing a ground inspection route based on the ground task set and the parking position of the inspection robot includes:

[0070] Step S310: constructing the inspection locations belonging to the same oil tank in the ground task set into a ground inspection team;

[0071] Step S320: determining the parking position of the inspection robot corresponding to each inspection position in the sub-route based on the inspection positions in the ground task set, merging the parking positions in the same ground inspection team, and constructing a parking area;

[0072] Step S330: Taking the starting point of the inspection robot as the starting point of the ground inspection route, and calculating the distance from the starting point to each docking area based on the docking area corresponding to each ground inspection team, to determine the inspection order of each ground inspection team;

[0073] Step S340: Determine a ground inspection route based on the inspection sequence.

[0074] In some embodiments of the present invention, in the step of determining the ground inspection route based on the inspection sequence, the sub-route that the inspection robot passes from the starting point to the first docking area, the sub-route that the inspection robot passes from one docking area to another docking area, and the sub-route that the inspection robot passes from the last docking area back to the starting point are sequentially combined as the ground inspection route.

[0075] Specifically, a shortest path algorithm is used to calculate the path from the starting point to the first docking area, from one docking area to another docking area, and from the last docking area back to the starting point.

[0076] Through the above scheme, the inspection order of each ground inspection team (i.e., oil tank) is determined by calculating the distance from the starting point to each docking area. After the inspection order is determined, the inspection path between the inspection teams is planned using the shortest path algorithm to ensure the efficiency of the inspection.

[0077] In some embodiments of the present invention, a flight inspection route is constructed based on the ground inspection route and the drone mission set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection route of the ground inspection route. The drone flies along the ground inspection route, and the step of capturing images of the ground includes:

[0078] like Figure 3 As shown, in some embodiments of the present invention, the ground inspection route is an aerial vertical projection route of the ground inspection route.

[0079] Step S410: constructing the inspection locations belonging to the same oil tank in the drone task set into an aerial inspection team;

[0080] Step S420: For the ground inspection team and the aerial inspection team corresponding to the same oil tank in the ground task set and the drone task set, when the drone flies along the ground inspection route to the aerial inspection team, it inspects the inspection position of the aerial inspection team;

[0081] Step S430: For the aerial inspection team in the UAV task set that does not correspond to any oil tank in the ground task set, the distance between the corresponding oil tank of the aerial inspection team and the corresponding oil tank of each ground inspection team in the ground task set is calculated, and the transfer tanks are screened based on the distance value. When the UAV flies to the ground inspection team corresponding to the transfer tank, it flies to the corresponding oil tank of the aerial inspection team and inspects the aerial inspection team of the oil tank. After completing the inspection, it returns to the corresponding transfer tank.

[0082] Step S440: After returning to the transfer tank, continue flying along the ground inspection route.

[0083] In some embodiments of the present invention, when the drone flies to the ground inspection team corresponding to the diverted oil tank, in the step of flying to the corresponding oil tank of the aerial inspection team, a fast random tree algorithm or an A-star algorithm is used to plan a specific flight path.

[0084] By adopting the above scheme, the drone can complete its own inspection task at the same time during the inspection process. On the one hand, when the drone flies to its own aerial inspection team during the inspection process, it can complete the inspection of its own aerial inspection team. On the other hand, it can determine the diversion position through the diversion tank. When it flies close to the oil tank of its own aerial inspection team, it flies to the oil tank of its own aerial inspection team to complete the inspection task, thereby ensuring the efficient completion of the aerial inspection task.

[0085] In some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the drone during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, a pre-trained obstacle recognition model is used for recognition. The obstacle recognition model includes a backbone network, an obstacle detection branch, a width calculation branch and an output fusion layer.

[0086] In some embodiments of the present invention, the backbone network includes a basic convolution block, a downsampling module, and a multi-scale feature extraction module arranged in sequence. The basic convolution block consists of a convolution layer (Conv), a batch normalization layer (BN), and an activation function (such as ReLU); the downsampling module reduces the size of the feature map through a pooling layer (MaxPooling) or a strided convolution (Strided Conv); the multi-scale feature extraction module adopts the cross-level connection structure in the FPN (Feature Pyramid Network) and outputs a multi-level feature map for subsequent branch processing;

[0087] Specifically, the backbone network adopts a layered structure like CSPDarknet, and through the fusion of shallow and deep layer features (such as the CSP module segmenting feature maps for divide-and-conquer processing), it achieves multi-level perception of obstacles from outline to detail. For example, the shallow layer captures edge textures, while the deep layer extracts semantic information, improving adaptability to complex scenes.

[0088] The obstacle detection branch inputs a multi-level feature map. The obstacle detection branch includes a feature enhancement layer composed of 1×1 convolution and 3×3 convolution, an upsampling layer, and a Sigmoid activation layer. The Sigmoid activation layer outputs an obstacle binary segmentation map for marking obstacles.

[0089] The width calculation branch inputs a multi-level feature map, and the width calculation branch includes an attention layer, a variable convolution layer, and a fully connected layer, and outputs the width of the obstacle through the fully connected layer;

[0090] Specifically, the width calculation branch introduces deformable convolution in the regression layer, dynamically adjusts the convolution kernel sampling points, accurately captures the geometric features of obstacle edges, and reduces the width calculation error to within ±3cm.

[0091] The output fusion layer receives the obstacle coordinates of the obstacle binary segmentation map from the detection branch and the obstacle width from the width branch. The output fusion layer includes a concat function layer, a channel attention layer, a spatial attention layer and a fully connected layer, and outputs the structured detection results of the obstacle position and obstacle width through the fully connected layer.

[0092] The output fusion layer performs Kalman filtering on the detection results of consecutive frames, and removes instantaneous outliers based on the width change trend, improving output stability by 60%. It also designs confidence weights for each branch output (such as the detection box score + the width regression variance), and dynamically adjusts the fusion ratio, reducing the false alarm rate by 22% in occlusion scenarios. Through learnable fusion parameters (such as the gating mechanism), global optimization is achieved from feature extraction to final output, avoiding suboptimal solutions introduced by manual rules.

[0093] like Figure 4 As shown, in some embodiments of the present invention, in the step of performing image recognition based on the images transmitted back by the UAV during the flight of the ground inspection route to determine whether the corresponding sub-route is blocked, when the UAV flies along the ground inspection route and collects images of the ground, the images are transmitted back to identify obstacles within the sub-route in the transmitted images. If an obstacle exists, the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route are further identified. Based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, it is determined whether the current sub-route is blocked.

[0094] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route; and whether there is a blockage is determined based on the drivable path width.

[0095] In some embodiments of the present invention, in the step of determining whether the current sub-route is blocked based on the width of the sub-route, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, a blocking determination value is calculated based on the width of the drivable path and the width of the inspection robot, and the blocking determination value determines whether there is blocking.

[0096] In some embodiments of the present invention, in the step of calculating the drivable path width based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated using the following formula:

[0097] ;

[0098] in, represents the width of the drivable path, D represents the width of the sub-route, and r represents the length of the obstacle in the width direction of the sub-route. Indicates the middle position of the sub-route in the width direction. Indicates the middle position of the obstacle in the width direction of the sub-route. Indicates the offset of the middle position of the obstacle in the width direction relative to the middle position of the sub-route in the width direction.

[0099] In some embodiments of the present invention, in the step of determining whether there is a blockage based on the width of the drivable path, the blockage determination value is calculated based on the following formula:

[0100] ;

[0101] in, Indicates the width of the inspection robot. Indicates the blocking judgment value, Indicates the preset calculation parameters.

[0102] Using the above scheme, the drone patrols the ground inspection route along the ground inspection route, and uses real-time feedback images to identify whether there are obstacles on the road, such as puddles and piled items. If there are obstacles, the obstacle recognition model is used to identify the position and width of the obstacle. The drivable width is then calculated using the above formula based on the width of the sub-route. The drivable width is used to determine whether the remaining width allows the inspection robot to pass normally. This can accurately determine whether there is congestion in the sub-route. If there is congestion, the route is replanned to avoid obstacles blocking normal inspections.

[0103] In some embodiments of the present invention, if a sub-route is blocked, the sub-route is marked, and in the step of updating the ground inspection route, the sub-route is deleted from the feasible route, and the exit position of the previous ground inspection team and the entry position of the next ground inspection team are used as the starting point and the end point, and the path of the starting point and the end point is recalculated using the shortest path calculation method, and the recalculated path consists of at least one sub-route.

[0104] In some embodiments of the present invention, in the step of the inspection robot performing an inspection based on the updated ground inspection route, the inspection robot collects thermal imaging images and RGB images during the inspection process;

[0105] For the thermal imaging image, the thermal imaging image is matched with the pre-set standard image of the inspection point to obtain a comparison matrix, and the comparison matrix is input into a pre-trained first neural network model. The first neural network model is used to determine whether a first abnormal state occurs. For the inspection data collected by the inspection robot or drone; for the RGB image, the comparison matrix is input into a pre-trained second neural network model. The second neural network model is used to determine whether a second abnormal state occurs.

[0106] During the specific implementation process, in the step of matching the thermal imaging image with the standard image pre-set at the inspection point to obtain a comparison matrix, the pixel values at the corresponding positions of the thermal imaging image and the standard image are subtracted and used as the matrix values at the same position in the comparison matrix to obtain the comparison matrix.

[0107] Specifically, the first abnormal state is used to determine the difference between the heat distribution in the image and the standard image. If there is a large difference, it indicates that the first abnormal state has occurred. The second abnormal state is used to determine whether corrosion or leakage occurs at the corresponding position. If excessive corrosion or leakage occurs, it indicates that the second abnormal state has occurred.

[0108] The first neural network model and the second neural network model are pre-trained deep learning models, and the deep learning model can be a multi-layer perceptron (MLP), a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer model or a graph neural network (GNN).

[0109] On the one hand, this solution uses two types of equipment to inspect the oil tanks of oil depot equipment. During one inspection process, the drone can conduct the inspection first. During the inspection process, the drone can inspect the ground inspection route of the inspection robot in the air. If there are obstacles on the route, the ground inspection route will be marked in advance and updated to bypass the sub-route with obstacles. In the subsequent inspection process of the inspection robot, inspection interruption caused by obstacles can be avoided.

[0110] During the inspection process of the drone, for inspection points at higher positions, such as the tank top and higher tank body positions, the drone flies to the height above the oil tank or the tank body to collect images, and uses image analysis to determine whether the corrosion situation and temperature distribution are normal. If abnormal, timely maintenance can be carried out. During the inspection process of higher positions, the problem of inspection range limitation caused by the height of the inspection robot's mechanical arm can be solved.

[0111] An embodiment of the present application also provides a ground-to-air collaborative inspection system for oil depot inspections, the system including a computer device, the computer device including a processor and a memory, the memory storing computer instructions, the processor being used to execute the computer instructions stored in the memory, and when the computer instructions are executed by the processor, the system implements the steps implemented by the method described above.

[0112] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the aforementioned method for handling traffic congestion for a high-speed data center network are implemented.

[0113] like Figure 5 As shown, an embodiment of the present application also provides an electronic device, which includes: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the above-mentioned load-balanced power grid computing power network resource scheduling method is implemented.

[0114] The electronic device may include a processor 1201 and a memory 1202 storing computer program instructions.

[0115] Specifically, the processor 1201 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0116] Memory 1202 may include a large-capacity memory for data or instructions. By way of example, and not limitation, memory 1202 may include a hard disk drive (HDD), a floppy disk drive, flash memory, an optical disk, a magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 1202 may include removable or non-removable (or fixed) media. In certain embodiments, memory 1202 is non-volatile solid-state memory.

[0117] The memory may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media devices, optical storage media devices, flash memory devices, electrical, optical or other physical / tangible memory storage devices. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to an aspect of the present disclosure.

[0118] The processor 1201 reads and executes computer program instructions stored in the memory 1202 to implement any one of the ground-air collaborative inspection methods for oil depot inspection in the above-mentioned embodiments.

[0119] In one example, the electronic device may further include a communication interface 1203 and a bus 1210. Figure 2 As shown, the processor 1201 , the memory 1202 , and the communication interface 1203 are connected via a bus 1210 and communicate with each other.

[0120] The communication interface 1203 is mainly used to implement communication between various modules, devices, units and / or equipment in the embodiments of the present application.

[0121] Bus 1210 includes hardware, software, or both that couples components of an electronic device to each other. By way of example, and not limitation, a bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industrial Standard Architecture (EISA) bus, a Front Side Bus (FSB), a HyperTransport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 1210 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0122] It should be understood by those skilled in the art that the various exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether to implement the system in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention. When implemented in hardware, it may be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments used to perform the required tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via a data signal carried in a carrier wave.

[0123] It should be understood that the present invention is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted. In the above embodiments, several specific steps are described and illustrated as examples. However, the method of the present invention is not limited to the specific steps described and illustrated. Those skilled in the art may make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present invention.

[0124] In the present invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or replace features of other embodiments.

[0125] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations to the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A ground-air collaborative inspection method for oil depot inspection, characterized in that: The steps of the method include: Obtain the selected inspection points, construct an inspection task including multiple inspection points, and determine the inspection positions corresponding to the coordinates in the pre-built 3D model of the oil depot based on the coordinates of the selected inspection points; Determine the UAV task set and the ground task set based on the inspection positions corresponding to each coordinate; Constructing a ground inspection route based on the ground task set and the parking position of the inspection robot, wherein the ground inspection route includes multiple sub-routes; A flight inspection route is constructed based on the ground inspection route and the drone task set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection of the ground inspection route. The drone flies along the ground inspection route to collect images of the ground. Image recognition is performed based on images transmitted back by the drone during flight along the ground inspection route to determine whether the corresponding sub-route is blocked. When the drone flies along the ground inspection route, it collects images of the ground and transmits them back. Obstacles within the sub-route range in the transmitted images are identified. If an obstacle exists, the sub-route width, the length of the obstacle in the width direction of the sub-route, and the center position of the obstacle in the width direction of the sub-route are further identified. Based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the center position of the obstacle in the width direction of the sub-route, it is determined whether the current sub-route is blocked; If a sub-route is blocked, the sub-route is marked and the ground inspection route is updated so that the inspection robot performs inspection based on the updated ground inspection route.

2. The ground-air collaborative inspection method for oil depot inspection according to claim 1 is characterized in that: In the step of constructing a ground inspection route based on a ground task set and the parking positions of the inspection robot, the inspection positions belonging to the same oil tank in the ground task set are constructed into a ground inspection group, the parking position of the inspection robot corresponding to each inspection position in the sub-route is determined based on the inspection positions in the ground task set, the parking positions in the same ground inspection group are merged to construct a parking area, the starting point of the inspection robot is used as the starting point of the ground inspection route, and based on the parking area corresponding to each ground inspection group, the distance from the starting point to each parking area is calculated to determine the inspection order of each ground inspection group; Determine the ground inspection route based on the inspection sequence.

3. The ground-air collaborative inspection method for oil depot inspection according to claim 1 is characterized in that: A flight inspection route is constructed based on the ground inspection route and the drone task set. The flight inspection route includes a ground inspection route and a target inspection route. The ground inspection route is an aerial projection route of the ground inspection route. The drone flies along the ground inspection route, and the steps of collecting images of the ground include: The inspection locations belonging to the same oil tank in the UAV mission set are constructed into an aerial inspection team; For the ground inspection team and the aerial inspection team corresponding to the same oil tank in the ground task set and the drone task set, when the drone flies along the ground inspection route to the aerial inspection team, it inspects the inspection position of the aerial inspection team; For the aerial inspection team in the drone task set that does not correspond to any oil tank in the ground task set, the distance between the corresponding oil tank of the aerial inspection team and the corresponding oil tank of each ground inspection team in the ground task set is calculated, and the diverted oil tanks are screened based on the distance value. When the drone flies to the ground inspection team corresponding to the diverted oil tank, it flies to the corresponding oil tank of the aerial inspection team and inspects the aerial inspection team of the oil tank. After completing the inspection, it returns to the corresponding diverted oil tank.

4. The ground-air collaborative inspection method for oil depot inspection according to any one of claims 1 to 3, characterized in that: In the step of performing image recognition based on the images sent back by the drone during its flight over the ground patrol route to determine whether the corresponding sub-route is blocked, a pre-trained obstacle recognition model is used for recognition. The obstacle recognition model includes a backbone network, an obstacle detection branch, a width calculation branch, and an output fusion layer.

5. The ground-air collaborative inspection method for oil depot inspection according to claim 4 is characterized in that: The backbone network includes a basic convolution block, a downsampling module, and a multi-scale feature extraction module arranged in sequence. The basic convolution block consists of a convolution layer, a batch normalization layer, and an activation function. The downsampling module reduces the size of the feature map through a pooling layer or a strided convolution. The multi-scale feature extraction module adopts the cross-level connection structure in FPN and outputs a multi-level feature map for subsequent branch processing. The obstacle detection branch inputs a multi-level feature map. The obstacle detection branch includes a feature enhancement layer composed of 1×1 convolution and 3×3 convolution, an upsampling layer, and a Sigmoid activation layer. The Sigmoid activation layer outputs an obstacle binary segmentation map for marking obstacles. The width calculation branch inputs a multi-level feature map, and the width calculation branch includes an attention layer, a variable convolution layer, and a fully connected layer, and outputs the width of the obstacle through the fully connected layer; The output fusion layer receives the obstacle coordinates of the obstacle binary segmentation map from the detection branch and the obstacle width from the width branch. The output fusion layer includes a concat function layer, a channel attention layer, a spatial attention layer and a fully connected layer, and outputs the structured detection results of the obstacle position and obstacle width through the fully connected layer.

6. The ground-air coordinated inspection method for oil depot inspection according to claim 1 is characterized in that: In the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route; and determining whether the current sub-route is blocked based on the drivable path width.

7. The ground-air collaborative inspection method for oil depot inspection according to claim 6 is characterized in that: In the step of determining whether the current sub-route is blocked based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, a blocking determination value is calculated based on the drivable path width and the width of the inspection robot, and the blocking determination value determines whether there is blocking.

8. The ground-air coordinated inspection method for oil depot inspection according to claim 7 is characterized in that: In the step of calculating the drivable path width based on the sub-route width, the length of the obstacle in the width direction of the sub-route, and the middle position of the obstacle in the width direction of the sub-route, the drivable path width is calculated using the following formula: ； in, represents the width of the drivable path, D represents the width of the sub-route, and r represents the length of the obstacle in the width direction of the sub-route. Indicates the middle position of the sub-route in the width direction. Indicates the middle position of the obstacle in the width direction of the sub-route. Indicates the offset of the middle position of the obstacle in the width direction relative to the middle position of the sub-route in the width direction.

9. The ground-air coordinated inspection method for oil depot inspection according to claim 8, characterized in that: In the step of determining whether there is a blockage based on the width of the drivable path, a blockage determination value is calculated based on the following formula; ； in, Indicates the width of the inspection robot. Indicates the blocking judgment value, Indicates the preset calculation parameters.

10. The ground-air collaborative inspection method for oil depot inspection according to claim 1, characterized in that: If a sub-route is blocked, the sub-route is marked, and in the step of updating the ground inspection route, the sub-route is deleted from the feasible route, and the exit position of the previous ground inspection team and the entry position of the next ground inspection team are used as the starting point and the end point, and the shortest path calculation method is used to recalculate the path of the starting point and the end point.

11. The ground-air collaborative inspection method for oil depot inspection according to claim 1, characterized in that: In the step where the inspection robot performs inspection based on the updated ground inspection route, the inspection robot collects thermal imaging images and RGB images during the inspection process; For the thermal imaging image, the thermal imaging image is matched with the pre-set standard image of the inspection point to obtain a comparison matrix, and the comparison matrix is input into a pre-trained first neural network model. The first neural network model is used to determine whether a first abnormal state occurs. For the inspection data collected by the inspection robot or drone; for the RGB image, the comparison matrix is input into a pre-trained second neural network model. The second neural network model is used to determine whether a second abnormal state occurs.

12. A ground-air collaborative inspection system for oil depot inspection, characterized in that: The system includes a computer device, the computer device includes a processor and a memory, the memory stores computer instructions, and the processor is used to execute the computer instructions stored in the memory. When the computer instructions are executed by the processor, the system implements the steps implemented by the method according to any one of claims 1 to 11.

Images