Landing method and device of flight equipment, image processing method and device, equipment and medium

Abstract

The invention provides a landing method and device of flight equipment, an image processing method and device, equipment and a medium. The method is applied to the flight equipment, and comprises the following steps: carrying out inspection on an inspection target in an inspection place based on a first map of the inspection place and the inspection target in a preset inspection task; under the condition that the preset inspection task is completed, or under the condition that the flight equipment is abnormal, the flight equipment flies to the sky of a first assembly of land equipment; the first assembly is used for parking the flight equipment; and the height between the flight equipment and the first assembly is determined, and the flight equipment is guided to land on the first assembly according to the height.

Description

Technical Field

[0001] The present application relates to the field of robot control technology, and is related to but not limited to a landing method for a flying device, an image processing method and apparatus, equipment, and medium. Background Art

[0002] At present, the field of computer room inspection is developing in the direction of intelligence, so computer room inspection robots have emerged. However, inspection drones can only land through a single visual sign, which makes it difficult to achieve high-precision, smooth and autonomous landing. This is because when the drone is close to the shutdown disk based on visual guidance, it cannot focus on the shutdown disk, so it cannot achieve the ranging and positioning function, and can only "smash" the shutdown disk in a vertical state, thus damaging the machine. Summary of the invention

[0003] The present application provides a method for landing a flying device, an image processing method, a device, a device, and a medium, wherein:

[0004] In a first aspect, an embodiment of the present application provides a landing method for a flying device, which is applied to a flying device, and includes: based on a first map of an inspection site and an inspection target in a preset inspection task, inspecting the inspection target at the inspection site; when the preset inspection task is completed, or when an abnormality occurs in the flying device, flying to the sky above a first component of a land device; the first component is used to park the flying device; determining the height between the flying device and the first component, and guiding the flying device to land on the first component according to the height.

[0005] In a second aspect, an embodiment of the present application provides an image processing method, the method comprising: inputting the collected image data of the inspection target into the backbone network of the trained second network model for feature extraction to obtain a first feature map, a second feature map and a third feature map; performing feature fusion on the result of the first scale transformation of the third feature map and the second feature map to obtain a fourth feature map; performing feature fusion on the result of the second scale transformation of the fourth feature map and the first feature map to obtain a fifth feature map; wherein the first scale transformation and the second scale transformation are used to enlarge the size of the feature map; performing feature fusion on the result of the third scale transformation of the fifth feature map, the fourth feature map and the second feature map to obtain a sixth feature map; performing feature fusion on the result of the fourth scale transformation of the sixth feature map and the third feature map to obtain a seventh feature map; wherein the third scale transformation and the fourth scale transformation are used to reduce the size of the feature map; based on the fifth feature map, the sixth feature map and the seventh feature map, determine the fault type of the inspection target.

[0006] In a third aspect, an embodiment of the present application provides a landing device for a flying device, the device comprising: an inspection module, configured to inspect the inspection targets in a first map of the inspection site and a preset inspection task at the inspection site; a flight module, configured to fly to the sky above a first component of a land device upon completion of the preset inspection task or in the event of an abnormality in the flying device; the first component is used to park the flying device; a landing module, configured to determine the height between the flying device and the first component, and guide the flying device to land on the first component according to the height.

[0007] In a fourth aspect, an embodiment of the present application provides an image processing device, the device comprising: a feature extraction module, configured to input the collected image data of the inspection target into the backbone network of the trained second network model for feature extraction, and obtain a first feature map, a second feature map and a third feature map; a first feature fusion module, configured to perform feature fusion on the result of the first scale transformation of the third feature map and the second feature map to obtain a fourth feature map; a second feature fusion module, configured to perform feature fusion on the result of the second scale transformation of the fourth feature map and the first feature map to obtain a fifth feature map; wherein the first scale transformation and the second scale transformation are used to enlarge the size of the feature map; a third feature fusion module, configured to perform feature fusion on the result of the third scale transformation of the fifth feature map, the fourth feature map and the second feature map to obtain a sixth feature map; a fourth feature fusion module, configured to perform feature fusion on the result of the fourth scale transformation of the sixth feature map and the third feature map to obtain a seventh feature map; wherein the third scale transformation and the fourth scale transformation are used to reduce the size of the feature map; a determination module, configured to determine the fault type of the inspection target based on the fifth feature map, the sixth feature map and the seventh feature map.

[0008] In a fifth aspect, an embodiment of the present application provides a flight device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the first aspect of the embodiment of the present application is implemented.

[0009] In a sixth aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the method described in the second aspect of the embodiment of the present application is implemented.

[0010] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method provided in the embodiment of the present application.

[0011] In an eighth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the method provided in the embodiment of the present application.

[0012] It can be understood that in the embodiment of the present application, based on the first map of the inspection site and the inspection targets in the preset inspection tasks, the inspection targets are inspected at the inspection site; when the preset inspection tasks are completed, or when an abnormality occurs in the flying device, the flight is flown to the sky above the first component of the land equipment; the first component is used to park the flying device; the height between the flying device and the first component is determined, and the flying device is guided to land on the first component according to the height. In this way, by determining the height between the flying device and the first component, the flying device is guided to land on the first component according to the height; thereby avoiding the problem that the flying device cannot focus on the first component, causing the flying device to "smash" the first component in a vertical state.

[0013] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0015] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0016] Figure 1 A schematic diagram of a method for landing a flying device according to an embodiment of the present application;

[0017] Figure 2 A schematic diagram of an implementation flow of an inspection provided in an embodiment of the present application;

[0018] Figure 3 A schematic diagram of a differential value of a flight device attitude deflection angle provided in an embodiment of the present application;

[0019] Figure 4 A schematic diagram of a process flow for implementing a flying device to fly above a first component of a land device provided in an embodiment of the present application;

[0020] Figure 5 A schematic diagram of determining the height between the flying device and the first component provided in an embodiment of the present application;

[0021] Figure 6 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application;

[0022] Figure 7 A schematic diagram of an implementation flow of training a second network model provided in an embodiment of the present application;

[0023] Figure 8 A schematic diagram of a network structure of a second network model provided in an embodiment of the present application;

[0024] Fig. 9 A schematic diagram of a process flow for implementing a first component of a flying device landing on a land device provided in an embodiment of the present application;

[0025] Fig.10 A schematic diagram of the structure of a landing device of a flying device provided in an embodiment of the present application;

[0026] Fig.11 A schematic diagram of a heterogeneous inspection robot device for a computer room provided in an embodiment of the present application;

[0027] Fig.12 A rear side view of a heterogeneous inspection robot for a computer room provided in an embodiment of the present application;

[0028] Fig.13 A front view of a heterogeneous inspection robot for a computer room provided in an embodiment of the present application

[0029] Fig.14 A schematic diagram of an inspection drone in a heterogeneous inspection robot for a computer room provided in an embodiment of the present application, waiting on a landing plate;

[0030] Fig.15 An internal hardware connection diagram of a heterogeneous inspection robot for a computer room provided in an embodiment of the present application;

[0031] Fig.16 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0032] Fig.17 A schematic diagram of the structure of the flight equipment provided in the embodiment of the present application;

[0033] Fig.18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the specific technical solution of the present application will be further described in detail below in conjunction with the drawings in the embodiments of the present application. The following embodiments are used to illustrate the present application, but are not used to limit the scope of the present application.

[0035] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0036] In the following description, reference is made to “some embodiments”, “this embodiment”, “embodiments of the present application” and examples, etc., which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] The descriptions of “first, second, third” etc. that appear in the embodiments of the present application are only for illustration and distinction of the description objects. There is no distinction of order, nor do they indicate any special limitation on the number of devices in the embodiments of the present application, and cannot constitute any limitation on the embodiments of the present application.

[0038] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies or terms of the embodiments of the present application. The following related technologies or related terms as optional solutions can be combined with the technical solutions of the embodiments of the present application at will, and they all belong to the protection scope of the embodiments of the present application.

[0039] In a related technology, a machine room inspection robot is provided; the robot includes a round shell, a semicircular cover plate fixedly connected to the top of the round shell by bolts, and a central processing system, and the top of the semicircular cover plate is fixedly connected to a square top, the top of the square top is fixedly connected to a camera, the top of the square top and the front of the camera is fixedly connected to an air parameter sensor, the front of the top of the square top is fixedly connected to a laser radar, and the front of the square top is fixedly connected to a depth camera. The machine room inspection robot can realize functions such as autonomous inspection, automatic fault identification, automatic generation of inspection reports, and remote control of the robot to a designated position in the machine room. At the same time, through the embedded 5G communication module, the inspection robot can realize the real-time transmission of 5G high-definition video and massive data, as well as the remote visual intercom communication function; this technology greatly improves the inspection efficiency through intelligent inspection settings, avoids fault omissions, and meets actual usage needs.

[0040] In another related technology, in order to solve the problem that the long-term use of the inspection robot will cause the camera lens to be covered with dust, resulting in unclear collected images and errors in the collected data, an artificial intelligence inspection robot supporting deep learning is proposed; the robot includes a body, and the body is provided with a walking module, an image acquisition module, an environment acquisition module, a cleaning module, a robot main control module, a network module, a lighting module, a control panel module, an alarm module, a positioning module and a power module, and the image acquisition module is a camera installed on the front of the body. The artificial intelligence computer room inspection robot supporting the working principle of deep learning is provided to ensure the stable operation of the computer room, and the cleaning module is used to deal with the dust on the inspection robot camera to avoid problems such as unclear collected images caused by the lens being covered with dust, thereby effectively avoiding the problem of errors in collected data.

[0041] However, the above technical solution has the following three shortcomings:

[0042] (1) Using a single land wheeled robot. When completing autonomous inspection tasks, its field of view is greatly limited, that is, the sensing area is small. Wheeled robots have limitations in inspecting target areas, such as inspections at higher positions, that is, the efficiency of completing tasks is low, and it is difficult to meet daily inspection tasks.

[0043] (2) It does not involve the use of deep reinforcement learning methods to achieve intelligent perception and advanced decision-making of the inspection robot, that is, the computer room environment is perceived through frameworks such as deep learning, and the advanced motion decision-making of the inspection robot is completed through reinforcement learning.

[0044] (3) It does not involve the autonomous landing method of the inspection drone based on multi-sensor fusion. However, the inspection drone can only land with the guidance of a single visual sign, which makes it difficult to achieve a high-precision, smooth and autonomous landing. This is because when the drone is close to the parking disk based on visual guidance alone, it cannot focus on the parking disk, so it cannot achieve the ranging and positioning function, and can only "smash" the parking disk in a vertical state, thus damaging the machine.

[0045] In view of this, an embodiment of the present application provides a method for landing a flying device. Figure 1 A schematic diagram of a method for implementing a landing of a flying device provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps 101 to 103:

[0046] Step 101, based on a first map of the inspection area and an inspection target in a preset inspection task, inspect the inspection target at the inspection area;

[0047] Step 102, when the preset inspection task is completed, or when an abnormality occurs in the flying device, flying to the air above the first component of the land device; the first component is used to park the flying device;

[0048] Step 103: determine the height between the flying device and the first component, and guide the flying device to land on the first component according to the height.

[0049] It can be understood that in the embodiment of the present application, based on the first map of the inspection site and the inspection targets in the preset inspection tasks, the inspection targets are inspected at the inspection site; when the preset inspection tasks are completed, or when an abnormality occurs in the flying device, the flight is flown to the sky above the first component of the land equipment; the first component is used to park the flying device; the height between the flying device and the first component is determined, and the flying device is guided to land on the first component according to the height. In this way, by determining the height between the flying device and the first component, the flying device is guided to land on the first component according to the height; thereby avoiding the problem that the flying device cannot focus on the first component, causing the flying device to "smash" the first component in a vertical state.

[0050] The following describes further optional implementations and related terms of each of the above steps.

[0051] In step 101, based on a first map of the inspection site and an inspection target in a preset inspection task, the inspection target is inspected at the inspection site.

[0052] It should be understood that in the embodiments of the present application, the inspection site is not limited. In some embodiments, the inspection site includes at least one of the following: a data center, a power distribution area, an industrial plant, a warehouse, a shopping mall, an office building, a forest, a dam, a mining area, a station, a track, an explosion-proof area, a chemical area, a bridge, a road, a pipeline, a substation, a transmission line, etc.

[0053] In the embodiments of the present application, the first map is not limited. In some embodiments, the first map is constructed by the land device and sent to the flying device; in other embodiments, the first map is sent to the flying device by a cloud server; in still other embodiments, the first map is stored in the flying device.

[0054] In the embodiment of the present application, the first map is updated based on changes in the environment, that is, the first map is changing rather than being constant.

[0055] In some embodiments, the method also includes: determining a first map; wherein, determining the first map includes: determining first map data collected by land equipment; determining altitude information of flying equipment; determining second map data collected by flying equipment; and determining the first map based on the first map information, altitude information, and second map information.

[0056] For example, as a possible implementation method, the inspection site is the machine room, and the land wheeled robot (i.e., an example of land equipment) uses laser radar, depth camera and other perception sensors based on the Simultaneous Localization and Mapping (SLAM) algorithm to build the machine room environment map, and share the map with the inspection drone. The inspection drone (i.e., an example of flying equipment) takes off, and the land wheeled robot first receives the altitude information of the inspection drone after takeoff in real time, as well as the environmental information perceived by the inspection drone's field of view. The land wheeled robot integrates the altitude information and environmental information with the original established map to update the environmental map.

[0057] In some embodiments, the inspection of the inspection target includes: collecting image data of the inspection target, and sending the image data to the land device, so that the land device determines the fault type of the inspection target based on the image data through a trained second network model.

[0058] It should be understood that in the embodiments of the present application, the image data is not limited. In some embodiments, the image data includes: a real-time video stream of the inspection target collected by the flight equipment, that is, the image data includes: an image set of multiple image frames.

[0059] In the embodiment of the present application, the fault type is not limited, and the inspection location and the inspection target are different, and the fault type is also different. The fault type is determined by processing the image data collected by the flight equipment, so the fault type includes: a fault type that can be determined by visualization.

[0060] In some embodiments, when the inspection site is a computer room and the inspection target is a cabinet, the fault type includes but is not limited to at least one of the following: abnormal display light of the cabinet, abnormal temperature and humidity indicator light of the cabinet, abnormal equipment abnormality indicator light of the cabinet, the cabinet door of the cabinet is open, the ground / floor where the cabinet is located is damaged, abnormal instruments of the elevated HVAC equipment in the computer room, broken cables, open flames, smoke, people falling, etc.

[0061] In some embodiments, based on the image data, the fault type of the inspection target is determined through a trained second network model, including: receiving image data of the inspection target collected by a flight device; inputting the image data into the trained second network model to obtain the fault type of the inspection target.

[0062] In other embodiments, based on the image data, the fault type of the inspection target is determined through a trained second network model, including: sending the image data of the inspection target collected by the received flight equipment to a cloud server, so that the cloud server inputs the image data into the trained second network model to obtain the fault type of the inspection target.

[0063] In some embodiments, Figure 2 A schematic diagram of an implementation flow of an inspection provided in an embodiment of the present application is shown in FIG. Figure 2 As shown, the inspection of the inspection target can be achieved through the following steps 201 to 203:

[0064] Step 201, obtaining status information of the flight equipment;

[0065] Step 202, sending the state information to a cloud device, so that the cloud device obtains a flight decision of the flight device based on the state information through a trained third network model; and sending the flight decision to the flight device; the trained third neural network model is obtained through reinforcement learning training;

[0066] Step 203, executing the received flight decision.

[0067] It can be understood that in the embodiment of the present application, the state information of the flight device is obtained; the state information is sent to the cloud device, so that the cloud device obtains the flight decision of the flight device based on the state information through the trained third network model; and the flight decision is sent to the flight device; the trained third neural network model is obtained through reinforcement learning training; and the received flight decision is executed. In this way, since the trained third network model is obtained through reinforcement learning training, it is beneficial to make the training data of the flight decision model based on reinforcement learning more consistent with Markov properties, so that the obtained flight decision is better.

[0068] It should be understood that in the embodiments of the present application, the third network model is not limited. In some embodiments, the trained third network model includes at least one of the following neural network models: a recurrent convolutional neural network, a generative adversarial convolutional neural network, a long short-term memory network, a feedforward neural network, a multi-layer perceptron, a convolutional neural network including an attention mechanism, a deep belief network, etc.

[0069] In some embodiments, the trained third network model is trained based on at least one of the following information: the rotational speed of the cable shaft, the differential value of the attitude angle of the flying device, the delay value of the status information, and the distance information between the power module of the flying device and the flying device on the first map; wherein, the cable shaft is used to connect the flying device with the land device so that the land device can supply power to the flying device.

[0070] It can be understood that in the embodiment of the present application, the third network model is trained through the rotation speed of the cable shaft, the differential component of the attitude deflection angle of the flight device, the delay of the state information, and the distance information between the power module of the flight device and the flight device on the first map to obtain the trained third network model. In this way, the trained third network model conforms to the Markov property, which is beneficial to obtain a better flight decision.

[0071] In some embodiments, the difference in the attitude angle of the flying device is the difference between the attitude angle of the flying device at a current moment and the attitude angle of the flying device at a previous moment.

[0072] It should be understood that the inertia of the quadcopter drone is large and is greatly affected by wind resistance. The quadcopter drone itself is a typical under-actuated system and cannot independently control position and attitude. By controlling the speed of the four drive motors of the quadcopter drone to control lift and torque, the attitude angle of the quadcopter drone is changed, and then the position of the quadcopter drone is changed. This is also one of the reasons why its action execution has hysteresis. In addition, the system belongs to a heterogeneous collaborative system, and the interaction of different agents (i.e., flight equipment and land equipment) increases the hysteresis. After observing the current state, the deep reinforcement learning agent of the quadcopter drone needs a certain amount of calculation time to make a flight decision. After making a flight decision, due to the wind resistance and the inertia of the drone propeller, it also takes time for the actual action to be executed. The speed of the cable shaft also affects the execution effect of the flight decision to a certain extent. Therefore, the speed of the cable shaft, the differential value of the attitude angle of the flight equipment, the delay amount of the state information, and the distance information between the power module of the flight equipment and the flight equipment on the first map are beneficial to make the training data of the trained third network model have Markov properties.

[0073] For example, as a possible implementation method, in order to realize the coordinated autonomous intelligent control of heterogeneous inspection robots and optimize the parameters in the robot state information that do not conform to the Markov property, a collaborative path control method for the inspection robot in the computer room based on deep reinforcement learning is proposed. The method is obtained by training through the following steps:

[0074] Step 11: Initialize action networks B1, B2 and evaluation network B1′ and B2 ′ network parameters.

[0075] In the process of initializing the network parameters, if it is the first training, the network weights are randomly initialized. If it is not the first training, the network is initialized to the network parameters saved at the end of the previous test. The parameter i is used to count the update of the decision network. The target network is updated only after the decision network is updated n times. The sampling interval of environmental data for the land wheeled robot and the inspection drone is set to T. The decision network is updated once every T, and the target network is updated once every s·Ts seconds. The decision network and the target network have the same structure, but the update method and update frequency are different.

[0076] Step 12: Sample status data from the experience pool to obtain the current status information M of the inspection drone t The current state information N of the land wheeled robot t , M t Including the current rotation speed of the power cable shaft, the location information of the inspection drone, speed information, detected obstacle information, the rudder angle position of the drone at the previous moment, and the power information output by the rotor propeller; N t It includes the current position information, speed information, detected obstacle information, motor rudder angle position of the land wheeled robot at the previous moment, and the distance information between it and the inspection drone on the shared map (i.e., an example of the first map).

[0077] Step 13: Introduce the difference of the drone attitude angle θ and the delay of the state information to reconstruct the state and obtain M t ′ In view of the large inertia of the inspection rotor UAV, the differential component of the UAV attitude angle θ is introduced. Figure 3 A schematic diagram of a differential component of a flight device attitude deflection angle provided in an embodiment of the present application, such as Figure 3 As shown, the attitude deflection angle is the angle between the coordinate plane L1 where the inspection drone is located and the reference plane L2, and the reference plane L2 is a plane parallel to the first component plane L3 and passing through the centroid O of the inspection drone. In view of the hysteresis of the heterogeneous collaborative system, the delay amount of the state information is introduced into the state information. In a discrete system, such a state reconstruction operation makes the state data that does not conform to the Markov property conform to the Markov property to a certain extent.

[0078] Step 14: Set state M t ′ and N t ′ Substitute into the decision network B and according to the strategy π(a|s), get action a m 、a n and reward r m、r n , and set the rewards obtained by heterogeneous inspection robots in detail to prevent the low efficiency of reinforcement learning training caused by the sparse reward problem. Among them, the decision network includes the action network and the evaluation network.

[0079] Step 15: Execute the action and enter the next state M t+1 and N t+1 , and preprocess to obtain state M t+1 ′ and N t+1 ′ .

[0080] Step 16: (M t ′ ,N t ′ ,M t+1 ′ ,N t+1 ′ ,a m ,a n ,r m ,r n ) and sampling priority are saved as a string of experience data to the experience pool.

[0081] Step 17: Randomly select k strings of data and put them into the target network based on the principle that sampling priority is sampling probability, and obtain the loss function loss.

[0082] Step 18: Update the decision network B by loss loss, j=0, j++.

[0083] Step 19: If j>=s, update the target network B once through the parameters of the decision network B. ′ .

[0084] Step 10: Determine whether to end the training. If not, return to step 12; if yes, end the training.

[0085] In step 102, when the preset inspection task is completed, or when an abnormality occurs in the flying device, the flying device is flown to the air above the first component of the land device; the first component is used to park the flying device.

[0086] It should be understood that in the embodiments of the present application, there is no limitation on the abnormality of the flight device. In some embodiments, the abnormality of the flight device includes: insufficient power of the flight device, or failure of the flight device; wherein the failure includes: signal interference to the flight device, electromagnetic interference to the flight device, damage to the propeller of the flight device, overheating of the motor of the flight device, sensor failure of the flight device, error in the flight control software of the flight device, etc.

[0087] It should be understood that in the embodiments of the present application, the flying device is not limited. In some embodiments, the flying device includes: a rotary wing drone. In the embodiments of the present application, the land equipment is not limited. In some embodiments, the land equipment includes: a wheeled robot. In the embodiments of the present application, the first component is not limited. In some embodiments, the first component includes: a helipad of the land equipment; wherein the helipad is used to park the flying device.

[0088] In some embodiments, Figure 4 A schematic diagram of a process flow for implementing a flying device to fly above a first component of a land device provided in an embodiment of the present application is as follows: Figure 4 As shown, the following steps 401 to 404 can be used to fly to the air above the first component;

[0089] Step 401, determining first location information of the land device;

[0090] Step 402, flying to the airspace above the land equipment based on the first position information;

[0091] Step 403, determining second location information of the first component;

[0092] Step 404: Fly to the airspace above the first component based on the second position information.

[0093] It can be understood that in the embodiment of the present application, the flight is first performed over the land device based on the determined first position information of the land device; and then the flight is performed over the first component based on the second position information of the first component. In this way, the flight is performed over the first component of the land device based not only on the first position information of the land device but also on the second position information of the first component, so that the flight result is accurate, which is beneficial to improve the accuracy of landing on the first component.

[0094] It should be understood that in the embodiments of the present application, there is no limitation on the determination of the first position information of the land device. In some embodiments, the first position information is sent by the land device to the flying device. In the embodiments of the present application, there is no limitation on the determination of the second position information of the first component. In some embodiments, the second position information is detected by the flying device; in other embodiments, the second position information is sent by the land device to the flying device.

[0095] In some embodiments, the flight to the sky above the first component of the land equipment includes but is not limited to combining the global map location information with deep learning detection and tracking algorithms to detect and lock the first component, track and accompany the first component, fly to the sky above the first component of the land equipment, and then enter the next step of autonomous landing.

[0096] In some embodiments, the method further includes: after flying over a first component of the land equipment, detecting and selecting the first component based on a trained first network model; locking the selected first component to track the first component and lower the flight altitude.

[0097] It can be understood that in the embodiment of the present application, after flying over the first component of the land equipment, the first component is detected and selected based on the trained first network model; the selected first component is locked to track the first component and reduce the flight altitude. In this way, the first component is detected and selected based on the trained first network model, thereby effectively solving the problem of manually selecting the first component.

[0098] It should be understood that in the embodiments of the present application, the trained first network model is not limited, and the trained first network model is used to detect and select the first component. In some embodiments, the trained first network model includes at least one of the following neural network models: a recurrent convolutional neural network, a generative adversarial convolutional neural network, a long short-term memory network, a feedforward neural network, a multi-layer perceptron, a convolutional neural network including an attention mechanism, a deep belief network, etc.

[0099] In the embodiment of the present application, the specific implementation method of locking the first component selected in the box to track the first component is not limited. In some embodiments, the first component selected in the box is locked through a kernel correlation filtering framework to track the first component and reduce the flight height.

[0100] In step 103 , the height between the flying device and the first component is determined, and the flying device is guided to land on the first component according to the height.

[0101] It should be understood that in the embodiments of the present application, the specific implementation method for determining the height between the flying device and the first component is not limited. In some embodiments, the height is sent by the land device to the flying device. In other embodiments, the height is determined by measuring the height between the land device and the flying device using a ranging sensor.

[0102] It should be understood that in the embodiments of the present application, the distance measuring sensor is not limited. In some embodiments, the distance measuring sensor is an ultrasonic distance measuring sensor; in other embodiments, the distance measuring sensor is a laser distance measuring sensor; in still other embodiments, the distance measuring sensor is an infrared distance measuring sensor.

[0103] For example, as a possible implementation method: in order to realize the autonomous collaborative operation of the inspection UAV (i.e., an example of flying equipment) and the mother land wheeled robot (i.e., an example of land equipment), the inspection UAV needs to realize autonomous take-off and landing on the dynamic mother land wheeled robot. The specific steps of the inspection UAV autonomous landing method based on multi-sensor fusion are as follows:

[0104] Step 21, the mother land wheeled robot shares the indoor map (i.e., an example of the first map) with the inspection drone through the communication module, and obtains positioning data from each other.

[0105] In the embodiment of the present application, mutually obtaining positioning data means that the mother land wheeled robot obtains the positioning data of the inspection drone, and the inspection drone obtains the positioning data of the mother land wheeled robot. In this way, not only the inspection drone flies to the location of the mother land wheeled robot, but the mother land wheeled robot can also move toward the inspection drone to improve landing efficiency.

[0106] Step 22, the inspection drone cruises toward the target location based on the positioning data and visual detection guidance of the mother land wheeled robot, and arrives above the apron of the mother land wheeled robot.

[0107] Step 23, use the visual detection algorithm integrated with deep learning and the kernel correlation filter framework to accurately detect and track the landing pad, and use binocular ranging to solve the distance Z between the land wheeled robot and the landing pad; and dynamically track the distance Z and reduce the flight altitude.

[0108] First, the YOLOv7 (You Only Look Once version 7) target detection algorithm is used to detect and autonomously select the landing pad, and then the Kernel Correlation Filter (KCF) algorithm is used to lock and track the detected and selected apron. In this way, the problem that KCF requires manual selection and tracking of the target object can be effectively solved, and autonomous detection, selection and tracking of the apron can be achieved. KCF is a key step in solving the problem that it is difficult to accurately and autonomously land on the mother aircraft landing disk based on the coordinate information shared by the map, because the real-time shared map (i.e., an example of the first map) only provides the coordinates of the entire mother aircraft land wheeled robot in the map, allowing the inspection drone to quickly reach the top of the mother aircraft land wheeled robot, but it cannot accurately determine the location of the specific parts on the mother aircraft land wheeled robot, that is, the landing pad location.

[0109] Furthermore, the distance Z is calculated by the following formula:

[0110]

[0111] Among them, the target point M is two points M on the imaging plane l 、M r The coordinates of x are l , x r , where x l -x r represents the parallax, T represents the baseline distance between the two cameras, and f represents the focal length.

[0112] Step 14, determine whether the height between the inspection UAV and the apron is less than the height threshold. If so, start the infrared ranging module for close-range ranging to guide the inspection UAV to land autonomously; if not, update the indoor shared real-time map information and return to step 22 or step 23, re-detect positioning and tracking ranging until the inspection UAV completes autonomous landing.

[0113] Among them, when it is judged that the height between the inspection drone and the apron is less than the height threshold, the distance D is calculated by the following formula:

[0114]

[0115] Among them, Figure 5 As shown, L refers to the distance between the position where the reflected light falls on the detector plane and the center of the plane; f is the focal length of the light-receiving lens; and x is the distance between the filter and the center of the infrared light emitter.

[0116] It should be understood that in the embodiment of the present application, when the height between the inspection drone and the apron is greater than or equal to the height threshold, the process returns to step 22 because the inspection drone may deviate due to obstacles during its flight to the mother land wheeled robot, i.e., it may not be guided to the correct position. The process returns to step 23 because the inspection drone has not landed at the height threshold.

[0117] In the embodiment of the present application, real-time shared indoor maps, visual detection guidance, ultra-wideband and laser ranging are combined, and different ground-fixed altitude ranging schemes are adopted from far to near according to the position of the rotorcraft UAV from the landing pad to achieve high-precision, smooth and autonomous landing of the inspection UAV without human intervention.

[0118] The above-mentioned method for autonomous landing of inspection drones based on multi-sensor fusion is still applicable to wireless inspections without using power cables for drones. The wired power supply solution adopted in the embodiment of the present application for the drones with weak endurance does not limit the method for autonomous landing of inspection drones based on multi-sensor fusion provided in the embodiment of the present application.

[0119] The present application provides an image processing method. Figure 6 A schematic diagram of an implementation flow of an image processing method provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the method includes the following steps 601 to 606:

[0120] Step 601, inputting the collected image data of the inspection target into the backbone network of the trained second network model for feature extraction to obtain a first feature map, a second feature map and a third feature map;

[0121] Step 602, performing feature fusion on the result of performing the first scale transformation on the third feature map and the second feature map to obtain a fourth feature map;

[0122] Step 603, performing feature fusion on a result of performing a second scale transformation on the fourth feature map and the first feature map to obtain a fifth feature map; wherein the first scale transformation and the second scale transformation are used to enlarge the size of the feature map;

[0123] Step 604, performing feature fusion on the result of performing the third scale transformation on the fifth feature map, the fourth feature map, and the second feature map to obtain a sixth feature map;

[0124] Step 605, performing feature fusion on a result of performing a fourth scale transformation on the sixth feature map and the third feature map to obtain a seventh feature map; wherein the third scale transformation and the fourth scale transformation are used to reduce the size of the feature map;

[0125] Step 606: Determine the fault type of the inspection target based on the fifth characteristic graph, the sixth characteristic graph, and the seventh characteristic graph.

[0126] It can be understood that in the embodiment of the present application, the backbone network of the trained second network model is used to perform feature extraction on the collected image data to obtain a first feature map, a second feature map and a third feature map; a feature map obtained by feature extraction is fused with other feature maps to obtain fused feature maps of different scales, and then the fault type of the inspection target is determined based on the fused feature map; thereby, it is beneficial to improve the accuracy of the determined fault type of the inspection target and reduce the speed of determining the fault type of the inspection target.

[0127] It should be understood that in the embodiments of the present application, the backbone network is not limited, and the backbone network is used to extract features from the image data. In some embodiments, the backbone network can be at least one of the following: MobileNetwork, EfficientNet, Residual Network (ResNet), Visual Geometry Group Network (VGG), GhostNet, Google net, Inception Network, etc.

[0128] It should be understood that in the embodiment of the present application, the first scale transform and the second scale transform are not limited, the first scale transform and the second scale transform are used to enlarge the size of the feature map, and the first scale transform and the second scale transform may be the same or different. In some embodiments, the first scale transform and the second scale transform include at least one of the following: deconvolution, upsampling (for example: bilinear interpolation, nearest neighbor interpolation), pooling, etc.

[0129] In the embodiment of the present application, the third scale transform and the fourth scale transform are not limited, and the third scale transform and the fourth scale transform are used to reduce the size of the feature map, and the third scale transform and the fourth scale transform may be the same or different. In some embodiments, the third scale transform and the fourth scale transform include at least one of the following: convolution (for example, increasing the stride of the convolution kernel, using a large-size convolution kernel, not using padding or less padding, etc.), maximum pooling, average pooling, global pooling, downsampling (for example, average method, Gaussian blur method, etc.), using 1×1 convolution kernel, hole convolution, etc.

[0130] In some embodiments, the trained second network model also includes: a feature extraction module, a prediction module; determining the fault type of the inspection target based on the fifth feature map, the sixth feature map and the seventh feature map, includes: inputting the fifth feature map, the sixth feature map and the seventh feature map into the feature extraction module for feature extraction, respectively, to obtain an eighth feature map corresponding to the fifth feature map, a ninth feature map corresponding to the sixth feature map and a tenth feature map corresponding to the seventh feature map; inputting the eighth feature map, the ninth feature map and the tenth feature map into the prediction module to obtain the fault type of the inspection target.

[0131] It can be understood that in the embodiment of the present application, the fifth characteristic map, the sixth characteristic map and the seventh characteristic map are respectively input into the feature extraction module for feature extraction, and the eighth characteristic map corresponding to the fifth characteristic map, the ninth characteristic map corresponding to the sixth characteristic map and the tenth characteristic map corresponding to the seventh characteristic map are obtained; the eighth characteristic map, the ninth characteristic map and the tenth characteristic map are input into the prediction module to obtain the fault type of the inspection target. In this way, after the feature fusion is performed, the enhanced feature extraction is also performed, and then the feature map obtained by the enhanced feature extraction is input into the prediction module to obtain the fault type of the inspection target, which is beneficial to improve the accuracy of the predicted fault type.

[0132] It should be understood that the feature extraction module is not limited in the embodiments of the present application. In some embodiments, the feature extraction module is: an improved receptive field block (RFB).

[0133] In some embodiments, the fifth feature map, the sixth feature map, and the seventh feature map are respectively input into the feature extraction module for feature extraction to obtain an eighth feature map corresponding to the fifth feature map, a ninth feature map corresponding to the sixth feature map, and a tenth feature map corresponding to the seventh feature map, including: upsampling the i-th feature map, performing a first 1×1 convolution on the upsampling result, performing a second 1×1 convolution, a 1×3 convolution, a 3×1 convolution, and a first 3×3 convolution on the result of the first 1×1 convolution, and performing the second 1×1 convolution on the result of the upsampling. The result of the 1×3 convolution, the result of the 3×1 convolution, and the result of the first 3×3 convolution are respectively subjected to a second 3×3 convolution, and the result of the second 1×1 convolution, the result of the 1×3 convolution, the result of the 3×1 convolution, and the result of the first 3×3 convolution are respectively subjected to a third 1×1 convolution to obtain an eleventh feature map; the eleventh feature map is feature fused with the result of the upsampling to obtain a twelfth feature map, and the twelfth feature map is input into the ReLU activation layer to obtain the i+3th feature map; wherein, i is 5 or 6 or 7.

[0134] In some embodiments, the fifth feature map, the sixth feature map and the seventh feature map are respectively input into the feature extraction module for feature extraction to obtain an eighth feature map corresponding to the fifth feature map, a ninth feature map corresponding to the sixth feature map and a tenth feature map corresponding to the seventh feature map, including: performing a first 1×1 convolution on the i-th feature map, performing a first 3×3 convolution, a second 3×3 convolution and a 5×5 convolution on the result of the first 1×1 convolution, performing a third 3×3 convolution on the result of the second 3×3 convolution, performing a fourth 3×3 convolution on the result of the 5×5 convolution, performing a first feature fusion on the result of the first 3×3 convolution, the result of the third 3×3 convolution and the result of the fourth 3×3 convolution to obtain a thirteenth feature map, performing a second 1×1 convolution on the thirteenth feature map to obtain a fourteenth feature map, performing a second feature fusion on the fourteenth feature map and the i-th feature map to obtain a fifteenth feature map; inputting the fifteenth feature map into the ReLU activation layer to obtain an i+3th feature map; wherein, i is 5 or 6 or 7.

[0135] In the embodiment of the present application, the prediction module is not limited, and the prediction module is used to predict the fault type of the inspection target. In some embodiments, the prediction module includes at least one of the following: YOLOv5, convolutional neural network, deep separable convolution, multi-scale feature fusion, attention mechanism, recurrent neural network (RNN), long short-term memory network (LSTM), integrated learning method, etc.

[0136] For example, in some embodiments, Figure 7 A schematic diagram of an implementation flow of training a second network model provided in an embodiment of the present application is as follows: Figure 7 As shown, the second network model can be trained through the following steps 701 to 706:

[0137] Step 701, input a training image;

[0138] Step 702, backbone network extracts features;

[0139] Step 703, feature fusion;

[0140] Step 704, the detection head predicts the output;

[0141] Step 705, whether the average ΔLOSS is less than 0.001; if so, execute step 706, otherwise execute step 701;

[0142] Step 706, terminate the training and output the optimal prediction weight.

[0143] For example, as a possible implementation, in order to achieve autonomous high-precision fault detection, Figure 8A schematic diagram of a network structure of a second network model provided in an embodiment of the present application, such as Figure 8 As shown, 801 represents a training image, 802 represents a backbone network, 803 represents a feature fusion and enhanced feature extraction module, module 803 includes a feature fusion module 8031 ​​and an enhanced feature extraction module 8032, and 804 represents a prediction module. Among them, Conv represents convolution, Shortcut represents direct connection, Conv2d represents two-dimensional convolution, CA-Bneck represents a neural network module that combines a channel attention mechanism and a bottleneck structure; the outputs Cls, Reg, and obj of the prediction module 804 represent three different prediction components respectively; specifically, Cls represents category prediction (Classification, Cls), Reg represents bounding box regression (bounding box Regression, Reg), and obj represents objectness probability (objectness, obj).

[0144] The specific training steps of the second network model include: inputting the real-time video stream data (i.e., training image 801) sent back by the inspection drone; sending the training data to the backbone network MobileNetv3 802 to obtain three effective feature layers Q i , i = 1, 2, 3; the three effective features Q i As the input of module 8031, enhanced feature extraction and fusion are performed to obtain effective features F j , j = 1, 2, 3; then F j Enter module 8032 to obtain the final effective feature L n , n=1, 2, 3; finally, the prediction module 804 outputs the predicted fault result (ie, an example of the fault type); sets the model training termination condition, terminates the model training and outputs the optimal prediction weight file for fault identification and detection.

[0145] Among them, module 8031 ​​is a multi-scale feature fusion network, and the backbone network MobileNetv3 is a lightweight convolutional neural network. The network adopts multiple optimization strategies, combining 3×3 standard convolution and bottleneck structure (Bneck) to enhance feature extraction, and adding the maximum pooling layer, and using 1×1 convolution blocks instead of fully connected layers to reduce the number of parameters and complexity. In addition, MobileNetV3 also introduces the Squeeze-and-Excitation Networks (SE) module to improve the recognition ability of the model.

[0146] In the embodiment of the present application, the structural specifications of the backbone network provided are shown in the following table.

[0147] Table 1 Structural specifications of the backbone network

[0148] enter Operator Parameter quantity Output Step Length <![CDATA[224 2 ×3]]> conv2d - 16 2 <![CDATA[112 2 ×16]]> Bneck, 3×3 16 16 1 <![CDATA[112 2 ×16]]> Bneck, 3×3 64 24 2 <![CDATA[56 2 ×24]]> Bneck, 3×3 72 24 1 <![CDATA[56 2 ×24]]> Bneck, 5×5 72 40 2 <![CDATA[28 2 ×40]]> Bneck, 5×5 120 40 1 <![CDATA[28 2 ×40]]> Bneck, 5×5 120 40 1 <![CDATA[28 2 ×40]]> Bneck, 3×3 240 80 2 <![CDATA[14 2 ×80]]> Bneck, 3×3 200 80 1 <![CDATA[14 2 ×80]]> Bneck, 3×3 184 80 1 <![CDATA[14 2 ×80]]> Bneck, 3×3 184 80 1 <![CDATA[14 2 ×80]]> Bneck, 3×3 480 112 1 <![CDATA[14 2 ×112]]> Bneck, 3×3 672 112 1 <![CDATA[14 2 ×112]]> Bneck, 5×5 672 160 2 <![CDATA[7 2 ×160]]> Bneck, 5×5 960 160 1

[0149] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0150] The embodiment of the present application provides a method for intelligently controlling a heterogeneous inspection robot in a computer room based on deep reinforcement learning, which autonomously realizes intelligent perception detection and advanced motion decision-making of the heterogeneous inspection robot. It is achieved through the following technical solutions:

[0151] Step 31: Start the robot, send commands to start each node and start inspection through the development host, receive the inspection location, start the power cable shaft and rotate in coordination with the take-off height of the inspection drone to release the cable of a reasonable length.

[0152] Step 32: The land wheeled robot uses laser radar, depth camera and other perception sensors based on the SLAM algorithm to create a computer room environment map and share the map with the inspection drone.

[0153] Step 33: The inspection UAV takes off. The land wheeled robot first receives the altitude information of the inspection UAV after takeoff in real time, as well as the environment perceived by the inspection UAV. The land wheeled robot integrates it with the original map and updates the environment map.

[0154] Step 34: According to the inspection location and the updated map, the inspection robot autonomously navigates to the inspection cabinet according to the collaborative path control method of the computer room inspection robot based on deep reinforcement learning.

[0155] Step 35: Detect the temperature and humidity of the cabinet through the temperature and humidity sensor, obtain the real-time video stream of the current cabinet through the visual sensor, and the inspection drone transmits the collected temperature and humidity information and the real-time video stream of the cabinet back to the land wheeled robot.

[0156] Step 36: The land wheeled robot sends the received real-time data to the trained prediction weights of the deep learning-based fault identification method for detection and identification, and outputs the detection results to the display screen and the development host. In this way, the inspection of all cabinets and chassis is completed and the detection report is output autonomously. After evaluation by engineers, it is uploaded and stored in the cloud.

[0157] Step 37: After the inspection is completed, the inspection UAV autonomously completes autonomous landing according to the inspection UAV autonomous landing method based on multi-sensor fusion.

[0158] Fig. 9 A schematic diagram of a process flow for implementing a first component of a flying device landing on a land device provided in an embodiment of the present application is as follows: Fig. 9 As shown, the first component of the land equipment can be landed through the following steps 901 to 908:

[0159] Step 901, the inspection drone approaches the mother machine (i.e., land equipment) based on the shared indoor map and visual detection guidance;

[0160] Step 902, reaching above the mother machine shutdown disk (i.e., the first component);

[0161] Step 903, integrating deep learning detection and kernel correlation filtering to autonomously lock and track the downed disk;

[0162] Step 904, binocular distance measurement, autonomously lowering the flight altitude;

[0163] Step 905, whether the distance is less than the threshold; if so, execute step 907, otherwise execute step 906;

[0164] Step 906, updating the shared indoor real-time SLAM map information;

[0165] Step 907, turn on the infrared ranging module to measure the distance at close range and guide the autonomous landing.

[0166] In an embodiment of the present application, the above method is applicable to data center security technologies including at least one of the following: information technology (IT) equipment safety, dynamic environment equipment safety, electrical safety, data security, building safety, fire safety, and personnel safety.

[0167] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.; or, steps in different embodiments may be combined into a new technical solution.

[0168] Based on the foregoing embodiments, an embodiment of the present application provides an access control device, which includes the modules included and the units included in the modules, which can be implemented by a processor; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be an AI acceleration engine (such as NPU, etc.), GPU, central processing unit (CPU), microprocessor (MPU), digital signal processor (DSP) or field programmable gate array (FPGA), etc.

[0169] Fig.10 A schematic diagram of the structure of a landing device of a flying device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the device 100 includes an inspection module 1001, a flight module 1002 and a landing module 1003, wherein:

[0170] The inspection module 1001 is configured to inspect the inspection target at the inspection site based on a first map of the inspection site and the inspection target in a preset inspection task;

[0171] The flight module 1002 is configured to fly to the air above the first component of the land equipment when the preset inspection task is completed or when an abnormality occurs to the flight equipment; the first component is used to park the flight equipment;

[0172] The landing module 1003 is configured to determine the height between the flying device and the first component, and guide the flying device to land on the first component according to the height.

[0173] In some embodiments, the flight module 1002 is configured to determine the first position information of the land equipment; based on the first position information, fly above the land equipment; determine the second position information of the first component; based on the second position information, fly above the first component.

[0174] In some embodiments, the device 100 also includes a detection and selection module and a locking and tracking module; wherein the detection and selection module is configured to fly over the first component of the land equipment, and then detect and select the first component based on the trained first network model; the locking and tracking module is configured to lock the selected first component to track the first component and lower the flight altitude.

[0175] In some embodiments, the inspection module 1001 is configured to collect image data of the inspection target and send the image data to the land device, so that the land device determines the fault type of the inspection target based on the image data through a trained second network model.

[0176] In some embodiments, the inspection module 1001 is configured to obtain status information of the flight equipment; send the status information to a cloud device so that the cloud device obtains a flight decision of the flight equipment based on the status information through a trained third network model; and send the flight decision to the flight equipment; the trained third neural network model is obtained through reinforcement learning training; and execute the received flight decision.

[0177] In some embodiments, the trained third network model is trained based on at least one of the following information: the rotational speed of the cable shaft, the differential value of the attitude angle of the flying device, the delay of the status information, and the distance information between the power module of the flying device and the flying device on the first map; wherein the cable shaft is used to connect the flying device with the land device so that the land device can supply power to the flying device.

[0178] An embodiment of the present application provides a heterogeneous inspection robot device for a computer room, which can achieve highly complementary performance compared to a traditional single robot and make up for the limitations of a single robot in completing tasks.

[0179] The embodiment of the present application provides a heterogeneous inspection robot device for a computer room, including a land wheeled robot and an airborne inspection quad-rotor drone, wherein the body of the land wheeled robot includes a car box, a laser radar, a bracket, a depth camera, a display screen, a drone power supply cable shaft frame and cables, a drone cellular shutdown disk, and a communication module. The airborne inspection quad-rotor drone is equipped with sensing sensors such as temperature and humidity sensors, visual sensors, signal receivers, infrared ranging sensors, etc., is equipped with a motor drive for motion control, and is equipped with a fifth generation mobile communication technology (5G) and a wireless network (WiFi) communication module for communication.

[0180] In some embodiments, the body compartment includes a computing board and a battery for computing and powering the entire heterogeneous inspection robot system.

[0181] In some embodiments, the laser radar is located directly in front of the surface of the vehicle body and is a navigation obstacle avoidance sensor used to sense the computer room environment and create a real-time map.

[0182] In some embodiments, the depth camera is fixed on a support stand with four fixing columns, and is used for visual target detection, combined with a laser radar to perceive the computer room environment and establish a computer room map.

[0183] In some embodiments, the display screen is located on a support platform below the depth camera and above the laser radar, and is used to display a real-time map of the computer room and inspection statistics results.

[0184] In some embodiments, the drone power supply cable axis bracket and the cable are located directly behind the support platform under the depth camera, that is, directly behind the display screen, for continuous power supply to the inspection quad-rotor drone, eliminating the disadvantage of poor drone endurance.

[0185] In some embodiments, the drone honeycomb parking disk is fixed with four brackets and placed directly behind the vehicle body for autonomous landing and parking of the patrol quad-rotor drone. The honeycomb structure landing disk can not only effectively prevent the sliding of the drone during autonomous landing and improve the stability of the system, but also has more obvious features, which is conducive to detection by intelligent algorithms.

[0186] A heterogeneous inspection robot device for a computer room, such as Figures 11 to 15 , including a mother land wheeled robot 1 and an airborne inspection quad-rotor drone 2, which are sub-robots that constitute the heterogeneous inspection robot device in the computer room.

[0187] The mother land wheeled robot 1 adopts the Ackerman kinematic model. The motion control module of this embodiment adopts STMicroelectronics 32-bit series microcontroller chip (ST Microelectronics 32-bit Microcontroller, STM32) as the main control board for the steering servo 27 and the left and right wheel servos 28 and 29. Through the pin interface, it can realize the transistor-transistor logic (Transistor-Transistor Logic, TTL) serial communication with the computing board NX development board (NVIDIA Jetson Xavier NX, Jetson nx). The body includes a car box 3 for the body shell; the communication module 12 is used to communicate with the development host and the inspection drone; the power supply and computing board 13 are located inside the car box for powering the robot system and providing computing power. The display screen 6 is located on the bracket platform above the laser radar below the depth camera and is used to display the real-time map of the computer room. The connection diagram of each part with the power supply and computing board is as shown in the figure Fig.16 As shown, DCDC represents a DC-DC converter, and openMV4 represents the fourth-generation OpenMV machine vision development board.

[0188] The laser radar 4 is fixed by a fixing ring 14 in front of the vehicle body surface and connected to the computing board via a Universal Serial Bus (USB). It is a navigation obstacle avoidance sensor used to sense the environment of the machine room and establish a real-time map.

[0189] The depth camera 8 is fixed on the support platform 7 with a U-shaped ladder-shaped fixing platform 15 and connected to the computing board with a USB, and is used for visual target detection, combined with a laser radar to perceive the computer room environment and establish a computer room map.

[0190] The support platform 7 is fixed on the support platform 5 by four fixing columns identical to 16 , and the support platform 5 is fixed on the vehicle box 3 and the edge of the honeycomb type parking disk 11 by 17 , 18 , 19 , 20 .

[0191] The UAV power supply cable shaft frame 9 and the power supply cable 10 are fixed on the support platform 5 with two triangular seats 21 and 22, and the cable 10 is connected to the UAV for continuous power supply to the inspection quad-rotor UAV, eliminating the disadvantage of poor endurance of the UAV.

[0192] The drone honeycomb parking disk 11 is fixed on the car body 3 with four brackets 23, 24, 25, and 26, and is placed directly behind the car body 3. It is used to inspect the autonomous landing and parking of the quad-rotor drone 2. The honeycomb landing disk 11 can not only effectively prevent the sliding of the drone during autonomous landing and improve the stability of the system, but also has more obvious features, which is conducive to its detection by intelligent algorithms.

[0193] Fig.16 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application is shown in FIG. Fig.16 As shown, the device 160 includes a feature extraction module 1601, a first feature fusion module 1602, a second feature fusion module 1603, a third feature fusion module 1604, a fourth feature fusion module 1605 and a determination module 1606, wherein:

[0194] The feature extraction module 1601 is configured to input the collected image data of the inspection target into the backbone network of the trained second network model to extract features, and obtain a first feature map, a second feature map and a third feature map;

[0195] A first feature fusion module 1602 is configured to perform feature fusion on a result of performing a first scale transformation on the third feature map and the second feature map to obtain a fourth feature map;

[0196] A second feature fusion module 1603 is configured to perform feature fusion on a result of performing a second scale transformation on the fourth feature map and the first feature map to obtain a fifth feature map; wherein the first scale transformation and the second scale transformation are used to enlarge the size of the feature map;

[0197] A third feature fusion module 1604 is configured to perform feature fusion on the result of performing the third scale transformation on the fifth feature map, the fourth feature map, and the second feature map to obtain a sixth feature map;

[0198] The fourth feature fusion module 1605 is configured to perform feature fusion on the result of performing the fourth scale transformation on the sixth feature map and the third feature map to obtain a seventh feature map; wherein the third scale transformation and the fourth scale transformation are used to reduce the size of the feature map;

[0199] The determination module 1606 is configured to determine the fault type of the inspection target based on the fifth characteristic map, the sixth characteristic map and the seventh characteristic map.

[0200] In some embodiments, the determination module 1606 is configured to input the fifth characteristic map, the sixth characteristic map and the seventh characteristic map respectively into the feature extraction module for feature extraction to obtain an eighth characteristic map corresponding to the fifth characteristic map, a ninth characteristic map corresponding to the sixth characteristic map and a tenth characteristic map corresponding to the seventh characteristic map; and input the eighth characteristic map, the ninth characteristic map and the tenth characteristic map into the prediction module to obtain the fault type of the inspection target.

[0201] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0202] It should be noted that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of a software functional unit. It may also be implemented in the form of a combination of software and hardware.

[0203] It should be noted that in the embodiments of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable the flight equipment / electronic equipment to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk or an optical disk. In this way, the embodiments of the present application are not limited to any specific combination of hardware and software.

[0204] The present application embodiment provides a flying device, Fig.17 A schematic diagram of the structure of the flight equipment provided in the embodiment of the present application, such as Fig.17 As shown, the flying device 170 includes a memory 1701 and a processor 1702 , wherein the memory 1701 stores a computer program that can be run on the processor 1702 , and the processor 1702 implements the steps in the method provided in the above embodiment when executing the program.

[0205] It should be noted that the memory 1701 is configured to store instructions and applications executable by the processor 1702, and can also cache data to be processed or processed by the processor 1702 and various modules in the flight equipment 170 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0206] An embodiment of the present application provides an electronic device, Fig.18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig.18 As shown, the electronic device 180 includes a memory 1801 and a processor 1802, wherein the memory 1801 stores a computer program that can be run on the processor 1802, and the processor 1802 implements the steps in the method provided in the above embodiment when executing the program.

[0207] It should be noted that the memory 1801 is configured to store instructions and applications executable by the processor 1802, and can also cache data to be processed or already processed by the processor 1802 and various modules in the electronic device 180 (for example, image data, audio data, voice communication data, and video communication data), which can be implemented through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0208] In some embodiments, the electronic device may be a land device or a cloud device.

[0209] An embodiment of the present application provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps in the method provided in the above embodiment are implemented.

[0210] An embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute the steps of the method provided in the above method embodiment.

[0211] It should be noted here that the description of the above flight equipment / electronic equipment, storage medium and computer program product embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the flight equipment / electronic equipment, storage medium and computer program product embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0212] It should be understood that "one embodiment" or "an embodiment" or "some embodiments" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in one embodiment" or "in some embodiments" appearing throughout the specification may not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The above-mentioned sequence numbers of the embodiments of the present application are only for description and do not represent the advantages and disadvantages of the embodiments. The above description of each embodiment tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. For the sake of brevity, this article will not repeat them.

[0213] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, object A and / or object B can represent three situations: object A exists alone, object A and object B exist at the same time, and object B exists alone.

[0214] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0215] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as: multiple modules or components can be combined, or can be integrated into another system, or some features can be ignored, or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or modules can be electrical, mechanical or other forms.

[0216] The modules described above as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules; they may be located in one place or distributed on multiple network units; some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0217] In addition, all functional modules in the embodiments of the present application may be integrated into one processing unit, or each module may be a separate unit, or two or more modules may be integrated into one unit; the above-mentioned integrated modules may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0218] A person skilled in the art can understand that all or part of the steps of implementing the above method embodiment can be completed by hardware related to program instructions, and the aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above method embodiment; and the aforementioned storage medium includes: mobile storage devices, read-only memories (ROM), magnetic disks or optical disks, etc., various media that can store program codes.

[0219] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling the flight equipment / electronic equipment to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks or optical disks.

[0220] The methods disclosed in several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0221] The features disclosed in several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0222] The features disclosed in several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0223] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for landing a flying device, characterized in that: The method is applied to a flight device, and the method comprises: Based on a first map of the inspection site and an inspection target in a preset inspection task, inspecting the inspection target at the inspection site; When the preset inspection task is completed or when an abnormality occurs to the flying device, the aircraft flies to the air above the first component of the land equipment; the first component is used to park the flying device; The height between the flying device and the first component is determined, and the flying device is guided to land on the first component according to the height.

2. The method according to claim 1, characterized in that The flying to the air above the first component of the land equipment comprises: Determining first position information of the land device; Based on the first position information, fly to the air above the land device; determining second position information of the first component; Based on the second position information, fly to the airspace above the first component.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: After flying over a first component of the land device, the first component is detected and selected based on the trained first network model; The first component selected in the frame is locked to track the first component and reduce the flying height.

4. The method according to claim 1, characterized in that: The inspecting the inspection target includes: Image data of the inspection target is collected, and the image data is sent to the land device, so that the land device determines the fault type of the inspection target based on the image data through the trained second network model.

5. The method according to claim 1, characterized in that The inspecting the inspection target includes: Obtaining status information of the flight equipment; sending the state information to a cloud device so that the cloud device obtains a flight decision of the flight device based on the state information through a trained third network model; and sending the flight decision to the flight device; the trained third neural network model is obtained through reinforcement learning training; The received flight decision is executed.

6. The method according to claim 5, characterized in that The trained third network model is trained based on at least one of the following information: the rotational speed of the cable shaft, the differential value of the attitude angle of the flying device, the delay value of the status information, and the distance information between the power module of the flying device and the flying device on the first map; wherein the cable shaft is used to connect the flying device with the land device so that the land device can supply power to the flying device.

7. An image processing method, characterized in that: The method comprises: Inputting the collected image data of the inspection target into the backbone network of the trained second network model for feature extraction to obtain a first feature map, a second feature map and a third feature map; Performing feature fusion on a result of performing a first scale transformation on the third feature map and the second feature map to obtain a fourth feature map; Performing feature fusion on a result of performing a second scale transformation on the fourth feature map and the first feature map to obtain a fifth feature map; wherein the first scale transformation and the second scale transformation are used to enlarge the size of the feature map; Performing feature fusion on a result of performing a third scale transformation on the fifth feature map, the fourth feature map, and the second feature map to obtain a sixth feature map; A result of performing a fourth scale transformation on the sixth feature map is subjected to feature fusion with the third feature map to obtain a seventh feature map; wherein the third scale transformation and the fourth scale transformation are used to reduce the size of the feature map; Based on the fifth characteristic graph, the sixth characteristic graph, and the seventh characteristic graph, a fault type of the inspection target is determined.

8. The method according to claim 7, characterized in that The trained second network model further includes: a feature extraction module and a prediction module; the determining of the fault type of the inspection target based on the fifth feature map, the sixth feature map and the seventh feature map includes: Inputting the fifth feature map, the sixth feature map, and the seventh feature map into the feature extraction module for feature extraction, respectively, to obtain an eighth feature map corresponding to the fifth feature map, a ninth feature map corresponding to the sixth feature map, and a tenth feature map corresponding to the seventh feature map; The eighth characteristic map, the ninth characteristic map and the tenth characteristic map are input into the prediction module to obtain the fault type of the inspection target.

9. A landing device for a flying device, characterized in that: The device comprises: An inspection module is configured to inspect the inspection target at the inspection site based on a first map of the inspection site and an inspection target in a preset inspection task; The flight module is configured to fly to the air above the first component of the land equipment when the preset inspection task is completed or when an abnormality occurs to the flight equipment; the first component is used to park the flight equipment; The landing module is configured to determine the height between the flying device and the first component, and guide the flying device to land on the first component according to the height.

10. An image processing device, characterized in that: The device comprises: A feature extraction module is configured to input the collected image data of the inspection target into the backbone network of the trained second network model to extract features, thereby obtaining a first feature map, a second feature map, and a third feature map; A first feature fusion module is configured to perform feature fusion on a result of performing a first scale transformation on the third feature map and the second feature map to obtain a fourth feature map; A second feature fusion module is configured to perform feature fusion on a result of performing a second scale transformation on the fourth feature map and the first feature map to obtain a fifth feature map; wherein the first scale transformation and the second scale transformation are used to enlarge the size of the feature map; A third feature fusion module is configured to perform feature fusion on a result of performing a third scale transformation on the fifth feature map, the fourth feature map, and the second feature map to obtain a sixth feature map; a fourth feature fusion module, configured to perform feature fusion on a result of performing a fourth scale transformation on the sixth feature map and the third feature map to obtain a seventh feature map; wherein the third scale transformation and the fourth scale transformation are used to reduce the size of the feature map; A determination module is configured to determine the fault type of the inspection target based on the fifth characteristic map, the sixth characteristic map and the seventh characteristic map.

11. A flying device, comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to any one of claims 1 to 6 is implemented.

12. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor executes the program, the method according to claim 7 or 8 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented; or when the computer program is executed by a processor, the method according to claim 7 or 8 is implemented.

14. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the method according to any one of claims 1 to 6 is implemented, or the method according to claim 7 or 8 is implemented.

Images