Heterogeneous multi-machine ad hoc network establishment method and device, electronic equipment and medium

Abstract

The invention relates to the technical field of communication, in particular to a heterogeneous multi-machine ad hoc network establishment method and device, electronic equipment and a medium, and the method comprises the steps: obtaining a 3D point cloud map of an inspection task scene of an unmanned aerial vehicle, determining a relay candidate point location of a quadruped robot when the inspection target point location of the unmanned aerial vehicle is detected to be located in a preset region of the 3D point cloud map, and transmitting the relay candidate point location to the quadruped robot; and determining an optimal relay point location of the quadruped robot in the relay candidate point locations, controlling the quadruped robot to navigate to the optimal relay point location to establish an ad hoc network with the unmanned aerial vehicle and the ground control center, and ensuring that data transmission between the unmanned aerial vehicle and the ground control center is realized through the ad hoc network under the condition of no signal. And the inspection task of the unmanned aerial vehicle is smoothly executed.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a method, device, electronic device and medium for establishing a heterogeneous multi-machine self-organizing network. Background Art

[0002] The existing substation and transmission line inspections are generally carried out by drones. In complex terrain and non-line-of-sight (NLOS) conditions, the lack of communication signals may cause the drone to lose control, posing a safety risk. In addition, the lack of communication signals prevents the images and video streams of the inspection equipment from being transmitted back to the power grid management system, which results in the failure to successfully complete the substation and transmission line inspection task.

[0003] Ad hoc Network is a multi-hop, centerless, self-organizing wireless network, also known as multi-hop network, infrastructureless network or self-organizing network. The entire network has no fixed infrastructure, each node is mobile, and can dynamically maintain contact with other nodes in any way. In a centerless self-organizing network, due to the limited range of terminal wireless coverage, two user terminals that cannot communicate directly can use other nodes for packet forwarding. Each node is also a router, which together completes the function of discovering and maintaining routes to other nodes. Ad hoc networks are mainly used in mobile environments to ensure reliable communication between any two nodes in the network. The data flow in the network can include voice, data and multimedia information. However, it focuses on the routing planning between connected nodes, and cannot adapt to the situation where nodes are not connected, and build an ad hoc network through dynamic path planning and node movement. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a method, device, electronic equipment and medium for establishing a heterogeneous multi-machine self-organizing network, which ensures that in the absence of signals, data transmission between the UAV and the ground control center can be achieved through the self-organizing network, thereby ensuring the smooth execution of the UAV's inspection mission.

[0005] The first aspect of the present application provides a method for establishing a heterogeneous multi-machine ad hoc network, the method comprising: Obtain the 3D point cloud map of the drone's inspection mission scene; When it is detected that the inspection target point of the drone is in a preset area of ​​the 3D point cloud map, determining the relay candidate point of the quadruped robot; Determining an optimal relay point of the quadruped robot among the candidate relay points; The quadruped robot is controlled to navigate to the optimal relay point, and forms a self-organizing network with the UAV and the ground control center.

[0006] In an optional embodiment, determining the best relay point of the quadruped robot among the candidate relay points includes: Determine a first distance from the no-signal starting point of the drone to the candidate relay point, a second distance from the no-signal inspection point of the drone to the candidate relay point, and a third distance from the candidate relay point to the ground control center; Determine a first time required for the quadruped robot to navigate from the current position to the candidate relay point, and a second time required for the drone to fly from the current position to the no-signal starting point; The best relay point is determined from the candidate relay points based on the first distance, the second distance, the third distance, the first required time, and the second required time.

[0007] In an optional implementation, determining the best relay point from the candidate relay points based on the first distance, the second distance, the third distance, the first required time, and the second required time includes: The score of each candidate relay point is determined by the following formula: ; Wherein, S is the score of the candidate relay point, is the first distance, is the second distance, is the third distance, The time required for the first The time required for the second and is the factor weight; The candidate relay point corresponding to the lowest score is determined as the best relay point.

[0008] In an optional embodiment, the method further comprises: Obtaining global positioning information, visual positioning information and flight altitude information of the UAV; Acquiring acceleration data and angular velocity data of the drone, so as to determine the change position information of the drone based on the acceleration data and the angular velocity data; The fused positioning information of the UAV is determined based on the fusion of the global positioning information, the visual positioning information, the flight altitude information and the change position information.

[0009] In an optional implementation, the determining of the candidate relay points of the quadruped robot includes: Determining a list of ground plane points of the quadruped robot; Eliminate the no-signal starting point and the no-signal inspection point, and the shielding point corresponding to the ground control center from the ground plane point list; The remaining ground plane points in the ground plane point list are determined as the relay point candidate points.

[0010] In an optional embodiment, determining the ground plane point list of the quadruped robot includes: Acquire 3D point cloud data of the 3D point cloud map, and input the 3D point cloud data into a preset point classification model, so that the point classification model classifies the point type of the 3D point cloud data to obtain a point category corresponding to each point in the 3D point cloud data; Determine a ground plane point in the 3D point cloud data based on the point category; The ground plane point list is constructed based on the ground plane points.

[0011] In an optional embodiment, the step of obtaining a 3D point cloud map of the inspection mission scene of the drone includes: Acquiring the surface shape and position information of objects in the inspection task scene by using the drone; Converting the surface shape of the object and the position information of the object into point cloud data to be processed; Preprocessing the point cloud data to be processed, and performing point cloud registration on the point cloud data to be processed after the preprocessing is completed, to obtain the 3D point cloud data; The 3D point cloud map is constructed based on the 3D point cloud data.

[0012] A second aspect of the present application provides a heterogeneous multi-machine ad hoc network building device, the device comprising: The acquisition module is used to obtain the 3D point cloud map of the inspection mission scene of the drone; A first determination module is used to determine the relay candidate point of the quadruped robot when it is detected that the inspection target point of the drone is in a preset area of ​​the 3D point cloud map; A second determination module is used to determine the best relay point of the quadruped robot among the candidate relay points; A building module is used to control the quadruped robot to navigate to the optimal relay point and build a self-organizing network with the drone and the ground control center.

[0013] The third aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for establishing a heterogeneous multi-machine self-organizing network when executing the computer program.

[0014] A fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-mentioned heterogeneous multi-machine self-organizing network establishment method.

[0015] In summary, the heterogeneous multi-machine self-organizing network establishment method, device, electronic device and medium provided by the present application realize self-understanding of the drone inspection mission scene by introducing an environmental perception algorithm. When the drone performs the inspection mission and predicts that the drone will fly to an area without network signals to perform the inspection mission, the optimal relay point of the quadruped robot is calculated using a high-precision positioning algorithm and an evaluation formula. Then, while the drone performs the inspection mission, a dynamic programming navigation algorithm is used to control the quadruped robot to navigate to the optimal relay point, that is, Through high-precision positioning algorithm and dynamic planning navigation algorithm, the quadruped robot that controls heterogeneous multiple machines can build a self-organizing network, solving the problems of drone control failure and mission interruption caused by the lack of 4G / 5G communication network coverage when drones perform inspection tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a structural schematic diagram of a heterogeneous multi-machine self-organizing network establishment system shown in an embodiment of the present application; Figure 2 It is a schematic diagram of an application scenario of a method for establishing a heterogeneous multi-machine self-organizing network shown in an embodiment of the present application; Figure 3 It is a flow chart of a method for establishing a heterogeneous multi-machine self-organizing network shown in an embodiment of the present application; Figure 4 It is another flow chart of a method for establishing a heterogeneous multi-machine self-organizing network shown in an embodiment of the present application; Figure 5 It is a schematic diagram of the overall architecture of a RandLA-Net algorithm shown in an embodiment of the present application; Figure 6 is a schematic diagram of a multi-sensor fusion positioning algorithm based on factor graph optimization shown in an embodiment of the present application; Figure 7 is a schematic diagram of a dynamic planning navigation algorithm of a quadruped robot shown in an embodiment of the present application; Figure 8 It is a functional module diagram of a heterogeneous multi-machine self-organizing network building device shown in an embodiment of the present application; Fig. 9 It is a structural schematic diagram of an electronic device shown in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0018] The following will clearly and completely describe the concept, specific structure and technical effects of the present invention in combination with the embodiments and drawings, so as to fully understand the purpose, characteristics and effects of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, other embodiments obtained by technicians in this field without creative work are all within the scope of protection of the present invention. In addition, all the connection / connection relationships involved in the patent do not refer to the direct connection of components, but refer to the formation of a better connection structure by adding or reducing connection accessories according to the specific implementation situation. The various technical features in the invention can be combined interchangeably without conflicting with each other.

[0019] Reference Figure 1 , which is a structural diagram of a heterogeneous multi-machine self-organizing network building system shown in an embodiment of the present application.

[0020] The heterogeneous multi-machine ad hoc network building system 1 includes a drone 10, a quadruped robot 20 and a ground control center 30. The drone 10 is used to perform the inspection task of the inspection task scene, and the ground control center 30 is used to establish a communication connection with the drone 10 to control the drone 10 to perform the inspection task while obtaining the data collected by the drone 10 to understand the real-time situation of the inspection task scene. When the inspection task scene is in complex terrain and non-line of sight (NLOS) conditions, it is likely that the drone 10 will be out of control due to the lack of 4G / 5G signals, which poses a safety risk. In addition, the lack of 4G / 5G signals causes the images and video streams collected by the drone 10 to be unable to be transmitted back to the ground control center 10, thereby causing the inspection task of the drone 10 to fail to be successfully completed. Therefore, when it is determined that the drone 10 is in complex terrain and non-line of sight (NLOS) conditions, the quadruped robot 20 can be controlled to travel to the ad hoc network relay area where the ad hoc network can be realized to form an ad hoc network, thereby ensuring that the network of the drone 10 is unobstructed.

[0021] Refer to Figure 2When the drone 10 needs to go to the inspection point B to inspect the transmission line, it can be seen from the inspection trajectory of the drone 10 that the drone 10 will be in a signal-free flight from the inspection point A to the inspection point B, so it is necessary to build an ad hoc network to achieve continuous execution of the inspection task and ensure the flight safety of the drone 10. When there are no other drones nearby that can be deployed, the quadruped robot 20 can be controlled to navigate to the ad hoc network relay area to build an ad hoc network to ensure that the network of the drone 10 is unobstructed.

[0022] In some embodiments, the ground control center may be configured with electronic equipment and / or servers, and the electronic equipment and / or servers are provided with heterogeneous multi-machine self-organizing network building devices for implementing heterogeneous multi-machine self-organizing network building methods.

[0023] Reference Figure 3 As shown, it is a flow chart of a method for establishing a heterogeneous multi-machine self-organizing network according to an embodiment of the present application. The method for establishing a heterogeneous multi-machine self-organizing network includes the following steps.

[0024] S31, obtaining a 3D point cloud map of the inspection mission scene of the UAV.

[0025] Refer to Figure 4In some embodiments, the ground control center determines the inspection mission scenario of the drone, determines the specific requirements of the inspection mission based on the inspection mission scenario, and plans the flight path of the drone to ensure that the inspection mission scenario can be fully covered. After ensuring that the drone and the depth camera are in normal working condition, the drone is started and controlled to fly according to the pre-planned flight path. During the flight, the depth camera is controlled to capture the surface shape and object position information of the object in the inspection mission scenario in real time. After the acquisition is completed, the collected surface shape and object position information of the inspection mission scene are transmitted to the ground control center, so that the ground control center pre-processes the collected surface shape and object position information of the object, such as denoising, downsampling and other pre-processing operations, to improve the data quality and processing efficiency. Then, the surface shape and object position information of the object after pre-processing are converted into corresponding point cloud data (referred to as point cloud data to be processed). Alternatively, the surface shape and object position information of the object are first converted into point cloud data to be processed, and then the above-mentioned pre-processing is performed on the point cloud data to be processed. Furthermore, the point cloud data to be processed collected from different perspectives are aligned, for example, using algorithms such as iterative closest point (ICP) to minimize the distance error between different point clouds and gradually align the two point clouds. After completing the point cloud alignment, 3D point cloud data is obtained. Finally, the ground control center can use professional 3D point cloud processing software (such as CloudCompare, PCL, etc.) to convert 3D point cloud data into 3D point cloud maps. Among them, the 3D point cloud map can intuitively display the shape, position and spatial relationship of objects in the inspection task scene.

[0026] In other embodiments, when the depth camera collects the surface shape and position information of objects in the inspection task scene, the surface shape and position information of the objects can be directly converted into point cloud data to be processed, and the point cloud data to be processed can be transmitted to the ground control center. The depth camera is installed on the drone, and the depth camera is ensured to be stable and firm, so as to obtain stable surface shape and position information of objects during the flight of the drone.

[0027] S32, when it is detected that the inspection target point of the drone is in a preset area of ​​the 3D point cloud map, determining the relay candidate point of the quadruped robot.

[0028] Refer to Figure 4In some embodiments, the ground control center can obtain the location information of the drone in real time based on the 3D point cloud map, and detect whether the drone will be in a preset area of ​​the inspection task scene (for example, a complex terrain area or a non-line of sight (NLOS) occlusion area) during the inspection task based on the flight path corresponding to the inspection task of the drone. When it is determined that the inspection target point of the drone is in the preset area, the ground control center needs to use a dynamic planning method to intelligently plan the navigation path of the drone or the ground quadruped robot, and guide it to an area where self-organizing network relay communication can be established, so that the communication signal connection between the drone and the ground control center can be restored and maintained through the self-organizing network relay mechanism, thereby effectively solving the problem of drone control failure caused by lack of communication network coverage, and ensuring the safety of drone operation, and further ensuring that the drone can continuously and uninterruptedly perform inspection tasks. Among them, the preset area is pre-determined by the ground control center based on the 3D point cloud data, and refers to an area where there is a potential risk of communication signal interruption with the ground control center.

[0029] When it is detected that the inspection target point is in the preset area, the ground control center first detects whether there are other drones near the preset area and allows them to be deployed. When it is determined that there are no other drones or the drones are not allowed to be deployed, the quadruped robot near the preset area is determined to control the quadruped robot to navigate to the self-organizing network relay area for self-organizing networking, so as to ensure the smooth network of the drone.

[0030] In an optional implementation, the determining of the candidate relay points of the quadruped robot includes: Determining a list of ground plane points of the quadruped robot; Eliminate the no-signal starting point and the no-signal inspection point, and the shielding point corresponding to the ground control center from the ground plane point list; The remaining ground plane points in the ground plane point list are determined as the relay point candidate points.

[0031] In some embodiments, when a risk of communication signal interruption between the drone and the ground control center is detected, it is first necessary to determine which points on the ground the quadruped robot can travel. That is, an analysis is performed based on the 3D point cloud map to obtain the object type of each point in the 3D point cloud map to determine a list of ground plane points suitable for the quadruped robot to walk. Specifically, the 3D point cloud data in the 3D point cloud map is input into a preset point classification model so that the point classification model classifies the 3D point cloud data into point types to obtain the point category corresponding to each point in the 3D point cloud data, and the point category may include hills, jungles, buildings, and ground that can be walked on by the quadruped robot. A list of ground plane points is further constructed based on the points corresponding to the ground that can be walked on by the quadruped robot based on the point category.

[0032] Refer to Figure 4 and Figure 5In the embodiment of the present application, the environment perception algorithm (Random Sampling and Local Feature Aggregator Network, RandLA-Net) model is used as a point classification model. The ground control center first deploys electronic equipment and / or servers, pre-installs and deploys an end-to-end open source machine learning platform (e.g., TensorFlow) and the RandLA-Net open source library in the server, loads the open source platform and the trained RandLA-Net model, inputs the 3D point cloud data of the drone inspection scene into the RandLA-Net model, and uses the RandLA-Net model to classify the point type, and obtains the point category of each point cloud data (one point cloud data corresponds to one point), and then obtains a list of ground plane points suitable for quadruped robots to walk from the point cloud data according to the point category. Among them, RandLA-Net is a neural network structure for processing point cloud data, which uses random sampling (Random Sampling, RS) to reduce the point cloud density and reduce the amount of calculation. Although random sampling may lose some useful information, it is alleviated by extracting and retaining key information through local spatial encoding and pooling based on the attention mechanism. Moreover, RandLA-Net can extract and enhance point features using Local Feature Aggregator (LFA), which includes two key parts: Local Spatial Encoding and Attentive Pooling. Local spatial encoding creates a local coordinate system around each point to increase the network's understanding of the geometric structure of point cloud data. Attention pooling dynamically weights features according to the importance of each point, allowing the network to focus on points that are more important to the task, thereby improving the accuracy of feature extraction. In addition, RandLA-Net also includes a Dilated Residual Block (DRB), which increases the receptive field by introducing convolutional layers with different dilation rates, further improving the network's ability to understand point cloud data. By using RandLA-Net to achieve self-understanding of inspection task scenarios, large-scale point cloud data can be effectively processed through the design of random sampling, local feature integration, and dilated residual modules, which not only reduces computational complexity, but also maintains high accuracy and robustness.

[0033] After determining the ground plane point list of the quadruped robot, the ground control center needs to further screen the points in the ground plane point list to determine the relay candidate points of the quadruped robot. Specifically, the ground control center can use the high-precision positioning algorithm of the drone to obtain the current position information of the drone, as well as the current position information of the quadruped robot and the ground control center, to determine the drone's no-signal starting point and each no-signal inspection point, and confirm the corresponding occlusion point based on the current position information of the ground control center. Traverse the ground plane point list, delete the no-signal starting point, no-signal inspection point, and the ground control center has occlusion points, ensure that the relay candidate points used for communication and the drone and the ground control center can communicate without obstacles, and use the remaining ground plane points as the relay candidate points of the quadruped robot. Through the above optional implementation method, by combining the 3D point cloud map and the dynamic planning method, stable communication between the drone and the ground control center in a complex environment is achieved. When the drone enters the preset risk area, the ground control center can intelligently plan the navigation path of the quadruped robot and guide it to the area where self-organizing network relay communication can be established. By using the RandLA-Net model to classify points, a list of ground plane points suitable for the quadruped robot to walk is obtained, and then candidate relay points are further screened to ensure the communication signal connection between the UAV and the ground control center, solve the communication interruption problem, and ensure the continuous and uninterrupted execution of the UAV inspection mission.

[0034] S33, determining the best relay point of the quadruped robot among the candidate relay points.

[0035] When candidate relay points suitable for the quadruped robot to travel are selected from the list of ground plane points, the ground control center can further determine the best relay point of the quadruped robot from the candidate relay points, so that the quadruped robot can travel to the best relay point, build an ad hoc network with the UAV and the ground control center, and achieve the best communication signal.

[0036] In an optional embodiment, determining the best relay point of the quadruped robot among the candidate relay points includes: Determine a first distance from the no-signal starting point of the drone to the candidate relay point, a second distance from the no-signal inspection point of the drone to the candidate relay point, and a third distance from the candidate relay point to the ground control center; Determine a first time required for the quadruped robot to navigate from the current position to the candidate relay point, and a second time required for the drone to fly from the current position to the no-signal starting point; The best relay point is determined from the candidate relay points based on the first distance, the second distance, the third distance, the first required time, and the second required time.

[0037] In some embodiments, the ground control center can use multi-sensor fusion positioning to obtain the precise positioning of the drone in the 3D point cloud map, that is, to determine the position information of the drone in the 3D point cloud map (referred to as the first position information), and based on the first position information, the precise straight-line distance between it and each point can be calculated. Specifically, determine the no-signal starting point of the drone, and calculate the straight-line distance from the no-signal starting point to each candidate relay point to obtain the first distance, and determine each no-signal inspection point of the drone, and calculate the straight-line distance from each no-signal inspection point to each candidate relay point to obtain the second distance, and determine the position information of the ground control center (referred to as the second position information), and calculate the straight-line distance from each candidate relay point to the ground control center based on the second position information to obtain the third distance. Next, determine the position information of the quadruped robot (referred to as the third position information), calculate the time required for the quadruped robot to each candidate relay point based on the third position information, and obtain the first required time, and determine the time required for the drone to the no-signal starting point based on the first position information, and obtain the second required time. Further, the score of each candidate relay point is determined by the following evaluation formula: ; Among them, S is the score of each relay candidate point, is the first distance, is the second distance, is the third distance, When the first time is needed, For the second required time, and is the weight of each factor that affects the evaluation score, which is given by expert experience. The smaller the sum of the first distance, the second distance and the third distance, the better the candidate interruption point. The smaller the time difference between the first required time and the second required time, the better the candidate interruption point. In other words, the lower the score, the more suitable the candidate relay point. Therefore, the ground control center can determine that the candidate interruption point corresponding to the lowest score is the best relay point for the quadruped robot.

[0038] In an optional embodiment, the method further comprises: Obtaining global positioning information, visual positioning information and flight altitude information of the UAV; Acquiring acceleration data and angular velocity data of the drone, so as to determine the change position information of the drone based on the acceleration data and the angular velocity data; The fused positioning information of the UAV is determined based on the fusion of the global positioning information, the visual positioning information, the flight altitude information and the change position information.

[0039] Refer to Figure 4 In the embodiment of the present application, the drone integrates multiple sensors, including real-time kinematic (RTK), depth camera, inertial measurement unit (IMU), barometer, etc., and provides more stable, locally high-precision, and globally low-drift 6DOF (six degrees of freedom) positioning by fusing multi-sensor data to make up for the defects of a single sensor. First, an RTK-GPS receiver is used to receive signals from multiple satellites and calculate the global position information (including longitude, latitude, altitude) and flight speed of the drone. Among them, the sensor data collected by RTK-GPS is provided with centimeter-level accuracy and may be affected by electromagnetic interference and occlusion. At the same time, a depth camera is used to capture the flight environment image, and the visual odometer (VO) and visual inertial odometer (VIO) algorithms are used to process the flight environment image to obtain the visual positioning information of the drone. At the same time, the IMU can also provide data from the accelerometer and gyroscope, that is, acceleration data and angular velocity data, which are used to calculate the flight attitude, flight speed and flight position of the drone to obtain the change position information of the drone. At the same time, the barometer can also be used to provide the UAV's flight altitude information to assist in the data fusion of RTK-GPS and IMU. Although VIO can provide positioning based on the local coordinate system of the UAV, it cannot provide positioning in the global ECEF (Earth-centered Earth-fixed coordinate system), and is also affected by ambient lighting. Therefore, the ground control center can obtain the sensor data collected by the UAV's multiple sensors, including the UAV's global positioning information, visual positioning information, flight altitude information, and change position information, and fuse them based on the sensor data to obtain the UAV's fused positioning information. First, the ground control center can filter and denoise the sensor data to improve the data quality. Then refer to Figure 6 , each type of sensor data is taken as an observation data, and the corresponding factor nodes are defined. Then, the sensor data is fused using methods such as nonlinear optimization or extended Kalman filter (EKF), and the fused positioning result is output, that is, the fused positioning information of the UAV, and the fused position information is used as the first position information of the UAV.

[0040] In other embodiments, when the depth camera collects the flight environment image, the depth camera can pre-process the flight environment image, such as graying, edge detection, etc., to facilitate feature extraction. Further, sensor calibration can be performed, including using a calibration plate to calibrate the camera's internal parameters and distortion parameters to ensure that the conversion relationship between the pixel coordinates in the flight environment image and the coordinates in the real world is accurate. In addition, the accelerometer and gyroscope of the IMU can also be calibrated to eliminate the deviation and cross-axis effects between sensors, and the camera, IMU and other sensors are jointly calibrated to ensure that the data between sensors can be accurately aligned. Finally, sensor data fusion is performed, including using timestamps to synchronize data from different sensors to ensure that each sensor data describes the state at the same time point, and converting different sensor data to the same coordinate system, such as converting and aligning the global coordinates of the RTK-GPS with the local coordinates of the camera and IMU. Finally, nonlinear optimization (such as Bundle Adjustment, BA) or EKF filtering methods are used for sensor data fusion. The nonlinear optimization method is used to optimize the position and attitude of the UAV by minimizing the reprojection error. The EKF method is used to fuse the data of different sensors through prediction and update steps, and the state of the UAV is estimated. Finally, the fused position information of the UAV (including longitude, latitude, and altitude) is output.

[0041] Through the above optional implementation, high-precision and stable positioning of the UAV is achieved by fusing the multi-sensor data integrated in the UAV, including global positioning information, visual positioning information, flight altitude information and change position information. The data quality is improved through preprocessing steps such as filtering, denoising and sensor calibration. Sensor data fusion is performed using methods such as nonlinear optimization or extended Kalman filtering to obtain the fused positioning information of the UAV. This information has the characteristics of local high precision and global low drift, which effectively makes up for the defects of a single sensor, improves the positioning accuracy and stability of the UAV in complex environments, and provides reliable position information support for the inspection mission of the UAV.

[0042] S34, controlling the quadruped robot to navigate to the optimal relay point, and forming an ad hoc network with the UAV and the ground control center.

[0043] Refer to Figure 4 After determining the best relay point of the quadruped robot, the quadruped robot can start its own positioning and navigation system, receive the 3D point cloud map of the inspection mission scene sent by the ground control center, and the target point coordinates corresponding to the best relay point, so as to use the dynamic planning navigation algorithm of the quadruped robot to control the quadruped robot to navigate to the best relay point. Figure 7As shown, for example, a global planner (such as A* algorithm, Dijkstra algorithm, etc.) is used to plan an optimal path for a quadruped robot from its current position to the optimal relay point on a 3D point cloud map. When the quadruped robot reaches the optimal relay point, it starts the ad hoc network radio and starts the ad hoc network node search, and can complete the establishment of a dynamic ad hoc network with the ad hoc network radio of the drone and the ground control center. The drone obtains a communication connection with the ground control center through the quadruped robot communication radio relay, and successfully realizes data transmission.

[0044] Compared with the prior art, when the UAV performs an inspection mission and predicts that the UAV will fly to an area with no network signal to perform the inspection mission, the present application uses a high-precision positioning algorithm and an evaluation formula to calculate the optimal relay point of the quadruped robot. Then, while the UAV performs the inspection mission, a dynamic planning navigation algorithm is used to control the quadruped robot to navigate to the optimal relay point, thereby realizing the self-organizing network of the UAV, the quadruped robot and the ground control center. This ensures that in the absence of a 4G / 5G network, data transmission between the UAV and the ground control center can be achieved through the self-organizing network, thereby ensuring the smooth execution of the UAV's inspection mission.

[0045] Reference Figure 8 , which is a functional module diagram of a heterogeneous multi-machine self-organizing network building device shown in an embodiment of the present application.

[0046] In some embodiments, the heterogeneous multi-machine ad hoc network building device 80 may include multiple functional modules composed of computer program segments. The computer programs of the various program segments of the heterogeneous multi-machine ad hoc network building device 80 may be stored in the memory of the electronic device and executed by at least one processor to execute (see Figure 3 The function of forming a heterogeneous multi-machine self-organizing network is described. According to the functions performed, it can be divided into multiple functional modules. The functional modules may include: an acquisition module 801, a first determination module 802, a second determination module 803, a formation module 804 and a fusion module 805. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory. In this embodiment, the functions of each module will be described in detail in subsequent embodiments.

[0047] The acquisition module 801 is used to acquire a 3D point cloud map of the inspection mission scene of the drone.

[0048] The first determination module 802 is used to determine the relay candidate points of the quadruped robot when it is detected that the inspection target point of the drone is in a preset area of ​​the 3D point cloud map.

[0049] The second determination module 803 is used to determine the best relay point of the quadruped robot among the candidate relay points.

[0050] The building module 804 is used to control the quadruped robot to navigate to the optimal relay point and build an ad hoc network with the drone and the ground control center.

[0051] The second determination module 803 is also specifically used to: determine the first distance from the no-signal starting point of the UAV to the relay candidate point, the second distance from the no-signal inspection point of the UAV to the relay candidate point, and the third distance from the relay candidate point to the ground control center; determine the first required time for the quadruped robot to navigate from the current position to the relay candidate point, and the second required time for the UAV to fly from the current position to the no-signal starting point; determine the best relay point from the relay candidate points based on the first distance, the second distance, the third distance, the first required time and the second required time.

[0052] The second determining module 803 is further specifically configured to: The score of each candidate relay point is determined by the following formula: ; Wherein, S is the score of the candidate relay point, is the first distance, is the second distance, is the third distance, The time required for the first The time required for the second and is the factor weight; The candidate relay point corresponding to the lowest score is determined as the best relay point.

[0053] The fusion module 805 is used to: obtain the global positioning information, visual positioning information and flight altitude information of the UAV; obtain the acceleration data and angular velocity data of the UAV to determine the change position information of the UAV based on the acceleration data and the angular velocity data; and fuse the global positioning information, the visual positioning information, the flight altitude information and the change position information to determine the fused positioning information of the UAV.

[0054] The first determination module 802 is also specifically used to: determine the ground plane point list of the quadruped robot; remove the no-signal starting point and the no-signal inspection point, and the blocked point corresponding to the ground control center from the ground plane point list; and determine the remaining ground plane points in the ground plane point list as the relay point candidate points.

[0055] The first determination module 802 is also specifically used to: obtain 3D point cloud data of the 3D point cloud map, so as to input the 3D point cloud data into a preset point classification model, so that the point classification model classifies the point type of the 3D point cloud data to obtain the point category corresponding to each point in the 3D point cloud data; determine the ground plane points in the 3D point cloud data based on the point category; and construct the ground plane point list based on the ground plane points.

[0056] The acquisition module 801 is also specifically used to: acquire the surface shape and position information of objects in the inspection task scene through the drone; convert the surface shape and position information of the objects into point cloud data to be processed; pre-process the point cloud data to be processed, and perform point cloud registration on the point cloud data to be processed after the pre-processing is completed to obtain the 3D point cloud data; and construct the 3D point cloud map based on the 3D point cloud data.

[0057] It should be understood that the various variations and specific embodiments of the heterogeneous multi-machine self-organizing network building method provided in the above-mentioned embodiments are also applicable to the heterogeneous multi-machine self-organizing network building device of the present embodiment. Through the above-mentioned detailed description of the heterogeneous multi-machine self-organizing network building method, technical personnel in this field can clearly know the implementation method of the heterogeneous multi-machine self-organizing network building device in the present embodiment. For the sake of brevity of the specification, it will not be described in detail here.

[0058] See also Fig. 9 FIG. 1 is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. In a preferred embodiment of the present application, the electronic device 9 includes a memory 91 , at least one processor 92 and at least one communication bus 93 .

[0059] Those skilled in the art should understand that Fig. 9 The structure of the electronic device 9 shown does not constitute a limitation of the embodiments of the present application, and may be either a bus structure or a star structure. The electronic device 9 may also include more or less other hardware or software than shown in the figure, or a different component arrangement.

[0060] In some embodiments, the electronic device 9 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits, programmable gate arrays, digital processors, and embedded devices. The electronic device 9 may also include user equipment, which includes but is not limited to any electronic product that can interact with a user through a keyboard, mouse, remote control, touchpad, or voice control device, such as a personal computer, tablet computer, smart phone, digital camera, etc.

[0061] In the above embodiments provided in the present application, it should be understood that the disclosed methods, devices, computer-readable storage media, and electronic devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple components or modules can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or components or modules, which can be electrical, mechanical or other forms.

[0062] The components described as separate components may or may not be physically separated, and the components shown as components may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the components may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0063] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each component may exist physically separately, or two or more modules may be integrated into one module. The above integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0064] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0065] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0066] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0067] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A method for establishing a heterogeneous multi-machine ad hoc network, characterized in that: The method comprises: Obtain the 3D point cloud map of the drone's inspection mission scene; When it is detected that the inspection target point of the drone is in a preset area of ​​the 3D point cloud map, determining the relay candidate point of the quadruped robot; Determining an optimal relay point of the quadruped robot among the candidate relay points; The quadruped robot is controlled to navigate to the optimal relay point, and forms a self-organizing network with the UAV and the ground control center.

2. The method for establishing a heterogeneous multi-machine ad hoc network according to claim 1, characterized in that: Determining the best relay point of the quadruped robot among the candidate relay points includes: Determine a first distance from the no-signal starting point of the drone to the candidate relay point, a second distance from the no-signal inspection point of the drone to the candidate relay point, and a third distance from the candidate relay point to the ground control center; Determine a first time required for the quadruped robot to navigate from the current position to the candidate relay point, and a second time required for the drone to fly from the current position to the no-signal starting point; The best relay point is determined from the candidate relay points based on the first distance, the second distance, the third distance, the first required time, and the second required time.

3. The method for establishing a heterogeneous multi-machine ad hoc network according to claim 2, characterized in that: The determining the best relay point from the candidate relay points based on the first distance, the second distance, the third distance, the first required time, and the second required time comprises: The score of each candidate relay point is determined by the following formula: ； Wherein, S is the score of the candidate relay point, is the first distance, is the second distance, is the third distance, The time required for the first The time required for the second and is the factor weight; The candidate relay point corresponding to the lowest score is determined as the best relay point.

4. The method for establishing a heterogeneous multi-machine ad hoc network according to claim 2, characterized in that: The method further comprises: Obtaining global positioning information, visual positioning information and flight altitude information of the UAV; Acquiring acceleration data and angular velocity data of the drone, so as to determine the change position information of the drone based on the acceleration data and the angular velocity data; The fused positioning information of the UAV is determined based on the fusion of the global positioning information, the visual positioning information, the flight altitude information and the change position information.

5. The method for establishing a heterogeneous multi-machine ad hoc network according to claim 2, characterized in that: Determining the candidate relay points of the quadruped robot includes: Determining a list of ground plane points of the quadruped robot; Eliminate the no-signal starting point and the no-signal inspection point, and the shielding point corresponding to the ground control center from the ground plane point list; The remaining ground plane points in the ground plane point list are determined as the relay point candidate points.

6. The method for establishing a heterogeneous multi-machine ad hoc network according to claim 5, characterized in that: The determining of the ground plane point list of the quadruped robot comprises: Acquire 3D point cloud data of the 3D point cloud map, and input the 3D point cloud data into a preset point classification model, so that the point classification model classifies the point type of the 3D point cloud data to obtain a point category corresponding to each point in the 3D point cloud data; Determine a ground plane point in the 3D point cloud data based on the point category; The ground plane point list is constructed based on the ground plane points.

7. The method for establishing a heterogeneous multi-machine ad hoc network according to claim 6, characterized in that: The method of obtaining a 3D point cloud map of the inspection mission scene of the drone includes: Acquiring the surface shape and position information of objects in the inspection task scene by using the drone; Converting the surface shape of the object and the position information of the object into point cloud data to be processed; Preprocessing the point cloud data to be processed, and performing point cloud registration on the point cloud data to be processed after the preprocessing is completed, to obtain the 3D point cloud data; The 3D point cloud map is constructed based on the 3D point cloud data.

8. A heterogeneous multi-machine self-organizing network building device, characterized in that: The device comprises: The acquisition module is used to obtain the 3D point cloud map of the inspection mission scene of the drone; A first determination module is used to determine the relay candidate point of the quadruped robot when it is detected that the inspection target point of the drone is in a preset area of ​​the 3D point cloud map; A second determination module is used to determine the best relay point of the quadruped robot among the candidate relay points; A building module is used to control the quadruped robot to navigate to the optimal relay point and build a self-organizing network with the drone and the ground control center.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the method for establishing a heterogeneous multi-machine self-organizing network described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for establishing a heterogeneous multi-machine ad hoc network described in any one of claims 1 to 7 are implemented.

Images