A method and system for unmanned aerial vehicle and intelligent vehicle cooperative patrol

Abstract

The application belongs to the technical field of path planning, and discloses a method and system for cooperation patrol of an unmanned aerial vehicle and an intelligent vehicle. The method comprises the following steps: constructing an HWT-OS operating system; performing task scheduling, deployment, real-time scene mapping, target detection, path planning, obstacle avoidance and positioning of the unmanned aerial vehicle and the intelligent vehicle; and performing work by a robot system composed of the unmanned aerial vehicle and the intelligent vehicle in a land-air working interval range. The collected relevant information images in the land-air working interval range are identified and warned by using a digital twin technology, and are sent to a dispatch center for visualization, wherein the relevant information comprises to-be-identified dynamic target information and static target information of a certain area. The application better completes the realization of tasks such as distribution and search and rescue. The application breaks the current status of single unmanned aerial vehicle and intelligent vehicle work, and provides a technical solution for multi-robot cross-field work problems.

Description

Technical Field

[0001] This invention belongs to the field of path planning technology, and in particular relates to a method and system for collaborative patrol by unmanned aerial vehicles and intelligent vehicles. Background Technology

[0002] To date, drones and smart cars have become increasingly common in the public eye. From campus delivery by Cainiao smart cars to low-altitude drone logistics operations, drones and smart cars are being deployed in commercial use. Given that in today's robotics industry, whether in the field of smart delivery or in the fields of military and special rescue, most research is more inclined to single-machine operation or aerial and land-based swarm operations, both of which have their own unique limitations.

[0003] Existing technologies have developed control systems for quadcopter drone swarms, but in the absence of depth information, real-time path planning for drones based solely on images is ineffective.

[0004] Intelligent robots have developed rapidly and are widely used. Intelligent unmanned delivery vehicles have emerged. Compared with traditional delivery methods, unmanned delivery vehicles have a significant advantage in load capacity. A single unmanned delivery vehicle can typically carry over 100 kilograms. Besides intelligent vehicles, drones have also been used for applications such as three-dimensional spraying disinfection, drones for remote communication, and drones for transporting supplies. However, whether in the fields of intelligent delivery, military, or special rescue, most research focuses on single-unit operation or aerial and land-based swarm operations. Aerial and land operations have their own inherent limitations. Research on intelligent vehicles has consistently struggled to overcome its limitations in terms of work efficiency, sensor range, terrain traversal, and climbing ability. Drone research is also constrained by other major factors, such as endurance, payload capacity, and flight stability.

[0005] How to combine land and air cluster operations and establish an independent and highly portable robot system that can be further developed according to the functions, usage scenarios, and special requirements of different robots to solve the pain points of single land operations and single air operations, and better accomplish tasks such as delivery and search and rescue, is a major challenge to be solved in this field.

[0006] Based on the above analysis, the problems and defects of the existing technology are as follows: the existing drones on the market have small payload and short flight time, the unmanned vehicles have limited operating areas, and the automation of logistics and transportation is hindered. Summary of the Invention

[0007] To overcome the problems existing in related technologies, the present invention discloses an embodiment of a collaborative patrol system and method for unmanned aerial vehicles and intelligent vehicles.

[0008] The technical solution is as follows: a method for collaborative patrolling using drones and intelligent vehicles, including:

[0009] S1, build the HWT-OS operating system;

[0010] S2 performs task scheduling and allocation for drones and intelligent vehicles, real-time scene mapping, target detection, path planning, obstacle avoidance, and positioning.

[0011] S3 is a robotic system composed of drones and intelligent vehicles that operates within a land and air work area.

[0012] S4 uses digital twin technology to identify and warn of the spatial coordinates of dynamic and static target information images collected within the land and air working area, and sends them to the dispatch center for visualization.

[0013] In step S1, the HWT-OS operating system is constructed, including: deploying a part of the HWT-OS operating system on different robot terminals as different CPUs, and treating unassigned tasks as processes to be executed; deploying another part of the HWT-OS operating system on the central server, which is used to receive tasks submitted by clients that need to be completed by the robot, and to allocate tasks according to task priority and the robot terminal's own status.

[0014] Data communication between the task publishing client and the central server, and between the central server and the robot terminal, is conducted through a unified task message format, HWT_message.

[0015] The task message format includes: task issuing customer, task priority, and task receiving location; among which, the task issuing customer explains the source of the task, the task priority indicates the importance of the task, and the task receiving location explains the approximate scope of the task execution.

[0016] In the robot terminal, if the robot's task queue is full, the central server sets the corresponding state machine of the robot terminal to busy; if the robot's task queue is not full, the central server sets the corresponding state machine of the robot terminal to idle. The functional information of the robot terminal includes: the robot's working area and the robot's remaining battery power. The robot's working area is used to describe whether the robot can perform tasks within the target range, and the robot's remaining battery power is used to determine whether the robot can complete the task.

[0017] In step S2, task scheduling and allocation between the UAV and the intelligent vehicle, real-time scene mapping, target detection, path planning, obstacle avoidance, and localization are performed. This includes: real-time scene mapping based on a deep reinforcement discrimination SLAM autonomous mapping algorithm; introducing ORB-SLAM visual SLAM technology based on the Cartographer environment; combining laser SLAM and visual SLAM to perform loop closure detection and image optimization for map construction; loop closure detection detects previously visited locations during SLAM and establishes loop closure by identifying shared feature points or feature descriptors of the locations, correcting errors in previous position estimation; image optimization optimizes the estimated camera trajectory and map during SLAM using optimization algorithms.

[0018] Furthermore, the path planning adopts UAV autonomous local path planning based on enhanced discrimination, and UAV real-time local path planning based on pure image under the condition of missing depth information by the target tracking algorithm; the key process is transformed into a depth enhanced discrimination model, in which RLAgent is used to extract preprocessed environmental information and sensor information for training, so that RLAgent can use environmental information to enable the UAV to find relative position information and make path planning when GPS signal and target depth information are missing.

[0019] Furthermore, the intelligent vehicle establishes a three-layer circumscribed auxiliary circle around itself based on the environmental information collected by radar. When an obstacle enters the range of the auxiliary circle, the intelligent vehicle takes different obstacle avoidance measures according to the relative position of the obstacle in the auxiliary circle. In the grid map, the edges of obstacles are subjected to erosion and expansion processing. An adaptive Monte Carlo method is introduced to assist in localization, mapping the local information scanned by the sensor to the global relative static space. With the help of the data fed back by the LiDAR, the vehicle adjusts its own pose in a timely manner and sets up a multi-layer feedback mechanism to respond. The intelligent vehicle can avoid obstacles that appear at any time during the journey.

[0020] Furthermore, the target detection includes: constructing a deep discriminant network for the target to be identified based on the YOLOv7 model, combining the D455 depth sensing camera on the agent to detect the specific location of the target, performing depth completion based on the trained detection model, obtaining a dense depth map by combining the sparse depth map with the RGB image, and acquiring rich environmental representations for high-level scene understanding, perception and planning in subsequent tasks.

[0021] In step S4, digital twin technology is used to identify and warn of spatial coordinate positions, including:

[0022] Step 1: Collection, preprocessing, and dataset creation of image spatial coordinate location sample sets;

[0023] Step 2: Design the digital twin network structure;

[0024] Step 3, Digital Twin Network Parameter Identification;

[0025] Step 4: Perform random pre-inversion on the test set images to construct inversion trajectories;

[0026] Step 5: Use the fully twinned instances of the trained digital twin network to infer the inversion pattern in parallel and calculate the multi-trajectory classification variance matrix F;

[0027] Step 6: Based on the principle of maximum variance deviation, the object category and spatial coordinate location vector value are simultaneously predicted from the multi-trajectory classification variance matrix F.

[0028] In step 1, the collection, preprocessing, and dataset creation of the image spatial coordinate location sample set include:

[0029] (1) For the image patterns to be identified, set their respective standard positioning vector values;

[0030] (2) Collect image spatial coordinate position sample sets for each type of pattern to ensure that the number of spatial coordinate position sample sets is balanced;

[0031] (3) The collected spatial coordinate position sample set is sequentially normalized to invert the spatial coordinate position sample set to the normalized positioning vector value; the inverted spatial coordinate position positioning vector value of the object is denoted as... When the object's positioning vector value is in the y-axis direction When an object is inverted counterclockwise, its spatial coordinate position positioning vector value is negative; when inverted clockwise, its spatial coordinate position positioning vector value is positive. For the collected image spatial coordinate position sample set of the i-th type of pattern, the spatial coordinate position positioning vector value is arbitrary, and any inversion pattern is represented as: Normalized spatial coordinate location sample set for arbitrary inversion mode Adjust to the standard mode with a spatial coordinate positioning vector value of 0 degrees.

[0032] (4) Divide the preprocessed image spatial coordinate position sample set into training, validation and test sets in a ratio of 7:1:2;

[0033] (5) During training and testing, the input image is normalized into a square before input;

[0034] Step 2 involves designing the digital twin network structure, including:

[0035] (1) Design the digital twin network structure according to the task dataset size to obtain generalization performance on the test set;

[0036] (2) The digital twin network structure used for inversion pattern recognition is designed by manual design and automated machine discrimination method of neural architecture search, ensuring that the network size is adapted to the task dataset and achieves the best generalization performance on the test dataset.

[0037] (3) The first layer of the network uses 64 5-Siamese twin kernels of size 5, with a stride of 2, and the output size of the filled image edge feature map remains unchanged; the second layer of the network uses max pooling operation with a stride of 2, which reduces the network size by half; the third Siamese twin layer uses 64 5-Siamese twin kernels of size 5, with a stride of 2, and the output size of the filled image edge feature map remains unchanged; the fourth layer of the network uses max pooling operation with a stride of 2, which reduces the network size by half; the fifth, sixth and seventh layers of the network use a fully connected network; among them, the fifth layer receives the vector features after the feature map of the fourth layer is flattened, and has 240 neurons output; the sixth layer of the network has 168 neurons output; the seventh layer of the network has 20 neurons output, which correspond to the 20 categories of the multi-classification task respectively;

[0038] Step 3, digital twin network parameter identification, includes:

[0039] (1) Regression is used to process the outputs of multiple neurons in the digital twin network, cross-entropy loss is used as the network discrimination bias principle, and mini-batch gradient descent is used to discriminate network parameters.

[0040] (2) Input the normalized training space coordinate position sample set into the digital twin network model, calculate the structured loss using the cross-entropy loss function, update the network using mini-batch gradient descent, and use an adjustable discrimination rate during training;

[0041] Step 4, the step of constructing inversion trajectories by performing random pre-inversion on the test set images, includes:

[0042] (1) For an input image x with any localization vector value θ in the test set θ ∈R H×W×d Where H, W, and d represent the image height, width, and number of trajectories, respectively; equal-interval inversion is performed within the range of [-Φ, +Φ] degrees, with N inversion times and an inversion vector angle interval ΔΦ = 2Φ / N;

[0043] (2) The inverted models and their angles are as follows:

[0044]

[0045]

[0046] in, Φ and ΔΦ represent the expected range and accuracy of the inversion pattern recognition angle estimation, respectively. They are set according to the task requirements, with 0≤Φ≤π. ΔΦ is adjusted by setting the number of inversion trajectories N.

[0047] (3) Image inversion was performed using a bilinear interpolation algorithm. The angle intervals of the inversion vectors were set to 2 degrees and 0.2 degrees, respectively, and the estimation intervals Φ were set to π and π, respectively.

[0048] In step 5, the inversion pattern is inferred in parallel using a fully twinned instance of the trained digital twin network, and the multi-trajectory classification variance matrix F is calculated, including:

[0049] (1) After pre-inversion of the image, the copied object pattern in the nth inversion trajectory is used to perform feature extraction and category prediction using the trained digital twin network, as expressed in the following expression:

[0050]

[0051] Among them, F n It is the classification response of the nth trajectory;

[0052] (2) Arrange the classification variance outputs of all inverted trajectories into an inverted multi-trajectory classification variance matrix, the expression of which is:

[0053]

[0054] Where F∈R N×C T is the number of categories to be predicted;

[0055] In step 6, based on the principle of maximum variance deviation, the object category and spatial coordinate location vector value are simultaneously predicted from the multi-trajectory classification variance matrix F, including:

[0056] (1) Using classification variance as the final criterion for pattern recognition, the maximum value of the multi-trajectory response is adopted to simultaneously predict the object category and spatial coordinate location vector value. The expression is:

[0057]

[0058] (j,k)=argmax(F N×C )

[0059] (2) When using the classic digital twin network for supervised classification, the category and vector angle are predicted based on the principle of maximum deviation of classification variance;

[0060] (3) Perform the above process on the dataset to obtain the estimation performance of category and spatial coordinate location vector values.

[0061] Another object of the present invention is to provide a drone and intelligent vehicle cooperative patrol system, which implements the drone and intelligent vehicle cooperative patrol method, and the system includes:

[0062] Raspberry Pis were mounted on a smart wireless charging car and a quadcopter drone. By incorporating the Ros robot control system, the operation paths of the car and drone were synchronized, and the overall scheduling of work tasks was based on communication principles to achieve multi-dimensional connection and interaction.

[0063] The main control module interfaces with the multi-purpose extension components in the upper and lower layers to enable timely processing and response of data in the cloud server queue;

[0064] The central server, using a task storage queue as its primary data structure, is responsible for temporarily storing tasks submitted by task publishing clients; establishing a task waiting queue to store tasks initially allocated by the central server; and establishing a server task execution queue to store tasks currently being executed by each robot terminal. In the construction of the central server, a state machine is established for each robot terminal to save its current working state. If a robot terminal's task execution queue is full, the robot terminal communicates with the central server, and the central server sets the corresponding state machine to busy. If a robot terminal's task execution queue is not full, the robot terminal communicates with the central server, and the central server sets the corresponding state machine to idle. An information stack is established on the central server to store information about each robot terminal, including its current location and functional information. A socket interface is established in the central server to facilitate data communication between task publishing clients and the central server, and between the central server and the robot terminals, using computer network communication technology. The central server continuously communicates with each robot terminal, updating the state machine and information stack information in real time.

[0065] The D455 depth camera is used to continuously capture photos, calculate the difference to determine the distance, and perform 3D modeling and obstacle avoidance using four consecutive images.

[0066] A lidar is a radar system that uses laser beams to detect the position, velocity, and other characteristics of a target. The lidar emits a detection signal towards the target, then compares the received signal reflected back from the target with the emitted signal, processes it, and obtains relevant information about the target, including the target's distance, azimuth, altitude, velocity, attitude, and shape parameters.

[0067] The digital twin identification module uses digital twin technology to identify and warn of spatial coordinates of relevant information images collected within the land and air working area, and sends them to the dispatch center for visualization. The relevant information includes dynamic target information and static target information to be identified in a certain area.

[0068] Combining all the above technical solutions, the advantages and positive effects of this invention are as follows: The land-air collaborative concept proposed in this invention solves a series of problems such as the limited operating area of ​​ground robots, the limited aerial operation time of robots, and the limited payload. This invention comprehensively utilizes enhanced discrimination technology and operating system scheduling concepts to realize functions such as task scheduling and allocation on both land and air sides, real-time scene mapping and localization, autonomous navigation, target detection, and path planning, significantly improving the working range of the robot system.

[0069] This invention provides a better solution for tasks such as delivery and search and rescue. It breaks away from the current situation of single drones and intelligent vehicles operating independently, offering a technical solution for the problem of multi-robot cross-domain operations. This invention incorporates a land-air collaborative approach, providing key technical points for enabling multi-machine cooperation across domains on drones and intelligent vehicles.

[0070] This invention relates to an intelligent collaborative robot focused on collaborative control between land and air. Utilizing DRL in conjunction with the TD3 algorithm, it implements an independent local path planning algorithm, solving the problem of real-time path planning for UAVs based solely on images when depth information is lacking. This invention uses digital twin technology to identify and warn of spatial coordinates of relevant information images collected within the land-air working area, and sends this information to the dispatch center for visualization. The relevant information includes information on dynamic and static targets to be identified in a specific area. Attached Figure Description

[0071] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure;

[0072] Figure 1 This is a flowchart of the collaborative patrol method between drones and intelligent vehicles provided in an embodiment of the present invention;

[0073] Figure 2 This is a flowchart of spatial coordinate position identification using digital twin technology provided in an embodiment of the present invention;

[0074] Figure 3 This is a schematic diagram of a drone and intelligent vehicle collaborative patrol system provided in an embodiment of the present invention;

[0075] In the diagram: 1. Raspberry Pi; 2. Main control module; 3. Central server; 4. D455 depth camera; 5. LiDAR; 6. Digital twin recognition module. Detailed Implementation

[0076] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0077] This invention establishes an independent, highly portable robot operating system that combines land and air cluster operations, and develops technologies such as operating system collaborative scheduling, SLAM scene mapping, path planning, and target recognition.

[0078] To coordinate multi-agent task scheduling, this invention designs a more adaptable HWT-OS operating system to realize land-air collaborative task allocation and scheduling, solving the problems of short flight time and low payload for aerial operations and large limitations for ground operations.

[0079] Mapping and real-time localization are fundamental to path planning and obstacle avoidance. The architecture of SLAM (Simultaneous Localization and Mapping) enables multi-level acquisition of external information, accurately achieving robot positioning and elevation. Real-time images acquired through depth cameras provide excellent feedback on the robot's current location, helping it to plan appropriate paths and avoid obstacles.

[0080] In path planning, to reduce invalid calculations of repeated nodes and avoid excessive pre-computation due to the use of global path planning algorithms, this invention compares the differences in evaluation functions between the traditional Dijkstra's algorithm and the A* algorithm, and builds a driving control framework adapted to intelligent vehicles. This ensures the efficiency and stability of path planning in constantly changing and complex environments. Simultaneously, this invention introduces an adaptive Monte Carlo method to assist localization in the intelligent vehicle's control system, ensuring that the vehicle can avoid obstacles that appear at any time during travel. By fully combining global and local path planning, the information acquired by the hardware can be effectively decomposed, improving the actual performance of the intelligent robot.

[0081] Example 1, such as Figure 1 As shown, the drone and intelligent vehicle collaborative patrol method provided in this embodiment of the invention includes:

[0082] S1, build the HWT-OS operating system;

[0083] S2 performs task scheduling and allocation for drones and intelligent vehicles, real-time scene mapping, target detection, path planning, obstacle avoidance, and positioning.

[0084] S3 is a robotic system composed of drones and intelligent vehicles that operates within a land and air work area.

[0085] S4 uses digital twin technology to identify and warn of the spatial coordinates of dynamic and static target information images collected within the land and air working area, and sends them to the dispatch center for visualization.

[0086] In this embodiment of the invention, part of the HWT-OS operating system is deployed in different locations in the environment on different robot terminals as different CPUs, and unassigned tasks are treated as processes to be executed; the other part is a central server used to receive tasks submitted by clients that need to be completed by the robots, and to allocate tasks according to task priority and the robot terminal's own status.

[0087] In this embodiment of the invention, the HWT-OS operating system implementation process includes: data communication between the task publishing client and the central server, and between the central server and the robot terminal, is conducted through a unified task message format HWT_message. The task message format includes the task publishing client, task priority, and task receiving location. The task publishing client explains the source of the task, the task priority indicates the importance of the task, and the task receiving location explains the approximate execution range of the task.

[0088] If the task queue for a robot terminal is full, the central server sets the state machine corresponding to that robot terminal to busy. If the task queue for a robot terminal is not full, the central server sets the state machine corresponding to that robot terminal to idle.

[0089] The information stack stores the functional information of the robot terminal, including the robot's working area and the robot's remaining battery power. The robot's working area describes whether the robot can perform the task within the target range, and the robot's remaining battery power is used to determine whether the robot can complete the task.

[0090] In this embodiment of the invention, a SLAM autonomous mapping algorithm based on depth enhancement discrimination is used for real-time scene mapping, including:

[0091] Based on the Cartographer environment, we introduce the visual SLAM technology of ORB-SLAM;

[0092] By combining laser SLAM and visual SLAM, loop closure detection and image optimization are performed for map construction. Loop closure detection detects previously visited locations during SLAM and establishes loop closures by identifying shared feature points or feature descriptors of the locations, correcting errors in previous location estimation. Image optimization uses optimization algorithms to optimize the camera trajectory and map estimated during SLAM.

[0093] In this embodiment of the invention, an enhanced discrimination-based UAV autonomous local path planning and target tracking algorithm is used to perform real-time local path planning for UAVs based on pure images under the condition of missing depth information.

[0094] In this embodiment of the invention, the key process in the UAV autonomous local path planning and target tracking algorithm based on enhanced discrimination is transformed into a deep enhanced discrimination model, in which RLAgent is used to extract preprocessed environmental and sensor information for training, enabling RLAgent to use environmental information to help the UAV find relative position information and make path planning when GPS signals and target depth information are missing.

[0095] In this embodiment of the invention, the intelligent vehicle establishes a three-layer outer auxiliary circle around itself based on the environmental information collected by radar. When an obstacle enters the range of the auxiliary circle, the intelligent vehicle takes different obstacle avoidance measures according to the relative position of the obstacle in the auxiliary circle. At the same time, in the grid map, the edge of the obstacle is subjected to erosion and expansion processing.

[0096] By introducing an adaptive Monte Carlo method to assist in localization, the local information scanned by the sensors is mapped to a global, relatively static space. With the help of data fed back by the LiDAR, the vehicle adjusts its pose in a timely manner and sets up a multi-layered feedback mechanism to avoid obstacles that may appear at any time during its journey.

[0097] In this embodiment of the invention, the target detection includes:

[0098] Based on the YOLOv7 model, a deep discriminant network is constructed for the target to be identified. Combined with the D455 depth sensing camera on the agent, the specific location of the target is detected. Depth completion is performed based on the trained detection model. A dense depth map is obtained by combining the sparse depth map with the RGB image, and rich environmental representation is obtained for high-level scene understanding, perception and planning in subsequent tasks.

[0099] like Figure 2 As shown, step S4, which uses digital twin technology to identify spatial coordinates, includes:

[0100] S101, Image spatial coordinate location sample set collection, preprocessing and dataset creation;

[0101] S102, Design a digital twin network structure;

[0102] S102, Digital Twin Network Parameter Identification;

[0103] S104, Random pre-inversion is performed on the test set images to construct inversion trajectories;

[0104] S105, use fully twinned instances of the trained digital twin network to infer the inversion pattern in parallel and compute the multi-trajectory classification variance matrix F;

[0105] S106, based on the principle of maximum variance deviation, the object category and spatial coordinate location vector value are simultaneously predicted by the multi-trajectory classification variance matrix F.

[0106] In step S101, the collection, preprocessing, and dataset creation of the image spatial coordinate location sample set include:

[0107] (1) For the image pattern to be identified, set the standard localization vector values ​​for different categories of patterns respectively;

[0108] (2) Collect image spatial coordinate position sample sets for each type of pattern, and try to ensure that the number of spatial coordinate position sample sets is balanced among the types;

[0109] (3) The collected spatial coordinate position sample set is normalized sequentially, and the spatial coordinate position sample set is inverted to the normalized positioning vector value;

[0110] The collected spatial coordinate position sample set is sequentially normalized, and the spatial coordinate position sample set is inverted to normalized positioning vector values; the inverted spatial coordinate position positioning vector values ​​of the object are denoted as... When the object's positioning vector value is in the y-axis direction When an object is inverted counterclockwise, its spatial coordinate position positioning vector value is negative; when inverted clockwise, its spatial coordinate position positioning vector value is positive. For the collected image spatial coordinate position sample set of the i-th type of pattern, the spatial coordinate position positioning vector value is arbitrary, and any inversion pattern is represented as: Normalized spatial coordinate location sample set for arbitrary inversion mode Adjust to the standard mode with a spatial coordinate positioning vector value of 0 degrees.

[0111] (4) Divide the preprocessed image spatial coordinate position sample set into training, validation and test sets in a ratio of 7:1:2;

[0112] (5) During training and testing, the input image is normalized into a square before input;

[0113] In step S102, the design of the digital twin network structure includes:

[0114] (1) Design the digital twin network structure reasonably according to the size of the task dataset in order to obtain higher generalization performance on the test set;

[0115] (2) The digital twin network structure used for inversion pattern recognition can be designed manually or by automated machine discrimination methods such as neural architecture search. The core principle of the design is to ensure that the network size is adapted to the task dataset and achieves the best generalization performance on the test dataset.

[0116] (3) The first layer of the network uses 64 twin kernels of size 5, with a twin stride of 2, and fills the image edge feature map in a filling manner so that the output size remains unchanged;

[0117] The second layer of the network uses max pooling with a stride of 2, which reduces the network size by half.

[0118] The third twin layer of the network uses 64 twin kernels of size 5, with a twin stride of 2, and fills the image edge feature map using a padding method, keeping the output size unchanged;

[0119] The fourth layer of the network uses max pooling with a stride of 2, which reduces the network size by half.

[0120] The fifth, sixth, and seventh layers of the network use a fully connected network; among them, the fifth layer receives vector features from the flattened feature map of the fourth layer and has 240 neuron outputs.

[0121] The sixth layer of the network outputs from 168 neurons;

[0122] The seventh layer of the network has 20 neurons, each corresponding to one of the 20 categories in the multi-class classification task.

[0123] In step S103, the digital twin network parameter identification includes:

[0124] (1) Regression is used to process the outputs of multiple neurons in the digital twin network, cross-entropy loss is used as the network discrimination bias principle, and mini-batch gradient descent is used to discriminate network parameters.

[0125] (2) Input the standardized training space coordinate position sample set into the digital twin network model in small batches, calculate the structured loss using the cross-entropy loss function, and then update the network using mini-batch gradient descent; wherein, the discrimination rate during training is an adjustable discrimination rate;

[0126] In step S104, the step of randomly pre-inverting the test set images to construct the inversion trajectory includes:

[0127] (1) For an input image x with any localization vector value θ in the test set θ ∈R H×W×d Where H, W, and d represent the image height, width, and number of trajectories, respectively; equal-interval inversion is performed within the range of [-Φ, +Φ] degrees, with N inversion times and an inversion vector angle interval ΔΦ = 2Φ / N;

[0128] (2) The inverted models and their angles are as follows:

[0129]

[0130]

[0131] in, Φ and ΔΦ represent the expected range and accuracy of the inversion pattern recognition angle estimation, respectively. They are set according to the task requirements, with 0≤Φ≤π. ΔΦ is adjusted by setting the number of inversion trajectories N.

[0132] (3) Image inversion was performed using a bilinear interpolation algorithm. The angle intervals of the inversion vectors were set to 2 degrees and 0.2 degrees, respectively, and the estimation intervals Φ were set to π and π, respectively.

[0133] In step S105, the step of using fully twinned instances of the trained digital twin network to infer the inversion pattern in parallel and calculate the multi-trajectory classification variance matrix F includes:

[0134] (1) After pre-inversion of the image, the copied object pattern in the nth inversion trajectory is used to perform feature extraction and category prediction using the trained digital twin network, as expressed in the following expression:

[0135]

[0136] Among them, F n It is the classification response of the nth trajectory;

[0137] (2) Arrange the classification variance outputs of all inverted trajectories into an inverted multi-trajectory classification variance matrix, the expression of which is:

[0138]

[0139] Where F∈R N×C T is the number of categories to be predicted;

[0140] In step S106, the step of simultaneously predicting the object category and spatial coordinate location vector value from the F matrix according to the maximum variance deviation principle includes:

[0141] (1) Using classification variance as the final criterion for pattern recognition, the maximum value of multiple trajectory responses is used to simultaneously predict the object category and spatial coordinate location vector value:

[0142]

[0143] (j,k)=argmax(F N×C )

[0144] (2) When using the classic digital twin network for supervised classification, the category and vector angle prediction can be performed using the principle of maximum deviation of classification variance. Similarly, the principle of maximum deviation of similarity can also be used.

[0145] (3) Perform the above process on the dataset to obtain the estimation performance of category and spatial coordinate location vector values.

[0146] like Figure 3 As shown, an embodiment of the present invention provides a collaborative patrol system between a drone and an intelligent vehicle, comprising:

[0147] Raspberry Pi 1 was mounted on a smart wireless charging car and a quadcopter drone. Through the onboard Ros robot control system, the operation paths of the car and the drone were synchronized. Based on the communication principle, the overall scheduling of the work tasks was carried out in a multi-dimensional series and interaction.

[0148] Main control module 2 interfaces with the multi-purpose extension parts in the upper and lower layers to enable timely processing and response of data in the cloud server queue;

[0149] Central server 3 uses a task storage queue as its main data structure. It is responsible for temporarily storing tasks submitted by task publishing clients; establishing a task waiting queue to store tasks that have been initially allocated by the central server; and establishing a server task execution queue to store tasks that are currently being executed by each robot terminal.

[0150] In the construction of the central server, a state machine is established for each robot terminal to store the current working state of each robot terminal. If the robot task queue of a certain robot terminal is full, the robot terminal communicates with the central server, and the central server sets the state machine corresponding to the robot terminal to busy. If the robot task queue of a certain robot terminal is not full, the robot terminal communicates with the central server, and the central server sets the state machine corresponding to the robot terminal to idle.

[0151] The central server establishes an information stack to store information about each robot terminal. The information stack stores the current location information and functional information of each robot terminal.

[0152] A socket interface is established in the central server, and data communication between the task publishing client and the central server, and between the central server and the robot terminals is realized with the help of computer network communication technology. The central server continuously communicates with each robot terminal to update the state machine and information stack information in real time.

[0153] The D455 depth camera 4 is used to continuously capture photos, calculate the difference to realize the distance, and use four consecutive images for 3D modeling and obstacle avoidance.

[0154] A lidar 5 is a radar system that uses emitted laser beams to detect the position, velocity, and other characteristics of a target. The lidar emits a detection signal towards the target, and then compares the received signal reflected back from the target (the target echo) with the emitted signal. After processing, it obtains relevant information about the target, including the target's distance, azimuth, altitude, velocity, attitude, and shape parameters.

[0155] The digital twin identification module 6 uses digital twin technology to identify and warn of spatial coordinates of relevant information images collected within the land and air working area, and sends them to the dispatch center for visualization. The relevant information includes dynamic target information and static target information to be identified in a certain area.

[0156] Example 2: To further illustrate the implementation process of the above technical solution, the relevant technical solution is described as follows.

[0157] 1. HWT-OS Operating System Design:

[0158] 1.1 While multi-machine collaborative robots have achieved some significant progress, and large-scale development and application have begun in this field, their control in drone swarms, though achieving good coordination, also has obvious limitations. Research is currently confined to multi-machine swarm algorithms, only enabling robot collaboration within a single swarm of drones or intelligent vehicles. Research on cross-domain robot collaboration is still insufficient. It is clear that current multi-machine collaboration still faces many challenges, and swarm control requires further improvement.

[0159] To address this issue, the present invention specifically designs a multi-robot collaborative robot operating system to realize the allocation of tasks among robots, solve the problem of collaborative operation among robots, and provide theoretical guidance and technical support for large-scale robot swarm operations.

[0160] 1.2 System Design Overview: The design of the robot's operating system should mainly cover two parts: the first part is the different robot terminals deployed in different locations in the environment; and the second part is the central server designed according to the requirements, which is used to receive the tasks submitted by the client that the robot needs to complete, and to allocate tasks according to task priority and the robot terminal's own status.

[0161] The core idea of ​​the system is to abstract different robot terminals as different CPUs in an operating system; to abstract unassigned tasks as processes to be executed in an operating system; and to abstract the central server as the kernel of an operating system. Tasks are then rationally allocated to each robot terminal through the scheduling of the central server, achieving faster and more efficient task execution.

[0162] Based on relevant information, this invention names this operating system, which integrates core functions including collaborative scheduling, SLAM scene mapping, path planning, and target recognition, the HWT-OS system. Figure 3 As shown, in order to realize the allocation and scheduling of land-air collaborative tasks, this operating system, as an independently developed embedded operating system, imitates the Linux kernel and can realize a series of requirements such as data sharing, task allocation and priority scheduling of processes in microcomputers such as Raspberry Pi. Unlike traditional intelligent agent control without an original operating system, the HWT-OS has achieved coordination and unification of internal data structure, and has condensed the code ideas of developers into the same closed-loop system, effectively solving the problems of short endurance time and low load capacity of robots in aerial operations and great limitations in ground operations.

[0163] 1.3, Core Components of the System:

[0164] The hardware includes a Raspberry Pi, a microcomputer with multiple external interfaces. Due to its excellent support for common standards such as HDMI and USB, the Raspberry Pi has strong expandability and can be used for embedded development under the Linux system. In this invention, the Raspberry Pi serves as the main controller for the entire project, mounted on both a smart wireless charging vehicle and a quadcopter. Through the integrated ROS robot control system, the operational paths of the vehicle and the drone are synchronized, and the overall scheduling of work tasks is based on communication principles, enabling multi-dimensional connection and interaction.

[0165] To better facilitate the interface between the main control module and the multi-purpose extension components in the upper and lower layers, and to ensure timely processing and response to data in the cloud server queue, this invention considers the Jestonnano development board provided by NVIDIA as a superior main control option. As a more powerful microcomputer, the Jestonnano offers greater support for the development of applications within the AI ​​framework. In practical use, its default support for SSH services simplifies the bridge construction between the main control module and the server. Furthermore, as the main hub for multiple sensors and controllers, the Jestonnano's parallel processing capabilities perfectly match the requirements of this work.

[0166] In terms of software, as an advanced system development based on the ROS underlying architecture, the HWT-OS system not only retains the excellent features of ROS in hardware abstraction, low-level device control, simplified function interfaces, and reasonable multi-process planning, but also enriches the mode conversion between drones and intelligent vehicles to meet the operational needs of this project. This greatly improves the efficiency of information transmission between modules and ensures good compatibility with main control terminals with limited performance and storage.

[0167] Existing drone development is constrained by limited performance, and the relatively bloated operating system is unable to "show its full potential" on microcomputers. Its functionality is also limited in simple take-off and landing and multi-point positioning. In contrast, HWT-OS has excellent portability and can give developers ample creative space within the limited available resources.

[0168] In terms of data construction for the central server, the central server uses a task storage queue as its main data structure, which is responsible for temporarily storing tasks submitted by task publishing clients; a task waiting queue is established to store tasks that have been initially allocated by the central server; and a server task execution queue is established to store the tasks that are currently being executed by each robot terminal.

[0169] In the construction of the central server, a state machine is to be established for each robot terminal to store the current working state of each robot terminal. The state machine has two states: "busy" and "idle". If the robot task queue of a certain robot terminal is full, the robot terminal communicates with the central server, and the central server sets the state machine corresponding to the robot terminal to "busy". If the robot task queue of a certain robot terminal is not full, the robot terminal communicates with the central server, and the central server sets the state machine corresponding to the robot terminal to "idle".

[0170] The central server establishes an information stack to store information about each robot terminal. The information stack stores the current location information and functional information of each robot terminal.

[0171] A socket interface is established in the central server, and data communication between the task publishing client and the central server, and between the central server and the robot terminals is realized with the help of computer network communication technology. The central server continuously communicates with each robot terminal to update the state machine and information stack information in real time.

[0172] 1.4 The HWT-OS operating system implementation process includes: data communication between the task issuing client and the central server, and between the central server and the robot terminal, is conducted through a unified task message format HWT_message. The task message format includes the task issuing client, task priority, and task receiving location. The task issuing client explains the source of the task, the task priority indicates the importance of the task, and the task receiving location explains the approximate execution scope of the task.

[0173] If the task queue for a robot terminal is full, the central server sets the state machine corresponding to that robot terminal to "busy". If the task queue for a robot terminal is not full, the central server sets the state machine corresponding to that robot terminal to "idle".

[0174] The information stack stores the functional information of the robot terminal, including the robot's working area and the robot's remaining battery power. The robot's working area describes whether the robot can perform the task within the target range, and the robot's remaining battery power is used to determine whether the robot can complete the task.

[0175] 2. SLAM Simultaneous Localization and Mapping Design:

[0176] 2.1 Introduction to SLAM Algorithm: SLAM, short for Simultaneous Localization and Mapping, is a technique used to simultaneously achieve autonomous localization and map building. It is widely used in unmanned systems, robotics, and augmented reality. The SLAM algorithm allows machines to locate their own position in unknown environments in real time while simultaneously building a map of the environment, thereby enabling autonomous navigation and exploration.

[0177] SLAM technology solutions are broadly divided into two directions: laser SLAM and visual SLAM. This work combines the excellent front-end matching mechanisms of the two, improves measurement accuracy through feature extraction, and introduces loop closure detection and image optimization to ensure good mapping results in real-world working environments.

[0178] SLAM technology incorporates loop closure detection and image optimization to improve mapping accuracy, which is particularly prominent in simulated multi-vehicle applications. Loop closure detection refers to identifying previously visited locations during SLAM and establishing loop closures by recognizing shared feature points or feature descriptors, thereby correcting errors in previous location estimations. Image optimization, on the other hand, uses algorithms to refine the estimated camera trajectory and map during SLAM, improving the accuracy of localization and map construction. These loop closure detection and image optimization techniques can reduce error accumulation and prevent positional deviations caused by error accumulation.

[0179] 2.2 Components of SLAM Functionality:

[0180] 2.2.1 Hardware Requirements: The successful implementation of SLAM applications relies on the selection and debugging of hardware. The D455 depth sensing camera module and an outdoor LiDAR module were chosen as the core components. Their energy efficiency met the system's operational specifications, serving as the foundation for the development process. Combined with technological investment, a hardware platform for the SLAM application was built. This includes:

[0181] The D455 depth camera is a quad-camera system designed to capture images continuously from four cameras, calculate the difference between them to determine distance, and then use the four consecutive images for 3D modeling and obstacle avoidance.

[0182] Advantages of the D455 depth camera: Compared to the two-dimensional images acquired by ordinary cameras, depth cameras can obtain richer positional relationships between objects through distance information, thus clearly distinguishing the foreground and background to a large extent. The richer information captured by the depth camera allows for better performance in traditional applications such as target image segmentation, labeling, recognition, and tracking, with particularly significant efficiency improvements in recognition and tracking. The main components of a depth camera are relatively inexpensive, including CCDs and ordinary LEDs, making manufacturing processes simpler and facilitating mass production. It also offers stronger resistance to interference in complex environments.

[0183] A lidar (Light Detection and Ranging) system is a radar system that uses laser beams to detect the position, velocity, and other characteristics of a target. Lidar emits a detection signal (laser beam) towards the target, then compares the received signal reflected back from the target (target echo) with the emitted signal. After appropriate processing, it can obtain relevant information about the target, such as its distance, azimuth, altitude, velocity, attitude, and shape.

[0184] Advantages of LiDAR: LiDAR can achieve extremely high angular, range, and velocity resolution, and can track multiple targets simultaneously, something microwave radar simply cannot do. LiDAR offers excellent concealment and strong resistance to active interference. Laser light propagates in a straight line, has excellent directionality, and a very narrow beam, meaning it can only be received along its propagation path. Conventional microwave radar is typically bulky, with the entire system weighing several tons and antenna apertures reaching several meters or even tens of meters. LiDAR, on the other hand, is much lighter, more flexible, has a relatively simple structure, is easy to maintain, easy to operate, and readily available for secondary development.

[0185] In this invention, the combination of LiDAR and D455 depth camera helps the vehicle achieve obstacle avoidance and navigation functions. The LiDAR scans the surrounding environment of the vehicle, and real-time mapping is performed based on the feedback signals, thereby designing and improving the overall path planning of the project.

[0186] 2.2.2, Algorithm Selection Explanation:

[0187] Cartographer and Gmapping, as common SLAM mapping algorithms, are widely used in the research and development of robotics. Both rely on signal data acquired from LiDAR to build maps. Specifically, they collect the three-dimensional coordinates of the target object's surface and convert them into point cloud data. Based on this point cloud data, the robot estimates and corrects its own trajectory, thereby optimizing the accuracy and stability of the mapping process and ultimately achieving the completeness and accuracy requirements of radar mapping. Compared to the inherent limitations of the Gmapping algorithm in outdoor mapping, the Cartographer algorithm selected in this invention has the capability to comprehensively map data from multiple sensors, and its computational resource and hardware requirements are basically met, thus making it the primary research direction.

[0188] Building upon its basic compatibility with the Cartographer environment, this invention further introduces a visual SLAM technology called "ORB-SLAM". ORB-SLAM features an excellent loop closure detection algorithm, which can effectively prevent the accumulation of errors during information analysis. Especially in indoor applications, when there are many visual co-occurrences, this SLAM technology will significantly improve positioning accuracy and local stability.

[0189] By combining laser SLAM and visual SLAM, this work fully leverages the advantages of both laser and visual sensors during the mapping process, thereby improving positioning accuracy and stability. Laser sensors provide high-precision distance and angle information, suitable for both indoor and outdoor environments, while visual sensors offer rich visual feature information, excelling in handling textured areas and special situations within the environment.

[0190] 2.2.3, SLAM technology and hardware adaptation:

[0191] With the development of SLAM and related technologies, existing RRT autonomous exploration mapping algorithms can no longer meet the needs of autonomous mapping for robots in large-scale environments, especially in the field of autonomous driving, where all mapping work still requires manual operation. To address this, we propose a SLAM autonomous mapping algorithm based on depth reinforcement discrimination. This algorithm solves the problems of slow mapping and excessive memory consumption in RRT autonomous exploration mapping when the environment range is large. It can quickly and autonomously build target maps in unfamiliar environments by combining various SLAM methods.

[0192] 3. Path planning and obstacle avoidance system design:

[0193] Intelligent agents reduce uncertainty in the travel process by choosing the optimal path based on existing map information; this is the application value of path planning. For example, Dijkstra's algorithm, a representative of traditional path planning, ensures that connecting lines are globally optimal by expanding outwards from the starting point and transforming node attributes. This invention, during its development, compared the differences in evaluation functions between the traditional Dijkstra algorithm and the A* algorithm, and built a travel control framework adapted to intelligent vehicles, reducing invalid calculations for duplicate nodes and verifying the scientific validity of this method applied to global path planning. However, it also points out that using only global path planning algorithms leads to massive pre-computation, high memory usage, and difficulty in ensuring the efficiency and stability of path planning in constantly changing and complex environments.

[0194] Therefore, the intelligent vehicle control system of this invention introduces an adaptive Monte Carlo method to assist in localization. This method maps the local information scanned by sensors to a global, relatively static space, and uses data from the LiDAR to adjust its own pose in a timely manner. A multi-layered feedback mechanism is established to ensure that the intelligent vehicle can avoid obstacles that may appear during its journey. By fully integrating global and local path planning, the information acquired by the hardware can be effectively decomposed, improving the vehicle's actual performance.

[0195] 3.1 UAV Local Path Planning Based on Fast-Planner: To better achieve UAV path planning, this invention develops an autonomous UAV path planning algorithm based on the Fast-Planner system, suitable for the UAVs in this system. Fast-Planner is a highly efficient global path planning algorithm, primarily used to generate long-term planned paths for UAVs to minimize the overall path time and distance. This algorithm generates an optimal path between the starting point and the destination using prior map information and the current state of the UAV, and it is very fast in terms of running time, typically completing path planning within a few seconds. In this system, this is specifically reflected in real-time path planning and obstacle avoidance during UAV fixed-point tasks.

[0196] 3.2, Autonomous Local Path Planning and Target Tracking Algorithm for UAVs Based on Enhanced Discrimination. Unlike traditional local path planning tasks, this invention departs from the traditional AMCL algorithm and establishes a pure image-based local path planning algorithm based on depth enhancement discrimination. This algorithm effectively solves the problem of real-time local path planning for UAVs based on pure images under conditions where depth information is lacking.

[0197] This invention transforms the key processes into a deep reinforcement discrimination model, where RLAgent is used to extract preprocessed environmental and sensor information for training. This enables RLAgent to utilize environmental information to allow the UAV to accurately locate its relative position and plan its path even when GPS signals and target depth information are missing. The success rate reaches 93% in the absence of obstacles and 92% even with obstacles present.

[0198] 3.3 Excellent Obstacle Avoidance Algorithm: In terms of obstacle avoidance functionality, the intelligent vehicle establishes a three-layered outer auxiliary circle around itself based on environmental information collected by radar. When an obstacle enters the range of the auxiliary circle, the intelligent vehicle can take different obstacle avoidance measures according to the obstacle's relative position within the circle. Simultaneously, erosion and dilation processing is applied to the obstacle edges in the grid map to further improve the accuracy of obstacle avoidance.

[0199] Furthermore, the obstacle avoidance control scheme designed for this robot system features a relatively complete closed-loop structure. This system converts external environmental information into a unified format recognizable by the main controller, separates the effective parts of the information, and extracts key expected values ​​such as the intelligent vehicle's steering radius, the drone's motor speed, and the optimal take-off and landing position. Comparing these expected values ​​with actual parameters reveals multiple deviations. The system integrates these deviations and optimizes them uniformly to achieve a closed loop, significantly improving the obstacle avoidance success rate of both the drone and the intelligent vehicle.

[0200] Unlike existing delivery systems, which rely heavily on simple sensors or human intervention to handle obstacle information, current smart delivery vehicles prioritize safety for both the robot and its surroundings.

[0201] 4. Other model designs and innovations:

[0202] 4.1 Object Detection Model Based on YOLOv7: The YOLO series of algorithms, as a popular object detection algorithm, combines classification tasks with object localization regression problems using anchor boxes. It divides the original image into multiple regions for prediction, and uses NMS (Non-Maximum Suppression) to filter bounding boxes with small variance, thus determining the specific location of the target. This truly achieves real-time detection and localization in industrial applications, exhibiting high efficiency, flexibility, and good generalization performance. The new generation YOLOv7 extends the efficient long-range attention network and designs and uses planned reparameterization of the twin, thus avoiding the loss of necessary effective information during data analysis. It achieves more stable results than previous models while balancing speed and accuracy.

[0203] This invention constructs a deep discrimination network based on the YOLOv7 model to identify the target, and further improves the target's specific location detection process by combining it with the D455 depth sensing camera module on the agent. Specifically, in the advanced application development of the D455, this invention can perform depth completion based on the trained detection model, obtaining a dense depth map by combining a sparse depth map with an RGB image, thereby acquiring a rich environmental representation, which is helpful for higher-level scene understanding, perception, and planning in subsequent tasks.

[0204] 5. Visual Interface Design: Considering the design requirements of the host computer, this invention uses PyQt5 in the Python environment as the main framework. Leveraging its excellent design in network transmission, graphical control, and responsive feedback, the main framework for the host computer display in this project is constructed. Thanks to PyQt5's cross-platform support, the code built on the Windows system can run conflict-free on the Linux system, demonstrating the stability pursued in the development of this project.

[0205] This host computer adopts a traditional client-server distributed system architecture. The client collects multiple data from the camera and sensor components. The link mechanism in the main logic module transfers the analyzed data and key information. Combined with the user's key feedback, the data is displayed in a unified format on the host computer's visualization page.

[0206] Built upon the established TCP / IP communication protocol, this CS architecture provides comprehensive parsing of received frames from the host computer. The received frame header specifies the image length, image type number, and necessary return values. The main receiving part involves judging the received information based on the return values, verifying whether the length of the received image is equal to the image length specified in the frame header. If they are equal, the image transmission operation is completed. Furthermore, communication between processes and threads incorporates queues as a data structure. Their FIFO characteristics enable asynchronous message delivery and buffering, reducing pressure on processes or threads and improving system reliability and stability.

[0207] The user interface design adopts a "3+1" display architecture. Real-time images captured by multiple devices are vertically arranged on the left side of the page via an image transmission module. The server integrates the environmental information collected during their respective explorations into the same composite map. The host computer acquires the map at set transmission intervals and then displays it in the center of the page.

[0208] Through the above embodiments, it can be seen that the present invention has the following advantages:

[0209] At the architectural level: Intelligent vehicles and drones, through system-level technological interconnection and task processing mechanisms extended on the client-server architecture, achieve complementary advantages of two distinct intelligent agents in the logistics field. While ensuring a certain volume of delivery tasks, intelligent vehicles are compatible with a range of needs of drones, such as multi-point transport, energy replenishment, and target designation. This not only meets the diverse needs of customers but also deeply embodies the overall architectural concept of "land-air collaboration."

[0210] In terms of energy efficiency: Through a horizontal comparison of the advantages and disadvantages of various SLAM mapping algorithms, this work constructs a more adaptable mapping system. Drones and intelligent vehicles can project the acquired environmental information onto the same map components, which significantly improves the mapping efficiency compared to traditional technologies. With a good map, the intelligent vehicle's obstacle avoidance response in unfamiliar environments is more flexible, and the self-designed feedback mechanism algorithm gives the intelligent vehicle different turning radius options, ensuring the continuity of its movement process.

[0211] On the innovation front: The introduction of deep learning enhancement (DRL) and its integration with the control strategies of intelligent agents allows for a higher creative ceiling. Compared to traditional robots that lack discrimination capabilities and have fixed working positions, DRL enables robots to autonomously discern, explore, plan, and map, and autonomously identify and implement local path planning actions. The application of DRL in the model enhances the user experience.

[0212] Application Example 1: Community Logistics Delivery. With the rapid development of e-commerce, the workload of the express delivery industry is becoming increasingly heavy. At the same time, people's demand for timely package delivery is growing daily, amplifying the resulting logistics inconveniences. However, intelligent collaborative robots can greatly improve logistics convenience: A small vehicle and a drone work together. The vehicle transports goods to the ground floor, and then the drone carries the goods up to a higher floor for delivery, thus realizing the sorting function of the vehicle and the last-mile delivery work of the drone.

[0213] Application Example 2: Disaster relief efforts in areas such as earthquakes, fires, and mudslides. During natural disasters, the harsh environments often prevent rescuers from reaching severely affected areas and obtaining information about trapped individuals, significantly hindering rescue operations. Therefore, this invention utilizes the dexterity of collaborative robots, enabling them to easily enter confined spaces. Through land-air collaboration and enhanced autonomous identification capabilities, it collects information about trapped individuals and transmits it back to rescuers for execution.

[0214] Application Example 3: Special Broadcasting of Events and News Scenes: When broadcasting breaking news, due to venue limitations, it may be impossible to immediately present firsthand news details to the public. A ground-air collaborative intelligent robot can broadcast live key moments of events and specific situations at news scenes. This is mainly used from relatively remote perspectives. The robot and the drone form two perspectives—ground and aerial—for simultaneous broadcasting. Furthermore, the robot can provide power to the drone at any time, ensuring the speed and real-time nature of the broadcast.

[0215] Application Example 4: Military force building. Intelligent robots can perform patrol functions, reduce the burden on border defense, and closely monitor the security of territory and airspace.

[0216] Application example 5 shows that, in terms of prospects for promotion, it can be applied in environmental analysis, logistics organizations, and robot swarms, and also has broad prospects for economic benefits.

[0217] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0218] The information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0219] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments.

[0220] Based on the technical solutions described in the above embodiments of the present invention, the following application examples can be further proposed.

[0221] According to embodiments of this application, the present invention also provides a computer device comprising: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above-described method embodiments.

[0222] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps described in the various method embodiments above.

[0223] This invention also provides an information data processing terminal, which, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments. The information data processing terminal is not limited to mobile phones, computers, or switches.

[0224] This invention also provides a server that, when executed on an electronic device, provides a user input interface to implement the steps described in the above method embodiments.

[0225] This invention also provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0226] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographic device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0227] To further demonstrate the positive effects of the above embodiments, the present invention conducts the following experiments based on the above technical solutions.

[0228] 1. Experimental data recording:

[0229] 2022-09-28 23:00:46:

[0230] Begin establishing the transmission node to achieve connectivity between mavros and hwtOS;

[0231] 2022-09-30 03:31:10:

[0232] Create a topic publishing class, establish a drone master node to enable information sharing from Mavros to HWT topics, and improve the information sharing functionality from Mavros to HWT topics:

[0233] More classes can be shared simply by adding it directly to the Hwt_Htransform class;

[0234] Create a class to accept topics, specifying which topics need to be published from the HWT topic to the corresponding Mavros topic using (setxxxx);

[0235] Imagine how to establish a unified server and client architecture;

[0236] October 1, 2022, 08:48:36:

[0237] The initial release function has been implemented, and the SendTest node has been created for testing and improvement.

[0238] You can directly create the topic you want to receive in the Hwt_HSetTransform class to receive the settings you need;

[0239] Create the topic to be published directly in the Hwt_HSetTransform class, and publish the received topic data to mavros. It's best to unlock the drone before publishing, and then publish the information to hwt / px4 / Setmessage / xxx; otherwise, the drone may fail to unlock. When the drone is performing stationary flight, it needs to continuously publish information for it to continue moving forward.

[0240] October 2, 2022, 13:12:24:

[0241] I successfully created a project on GitHub and was able to clone and modify the Test_Send function. I changed the Sub format to Hwt format to create a px4 unlock node. After writing the clientserve file, I built the px4 node before unlocking and sent it in a loop to successfully enter offboard mode.

[0242] October 3, 2022, 12:03:47:

[0243] Begin establishing service communication functions to implement request and receive functionality.

[0244] October 4, 2022, 16:11:10:

[0245] Begin writing the ClientServe class; the initial class creation is complete. If additional request services are added during development, they can be appended to the Hwt_Hmain_Server.cpp / .h file, and the same logic will ensure successful requests. Guidelines for writing .launch files:

[0246] A total of three launch files were written;

[0247] The first node to be started from the three files is px4 / px4_stil / vision_to_mavros::t265_all_nodes;

[0248] The second node to be started is Hwt_Hmain_node;

[0249] The third node to start is Hwt_Hmain_Server;

[0250] October 7, 2022, 19:42:30:

[0251] Add a cv directory and compile using Python 3 to move hwt_Tranzxxxx_msgs to hwt_data. Other projects should then reference hwt_data to obtain the hwt data format.

[0252] When writing a CMake file, you need to add `add_dependencies(xxx${catkin_EXPORTED_TARGETS}hwt_data_gencpp)` to ensure that the `hwt_data` directory is compiled first, and then other files are compiled.

[0253] Use the command "catkin_make-DPYTHON_EXECUTABLE= / usr / bin / python3" to compile.

[0254] October 8, 2022, 20:06:50:

[0255] Add a rectangle detection test to the hwt_cv_detect package and successfully extract the rectangle. Next steps:

[0256] Combining color detection and rectangle detection;

[0257] First, perform color detection to extract the green shapes (after this step, only the green shapes in the inRange_hsv image will have white dots, while other shapes will have black dots);

[0258] Continue the operation in the inRange_hsv image, performing rectangle detection on it;

[0259] After the two-step detection is completed, the candidate boxes are added for filtering to obtain the final candidate boxes;

[0260] October 09, 2022, 19:27:12:

[0261] The `home_position_detect_test1.py` node was created, initially implementing detection for rectangles and colors. However, the rectangle selection effect is not good and further modifications are needed.

[0262] 2. Add a control module node:

[0263] November 20, 2022, 18:40:20:

[0264] The control module of HWT_OS has been improved, enabling the control of drones via external control commands using HWT_OS;

[0265] To be improved: Add Euler angles to Hwt_ht_position.

[0266] 3. Development of the car module:

[0267] October 15, 2022, 09:40:20:

[0268] A simulation environment for the car platform was built to explore and adapt the Gmapping mapping method. On the specified simulated car model, the mapping effect in the city met expectations, but the image noise in the outdoor mapping was too much, and the relevant parameters need to be further adjusted.

[0269] November 23, 2023, 15:40:00

[0270] Preliminary attempts at feature point detection and matching were made in a simulation environment, but many problems were encountered, including but not limited to: the number of feature point pairs could not meet the minimum requirements for matching, which meant that the Hamming distance metric could not be reflected in the matching operation; there were many conflicts between the available improved algorithms and the simulation environment, and there was a lack of relevant reference materials.

[0271] December 11, 2023, 22:00:00

[0272] To address the limitations of Cartographer's SLAM algorithm in terms of computing resources and hardware, team members collaborated to deeply optimize resource scheduling within the system, modifying configuration files such as urdf for the vehicle component in accordance with this algorithm.

[0273] January 5, 2023, 18:00:00

[0274] There are still many uncertainties in the path planning guidance. The built-in move_base function package requires the specification of relevant parameters of the car. At the same time, some necessary sensor data cannot be reflected in the constructed cost map, and the car failed to avoid local obstacles as expected. Further debugging is needed.

[0275] February 10, 2023, 12:00:00

[0276] The AMCL algorithm was used to adaptively optimize the pose, which effectively corrected the robot's real-time position on the map. Global and local path planning were coordinated and unified in actual results. By subscribing to relevant topics, the complete autonomous mapping effect was seen in the self-built Gazebo map.

[0277] March 5, 2023, 17:25:00

[0278] Significant progress has been made in "coordinate calculation" and "pixel addition". The relative positional relationship between two cars is calculated by transforming Euler angles and quaternions. Combined with feature points, the overlapping position of the two cars on the map is found and the orientation of multiple maps is unified. Thus, multiple maps are overlaid and added together to achieve the fusion and construction of the entire map.

[0279] March 11, 2023, 17:25:00

[0280] The TF tree connection graph for multi-vehicle collaboration has been planned in a relatively complete way. We attempted to add a namespace for each vehicle, changing the vehicle's wheel, lidar, and camera to qinzhou_x / xxxframe in the original frame. On this basis, by subscribing to the robot / state_link node of Gazebo, the global odom data of each vehicle is published to the corresponding topic under its own namespace, such as yingyi_0 / odom. The odomframe on the TF tree automatically publishes the data in / yingyi_0 / ackman_steels / odom to the TF tree topic through the steel package, thus realizing the linking of the TF tree.

[0281] March 28, 2023, 12:25:00

[0282] Multiple vehicles were successfully displayed in Gazebo.

[0283] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention and within the spirit and principles of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles, characterized in that, The method includes: S1, build the HWT-OS operating system; S2 performs task scheduling and allocation for drones and intelligent vehicles, real-time scene mapping, target detection, path planning, obstacle avoidance, and positioning. S3 is a robotic system composed of drones and intelligent vehicles that operates within a land and air work area. S4 uses digital twin technology to identify and warn of the spatial coordinates of dynamic and static target information images collected within the land and air working area, and sends them to the dispatch center for visualization. In step S4, digital twin technology is used to identify and warn of spatial coordinate positions, including: Step 1: Collection, preprocessing, and dataset creation of image spatial coordinate location sample sets; Step 2: Design the digital twin network structure; Step 3: Digital twin network parameter identification; Step 4: Perform random pre-inversion on the test set images to construct inversion trajectories; Step 5: Use fully twinned instances of the trained digital twin network to infer the inversion patterns in parallel and calculate the multi-trajectory classification variance matrix. ; Step 6: Based on the principle of maximum variance deviation, classify the variance matrix of the multi-trajectory system. Simultaneously, it predicts the object category and spatial coordinate location vector value; In step 1, the collection, preprocessing, and dataset creation of the image spatial coordinate location sample set include: (1) For the image patterns to be identified, set their respective standard positioning vector values; (2) Collect image spatial coordinate position sample sets for each type of pattern to ensure that the number of spatial coordinate position sample sets is balanced; (3) The collected spatial coordinate position sample set is normalized sequentially, and the spatial coordinate position sample set is inverted to the normalized positioning vector value; the inverted spatial coordinate position positioning vector value of the object is denoted as The object's positioning vector value is in In the axial direction When an object is inverted counterclockwise, its spatial coordinate position positioning vector value is negative; when inverted clockwise, its spatial coordinate position positioning vector value is positive. For the collected... The image spatial coordinate position sample set of the class pattern, where the spatial coordinate position positioning vector value is arbitrary, and the arbitrary inversion pattern is represented as: Normalized spatial coordinate location sample sets are used to retrieve arbitrary inversion models. Adjust to the standard mode with a spatial coordinate positioning vector value of 0 degrees. ; (4) Divide the preprocessed image spatial coordinate position sample set into training, validation and test sets in a ratio of 7:1:2; (5) During training and testing, the input image is normalized into a square before input.

2. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 1, characterized in that, In step S1, the HWT-OS operating system is constructed, including: deploying a part of the HWT-OS operating system on different robot terminals as different CPUs, and treating unassigned tasks as processes to be executed; deploying another part of the HWT-OS operating system on the central server, which is used to receive tasks submitted by clients that need to be completed by the robot, and to allocate tasks according to task priority and the robot terminal's own status.

3. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 2, characterized in that, Also includes: Data communication between the task publishing client and the central server, and between the central server and the robot terminal, is conducted through a unified task message format, HWT_message. The task message format includes: task issuing customer, task priority, and task receiving location; among which, the task issuing customer explains the source of the task, the task priority indicates the importance of the task, and the task receiving location explains the scope of task execution. In the robot terminal, if the robot's task queue is full, the central server sets the corresponding state machine of the robot terminal to busy; if the robot's task queue is not full, the central server sets the corresponding state machine of the robot terminal to idle. The functional information of the robot terminal includes: the robot's working area and the robot's remaining battery power. The robot's working area is used to describe whether the robot can perform tasks within the target range, and the robot's remaining battery power is used to determine whether the robot can complete the task.

4. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 1, characterized in that, In step S2, task scheduling and allocation between the UAV and the intelligent vehicle, real-time scene mapping, target detection, path planning, obstacle avoidance, and localization are performed. This includes: real-time scene mapping based on a deep reinforcement discrimination SLAM autonomous mapping algorithm; introducing ORB-SLAM visual SLAM technology based on the Cartographer environment; combining laser SLAM and visual SLAM to perform loop closure detection and image optimization for map construction; loop closure detection detects previously visited locations during SLAM and establishes loop closure by identifying shared feature points or feature descriptors of the locations, correcting errors in previous position estimation; image optimization optimizes the estimated camera trajectory and map during SLAM using optimization algorithms.

5. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 4, characterized in that, The path planning adopts UAV autonomous local path planning based on enhanced discrimination, and UAV real-time local path planning based on pure image under the condition of missing depth information by the target tracking algorithm. The key process is transformed into a depth enhanced discrimination model, in which RLAgent is used to extract preprocessed environmental information and sensor information for training. RLAgent uses environmental information to enable the UAV to find relative position information and make path planning when GPS signal and target depth information are missing.

6. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 4, characterized in that, The intelligent vehicle establishes a three-layer circumscribed auxiliary circle around itself based on environmental information collected by radar. When an obstacle enters the range of the auxiliary circle, the intelligent vehicle takes different obstacle avoidance measures according to the relative position of the obstacle in the auxiliary circle. In the grid map, the edges of obstacles are subjected to erosion and expansion processing. An adaptive Monte Carlo method is introduced to assist in localization, mapping the local information scanned by the sensor to the global relative static space. With the help of data fed back by LiDAR, the vehicle adjusts its pose in a timely manner and sets up a multi-layer feedback mechanism to avoid obstacles that may appear at any time during the journey.

7. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 4, characterized in that, The target detection includes: constructing a deep discriminant network based on the YOLOv7 model for the target to be identified; combining the D455 depth sensing camera on the agent to detect the specific location of the target; performing depth completion based on the trained detection model; obtaining a dense depth map by combining a sparse depth map with an RGB image; acquiring rich environmental representations for high-level scene understanding, perception, and planning in subsequent tasks.

8. The method for collaborative patrolling by unmanned aerial vehicles and intelligent vehicles according to claim 1, characterized in that, Step 2 involves designing the digital twin network structure, including: (1) Design the digital twin network structure according to the task dataset size to obtain the generalization performance on the test set; (2) The digital twin network structure used for inversion pattern recognition is designed by manual design and neural architecture search automated machine discrimination method to ensure that the network size is adapted to the task dataset and achieves the best generalization performance on the test dataset; (3) The first layer of the network uses 64 5-Siamese twin kernels with a stride of 2, and the output size of the filled image edge feature map remains unchanged; the second layer of the network uses max pooling with a stride of 2, which reduces the network size by half; the third Siamese twin layer uses 64 5-Siamese twin kernels with a stride of 2, and the output size of the filled image edge feature map remains unchanged; the fourth layer of the network uses max pooling with a stride of 2, which reduces the network size by half; the fifth, sixth and seventh layers of the network use a fully connected network; the fifth layer receives the vector features after the feature map of the fourth layer is flattened, and has 240 neurons output; the sixth layer of the network has 168 neurons output; the seventh layer of the network has 20 neurons output, which correspond to the 20 categories of the multi-classification task respectively; Step 3, digital twin network parameter identification, includes: (1) Regression is used to process the outputs of multiple neurons in the digital twin network, cross-entropy loss is used as the network discrimination bias principle, and mini-batch gradient descent is used to discriminate network parameters; (2) Input the standardized training space coordinate position sample set into the digital twin network model, calculate the structured loss using the cross-entropy loss function, update the network using mini-batch gradient descent, and use an adjustable discrimination rate during training; Step 4, which involves randomly pre-inverting the test set images to construct the inversion trajectory, includes: (1) For any location vector value in the test set Input image ;in, These represent the image height, width, and number of trajectories, respectively; in Perform equal-interval inversions within a certain range, with the number of inversions being [number missing]. Inversion vector angle interval ; (2) The inverted models and their angles are as follows: ； ； in, ; These represent the expected range and accuracy of the angle estimation for pattern recognition inversion, respectively, and are set according to task requirements. , By setting the number of inversion trajectories adjust; (3) Image inversion is achieved using bilinear interpolation algorithm, with the inversion vector angle intervals set to 2 degrees and 0.2 degrees respectively, and the estimation interval... Set to respectively and ; In step 5, the inversion patterns are inferred in parallel using fully twinned instances of the trained digital twin network, and the multi-trajectory classification variance matrix is ​​calculated. ,include: (1) After pre-inversion of the image, the object pattern is processed in the first... The replicas in each inversion trajectory are used for feature extraction and category prediction using a trained digital twin network, expressed as: ； in, It is the first The classification response of each trajectory; (2) Arrange the classification variance outputs of all inverted trajectories into an inverted multi-trajectory classification variance matrix, the expression of which is: ； in, , The number of categories to be predicted; In step 6, the variance matrix of the multi-trajectory classification is determined according to the principle of maximum variance deviation. Simultaneously predicting the object category and spatial coordinate location vector value, including: (1) Using classification variance as the final criterion for pattern recognition, the maximum value of the multi-trajectory response is adopted to simultaneously predict the object category and spatial coordinate location vector value. The expression is as follows: ； ； (2) When using the classic digital twin network for supervised classification, the category and vector angle are predicted based on the principle of maximum deviation of classification variance; (3) Perform steps (1)-(2) on the dataset respectively to obtain the estimation performance of category and spatial coordinate location vector values.

9. A collaborative patrol system combining unmanned aerial vehicles (UAVs) and intelligent vehicles, characterized in that: The system implements any one of the drone and intelligent vehicle collaborative patrol methods according to claims 1-8, and the system includes: Raspberry Pi (1) is mounted on a smart wireless charging car and a quadcopter. By using the Ros robot control system, the operation paths of the car and the drone are synchronized. Based on the communication principle, the overall scheduling of the work tasks is carried out in a multi-dimensional series and interaction. The main control module (2) interfaces with the multi-purpose extension parts in the upper and lower layers to enable timely processing and response of data in the cloud server queue; The central server (3) uses a task storage queue as its main data structure to temporarily store tasks submitted by task publishing clients; it establishes a task waiting queue to store tasks initially allocated by the central server; it establishes a server task execution queue to store tasks currently being executed by each robot terminal; in the construction of the central server, a state machine is established for each robot terminal to save the current working state of each robot terminal. If the robot execution task queue in a robot terminal is full, the robot terminal communicates with the central server, and the central server sets the state machine corresponding to the robot terminal to busy. If the robot execution task queue in a robot terminal is not full, the robot terminal communicates with the central server, and the central server sets the state machine corresponding to the robot terminal to idle; the central server establishes an information stack to store the information of each robot terminal. The information stack stores the current location information and function information of each robot terminal; a socket interface is established in the central server to realize data communication between task publishing clients and the central server, and between the central server and robot terminals using computer network communication technology. The central server continuously communicates with each robot terminal and updates the information of the state machine and information stack in real time. The D455 depth camera (4) is used to continuously take photos, calculate the difference to realize the distance, and use four consecutive images for 3D modeling and obstacle avoidance. Laser radar (5) is a radar system that uses laser beams to detect the position, speed and other characteristics of a target. The laser radar emits a detection signal to the target, and then compares the received signal reflected back from the target with the emitted signal. After processing, it obtains relevant information about the target, including the target distance, azimuth, altitude, speed, attitude and shape parameters. The digital twin identification module (6) uses digital twin technology to identify and warn of the spatial coordinates of relevant information images collected within the land and air working area, and sends them to the dispatch center for visualization. The relevant information includes dynamic target information and static target information to be identified in a certain area.

Images