Multi-device collaborative operation scheduling method, system and device and storage medium

Abstract

The invention relates to a multi-device cooperative operation scheduling method, system and device and a storage medium, and the method comprises the steps: obtaining an original point cloud and a local point cloud of a task area, the original point cloud being obtained through the scanning of an unmanned plane, and the local point cloud being obtained through the scanning of an execution device; obtaining an initial transformation matrix, converting the original point cloud and the local point cloud into a world coordinate system through the initial transformation matrix, and calculating an optimal transformation matrix; constructing a three-dimensional semantic map according to the original point cloud and the local point cloud which are converted into the world coordinate system, and determining position information of the execution equipment according to the optimal transformation matrix and the three-dimensional semantic map; according to the method, the task information and the equipment information of the execution equipment are obtained, the task information, the position information and the equipment information are evaluated through the preset dynamic scheduling model, the optimal execution equipment for executing the task is determined according to the evaluation result, the positioning precision is improved, and the leap-type enhancement of the scheduling performance is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-device path planning, in particular to a multi-device cooperative operation scheduling method and system, a device and a storage medium. BACKGROUND

[0002] With the rapid development of Internet of Things (IoT), robotics and distributed artificial intelligence, Multi-Agent Systems (MAS) have become a core technology in the fields of industrial automation, intelligent warehousing, disaster rescue, etc. Such systems are composed of multiple autonomous devices (such as drones, mobile robots, intelligent sensors, etc.) that work together to complete complex tasks.

[0003] However, the key technical bottleneck faced by traditional multi-device systems is the heterogeneity of environmental perception and the inconsistency of coordinate systems, which seriously restricts the efficiency and reliability of cooperation. For example, in traditional multi-device systems, each device (such as a drone or a robot dog) relies on its own SLAM to build a map independently, resulting in inconsistent map coordinate systems. When devices need to share environmental information, due to coordinate system deviations (usually > 30 cm), cooperative obstacle avoidance, path planning, and other functions fail; in addition, traditional polling or fixed priority scheduling does not consider the real-time state of devices, and high-load device tasks accumulate, and low-power devices frequently interrupt, resulting in low system task completion rate. Therefore, the existing multi-device system has the problems of collaboration failure caused by inconsistent cooperative positioning benchmarks and low system efficiency caused by static task allocation. SUMMARY

[0004] The present application provides a multi-device cooperative operation scheduling method, system, device and storage medium to at least solve the problems of collaboration failure caused by inconsistent cooperative positioning benchmarks and low system efficiency caused by static task allocation in the existing multi-device system in related technologies.

[0005] In a first aspect, the present application provides a multi-device cooperative operation scheduling method, which is applied in a multi-device cooperative operation system, the system comprising a drone, a cloud brain terminal and a plurality of execution devices, the drone and the execution devices being respectively installed with a perception device, the method comprising: obtaining a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is generated by scanning with the perception device on the drone, and the local point cloud map is generated by scanning with the perception device on the execution device; obtaining an initial transformation matrix, and converting the global point cloud map and the local point cloud map to a world coordinate system through the initial transformation matrix; constructing a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted to the world coordinate system, and determining position information of the execution device according to the three-dimensional semantic map; Obtain task information and device information of all execution devices, evaluate the task information, location information and device information through a preset dynamic scheduling model, and select an optimal execution device for executing a task according to an evaluation result.

[0006] Specifically, after the cloud brain terminal receives the global point cloud map, the method further comprises: Obtain a resolution requirement preset by a user, and assign a resolution requirement label to each three-dimensional voxel block in the global point cloud map according to the resolution requirement, wherein the resolution requirement label comprises a high-precision label and a low-precision label; Obtain a current network bandwidth load and a device accuracy of a sensing device on the execution device in real time, dynamically adjust the resolution requirement label of each three-dimensional voxel block according to the device accuracy and the current network bandwidth load, and obtain a target resolution of different three-dimensional voxel blocks; According to the target resolution, the global point cloud map is compressed by Octree, and an Octree compressed map is obtained and output.

[0007] Specifically, after determining the optimal execution device, the task information is sent to the optimal execution device, and an optimal global path is generated by the optimal execution device; wherein the generation of the optimal global path comprises: Determine a path starting point and a path ending point according to the location information and the task information, respectively; Voxelize the three-dimensional semantic map to generate a three-dimensional grid map; Perform path query on the three-dimensional grid map, the path starting point and the path ending point through a path finding and graph traversal algorithm, and output an optimal global path of the optimal execution device.

[0008] Specifically, during the generation of the optimal global path, the method further comprises: Detect the Euclidean distance between the optimal execution device and an obstacle in real time through a sensing device on the optimal execution device; If the difference between the Euclidean distance and the distance of the obstacle recorded in the three-dimensional grid map exceeds a preset threshold, trigger a dynamic window method to perform local path re-planning; After the optimal execution device avoids the obstacle, return to the global path and dynamically update the data corresponding to the obstacle in the three-dimensional grid map.

[0009] Specifically, the initial transformation matrix is obtained, and the global point cloud map and the local point cloud map are converted to a world coordinate system through the initial transformation matrix, comprising: Obtaining the longitude, latitude and height of the UAV through satellite positioning technology, and converting the longitude, latitude and height into world coordinates in a world coordinate system; Obtaining the pose data of the UAV through an inertial measurement device on the UAV, and calculating an initial transformation matrix according to the world coordinates and the pose data; Coarsely aligning the local point cloud map and the global point cloud map, and globally aligning the coarsely aligned local point cloud map and the global point cloud map and the initial transformation matrix through an ICP algorithm to generate an optimal transformation matrix; Converting the local point cloud map to a time coordinate system according to the optimal transformation matrix.

[0010] Specifically, the position information of the execution device is determined according to the three-dimensional semantic map, including: Obtaining real-time pose data of the execution device through a perception device on the execution device, and optimizing the real-time pose data through a Kalman filter and a multi-sensor fusion algorithm to obtain a current pose of the execution device; Updating the optimal transformation matrix according to the current pose to generate a latest transformation matrix and broadcast through a TF library; Determining the position of each execution device in the three-dimensional semantic map according to the latest transformation matrix to obtain the position information.

[0011] Specifically, the task information, position information and device information are evaluated through a preset dynamic scheduling model, and the optimal execution device for executing the task is selected according to the evaluation result, including: Obtaining the task position of the to-be-executed task from the task information, and screening according to the task position and the position information of all execution devices to obtain candidate execution devices within a preset range; Evaluating the device information of the candidate execution devices and the task to be executed through a scheduling evaluation model, and outputting a comprehensive score; Screening the candidate execution devices with a comprehensive score greater than a standard score value as the optimal execution device, and controlling the optimal execution device to execute the to-be-executed task.

[0012] Specifically, the construction of the scheduling evaluation model includes: Obtaining the capability matching degree, time cost, energy efficiency of the execution device, and the dynamic priority of the to-be-executed task; Respectively obtaining the weights of the capability matching degree, time cost, energy efficiency and dynamic priority through a dynamic learning mechanism; Constructing a scheduling evaluation model according to the capability matching degree, time cost, energy efficiency, dynamic priority and corresponding weights.

[0013] Specifically, if the comprehensive score of all candidate execution devices is not greater than the standard score, the method further comprises: decomposing the to-be-executed task into a plurality of to-be-executed sub-tasks, re-inputting the task information of the to-be-executed sub-tasks into the dynamic scheduling model, and screening out an optimal execution device combination.

[0014] In a second aspect, the application provides a multi-device collaborative operation scheduling system for implementing the multi-device collaborative operation scheduling method described in any one of the above aspects, the system comprising: a UAV provided with a first sensing device, configured to collect a global point cloud map of a task area through the first sensing device and send the global point cloud map to a cloud brain terminal; an execution device provided with a second sensing device, configured to collect a local point cloud map of the task area through the second sensing device and send the local point cloud map to the cloud brain terminal; a cloud brain terminal configured to obtain a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is obtained by scanning through a sensing device on the UAV, and the local point cloud map is obtained by scanning through a sensing device on the execution device; obtain an initial transformation matrix, and convert the global point cloud map and the local point cloud map to a world coordinate system through the initial transformation matrix; construct a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted to the world coordinate system, and determine position information of the execution device according to the three-dimensional semantic map; obtain task information and device information of all execution devices, evaluate the task information, the position information and the device information through a preset dynamic scheduling model, and select optimal execution devices for executing tasks according to an evaluation result; The execution device is further configured to perform global path rules and obstacle avoidance according to the received task information and the three-dimensional semantic map, and complete the received task.

[0015] Specifically, the cloud brain terminal comprises: a map construction module configured to receive a global point cloud map and perform dynamic resolution adjustment, and realize Octree compression; a coordinate system mapping module configured to map the point cloud map to a world coordinate system through an RTK+SLAM+ICP hybrid calibration algorithm; a scheduling decision module configured to run a dynamic scheduling model with a weight learning mechanism, and distribute tasks of the task information to each execution device; a ROS2 communication hub configured to realize data transmission between the cloud brain terminal and the execution device.

[0016] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-device cooperative operation scheduling method according to the first aspect when executing the computer program.

[0017] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having a computer program stored thereon, and the program is executable on a processor to implement the multi-device cooperative operation scheduling method according to the first aspect.

[0018] The multi-device cooperative operation scheduling method, system, device and storage medium provided by the embodiments of the present application have at least the following technical effects.

[0019] Through the world coordinate system dynamic mapping technology, the space reference of the multi-device is unified, the independent mapping error of the traditional SLAM is eliminated, and the positioning accuracy is improved significantly; based on the dynamic scheduling model (ability matching degree + path cost + energy efficiency + dynamic priority), the real-time optimal allocation is realized by combining the auction mechanism, and the scheduling performance is improved dramatically.

[0020] Details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more apparent. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings illustrated herein are used to provide further understanding of the present application, and constitute a part of the present application. The illustrative embodiments of the present application and their description serve to explain the present application, and do not constitute improper limitations on the present application. In the drawings: Figure 1 is a flowchart of the multi-device cooperative operation scheduling method in an embodiment of the present application; Figure 2 is a system architecture diagram of the multi-device cooperative operation system in an embodiment of the present application; Figure 3 is a flowchart of the scheduling algorithm in an embodiment of the present application; Figure 4 is a structural block diagram of the electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the present application more apparent, the present application is described and explained in the following with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not limit the present application. Based on the embodiments provided by the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of the present application.

[0023] It is apparent that the drawings in the following description merely show some examples or embodiments of the present application, and the present application can be applied to other similar situations without creative labor by those skilled in the art based on these drawings. In addition, it can be understood that although the efforts made in the development process can be complex and lengthy, some modifications, such as design, manufacture or production, etc. based on the technical content disclosed in the present application, are only routine technical means for those skilled in the art related to the content disclosed in the present application, and should not be understood as insufficient disclosure of the content disclosed in the present application.

[0024] Reference to "an embodiment" in this application means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment, nor are they necessarily mutually exclusive of one another. It is explicitly contemplated that embodiments described herein can be combined with other embodiments in a non- conflicting manner.

[0025] Unless otherwise defined, technical terms or scientific terms used in the present application shall have the ordinary meaning understood by one of ordinary skill in the art to which the present application pertains. The terms "a", "an", "one", "the", and similar terms in the present application do not denote a limitation on quantity but rather denote the presence of at least one of the referenced item. The terms "including", "comprising", "having" and their variants in the present application are intended to cover non-exclusive inclusion; for example, processes, methods, systems, products, or devices that include a series of steps or modules (units) are not limited to the listed steps or units, but can also include other steps or units not listed, or can also include other steps or units inherent to these processes, methods, products or devices. The terms "connected", "connected", "coupled" and similar terms in the present application are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. The term "multiple" in the present application means two or more. The term "and / or" describes the association relationship between the associated objects, which means that there can be three relationships, for example, "A and / or B" can mean that A exists alone, A and B exist together, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects. The terms "first", "second", "third" and the like in the present application are merely to distinguish similar objects, and do not represent a specific order for the objects.

[0026] ROS2: Robot Operating System 2 is an open-source middleware framework (non-traditional operating system) designed specifically for robots. Its core is to provide a communication infrastructure and development toolset to simplify the construction of complex robot systems.

[0027] Topic: Asynchronous communication channel, nodes send messages through publishers (Publishers), and subscribers (Subscribers) receive messages to achieve data flow (such as sensor data).

[0028] PCD graph: Full name Point Cloud Data, use this name uniformly, do not call "point cloud graph".

[0029] Cloud brain terminal: refers to the central processing server, not "cloud" or "control center".

[0030] Scheduler: specifically refers to the dynamic task allocation module, distinguished from the path planner.

[0031] A*: A Star algorithm is a commonly used path search and graph traversal algorithm. It has good performance and accuracy.

[0032] The current multi-device collaborative system has the following problems: 1. Insufficient positioning accuracy, traditional SLAM system relies on device's own sensor, positioning drift easily occurs in complex environment, leading to failure of device cooperation; 2. Map sharing lag, devices communicate through raw point cloud data, high bandwidth occupation (single frame PCD data up to hundreds of MB), poor real-time performance; 3. Low scheduling efficiency, task allocation adopts simple polling or fixed priority strategy, without considering real-time state of devices (such as power, position), task completion rate is about 30%.

[0033] Based on the above situation, the embodiments of the present application provide a multi-device collaborative operation scheduling method, system, device and storage medium.

[0034] In a first aspect, the embodiments of the present application provide a multi-device collaborative operation scheduling method, the scheduling method of the present embodiment is applied in a multi-device collaborative operation system, the system refers to Figure 2The system comprises a UAV (a perception layer), a cloud brain terminal, and a plurality of execution devices (an execution layer), and the UAV and the execution devices are respectively installed with a perception device. For example, the perception device of the embodiment is a UAV with a laser radar, which is used for environment scanning and raw data real-time collection, and performs three-dimensional laser scanning (scanning frequency 20 Hz) in a task area to generate a laser point cloud (global point cloud map, PCD format, accuracy ±2 cm), and the data is transmitted to the cloud brain terminal through a 5G / WiFi 6 transmission data of a ROS2 communication node. In the embodiment, the cloud brain terminal is a server cluster, and the cloud brain terminal comprises a point cloud processing engine, a coordinate mapping module, and a task scheduling core. The point cloud processing engine generates a three-dimensional semantic map from the global point cloud map and the global point cloud map processing. The coordinate mapping module generates a world coordinate system conversion matrix according to the global point cloud map and the global point cloud map. Then, the task scheduling core obtains task information, a three-dimensional semantic map, and a world coordinate system conversion matrix and comprehensively processes them to generate a device control instruction to control the execution device to execute a corresponding task. The execution device is a humanoid work device or a ground mobile execution device. After receiving the control instruction and the task instruction issued by the cloud brain terminal, the execution device can move on the ground to execute the task and perform path planning and real-time obstacle avoidance during the execution of the task, such as carrying out fine tasks such as sensor inspection, article transportation, switch operation, and device maintenance. In addition, the execution device is also installed with an IMU, an odometer, a depth camera (local positioning), a mechanical arm, a force feedback sensor, and other sensing devices. The environment data or point cloud data detected by these sensing devices can be fed back to the cloud brain terminal, so that the cloud brain terminal can update the three-dimensional semantic map and the conversion matrix in real time. In the preferred embodiment, the multi-device collaborative operation system of the embodiment also needs to rely on a UWB positioning base station and a GPS differential positioning module for auxiliary positioning.

[0035] As shown in Figure 1 The multi-device collaborative operation scheduling method of the embodiment is implemented by the following steps.

[0036] Step S1, obtaining a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is generated by scanning through a perception device on the UAV, and the local point cloud map is generated by scanning through a perception device on the execution device.

[0037] Since the global point cloud map data transmitted between devices has a large memory (single frame up to hundreds of MB level), causing network congestion, map update delay > 500ms, therefore, in a dynamic environment (such as a moving obstacle), the device cannot update the path in time, and the collision risk is increased by more than 40%, that is, the existing multi-device collaborative operation system has the problem of decision lag caused by poor real-time performance of map sharing. In order to solve the above technical problems, the ROS2 Topic is used to transmit the Octree compressed map in the embodiment, so that the data volume is reduced by 90% (reduced to 10MB / s), the delay is <100ms, and 100+ devices are supported for concurrent subscription.

[0038] Specifically, after the cloud brain terminal receives the global point cloud map, or before outputting the three-dimensional semantic point cloud map to the execution device, the point cloud map can also be compressed. Specifically, the resolution requirement preset by the user is obtained, the resolution requirement label is assigned to each three-dimensional voxel block in the global point cloud map according to the resolution requirement, the resolution requirement label includes a high-precision label and a low-precision label; the current network bandwidth load and the device accuracy of the perception device on the execution device are obtained in real time, the resolution of the three-dimensional voxel block marked with the resolution requirement label is dynamically adjusted according to the device accuracy and the current network bandwidth load, and the target resolution of different three-dimensional voxel blocks is obtained; and the global point cloud map is compressed according to the target resolution to obtain an Octree compressed map and output.

[0039] In more detail, the cloud brain terminal of the embodiment marks different regions or task types with resolution requirement labels, such as high requirement labels and low requirement labels (HIGH_DETAIL, LOW_DETAIL), according to the task target in the task decomposition stage, and at the same time considers the device sensor accuracy and the current network bandwidth load to improve the resolution of key regions (such as narrow passages and known obstacle dense areas) or interest regions (task specified regions). The cloud brain terminal maintains a dynamic resolution configuration table to record the target resolution of different regions (three-dimensional voxel blocks) of the map. When generating Octree by compressing the original PCD data, the global uniform resolution parameter is no longer used. For the regions marked as HIGH_DETAIL or the regions that need high precision according to the strategy, smaller voxel size (such as 0.05m) is used to construct the Octree subtree, and more point cloud details are retained; for the regions marked as LOW_DETAIL or the background / non-key regions, larger voxel size (such as 0.2m) is used to construct the Octree subtree, and the data volume is significantly reduced, and the resolution boundary region adopts a smooth transition strategy (such as using interpolation) to avoid map discontinuity.

[0040] The embodiment can further reduce the bandwidth occupation and storage requirement of the overall map data by Octree compression, while ensuring high-precision details in the key area, and realize on-demand allocation of computing and communication resources.

[0041] In a preferred embodiment, in addition to using the Octree compression method, the voxel grid of the point cloud map can also be down-sampled. Direct down-sampling can also achieve the effect of improving transmission speed and reducing storage requirements.

[0042] In step S2, an initial transformation matrix is obtained, and the global point cloud map and the local point cloud map are converted to a world coordinate system by the initial transformation matrix. Specifically, the longitude, latitude and height of the unmanned aerial vehicle are obtained by high-precision satellite positioning technology, and the longitude, latitude and height are converted into world coordinates in the world coordinate system; the pose data of the unmanned aerial vehicle is obtained by the inertial measurement device on the unmanned aerial vehicle, and the initial transformation matrix is calculated according to the world coordinates and the pose data; the local point cloud map and the global point cloud map are roughly aligned, and the local point cloud map and the global point cloud map after rough alignment and the initial transformation matrix are globally aligned by an ICP algorithm to generate an optimal transformation matrix; and the local point cloud map is converted to a time coordinate system according to the optimal transformation matrix.

[0043] In an embodiment, the formula for calculating the world coordinate system mapping is as follows:

[0044] wherein, P is a point in the world coordinate system, P is a point in the device local coordinate system, P is the optimal transformation matrix from the device coordinate system to the world coordinate system.

[0045] In the system provided in the present application, the transformation matrix is obtained by the following method . Specifically, when initialized, the unmanned aerial vehicle can obtain high-precision global positioning (longitude, latitude and height) by RTK, and combined with IMU pose data, an initial transformation matrix in the world coordinate system can be calculated. For the robotic dog and robot without RTK, positioning can be performed at startup by laser SLAM (such as LOAM, LIO-SAM, etc.) or visual SLAM (such as VINS), and then the global point cloud map scanned by the unmanned aerial vehicle is used to obtain the pose of the robotic dog / robot relative to the world coordinate system by point cloud matching (such as ICP algorithm) at the initial position.

[0046] When the robot dog / robot starts, its own SLAM system generates a local point cloud map. Then the local point cloud map is matched with the global point cloud map provided by the UAV, and the RTK initialization + SLAM local mapping + ICP real-time alignment algorithm is used. More specifically, the RTK module is used to obtain the longitude, latitude and height (WGS84 coordinates) of the UAV, which is converted into (x, y, z) in the world coordinate system (such as UTM), and the IMU attitude (quaternion q) is fused to directly construct an initial transformation matrix. Then the LIO-SAM algorithm is started to generate a local point cloud map and a real-time pose. Finally, the ICP algorithm is used for global alignment. The core of ICP is to minimize the distance between two point clouds, and KD-Tree is used to accelerate the nearest neighbor search. Through iterative solution, the optimal rotation matrix R and translation vector t, i.e. the optimal transformation matrix, are obtained.

[0047] During the operation of the system, real-time pose data of the execution device is obtained through the sensing device on the execution device, and the real-time pose data is optimized by using a Kalman filter and a multi-sensor fusion algorithm to obtain the current pose of the execution device; the optimal transformation matrix is updated according to the current pose to generate a latest transformation matrix and broadcast through a TF library; the positions of the execution devices in the three-dimensional semantic map are determined according to the latest transformation matrix to obtain the position information.

[0048] Specifically, during operation, each device updates its pose in real time through its own positioning system (IMU, odometry, laser / visual SLAM) and publishes a TF transformation. The cloud brain terminal maintains a global TF tree, which records the real-time transformation relationship of each device relative to the world coordinate system.

[0049] Therefore, the coordinate mapping of the embodiments of the present application has two stages in the operation of the system. In the initialization stage, the initial pose (transformation matrix) of the device is obtained, and the initial transformation matrix is obtained through point cloud matching; in the system operation stage, the real-time pose is updated and coordinated, and the robot dog or robot updates the transformation matrix (i.e. the pose of the device in the world coordinate system) in real time through the sensors and positioning algorithms of the device itself.

[0050] In a preferred embodiment, broadcasting and listening are performed through TF. Specifically, the transformation of each device relative to the world coordinate system is broadcast through the TF2 library of ROS2; each device can listen to the TF of other devices, so as to know the positions of other devices in the world coordinate system.

[0051] Step S3, constructing a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted into the world coordinate system, and determining the position information of the execution device according to the three-dimensional semantic map.

[0052] Specifically, the point cloud map is preprocessed, such as denoising and coarse downsampling. The denoising removes outliers by statistical filtering; the downsampling reduces the computational amount by filtering the voxel grid (such as 0.05m resolution). After preprocessing, semantic segmentation and semantic injection are performed. Specifically, a segmentation model is constructed by data labeling and training in advance, then online semantic labeling is performed, the trained model is deployed to an edge device (TensorRT acceleration), real-time segmentation of point cloud is performed, post-processing is performed, the label consistency is optimized by CRF (Conditional Random Field), and finally semantic fusion is performed to generate a three-dimensional semantic map by fusing the semantic and point cloud map.

[0053] In step S4, task information and device information of all execution devices are acquired, the task information, location information and device information are evaluated by a preset dynamic scheduling model, and an optimal execution device for executing a task is selected according to an evaluation result.

[0054] Specifically, a task location of a to-be-executed task is acquired from the task information, candidate execution devices within a preset range are acquired by screening according to the task location and location information of all execution devices, device information of the candidate execution devices and the to-be-executed task are evaluated by a scheduling evaluation model, and a comprehensive score is output; a candidate execution device with a comprehensive score greater than a standard score value is selected as an optimal execution device, and the optimal execution device is controlled to execute the to-be-executed task.

[0055] In the case where there is no candidate execution device with a comprehensive score greater than a preset threshold, that is, the comprehensive scores of all candidate execution devices are not greater than a standard score value, the method further includes: decomposing the to-be-executed task into a plurality of to-be-executed sub-tasks, re-inputting task information of the to-be-executed sub-tasks into the dynamic scheduling model, and screening an optimal execution device combination.

[0056] The construction of the scheduling evaluation model includes: acquiring a capability matching degree, a time cost, an energy consumption efficiency of the execution device, and a dynamic priority of a to-be-executed task; acquiring weights of the capability matching degree, the time cost, the energy consumption efficiency and the dynamic priority respectively by a dynamic learning mechanism; and constructing a scheduling evaluation model according to the capability matching degree, the time cost, the energy consumption efficiency, the dynamic priority and the corresponding weights.

[0057] The scheduling evaluation model of the embodiment of the application is a four-dimensional scheduling algorithm, and a formula of the scheduling evaluation model is:

[0058] wherein, is a weight of each dimension; capability matching degree , represents a device capability value, representing task demand value, representing ability weight; time cost , representing path planning length, representing average speed of equipment, representing terrain and obstacle coefficient; energy consumption efficiency , representing current power of equipment, representing energy consumption required by task and execution; dynamic priority , representing task priority, representing average priority of system.

[0059] For the dynamic weight learning mechanism, the weight is initialized first, ; then updated online , wherein the formula of the loss function is:

[0060] The calculation formula of gradient calculation is:

[0061] learning rate , task clustering is performed, and K-means is used for task feature clustering: clusters = kmeans(features = [moving distance, accuracy requirement, urgency level], k = 3), and independent weights are maintained for each type of task.

[0062] For more details, refer to Figure 3 , in the task scheduling process, the to-be-executed task is extracted from the task pool in a preset order (such as execution time sequence), it is judged whether the to-be-executed task is an urgent task, if yes, the to-be-executed task is put into a high-priority queue, and a suitable execution device is immediately selected for allocation. If it is not an urgent task, the comprehensive scores of all candidate devices are calculated, the device with the largest (max comprehensive score) score is selected according to the four-dimensional scheduling algorithm, and the task is allocated to the execution device. The execution device allocates the task to the execution device through the A* algorithm for global path planning. The four-dimensional scheduling algorithm of the embodiment dynamically allocates the task to the optimal device / device group, comprehensively considers four dimensions of device capability, path cost, power efficiency and task priority, realizes optimization of multi-device collaborative task scheduling, input parameters are device list (containing device information), environment map and task information, and output results are task allocation results (single device or device group).

[0063] The four-dimensional scheduling algorithm is realized through the following steps: First, set the algorithm base parameters and global variables, get the single device execution threshold, which is the threshold value of single device execution, get the system's available device set and navigation map service containing environmental information. Then input the task object to be allocated (such as containing location, capability requirement, energy consumption, time cost, task priority, etc.), and then call the spatial_filter() function to filter the devices near the task location (such as within 50 meters), output the candidate device list, if there is no suitable device, trigger the extended search (such as relaxing the distance limit). Then traverse each candidate device, respectively calculate the capability matching degree, time cost, energy efficiency and dynamic priority score of each candidate device, then calculate the comprehensive score Score according to the capability matching degree, time cost, energy efficiency and dynamic priority, and the corresponding weight, make a decision according to the comprehensive score, if the highest comprehensive score in all candidate devices is greater than the single device execution threshold, select the candidate device as the optimal single device execution. After getting the optimal execution device, call the A* algorithm for path planning, specifically, convert the semantic map to a 0.1m resolution navigation grid map, use the A* algorithm to calculate the optimal path from the starting point to the end point, then return the path point list or null (no path).

[0064] If the highest comprehensive score is not greater than the single device execution threshold, call the collaboration group allocation function. Specifically, first create a collaboration group object, decompose the task into atomic subtasks (such as "move → grab → transport"), traverse each subtask, select the most suitable device for the subtask (considering the matching degree of device capability and subtask), device reuse check to ensure that the device is not assigned to other subtasks of the same task, add to the collaboration group and assign the subtask. Finally, calculate the overall efficiency of the collaboration group (such as estimated completion time / single device optimal time), if the efficiency > 0.6, return the collaboration group, otherwise throw an exception of no suitable device.

[0065] In an embodiment, in order to solve the problem of lack of autonomous cooperation ability in complex scenes, first, the cloud brain terminal intelligently decomposes the composite task into atomic subtasks, uses a dynamic scheduling algorithm (multi-constraint cost function + auction mechanism), and comprehensively considers the characteristics of the subtasks (location, required ability, priority, and dependency relationship) and the real-time state of the devices to optimally allocate the subtasks to the most suitable device or device group. A global PCD map is used to preliminarily plan a globally optimal path (using the A* algorithm) from the current location to the task starting point or target point for each device that needs to move. The allocated subtasks, target point coordinates, and pre-planned paths are encapsulated into specific instructions, which are issued to the corresponding devices through a ROS2 Topic (such as / task_assignment). The devices receive and execute the task instructions issued by the cloud brain. During the execution process, the devices continuously perceive the surrounding local environment (such as real-time point clouds and detected dynamic obstacle positions / speeds) using their own sensors (laser radars, depth cameras, IMUs, etc.), perform real-time positioning (SLAM) and local obstacle detection, and report the above data and abnormal events to the cloud brain terminal in real time through a ROS2 Topic. Finally, the autonomous cooperation closed loop of "cloud decision-making-device execution-real-time feedback" is realized.

[0066] After determining the optimal execution device, the task information is sent to the optimal execution device, and an optimal global path is generated by the optimal execution device. Specifically, the path starting point and the path ending point are determined according to the location information and the task information, respectively; the three-dimensional semantic map is voxelized to generate a three-dimensional grid map; the three-dimensional grid map, the path starting point, and the path ending point are queried by a path finding and graph traversal algorithm to output the optimal global path of the optimal execution device.

[0067] During the generation of the optimal global path, the perception device on the optimal execution device detects the Euclidean distance between the optimal execution device and the obstacle in real time; if the difference between the Euclidean distance and the distance of the obstacle recorded in the three-dimensional grid map exceeds a preset threshold, the dynamic window method is triggered for local path re-planning; after the optimal execution device avoids the obstacle, it returns to the global path and dynamically updates the data corresponding to the obstacle in the three-dimensional grid map.

[0068] More specifically, in the process of path planning and obstacle avoidance, global path planning (A* algorithm) is used, and the real-time obstacle avoidance triggering condition is shown in the following formula:

[0069] When the obstacle avoidance trigger condition is met, the DWA (Dynamic Window Approach) local path re-planning is triggered. DWA (Dynamic Window Approach) is an algorithm for real-time local obstacle avoidance and motion planning of robots, which is designed for dynamic environment. Its core idea is to dynamically generate feasible trajectories in the velocity space and select the optimal path to avoid obstacles.

[0070] The implementation process of global path planning and real-time obstacle avoidance of the embodiment is described as follows: An A-star algorithm is used to calculate the optimal path in a three-dimensional grid map, support dynamic obstacle avoidance and real-time re-planning, input three-dimensional grid map, device starting point and task end point, and then output path point list or empty table. A-star algorithm is a classic heuristic path search algorithm, which is used to find the optimal path between two points in a graph or grid. It combines the completeness of Dijkstra algorithm and the efficiency of greedy best-first search, and is hailed as the cornerstone of "intelligent search".

[0071] A* algorithm (A-star algorithm) covers all possible moving directions in three-dimensional space through 26-neighborhood search; differentiates the physical rationality of straight line / diagonal movement through differentiated movement cost; and detects obstacles in real time and triggers re-planning through dynamic obstacle avoidance mechanism. When the obstacles in the local point cloud graph change (new obstacles are added or removed), the A-star algorithm is called again, and a new path is obtained immediately by inputting the updated map. Since the algorithm is completely based on the current grid state, it naturally supports real-time re-planning.

[0072] The multi-device cooperative operation scheduling method provided by the application has the following advantages: In order to solve the problem of cooperation failure caused by non-uniform reference of multi-device cooperative positioning, the world coordinate system is mapped to unify the space reference, so that all devices share the global coordinates of the same PCD map, and the positioning error is controlled to be less than 5 cm; In order to solve the problem of decision lag caused by poor real-time performance of map sharing, ROS2 Topic is used to transmit Octree compressed map, the data volume is reduced by 90% (to 10 MB / s), the delay is less than 100 ms, and 100+ devices can be subscribed concurrently; In order to solve the problem of low system efficiency caused by static task allocation, the dynamic scheduling algorithm (multi-constraint cost function + auction mechanism) is used to allocate tasks in real time by comprehensively considering the distance, power and ability matching degree of the device, the task completion rate is improved to 95%, and the response speed of high priority tasks is improved by 40%.

[0073] In summary, the multi-device cooperative operation scheduling method provided by the application has the effects of improving positioning accuracy, optimizing communication efficiency and improving scheduling performance.

[0074] In a second aspect, the embodiments of the present application provide a multi-device cooperative operation system for implementing the multi-device cooperative operation scheduling method of any of the above embodiments, and the system comprises: a UAV provided with a first sensing device, configured to collect a global point cloud map of a task area through the first sensing device and send the global point cloud map to a cloud brain terminal; an execution device provided with a second sensing device, configured to collect a local point cloud map of the task area through the second sensing device and send the local point cloud map to the cloud brain terminal; the cloud brain terminal, configured to acquire the global point cloud map and the local point cloud map of the task area, wherein the global point cloud map is acquired through scanning by the sensing device on the UAV, and the local point cloud map is acquired through scanning by the sensing device on the execution device; acquire an initial transformation matrix, and convert the global point cloud map and the local point cloud map to a world coordinate system through the initial transformation matrix; construct a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted to the world coordinate system, and determine position information of the execution device according to the three-dimensional semantic map; acquire task information and device information of all execution devices, evaluate the task information, the position information and the device information through a preset dynamic scheduling model, and select an optimal execution device for executing a task according to an evaluation result; the execution device is further configured to perform global path rules and obstacle avoidance according to the received task information and the three-dimensional semantic map, and complete the received task.

[0075] Specifically, the cloud brain terminal comprises: a map construction module, configured to receive the global point cloud map and perform dynamic resolution adjustment for execution, and realize Octree compression; a coordinate system mapping module, configured to map the point cloud map to the world coordinate system through an RTK+SLAM+ICP hybrid calibration algorithm; a scheduling decision module, configured to run a dynamic scheduling model with a weight learning mechanism, and distribute tasks of the task information to each execution device; a ROS2 communication hub, configured to realize data transmission between the cloud brain terminal and the execution device, such as publishing / subscription of topics such as management / map and / task.

[0076] It should be noted that the multi-device cooperative operation system provided in the embodiments is used to implement the above method embodiments, and details are not repeated. As used above, the terms "module", "unit", "sub-unit", and the like can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the above embodiments are preferably implemented in software, hardware, or a combination of software and hardware can also be implemented and conceived.

[0077] In a third aspect, an electronic device is provided, Figure 4 is a block diagram of an electronic device according to an exemplary embodiment. As shown, the electronic device can include a processor 11 and a memory 12 having stored computer program instructions. Figure 4

[0078] In particular, the processor 11 described above can include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement one or more embodiments of the present application.

[0079] ​The memory 12 can include a mass storage for data or instructions. By way of example and not limitation, the memory 12 can include a Hard Disk Drive (HDD), floppy disk drive, a Solid State Drive (SSD), flash memory, a Compact Disc Read Only Memory (CDROM), a Digital Versatile Disk (DVD), a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 12 can be removable and / or non-removable (or fixed) as appropriate. The memory 12 can be internal or external as appropriate. In particular embodiments, the memory 12 is a Non-Volatile memory. In particular embodiments, the memory 12 includes a Read-Only Memory (ROM) and a Random-Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended Data Out Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0080] The memory 12 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 11.

[0081] The processor 11 realizes the multi-device cooperative operation scheduling method in any of the above embodiments by reading and executing the computer program instructions stored in the memory 12.

[0082] In an embodiment, the electronic device can further include a communication interface 13 and a bus 10. As shown, the processor 11, the memory 12, and the communication interface 13 are connected through the bus 10 and complete communication with each other. Figure 4

[0083] The communication interface 13 is used to realize the communication between the modules, devices, units and / or devices in the embodiments of the present application. The communication interface 13 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, image / data processing workstations, etc.

[0084] ​Bus 10 includes hardware, software, or both, to couple components of electronic device to each other. Bus 10 includes, but is not limited to, at least one of a data bus, an address bus, a control bus, an expansion bus, a local bus, and the like. By way of example and not limitation, bus 10 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or the like, or a combination of two or more of these. Where appropriate, bus 10 can include one or more buses. Although the present application is described and illustrated with a particular bus, it is contemplated that any appropriate bus or interconnect can be used.

[0085] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having stored thereon a program, wherein the program is executed by a processor to implement the method for scheduling cooperative operation of multiple devices according to the first aspect.

[0086] More specifically, the computer readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0087] In possible implementation manners, the present application can also be implemented in the form of a program product, which comprises program codes for causing a terminal device to perform the steps of the method for implementing the multi-device cooperative operation scheduling provided by the first aspect when the program product is run on the terminal device.

[0088] The program codes for executing the present application can be written in any combination of one or more programming languages, and can be executed completely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or completely on a remote device.

[0089] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but it should be considered that any combination of the technical features is within the scope of the present disclosure as long as the combination does not result in contradictions.

[0090] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these are all within the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-device collaborative operation scheduling method, characterized in that: Applied in a multi-device collaborative operation system, the system includes a drone, a cloud brain terminal, and multiple execution devices, the drone and the execution devices are respectively equipped with a sensing device, and the method includes: Acquire a global point cloud map and a local point cloud map of the task area, wherein the global point cloud map is generated by scanning by a sensing device on the UAV, and the local point cloud map is generated by scanning by a sensing device on the execution device; Obtaining an initial transformation matrix, and transforming the global point cloud image and the local point cloud image into a world coordinate system using the initial transformation matrix; Constructing a three-dimensional semantic map based on the global point cloud map and the local point cloud map converted into a world coordinate system, and determining the position information of the execution device based on the three-dimensional semantic map; Obtain task information and device information of all execution devices, evaluate the task information, location information and device information through a preset dynamic scheduling model, and select the optimal execution device for executing the task based on the evaluation results.

2. The multi-device collaborative operation scheduling method according to claim 1, characterized in that: After the cloud brain terminal receives the global point cloud image, the method further includes: Obtaining a user-preset resolution requirement, and assigning a resolution requirement label to each three-dimensional voxel block in the global point cloud image according to the resolution requirement, wherein the resolution requirement label includes a high-precision label and a low-precision label; obtaining in real time the current network bandwidth load and the device accuracy of the sensing device on the execution device, and dynamically adjusting the resolution of the three-dimensional voxel blocks marked with the resolution requirement tag according to the device accuracy and the current network bandwidth load to obtain target resolutions for different three-dimensional voxel blocks; The global point cloud image is subjected to Octree compression according to the target resolution to obtain an Octree compressed map and output it.

3. The multi-device collaborative operation scheduling method according to claim 1, characterized in that: After determining the optimal execution device, the task information is sent to the optimal execution device, and an optimal global path is generated by the optimal execution device; wherein generating the optimal global path includes: Determine a path starting point and a path ending point according to the location information and the task information respectively; Performing voxel processing on the three-dimensional semantic map to generate a three-dimensional grid map; A path query is performed on the three-dimensional grid map, the path starting point and the path ending point through a path search and graph traversal algorithm, and an optimal global path of the optimal execution device is output.

4. The multi-device collaborative operation scheduling method according to claim 3, characterized in that: In the process of generating the optimal global path, the method further includes: detecting the Euclidean distance between the optimal execution device and an obstacle in real time by a sensing device on the optimal execution device; If the difference between the Euclidean distance and the obstacle distance recorded in the three-dimensional grid map exceeds a preset threshold, the dynamic window method is triggered to perform local path replanning; After the optimal execution device avoids the obstacle, it returns to the global path and dynamically updates the data corresponding to the obstacle in the three-dimensional grid map.

5. The multi-device collaborative operation scheduling method according to claim 1, characterized in that: The obtaining of an initial transformation matrix, and converting the global point cloud image and the local point cloud image into a world coordinate system using the initial transformation matrix, includes: Obtaining the latitude and longitude of the UAV through satellite positioning technology, and converting the latitude and longitude into world coordinates in a world coordinate system; Acquire the posture data of the UAV through an inertial measurement device on the UAV, and calculate an initial transformation matrix according to the world coordinates and the posture data; Coarsely aligning the local point cloud image with the global point cloud image, and performing global alignment processing on the roughly aligned local point cloud image, the global point cloud image, and the initial transformation matrix using an ICP algorithm to generate an optimal transformation matrix; The local point cloud image is converted into a time coordinate system according to the optimal transformation matrix.

6. The multi-device collaborative operation scheduling method according to claim 5, characterized in that: The determining the location information of the execution device according to the three-dimensional semantic map includes: Acquiring real-time posture data of the execution device through a sensing device on the execution device, and optimizing the real-time posture data using a Kalman filter and a multi-sensor fusion algorithm to obtain a current posture of the execution device; Update the optimal transformation matrix according to the current posture, generate the latest transformation matrix and broadcast it through the TF library; The position of each execution device in the three-dimensional semantic map is determined according to the latest transformation matrix to obtain the position information.

7. The multi-device collaborative operation scheduling method according to claim 1, characterized in that: The task information, location information, and device information are evaluated using a preset dynamic scheduling model, and the optimal execution device for executing the task is selected based on the evaluation results, including: Obtaining a task location of the task to be executed from the task information, and filtering according to the task location and location information of all execution devices to obtain candidate execution devices within a preset range; Evaluate the device information of the candidate execution device and the tasks to be executed through the scheduling evaluation model, and output a comprehensive score; A candidate execution device having a comprehensive score greater than a standard score is selected as an optimal execution device, and the optimal execution device is controlled to execute the task to be executed.

8. The multi-device collaborative operation scheduling method according to claim 7, characterized in that: The construction of the scheduling evaluation model includes: Obtaining the capability matching degree, time cost, energy efficiency of the execution device, and the dynamic priority of the task to be executed; Obtaining weights of the capability matching degree, time cost, energy efficiency and dynamic priority respectively through a dynamic learning mechanism; A scheduling evaluation model is constructed based on the capability matching degree, time cost, energy efficiency, dynamic priority and corresponding weights.

9. The multi-device collaborative operation scheduling method according to claim 7, characterized in that: If the combined scores of all candidate execution devices are not greater than the standard score, the method further includes: The task to be executed is decomposed into a plurality of subtasks to be executed, and the task information of the subtasks to be executed is re-inputted into the dynamic scheduling model to screen out the optimal execution device combination.

10. A multi-device collaborative operation scheduling system, characterized in that: A system for implementing the multi-device collaborative operation scheduling method according to any one of claims 1 to 9, comprising: A drone equipped with a first sensing device is used to collect a global point cloud map of the mission area through the first sensing device and send it to the cloud brain terminal; An execution device equipped with a second sensing device, configured to collect a local point cloud image of the task area through the second sensing device and send the image to the cloud brain terminal; The cloud brain terminal is used to obtain a global point cloud map and a local point cloud map of the task area; obtain an initial transformation matrix, and transform the global point cloud map and the local point cloud map into a world coordinate system through the initial transformation matrix; construct a three-dimensional semantic map based on the global point cloud map and the local point cloud map transformed into the world coordinate system, and determine the position information of the execution device based on the three-dimensional semantic map; obtain task information and device information of all execution devices, evaluate the task information, location information and device information through a preset dynamic scheduling model, and select the optimal execution device for executing the task based on the evaluation results; The execution device is further configured to implement global path rules and obstacle avoidance according to the received task information and the three-dimensional semantic map to complete the received task.

11. The multi-device collaborative operation scheduling system according to claim 10, characterized in that: The cloud brain terminal includes: A map construction module that receives a global point cloud map and performs dynamic resolution adjustment to achieve Octree compression. The coordinate system mapping module is used to map the point cloud image to the world coordinate system through the RTK+SLAM+ICP hybrid calibration algorithm; The scheduling decision module is used to run a dynamic scheduling model with a weighted learning mechanism to allocate tasks to various execution devices; ROS2 communication hub, used to realize data transmission between the cloud brain terminal and the execution device.

12. An electronic device, characterized in that: The system comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the multi-device collaborative operation scheduling method according to any one of claims 1 to 9 is implemented.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the multi-device collaborative operation scheduling method according to any one of claims 1 to 9 is implemented.

Images