Multi-device cooperative operation scheduling method, system, device and storage medium

Abstract

The application relates to a multi-device cooperative operation scheduling method, system, device and storage medium, comprising the following steps: acquiring an original point cloud and a local point cloud of a task area, wherein the original point cloud is scanned and acquired by a drone, and the local point cloud is scanned and acquired by an execution device; acquiring an initial transformation matrix, converting the original point cloud and the local point cloud to a world coordinate system through the initial transformation matrix, and calculating an optimal transformation matrix; constructing a three-dimensional semantic map according to the original point cloud and the local point cloud converted to the world coordinate system, and determining position information of the execution device according to the optimal transformation matrix and the three-dimensional semantic map; acquiring task information and device information of the execution device, evaluating the task information, the position information and the device information through a preset dynamic scheduling model, and determining an optimal execution device for executing a task according to an evaluation result, so that the positioning accuracy is improved, and the scheduling performance is greatly enhanced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of multi-device path planning, in particular to a multi-device cooperative operation scheduling method and system, a device and a storage medium. BACKGROUND

[0002] With the rapid development of Internet of Things (IoT), robotics and distributed artificial intelligence, Multi-Agent Systems (MAS) have become a core technology in the fields of industrial automation, intelligent warehousing, disaster rescue, etc. Such systems are composed of multiple autonomous devices (such as drones, mobile robots, intelligent sensors, etc.) that work together to complete complex tasks.

[0003] However, the key technical bottleneck faced by traditional multi-device systems is the heterogeneity of environmental perception and the inconsistency of coordinate systems, which seriously restricts the efficiency and reliability of cooperation. For example, in traditional multi-device systems, each device (such as drones, robotic dogs) relies on its own SLAM to build a map independently, resulting in inconsistent map coordinate systems. When devices need to share environmental information, due to coordinate system deviations (usually > 30 cm), cooperative obstacle avoidance, path planning, and other functions fail; in addition, traditional polling or fixed priority scheduling does not consider the real-time state of devices, and high-load device tasks accumulate, and low-power devices frequently interrupt, resulting in low system task completion rate. Therefore, the existing multi-device system has the problems of collaboration failure caused by inconsistent cooperative positioning benchmarks and low system efficiency caused by static task allocation. SUMMARY

[0004] The present application provides a multi-device cooperative operation scheduling method, system, device and storage medium to at least solve the problems of collaboration failure caused by inconsistent cooperative positioning benchmarks and low system efficiency caused by static task allocation in the existing multi-device system in related technologies.

[0005] In a first aspect, the present application provides a multi-device cooperative operation scheduling method, applied in a multi-device cooperative operation system, the system comprising a drone, a cloud brain terminal and a plurality of execution devices, the drone and the execution devices being respectively installed with a perception device, the method comprising:

[0006] obtaining a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is generated by scanning with the perception device on the drone, and the local point cloud map is generated by scanning with the perception device on the execution device;

[0007] obtaining an initial transformation matrix, and converting the global point cloud map and the local point cloud map to a world coordinate system through the initial transformation matrix;

[0008] According to the global point cloud map and the local point cloud map converted into a world coordinate system, a three-dimensional semantic map is constructed, and position information of the execution device is determined according to the three-dimensional semantic map;

[0009] Task information and device information of all execution devices are acquired, the task information, the position information and the device information are evaluated through a preset dynamic scheduling model, and an optimal execution device for executing a task is selected according to an evaluation result.

[0010] Specifically, after the cloud brain terminal receives the global point cloud map, the method further comprises:

[0011] Resolution requirements of a user are acquired, resolution requirement labels of each three-dimensional voxel block in the global point cloud map are allocated according to the resolution requirements, and the resolution requirement labels include high-precision labels and low-precision labels;

[0012] Current network bandwidth load and device accuracy of a sensing device on the execution device are acquired in real time, resolution requirement labels of each three-dimensional voxel block are dynamically adjusted according to the device accuracy and the current network bandwidth load, and target resolutions of different three-dimensional voxel blocks are obtained;

[0013] The global point cloud map is compressed according to the target resolutions, an Octree compressed map is obtained, and the Octree compressed map is output.

[0014] Specifically, after the optimal execution device is determined, the task information is sent to the optimal execution device, and an optimal global path is generated by the optimal execution device; wherein the optimal global path is generated by:

[0015] A path starting point and a path ending point are determined according to the position information and the task information, respectively;

[0016] The three-dimensional semantic map is voxelized to generate a three-dimensional grid map;

[0017] The three-dimensional grid map, the path starting point and the path ending point are queried by a path searching algorithm and a graph traversal algorithm, and an optimal global path of the optimal execution device is output.

[0018] Specifically, in the process of generating the optimal global path, the method further comprises:

[0019] The Euclidean distance between the optimal execution device and an obstacle is detected in real time by a sensing device on the optimal execution device;

[0020] If a difference between the Euclidean distance and a distance of the obstacle recorded in the three-dimensional grid map exceeds a preset threshold, a dynamic window method is triggered to perform local path re-planning;

[0021] After the optimal execution device avoids the obstacle, the optimal execution device returns to the global path and dynamically updates data corresponding to the obstacle in the three-dimensional grid map.

[0022] Specifically, the initial transformation matrix is obtained, and the global point cloud map and the local point cloud map are converted to a world coordinate system by the initial transformation matrix, including:

[0023] The longitude, latitude and height of the UAV are obtained by satellite positioning technology, and the longitude, latitude and height are converted into world coordinates in the world coordinate system;

[0024] The pose data of the UAV is obtained by the inertial measurement device on the UAV, and the initial transformation matrix is calculated according to the world coordinates and the pose data;

[0025] The local point cloud map and the global point cloud map are roughly aligned, and the local point cloud map and the global point cloud map after rough alignment and the initial transformation matrix are globally aligned by an ICP algorithm to generate an optimal transformation matrix;

[0026] The local point cloud map is converted to a time coordinate system according to the optimal transformation matrix.

[0027] Specifically, the position information of the execution device is determined according to the three-dimensional semantic map, including:

[0028] The real-time pose data of the execution device is obtained by the perception device on the execution device, and the real-time pose data is optimized by using a Kalman filter and a multi-sensor fusion algorithm to obtain the current pose of the execution device;

[0029] The optimal transformation matrix is updated according to the current pose to generate a latest transformation matrix and broadcast by a TF library;

[0030] The position of each execution device in the three-dimensional semantic map is determined according to the latest transformation matrix to obtain the position information.

[0031] Specifically, the task information, position information and device information are evaluated by a preset dynamic scheduling model, and the optimal execution device for executing the task is selected according to the evaluation result, including:

[0032] The task position of the task to be executed is obtained from the task information, and the candidate execution devices within a preset range are obtained according to the task position and the position information of all execution devices;

[0033] The device information of the candidate execution devices and the task to be executed are evaluated by a scheduling evaluation model, and a comprehensive score is output;

[0034] screening a candidate execution device with a comprehensive score greater than a standard score value as an optimal execution device, and controlling the optimal execution device to execute the to-be-executed task.

[0035] Specifically, the construction of the scheduling evaluation model comprises:

[0036] obtaining the capability matching degree, time cost, energy consumption efficiency of the execution device, and the dynamic priority of the to-be-executed task;

[0037] obtaining the weights of the capability matching degree, time cost, energy consumption efficiency and dynamic priority through a dynamic learning mechanism respectively;

[0038] constructing a scheduling evaluation model according to the capability matching degree, time cost, energy consumption efficiency, dynamic priority and corresponding weights.

[0039] Specifically, if the comprehensive score of all candidate execution devices is not greater than the standard score value, the method further comprises:

[0040] decomposing the to-be-executed task into a plurality of to-be-executed sub-tasks, re-inputting the task information of the to-be-executed sub-tasks into the dynamic scheduling model, and screening an optimal execution device combination.

[0041] In a second aspect, the present application provides a multi-device cooperative operation scheduling system for implementing the multi-device cooperative operation scheduling method of any one of the above aspects, the system comprising:

[0042] a UAV provided with a first sensing device, configured to collect a global point cloud map of a task area through the first sensing device and send the global point cloud map to a cloud brain terminal;

[0043] an execution device provided with a second sensing device, configured to collect a local point cloud map of the task area through the second sensing device and send the local point cloud map to the cloud brain terminal;

[0044] a cloud brain terminal, configured to obtain a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is obtained by scanning through a sensing device on the UAV, and the local point cloud map is obtained by scanning through a sensing device on the execution device; obtain an initial transformation matrix, and convert the global point cloud map and the local point cloud map to a world coordinate system through the initial transformation matrix; construct a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted to the world coordinate system, and determine the position information of the execution device according to the three-dimensional semantic map; obtain task information and device information of all execution devices, evaluate the task information, position information and device information through a preset dynamic scheduling model, and select an optimal execution device for executing a task according to the evaluation result;

[0045] The execution device is further configured to perform global path rules and obstacle avoidance according to the received task information and the three-dimensional semantic map, and complete the received task.

[0046] Specifically, the cloud brain terminal comprises:

[0047] a map construction module configured to receive a global point cloud map and perform dynamic resolution adjustment to realize Octree compression;

[0048] a coordinate system mapping module configured to map the point cloud map to a world coordinate system by using an RTK+SLAM+ICP hybrid calibration algorithm;

[0049] a scheduling decision module configured to run a dynamic scheduling model with a weight learning mechanism to assign tasks of the task information to the execution devices;

[0050] a ROS2 communication hub configured to realize data transmission between the cloud brain terminal and the execution devices.

[0051] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the multi-device cooperative operation scheduling method according to the first aspect.

[0052] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program executable by a processor to implement the multi-device cooperative operation scheduling method according to the first aspect.

[0053] The multi-device cooperative operation scheduling method, system, device and storage medium provided by the embodiments of the present application have at least the following technical effects.

[0054] Through the world coordinate system dynamic mapping technology, the spatial reference of the multiple devices is unified, the independent mapping error of the traditional SLAM is eliminated, and the positioning accuracy is improved significantly; based on the dynamic scheduling model (capability matching degree+path cost+energy efficiency+dynamic priority), the real-time optimal allocation is realized by combining the auction mechanism, and the scheduling performance is improved dramatically.

[0055] The details of one or more embodiments of the present application are presented in the following drawings and description to make other features, objects and advantages of the present application more clear and easy to understand. BRIEF DESCRIPTION OF DRAWINGS

[0056] The drawings described herein are used to provide further understanding of the present application, and form a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute an improper limitation on the present application. In the drawings:

[0057] Figure 1 is a flow chart of a multi-device cooperative operation scheduling method in an embodiment of the present application;

[0058] Figure 2 is a system architecture diagram of a multi-device cooperative operation system in an embodiment of the present application;

[0059] Figure 3 is a flow chart of a scheduling algorithm in an embodiment of the present application;

[0060] Figure 4 is a structural block diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0061] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application is described and explained below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts fall within the scope of the present application.

[0062] It is obvious that the accompanying drawings in the following description are only some examples or embodiments of the present application, and those of ordinary skill in the art can apply the present application to other similar scenarios without creative efforts based on the drawings. In addition, it can be understood that although the efforts made in the development process can be complex and lengthy, some designs, manufacturing or production changes based on the technical content disclosed in the present application are only routine technical means for those of ordinary skill in the art related to the content disclosed in the present application, and should not be understood as insufficient disclosure of the present application.

[0063] In the present application, "embodiment" means that the specific features, structures or characteristics described in conjunction with the embodiment can be included in at least one embodiment of the present application. The phrase appears at various places in the specification does not necessarily refer to the same embodiment, nor is it independent or alternative to other embodiments. Those of ordinary skill in the art explicitly and implicitly understand that the embodiments described in the present application can be combined with other embodiments without conflict.

[0064] Unless otherwise defined, technical terms and scientific terms used in the present application shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terms "a", "an", "one", "this", and similar referents in the context of describing the application are to be construed to be open-ended, referring to one or more than one, unless otherwise noted. The terms "including", "comprising", "having" and variations thereof in this application are meant to encompass the items listed thereafter, and any equivalent thereof, as well as additional items, not excluding other items. The terms "connected", "coupled", and similar referents in the context of this application are to be construed as be having a logical or physical relationship depending on the context in which it is used. The term "multiple" means two or more. The term "and / or" describes association between or among multiple items and means that the items can be taken individually or in any combination. The term " / " is generally used to indicate "or". The terms "first", "second", "third", and the like are used to distinguish between similar objects, not to denote a particular order.

[0065] ROS2: Robot Operating System 2 is an open-source middleware framework (non-traditional operating system) designed for robots. Its core is to provide a communication infrastructure and a set of development tools to simplify the construction of complex robot systems.

[0066] Topic: Asynchronous communication channel, nodes send messages through publishers, and subscribers receive messages, realizing data flow (such as sensor data).

[0067] PCD map: Full name Point Cloud Data, use this name uniformly, do not call "point cloud map".

[0068] Cloud brain terminal: refers to the central processing server, not called "cloud" or "control center".

[0069] Scheduler: specifically refers to the dynamic task allocation module, distinguished from the path planner.

[0070] A*: A Star algorithm is a commonly used path finding and graph traversal algorithm. It has good performance and accuracy.

[0071] The current multi-device collaborative system has the following problems:

[0072] 1. Insufficient positioning accuracy, traditional SLAM systems rely on device sensors, which are prone to positioning drift in complex environments, leading to failure of device cooperation;

[0073] 2. Map sharing lag, devices communicate through raw point cloud data, high bandwidth occupation (single frame PCD data up to 100MB), poor real-time performance;

[0074] 3. Low scheduling efficiency, task allocation uses simple polling or fixed priority strategy, without considering real-time state of devices (such as power, location), task completion rate is about 30% lower.

[0075] Based on the above situation, the embodiments of the present application provide a multi-device cooperative operation scheduling method, system, device and storage medium.

[0076] In a first aspect, the embodiments of the present application provide a multi-device cooperative operation scheduling method, the scheduling method of the present embodiment is applied in a multi-device cooperative operation system, the system refers to Figure 2 , the system includes a UAV (perception layer), a cloud brain terminal and a plurality of execution devices (execution layer), the UAV and the execution device are respectively installed with a perception device. For example, the perception device of the present embodiment is a UAV with a laser radar, which is used for environment scanning and real-time raw data acquisition, three-dimensional laser scanning (scanning frequency 20Hz) is performed in the task area, laser point cloud (global point cloud map, PCD format, accuracy ±2cm) is generated, and data is transmitted to the cloud brain terminal through the 5G / WiFi 6 of the ROS2 communication node. In the present embodiment, the cloud brain terminal is a server cluster, the cloud brain terminal includes a point cloud processing engine, a coordinate mapping module and a task scheduling core, the point cloud processing engine generates a three-dimensional semantic map from the global point cloud map and the global point cloud map, the coordinate mapping module generates a world coordinate system conversion matrix according to the global point cloud map and the global point cloud map, and then the task scheduling core obtains task information, a three-dimensional semantic map and a world coordinate system conversion matrix and performs comprehensive processing to generate a device control instruction to control the execution device to execute a corresponding task. The execution device is a humanoid work device or a ground mobile execution device, which can move on the ground to execute tasks and plan paths and avoid obstacles in real time during task execution after receiving the control instruction and task instruction issued by the cloud brain terminal, such as carrying out inspection, transporting goods, operating switches, and performing fine tasks such as device maintenance. In addition, the execution device is also installed with an IMU, an odometer, a depth camera (local positioning), a mechanical arm, a force feedback sensor and other sensing devices, and the environment data or point cloud data detected by these sensing devices can be fed back to the cloud brain terminal, so that the cloud brain terminal can update the three-dimensional semantic map and the conversion matrix in real time. In the preferred embodiment, the multi-device cooperative operation system of the present embodiment also needs to rely on a UWB positioning base station and a GPS differential positioning module for auxiliary positioning.

[0077] As shown in Figure 1 The multi-device cooperative operation scheduling method of the embodiment is implemented by the following steps.

[0078] In step S1, a global point cloud map of a task area and a local point cloud map are obtained, wherein the global point cloud map is generated by scanning with a perception device on the UAV, and the local point cloud map is generated by scanning with a perception device on the execution device.

[0079] Since the global point cloud data transmitted between devices has a large memory (up to hundreds of MB per frame), it causes network congestion and map update delay > 500ms, so in a dynamic environment (such as a moving obstacle), the device cannot update the path in time, and the collision risk is increased by more than 40%. That is, the existing multi-device cooperative operation system has the problem of decision lag caused by poor real-time performance of map sharing. In order to solve the above technical problems, the embodiment uses ROS2 Topic to transmit Octree compressed maps, so that the data volume is reduced by 90% (reduced to 10MB / s), the delay is <100ms, and 100+ devices are supported for concurrent subscription.

[0080] Specifically, after the cloud brain terminal receives the global point cloud map, or before outputting the three-dimensional semantic point cloud map to the execution device, the point cloud map can also be compressed. Specifically, the resolution requirement preset by the user is obtained, and resolution requirement labels are assigned to each three-dimensional voxel block in the global point cloud map according to the resolution requirement. The resolution requirement labels include high-precision labels and low-precision labels; the current network bandwidth load and the device accuracy of the perception device on the execution device are obtained in real time, and the resolution of the three-dimensional voxel block marked with the resolution requirement label is dynamically adjusted according to the device accuracy and the current network bandwidth load to obtain the target resolution of different three-dimensional voxel blocks; and the global point cloud map is compressed according to the target resolution to obtain an Octree compressed map and output the Octree compressed map.

[0081] In more detail, the cloud brain terminal of the embodiment labels different regions or task types with resolution requirement labels, such as a high requirement label and a low requirement label (HIGH_DETAIL, LOW_DETAIL), according to a task target in a task decomposition stage, and meanwhile considers device sensor accuracy and current network bandwidth load to improve resolution for key regions (such as narrow passages and known obstacle dense areas) or regions of interest (task designated regions). The cloud brain terminal maintains a dynamic resolution configuration table that records target resolutions of different regions (three-dimensional voxel blocks) of a map. When generating an Octree by compressing original PCD data, a global uniform resolution parameter is no longer used. For regions labeled as HIGH_DETAIL or regions that need high precision according to a strategy, smaller voxel sizes (such as 0.05 m) are used to construct Octree sub-trees to retain more point cloud details; for regions labeled as LOW_DETAIL or background / non-key regions, larger voxel sizes (such as 0.2 m) are used to construct Octree sub-trees to significantly reduce data volume, and a smooth transition strategy (such as using interpolation) is used for resolution boundary regions to avoid map discontinuity.

[0082] The embodiment can perform more powerful compression on non-key regions while guaranteeing high-precision details of key regions through Octree compression, further reduces bandwidth occupation and storage requirements of overall map data, and realizes on-demand allocation of computing and communication resources.

[0083] In a preferred embodiment, in addition to using the Octree compression method, the voxel grid of the point cloud map can also be down-sampled, and direct down-sampling can also achieve the effect of improving transmission speed and reducing storage requirements.

[0084] In step S2, an initial transformation matrix is obtained, and the global point cloud map and the local point cloud map are converted to a world coordinate system through the initial transformation matrix. Specifically, the longitude, latitude and height of the unmanned aerial vehicle are obtained through high-precision satellite positioning technology, and the longitude, latitude and height are converted into world coordinates in the world coordinate system; the pose data of the unmanned aerial vehicle is obtained through an inertial measurement device on the unmanned aerial vehicle, and an initial transformation matrix is calculated according to the world coordinates and the pose data; the local point cloud map and the global point cloud map are coarsely aligned, and the local point cloud map and the global point cloud map after coarse alignment and the initial transformation matrix are globally aligned through an ICP algorithm to generate an optimal transformation matrix; and the local point cloud map is converted to a time coordinate system according to the optimal transformation matrix.

[0085] In an embodiment, the formula for calculating the world coordinate system mapping is as follows:

[0086]

[0087] wherein, Pworldis a point in the world coordinate system, Pdeviceis a point in the local coordinate system of the execution device, Tworld_deviceis the optimal transformation matrix from the device coordinate system to the world coordinate system.

[0088] In the system provided in the present application, the transformation matrix is obtained by the following method . Specifically, during initialization, the unmanned aerial vehicle can obtain high-precision global positioning (longitude, latitude, height) through RTK, and combined with IMU attitude data, the initial transformation matrix in the world coordinate system can be calculated. For the robotic dog and robot without RTK, positioning can be performed at startup through laser SLAM (such as LOAM, LIO-SAM, etc.) or visual SLAM (such as VINS), and then the global point cloud map scanned by the unmanned aerial vehicle is used to obtain the pose of the robotic dog / robot relative to the world coordinate system through point cloud matching (such as ICP algorithm) at the initial position.

[0089] When the robotic dog / robot starts, its own SLAM system generates a local point cloud map. Then the local point cloud map is matched with the global point cloud map provided by the unmanned aerial vehicle, and the RTK initialization + SLAM local mapping + ICP real-time alignment algorithm is used. More specifically, the longitude, latitude and height (WGS84 coordinates) of the unmanned aerial vehicle are obtained using the RTK module, converted into (x, y, z) in the world coordinate system (such as UTM), fused with the IMU attitude (quaternion q), and an initial transformation matrix is directly constructed. Then the LIO-SAM algorithm is started to generate a local point cloud map and a real-time pose. Finally, global alignment is performed using the ICP algorithm. The core of ICP is to minimize the distance between two point clouds, KD-Tree is used to accelerate the nearest neighbor search, and through iterative solving, the optimal rotation matrix R and translation vector t, i.e. the optimal transformation matrix, are obtained.

[0090] During the running of the system, real-time pose data of the execution device is obtained through a perception device on the execution device, and the real-time pose data is optimized using a Kalman filter and a multi-sensor fusion algorithm to obtain a current pose of the execution device; the optimal transformation matrix is updated according to the current pose, a latest transformation matrix is generated and broadcast through a TF library; the positions of the execution devices in the three-dimensional semantic map are determined according to the latest transformation matrix, and the position information is obtained.

[0091] Specifically during running, each device updates its pose in real time through its own positioning system (IMU, odometry, laser / visual SLAM) and publishes a TF transformation. The cloud brain terminal maintains a global TF tree, recording the real-time transformation relationship of each device relative to the world coordinate system.

[0092] Therefore, the coordinate mapping of the embodiment of the present application has two stages in the system running. In the initialization stage, the initial pose (transformation matrix) of the device is obtained, and the initial transformation matrix is obtained through point cloud matching; in the system running stage, real-time pose updating and cooperation, the robot or the robot dog updates the transformation matrix (i.e. the pose of the device in the world coordinate system) in real time through the sensor and the positioning algorithm of the device itself.

[0093] In a preferred embodiment, broadcasting and listening are performed through TF, specifically, the transformation of each device relative to the world coordinate system is broadcasted through the TF2 library of ROS2; each device can listen to the TF of other devices, so as to know the positions of other devices in the world coordinate system.

[0094] Step S3, constructing a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted to the world coordinate system, and determining the position information of the execution device according to the three-dimensional semantic map.

[0095] Specifically, the point cloud map is preprocessed, such as denoising and coarse processing of downsampling. Denoising is to remove outliers through statistical filtering; downsampling is to reduce the calculation amount of the voxel grid (such as 0.05m resolution). After preprocessing, semantic segmentation and semantic injection are performed, specifically, a segmentation model is constructed in advance through data labeling and training, then online semantic labeling is performed, the trained model is deployed to the edge device (TensorRT acceleration), real-time segmentation of the point cloud is performed, post-processing is performed, the label consistency is optimized through CRF (Conditional Random Field), and finally semantic fusion is performed to generate a three-dimensional semantic map by fusing the semantic and point cloud map.

[0096] Step S4, obtaining task information and device information of all execution devices, evaluating the task information, position information and device information through a preset dynamic scheduling model, and selecting an optimal execution device for executing a task according to the evaluation result.

[0097] Specifically, the task position of the to-be-executed task is obtained from the task information, the candidate execution devices within a preset range are obtained by screening according to the task position and the position information of all execution devices; the device information of the candidate execution devices and the task to be executed are evaluated through the scheduling evaluation model, and a comprehensive score is output; the candidate execution devices with a comprehensive score greater than a standard score value are selected as optimal execution devices, and the optimal execution devices are controlled to execute the to-be-executed task.

[0098] In the case where there is no candidate execution device with a comprehensive score greater than a preset threshold, i.e. the comprehensive scores of all candidate execution devices are not greater than a standard score value, the method of the present application further comprises: decomposing the to-be-executed task into a plurality of to-be-executed sub-tasks, re-inputting the task information of the to-be-executed sub-tasks into the dynamic scheduling model, and screening an optimal execution device combination.

[0099] The construction of the scheduling evaluation model comprises: acquiring the capability matching degree, time cost, energy consumption efficiency of the execution device, and dynamic priority of the task to be executed; acquiring the weights of the capability matching degree, time cost, energy consumption efficiency and dynamic priority respectively through a dynamic learning mechanism; and constructing a scheduling evaluation model according to the capability matching degree, time cost, energy consumption efficiency, dynamic priority and corresponding weights.

[0100] The scheduling evaluation model of the embodiment of the present application is a four-dimensional scheduling algorithm, and the formula of the scheduling evaluation model is:

[0101]

[0102] wherein, is the weight of each dimension;

[0103] capability matching degree , represents a device capability value, represents a task demand value, represents a capability weight;

[0104] time cost , represents a path planning length, represents an average speed of a device, represents a terrain and obstacle coefficient;

[0105] energy consumption efficiency , represents a current power of a device, represents energy consumption required by a task and execution;

[0106] dynamic priority , represents a task priority, represents an average priority of a system.

[0107] For the dynamic weight learning mechanism, the weight is initialized as ; then is updated online , wherein the formula of the loss function is:

[0108]

[0109] The calculation formula of the gradient calculation is:

[0110]

[0111] learning rate Then task clustering is performed, and the task features are clustered using K-means: clusters = kmeans(features=[moving distance, accuracy requirement, urgency level], k=3), and an independent weight is maintained for each type of task.

[0112] In more detail, with reference to Figure 3 In the task scheduling process, a to-be-executed task is first extracted from the task pool in a preset order (such as an execution time sequence), and it is determined whether the to-be-executed task is an urgent task. If so, the to-be-executed task is placed in a high-priority queue, and an appropriate execution device is immediately selected for distribution. If it is not an urgent task, the comprehensive scores of all candidate devices are calculated, and the device with the largest score (max comprehensive score) is selected according to the four-dimensional scheduling algorithm for task distribution. Then, the execution device assigned to the task performs global path planning through the A* algorithm. The four-dimensional scheduling algorithm of this embodiment dynamically assigns tasks to the optimal device / device group, comprehensively considers four dimensions of device capability, path cost, power efficiency, and task priority, optimizes multi-device collaborative task scheduling, and the input parameters are a device list (containing device information), an environment map, and task information. The output result is a task distribution result (single device or device group).

[0113] The four-dimensional scheduling algorithm is implemented through the following steps:

[0114] First, the basic parameters and global variables of the algorithm are set, the single-device execution threshold is obtained, and the system all available device set and the navigation map service containing environment information are obtained. Then, the to-be-distributed task object (such as containing location, capability requirement, energy consumption, time cost, task priority, etc.) is input, and the spatial_filter() function is called to filter the devices near the task location (such as within 50 meters), and the candidate device list is output. If there is no suitable device, an extended search is triggered (such as relaxing the distance limit). Then, each candidate device is traversed, and the scores of the capability matching degree, time cost, energy efficiency, and dynamic priority of each candidate device are calculated, respectively. Then, the comprehensive score Score is calculated according to the capability matching degree, time cost, energy efficiency, and dynamic priority, and the corresponding weight, and a decision output is made according to the comprehensive score. If the highest comprehensive score in all candidate devices is greater than the single-device execution threshold, the candidate device is selected as the optimal single-device execution. After obtaining the optimal execution device, the A* algorithm is called for path planning. Specifically, the semantic map is converted into a 0.1m resolution navigation grid map, the A* algorithm is used to calculate the optimal path from the starting point to the ending point, and then the path point list or null value (no path) is returned.

[0115] If the highest comprehensive score is not greater than the single-device execution threshold, a collaboration group assignment function is called. Specifically, a collaboration group object is first created, the task is decomposed into atomic subtasks (such as “move → grab → transport”), each subtask is traversed, the most suitable device is selected for the subtask (considering the matching degree of device capability and subtask), device reuse checking ensures that the device is not assigned to other subtasks of the same task, the subtask is added to the collaboration group and assigned. Finally, the overall efficiency of the collaboration group is calculated (such as estimated completion time / single-device optimal time), if the efficiency > 0.6, the collaboration group is returned, otherwise an exception of no suitable device is thrown.

[0116] In an embodiment, in order to solve the problem of lack of autonomous collaboration capability in complex scenarios, the cloud brain terminal first intelligently decomposes the composite task into atomic subtasks, uses a dynamic scheduling algorithm (multi-constraint cost function + auction mechanism), and comprehensively considers the characteristics of the subtasks (location, required capability, priority, dependency relationship) and the real-time state of the devices, optimally assigns the subtasks to the most suitable device or device group, uses the global PCD map to preliminarily plan a globally optimal path (using the A* algorithm) for each device that needs to move from the current location to the task starting point or target point, encapsulates the assigned subtasks, target point coordinates, pre-planned path, etc. into specific instructions, and issues them to the corresponding device through the ROS2 Topic (such as / task_assignment). The device receives and executes the task instructions issued by the cloud brain. During execution, the device continuously perceives the surrounding local environment (such as real-time point cloud, detected dynamic obstacle position / speed) using its own sensors (laser radar, depth camera, IMU, etc.), performs real-time positioning (SLAM) and local obstacle detection, and reports the above data and abnormal events to the cloud brain terminal in real time through the ROS2 Topic. Finally, the autonomous collaboration closed loop of “cloud decision-making-device execution-real-time feedback” is realized.

[0117] After determining the optimal execution device, the task information is sent to the optimal execution device, and an optimal global path is generated by the optimal execution device. Specifically, the path starting point and the path ending point are determined according to the location information and the task information respectively; the three-dimensional semantic map is voxelized to generate a three-dimensional grid map; the three-dimensional grid map, the path starting point and the path ending point are queried by a path finding and graph traversal algorithm, and the optimal global path of the optimal execution device is output.

[0118] In the process of generating the optimal global path, the Euclidean distance between the optimal execution device and the obstacle is detected in real time by the sensing device on the optimal execution device; if the difference between the Euclidean distance and the distance of the obstacle recorded in the three-dimensional grid map exceeds a preset threshold, the dynamic window method is triggered to perform local path replanning; after the optimal execution device avoids the obstacle, the global path is returned, and the data corresponding to the obstacle in the three-dimensional grid map is dynamically updated.

[0119] In more detail, in the process of path planning and obstacle avoidance, global path planning (A* algorithm) is adopted, and the real-time obstacle avoidance triggering condition is shown in the following formula:

[0120]

[0121] When the obstacle avoidance triggering condition is met, the DWA (Dynamic Window Approach) local path replanning is triggered. DWA (Dynamic Window Approach) is an algorithm for real-time local obstacle avoidance and motion planning of robots, which is designed for dynamic environment. Its core idea is to dynamically generate feasible trajectories in the velocity space and select the optimal path to avoid obstacles.

[0122] The implementation process of global path planning and real-time obstacle avoidance of the embodiment is described as follows:

[0123] A-star algorithm is adopted to calculate the optimal path in the three-dimensional grid map, support dynamic obstacle avoidance and real-time replanning, input three-dimensional grid map, device starting point and task end point, and then output path point list or empty table. A-star algorithm is a classic heuristic path search algorithm, which is used to find the optimal path between two points in a graph or grid. It combines the completeness of Dijkstra algorithm and the efficiency of greedy best-first search, and is praised as the cornerstone of "intelligent search".

[0124] A* algorithm (A-star algorithm) covers all possible moving directions in three-dimensional space through 26-neighborhood search; differentiates the physical rationality of straight line / diagonal movement through differentiated movement cost; and detects obstacles in real time and triggers replanning through dynamic obstacle avoidance mechanism. When the obstacles in the local point cloud graph change (new obstacles are added or removed), the A-star algorithm is called again by inputting the updated map to immediately obtain a new path. Since the algorithm is completely based on the current grid state, it naturally supports real-time replanning.

[0125] The multi-device cooperative operation scheduling method provided in the application has the following advantages:

[0126] In order to solve the problem of cooperation failure caused by the non-uniform reference of multi-device cooperative positioning, the spatial reference is unified through the world coordinate system mapping, so that all devices share the global coordinates of the same PCD map, and the positioning error is controlled within <5cm;

[0127] In order to solve the problem of decision lag caused by poor real-time performance of map sharing, ROS2 Topic is used to transmit Octree compressed map, the data volume is reduced by 90% (to 10MB / s), the delay is <100ms, and 100+ devices are supported for concurrent subscription;

[0128] In order to solve the problem of low system efficiency caused by static task allocation, a dynamic scheduling algorithm (multi-constraint cost function + auction mechanism) is used to allocate tasks in real time by comprehensively considering the distance, power and capability matching degree of the device, the task completion rate is improved to 95%, and the response speed of high-priority tasks is increased by 40%.

[0129] In summary, the multi-device cooperative operation scheduling method provided by the application has the effects of improving positioning accuracy, optimizing communication efficiency and improving scheduling performance.

[0130] In a second aspect, the embodiments of the application provide a multi-device cooperative operation system for implementing the multi-device cooperative operation scheduling method of any one of the above embodiments, the system comprising:

[0131] An unmanned aerial vehicle provided with a first sensing device, configured to collect a global point cloud map of a task area through the first sensing device and send the global point cloud map to a cloud brain terminal;

[0132] An execution device provided with a second sensing device, configured to collect a local point cloud map of the task area through the second sensing device and send the local point cloud map to the cloud brain terminal;

[0133] A cloud brain terminal configured to obtain a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is obtained by scanning through a sensing device on the unmanned aerial vehicle, and the local point cloud map is obtained by scanning through a sensing device on the execution device; obtain an initial transformation matrix, and convert the global point cloud map and the local point cloud map to a world coordinate system through the initial transformation matrix; construct a three-dimensional semantic map according to the global point cloud map and the local point cloud map converted to the world coordinate system, and determine position information of the execution device according to the three-dimensional semantic map; obtain task information and device information of all execution devices, evaluate the task information, the position information and the device information through a preset dynamic scheduling model, and select an optimal execution device for executing the task according to an evaluation result;

[0134] The execution device is further configured to perform global path rules and obstacle avoidance according to the received task information and the three-dimensional semantic map, and complete the received task.

[0135] Specifically, the cloud brain terminal comprises:

[0136] a map construction module configured to receive a global point cloud map and perform dynamic resolution adjustment and Octree compression;

[0137] a coordinate system mapping module configured to map the point cloud map to a world coordinate system by using an RTK+SLAM+ICP hybrid calibration algorithm;

[0138] a scheduling decision module configured to run a dynamic scheduling model with a weighted learning mechanism to assign tasks to each execution device;

[0139] a ROS2 communication hub configured to realize data transmission between the cloud brain terminal and the execution device, such as publishing / subscription of topics such as management / map and / task.

[0140] It should be noted that the multi-device cooperative operation system provided in the embodiment is used to implement the above method embodiments, and the repeated description is omitted. As used above, the terms "module", "unit", "sub-unit", and the like can be a combination of software and / or hardware that implements a predetermined function. Although the above embodiments describe the device preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and contemplated.

[0141] In a third aspect, the embodiments of the present application provide an electronic device, Figure 4 is a block diagram of an electronic device according to an exemplary embodiment. As Figure 4 shown, the electronic device can include a processor 11 and a memory 12 storing computer program instructions.

[0142] Specifically, the above-mentioned processor 11 can include a central processing unit (CPU), or a specific integrated circuit (Application Specific Integrated Circuit, referred to as ASIC), or can be configured to implement one or more integrated circuits of the embodiments of the present application.

[0143] The memory 12 can include a mass storage for data or instructions. By way of example and not limitation, the memory 12 can include a Hard Disk Drive (HDD), floppy disk drive, a Solid State Drive (SSD), flash memory, a Compact Disc Read Only Memory (CDROM), a Digital Versatile Disk (DVD), a magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. The memory 12 can be removable and / or non-removable (or fixed) as appropriate. The memory 12 can be internal or external as appropriate. In particular embodiments, the memory 12 is a Non-Volatile memory. In particular embodiments, the memory 12 includes a Read-Only Memory (ROM) and a Random-Access Memory (RAM). The ROM can be a mask-programmed ROM, a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically EPROM (EEPROM), an Electrically Alterable ROM (EAROM), or a FLASH, or a combination of two or more of these, as appropriate. The RAM can be a Static Random-Access Memory (SRAM) or a Dynamic Random-Access Memory (DRAM), which can be a Fast Page Mode Dynamic Random-Access Memory (FPMDRAM), an Extended Data Out Dynamic Random-Access Memory (EDODRAM), a Synchronous Dynamic Random-Access Memory (SDRAM), or the like, as appropriate.

[0144] The memory 12 can be used to store or buffer various data files required for processing and / or communication, and possible computer program instructions executed by the processor 11.

[0145] The processor 11 reads and executes the computer program instructions stored in the memory 12 to implement any of the multi-device cooperative operation scheduling methods in the above embodiments.

[0146] In an embodiment, the electronic device can further include a communication interface 13 and a bus 10. As shown in the figure, the processor 11, the memory 12, and the communication interface 13 are connected through the bus 10 and complete communication with each other. Figure 4

[0147] The communication interface 13 is used to realize the communication between various modules, devices, units and / or devices in the embodiments of the present application. The communication interface 13 can also realize data communication with other components, such as external devices, image / data acquisition devices, databases, external storage, and image / data processing workstations, etc.

[0148] ​Bus 10 includes hardware, software, or both, to couple components of electronic device to each other. Bus 10 includes, but is not limited to, at least one of a data bus, an address bus, a control bus, an expansion bus, a local bus, and the like. By way of example and not limitation, bus 10 can include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand (IB) interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or the like, or a combination of two or more of these. Where appropriate, bus 10 can include one or more buses. Although the present application is described and illustrated with a particular bus, it is contemplated that any appropriate bus or interconnect can be used.

[0149] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, having stored thereon a program, wherein the program is executed by a processor to implement the method for scheduling cooperative operation of multiple devices according to the first aspect.

[0150] More specifically, the computer readable storage medium can include, but is not limited to, a portable disc, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any appropriate combination of the above.

[0151] In possible implementation manners, the present application can also be implemented in the form of a program product, which comprises program codes for causing a terminal device to perform the steps of the method for implementing the multi-device cooperative operation scheduling provided by the first aspect when the program product is run on the terminal device.

[0152] The program codes for executing the present application can be written in any combination of one or more programming languages, and can be executed completely on a user device, partially on a user device, as a stand-alone software package, partially on a user device and partially on a remote device, or completely on a remote device.

[0153] The technical features of the above-described embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described, but it should be considered that any combination of the technical features is within the scope of the present disclosure as long as the combination does not result in contradictions.

[0154] The above-described embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A multi-device collaborative operation scheduling method, characterized in that, The application is applied to a multi-device cooperative operation scheduling system, the system comprises a UAV, a cloud brain terminal and a plurality of execution devices, the UAV and the execution devices are respectively installed with sensing devices, and the method comprises: The cloud brain terminal acquires a global point cloud map and a local point cloud map of a task area, wherein the global point cloud map is generated by scanning with the sensing device on the UAV, and the local point cloud map is generated by scanning with the sensing device on the execution device; The longitude, latitude and height of the UAV are acquired by satellite positioning technology, and are converted into world coordinates in a world coordinate system; the pose data of the UAV are acquired by the inertial measurement device on the UAV, and an initial transformation matrix is calculated according to the world coordinates and the pose data; the local point cloud map and the global point cloud map are roughly aligned, and the local point cloud map and the global point cloud map after rough alignment and the initial transformation matrix are globally aligned by an ICP algorithm to generate an optimal transformation matrix; the local point cloud map is converted into the world coordinate system according to the optimal transformation matrix; A three-dimensional semantic map is constructed according to the global point cloud map and the local point cloud map converted into the world coordinate system, and the position information of the execution device is determined according to the three-dimensional semantic map; Task information and device information of all execution devices are acquired, the task position of a task to be executed is acquired from the task information, the candidate execution devices within a preset range are acquired by screening according to the task position and the position information of all execution devices; the device information of the candidate execution devices and the task to be executed are evaluated by a scheduling evaluation model, and a comprehensive score is output; the candidate execution devices with a comprehensive score greater than a standard score value are screened as optimal execution devices, and the optimal execution devices are controlled to execute the task to be executed; The construction of the scheduling evaluation model comprises: The capability matching degree, time cost, energy consumption efficiency of the execution device, and the dynamic priority of the task to be executed are acquired; The weights of the capability matching degree, time cost, energy consumption efficiency and dynamic priority are respectively acquired by a dynamic learning mechanism; The scheduling evaluation model is constructed according to the capability matching degree, time cost, energy consumption efficiency, dynamic priority and corresponding weights.

2. The method of claim 1, wherein, After the cloud brain terminal acquires the global point cloud map, the method further comprises: The resolution requirement preset by the user is acquired, and resolution requirement labels are assigned to each three-dimensional voxel block in the global point cloud map according to the resolution requirement, wherein the resolution requirement labels comprise high-precision labels and low-precision labels; The current network bandwidth load and the device accuracy of the sensing device on the execution device are acquired in real time, the resolution of the three-dimensional voxel block marked with the resolution requirement label is dynamically adjusted according to the device accuracy and the current network bandwidth load, and the target resolution of different three-dimensional voxel blocks is obtained; The global point cloud map is compressed according to the target resolution to obtain an Octree compressed map and output.

3. The method of claim 1, wherein, After determining the optimal execution device, the task information is sent to the optimal execution device, and an optimal global path is generated by the optimal execution device; wherein the generation of the optimal global path comprises: determining a path starting point and a path ending point according to the position information and the task information; performing voxelization processing on the three-dimensional semantic map to generate a three-dimensional grid map; performing path query on the three-dimensional grid map, the path starting point and the path ending point through a path finding and graph traversal algorithm, and outputting an optimal global path of the optimal execution device.

4. The method of claim 3, wherein, In the process of generating the optimal global path, the method further comprises: detecting a Euclidean distance between the optimal execution device and an obstacle in real time through a sensing device on the optimal execution device; if a difference between the Euclidean distance and a distance of the obstacle recorded in the three-dimensional grid map exceeds a preset threshold, triggering a dynamic window method to perform local path re-planning; after the optimal execution device avoids the obstacle, returning to the global path and dynamically updating data corresponding to the obstacle in the three-dimensional grid map.

5. The method of claim 1, wherein, The determination of the position information of the execution device according to the three-dimensional semantic map comprises: obtaining real-time pose data of the execution device through a sensing device on the execution device, and optimizing the real-time pose data through a Kalman filter and a multi-sensor fusion algorithm to obtain a current pose of the execution device; updating the optimal transformation matrix according to the current pose, generating a latest transformation matrix and broadcasting through a TF library; determining the position of each execution device in the three-dimensional semantic map according to the latest transformation matrix to obtain the position information.

6. The method of claim 1, wherein, If the comprehensive score of all candidate execution devices is not greater than a standard score value, the method further comprises: decomposing the task to be executed into a plurality of sub-tasks to be executed, re-inputting task information of the sub-tasks to be executed into the scheduling evaluation model, and screening an optimal execution device combination.

7. A multi-device cooperative operation scheduling system, characterized by, The system is used for implementing the multi-device cooperative operation scheduling method according to any one of claims 1-6, and comprises: a drone provided with a first sensing device, which is used for collecting a global point cloud map of a task area through the first sensing device and sending the global point cloud map to a cloud brain terminal; an execution device provided with a second sensing device, which is used for collecting a local point cloud map of the task area through the second sensing device and sending the local point cloud map to the cloud brain terminal; wherein the execution device is a plurality of execution devices. The cloud brain terminal is used to acquire a global point cloud map and a local point cloud map of a task area; the longitude, latitude and height of the UAV are acquired through satellite positioning technology, and the longitude, latitude and height are converted into world coordinates in a world coordinate system; the pose data of the UAV are acquired through an inertial measurement device on the UAV, and an initial transformation matrix is calculated according to the world coordinates and the pose data; the local point cloud map and the global point cloud map are coarsely aligned, and the local point cloud map and the global point cloud map after coarse alignment and the initial transformation matrix are globally aligned through an ICP algorithm to generate an optimal transformation matrix; the local point cloud map is converted into the world coordinate system according to the optimal transformation matrix; a three-dimensional semantic map is constructed according to the global point cloud map and the local point cloud map converted into the world coordinate system, and the position information of the execution device is determined according to the three-dimensional semantic map; task information and device information of all execution devices are acquired, the task position of a task to be executed is acquired from the task information, the task position and the position information of all execution devices are screened to acquire candidate execution devices within a preset range; the device information of the candidate execution devices and the task to be executed are evaluated through a scheduling evaluation model, and a comprehensive score is output; the candidate execution devices with a comprehensive score greater than a standard score value are screened as optimal execution devices, and the optimal execution devices are controlled to execute the task to be executed. The execution device is further used to perform global path planning and obstacle avoidance according to the received task information and the three-dimensional semantic map, and complete the received task.

8. The multi-device collaboration scheduling system of claim 7, wherein, The cloud brain terminal comprises: A map construction module is used to receive a global point cloud map and perform dynamic resolution adjustment to realize Octree compression. A coordinate system mapping module is used to map the global point cloud map and the local point cloud map into a world coordinate system through an RTK+SLAM+ICP hybrid calibration algorithm. A scheduling decision module is used to run a scheduling evaluation model with a weight learning mechanism to assign tasks corresponding to task information to each execution device. A ROS2 communication hub is used to realize data transmission between the cloud brain terminal and the execution devices.

9. An electronic device, comprising: The program is executed by the processor to realize the multi-device cooperative operation scheduling method according to any one of claims 1 to 6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The program is executed by the processor to realize the multi-device cooperative operation scheduling method according to any one of claims 1 to 6.

Images