A multi-modal intelligent dynamic scheduling method and system for intelligent deduction relay stations

Abstract

The application provides a multi-modal intelligent dynamic scheduling method and system for an intelligent deduction relay station, which comprises the following steps: constructing robot multi-source state information, performing dynamic task allocation and load balancing based on the information, adaptively switching the robot operation mode through a hierarchical state machine architecture, utilizing a distributed infrared laser monitoring network to gridize the target area, monitoring the target area, triggering congestion early warning, and executing response strategies, and a scheduling system suitable for the method. The application achieves the technical effects of more reasonably allocating robot tasks, flexibly switching operation modes, timely monitoring and processing regional congestion, thereby improving the robot scheduling efficiency and stability of the intelligent deduction relay station.

Description

Technical Field

[0001] This invention relates to the field of robot scheduling technology, and in particular to a multimodal intelligent dynamic scheduling method and system for intelligent performance stations. Background Technology

[0002] With the large-scale deployment of service robots in commercial scenarios, task allocation, state switching, and anomaly response technologies based on centralized scheduling platforms have been widely applied in structured environments such as warehousing and logistics, and cleaning and inspection. The application of these technologies has significantly improved the level of operational automation, enabling robots to efficiently perform tasks that previously required significant human labor. It has also improved human-robot collaboration efficiency, freeing humans from repetitive and tedious tasks and allowing them to focus more on more creative and decision-making work. These technologies are also involved to some extent in typical application scenarios such as shopping mall guidance and delivery services, hospital medicine / sample delivery, and airport terminal passenger guidance and baggage transfer, but some shortcomings still exist.

[0003] In the past, to solve related problems, a task allocation mechanism with a static task queue and a fixed priority scheduling strategy was usually adopted. This mechanism allocates tasks according to a pre-set task order and priority, which to a certain extent ensures the orderly execution of tasks. At the same time, simple mode switching logic based on preset time thresholds or single-point infrared sensor triggers is also widely used. Mode switching is achieved by setting a specific time or relying on the signal of a single-point infrared sensor. In addition, a centralized area monitoring architecture that relies on a central server to uniformly process video streams and identify congestion is also a common approach. The central server collects video information from various areas and performs unified analysis and processing to determine whether there is congestion.

[0004] However, these existing technologies have significant drawbacks. Task allocation cannot be dynamically adjusted based on the robot's real-time battery level, spatial location, and environmental accessibility weights, resulting in severely uneven load distribution, high single-robot overload rates, and low utilization of nearby idle resources. Mode switching only responds to preset times or coarse-grained sensor signals, failing to autonomously activate adaptive behavior strategies based on scene semantic features, leading to delayed mode response, inaccurate strategy matching, and erroneous execution of non-whitelisted instructions. Congestion warnings in key areas rely on a single sensor and global video analysis, resulting in high response latency, high false alarm rates, a lack of tiered intervention capabilities, and an inability to achieve localized real-time path replanning and task circuit breaking. Summary of the Invention

[0005] This invention solves the problems of low scheduling efficiency and poor stability in existing technologies, and proposes a multimodal intelligent dynamic scheduling method and system for intelligent performance stations.

[0006] To achieve the above objectives, the following technical solution is proposed:

[0007] A multimodal intelligent dynamic scheduling method for intelligent interactive stations includes the following steps: S1, constructing multi-source state information for each robot, wherein the multi-source state information includes real-time battery level, positioning coordinates, task queue length, and dynamic weight values ​​based on passable grids in a three-dimensional environment topology map; S2. Based on the multi-source state information, dynamic task allocation and load balancing are performed for several robots through the task allocation module deployed in the edge computing unit. S3. In response to external commands, the robot's operating mode is adaptively switched through a hierarchical state machine architecture. The hierarchical state machine architecture includes a state machine, a scene recognition module, and a semantic verification module. The state machine is defined as a station mode, a performance mode, and a free interaction mode. The station mode, performance mode, and free interaction mode are mutually exclusive. S4. The target area is monitored in a grid pattern by a distributed infrared laser monitoring network deployed in key areas. Based on the monitoring results, different levels of congestion warnings are triggered, and corresponding robot response strategies are executed in conjunction. The response strategies include path replanning and task dispatch circuit breaker.

[0008] By adopting the above technical solutions, we have achieved a reduction in single-machine task overload rate, an increase in the task acceptance rate of nearby idle robots, a shortening of mode switching response time, a reduction in instruction semantic error execution rate, a reduction in the occurrence rate of congestion in key areas, an improvement in the accuracy of central control console alarms, and a reduction in the delay of new task dispatching circuit breakers.

[0009] Preferably, in step S3, the adaptive switching of the robot's operating mode through a hierarchical state machine architecture specifically includes the following steps: S3.1 The scene recognition module processes the point cloud data of the lidar on the robot to identify the current scene features, and triggers scene tags when it identifies preset high-density crowd features; S3.2 The semantic verification module receives the speech-to-text instruction and determines whether to execute the instruction by calculating the similarity between the semantic vector of the instruction and the pre-stored instruction vector in the whitelist rule base corresponding to the current mode.

[0010] By adopting the above technical solutions, multi-source state information for each robot is constructed, and dynamic task allocation and load balancing are performed based on this information. A hierarchical state machine architecture with a state machine, scene recognition module, and semantic verification module is used to adaptively switch robot operation modes. The state machine defines mutually exclusive station, performance, and free interaction modes, which facilitates accurate entry into the corresponding mode according to different conditions. The scene recognition module processes LiDAR point cloud data to trigger scene tags, which can accurately identify scenes such as high-density crowds. The semantic verification module determines whether to execute an instruction by calculating the similarity between the instruction semantic vector and the pre-stored instruction vector in the whitelist rule base, which shortens the mode switching response time and reduces the instruction semantic error execution rate. In addition, a distributed infrared laser monitoring network is used to perform grid-based monitoring of the target area and trigger different levels of congestion warnings and execute corresponding response strategies, which can reduce the congestion rate in key areas and improve the accuracy of alarms from the central control console.

[0011] Preferably, the scene recognition module identifies current scene features by processing point cloud data from the lidar mounted on the robot, specifically including the following steps: S3.1.1 Perform voxel meshing on the current scan frame of the lidar; S3.1.2 Count the number of point clouds within voxels centered on the robot and within a preset radius, and generate a point cloud density heatmap; S3.1.3 Calculate the average point cloud density of the central region based on the temporal variation of the point cloud density heatmap within the sliding time window; S3.1.4 If the average point cloud density continues to exceed the first density threshold, the "high-density crowd" scene label is triggered.

[0012] By adopting the above technical solution, the current scene characteristics can be accurately identified. When the characteristics of a high-density crowd are identified, the scene tag is triggered to realize the mode switching based on the scene characteristics, ensure the real-time response of the mode, and reduce the error rate of instruction semantic execution.

[0013] Preferably, step S3.3 specifically includes the following steps: S3.2.1 Input the voice command into the lightweight natural language processing model and output the command semantic vector; S3.2.2 Obtain at least one pre-stored rule semantic vector from the whitelist rule base corresponding to the robot's current mode; S3.2.3 Calculate the cosine similarity between the instruction semantic vector and each of the rule semantic vectors; S3.2.4 If the maximum cosine similarity value is not less than the preset similarity threshold, then execute the voice command; otherwise, refuse to execute and return an error.

[0014] By adopting the above technical solution, voice commands are input into a lightweight natural language processing model to obtain command semantic vectors. Cosine similarity is calculated between the command and the regular semantic vectors. The decision on whether to execute the command is based on the similarity value. This not only ensures the real-time response of the robot to commands in different operating modes, but also improves the safety of command execution and reduces the error rate of command semantic execution.

[0015] Preferably, the distributed infrared laser monitoring network includes several composite sensing nodes deployed in a three-dimensional grid in key areas, each composite sensing node including an infrared pyroelectric sensor and a laser ranging module; the triggering of different levels of congestion warnings based on the monitoring results specifically includes the following steps: S4.1 Each composite sensing node performs fusion processing on infrared pyroelectric signals and laser ranging signals locally, and counts the number of valid human targets identified within the sliding window with a preset time length. S4.2 When the number of valid human targets in the first consecutive sliding windows is not less than the first number threshold, a level one congestion warning is triggered. S4.3 When a Level 1 congestion warning is triggered, if the number of valid human targets in a second consecutive number of sliding windows is not less than a second number threshold that is higher than the first number threshold, a Level 2 congestion warning is triggered.

[0016] By adopting the above technical solutions, constructing multi-source state information and performing dynamic task allocation and load balancing, the overload rate of single-machine tasks can be reduced, and the task acceptance rate of nearby idle robots can be improved. Utilizing a hierarchical state machine architecture for adaptive switching of operating modes can compress mode switching response time and reduce the error rate of instruction semantics. Through the three-dimensional grid-like composite sensing nodes of the distributed infrared laser monitoring network, the infrared pyroelectric signal and laser ranging signal are fused and processed. The number of valid human targets is counted according to a preset sliding window, triggering different levels of congestion warnings. This can reduce the congestion rate in key areas, improve the alarm accuracy of the central control console, and reduce the delay of new task dispatching.

[0017] Preferably, if the robot receives a Level 1 congestion warning, it immediately invokes a path planning algorithm to replan the path in order to avoid the area corresponding to the warning grid coordinates.

[0018] Preferably, if the robot and the central control station receive a level-two congestion warning, the task assignment circuit breaker mechanism is triggered to suspend the assignment of new tasks to the affected robot.

[0019] By adopting the above technical solutions, if the robot receives a Level 1 congestion warning, it will immediately perform path replanning to avoid the warning area. When the robot and the central control station receive a Level 2 congestion warning, the task assignment circuit breaker mechanism will be triggered to suspend the assignment of new tasks to the affected robot. This can reduce the congestion rate in key areas, improve the alarm accuracy of the central control station, and reduce the delay of new task assignment circuit breaker.

[0020] Preferably, step S4.1 specifically includes the following steps: S4.1.1 Filter the output signal of the infrared pyroelectric sensor to eliminate environmental temperature drift interference; S4.1.2 Determine that the amplitude of the filtered pyroelectric signal exceeds the first voltage threshold and the duration exceeds the first time threshold; S4.1.3 Determine that the distance value measured by the laser ranging module is within the preset effective human body distance range, and that the variance of the measured distance is less than the preset variance threshold. S4.1.4 When the judgment conditions of the laser ranging signal in steps S4.1.2 and S4.1.3 are met simultaneously, it is determined that a valid human target has been identified.

[0021] By adopting the above technical solutions, filtering the output signal of the infrared pyroelectric sensor can eliminate environmental temperature drift interference and improve signal accuracy. Setting voltage and time thresholds to judge the pyroelectric signal and setting distance interval and variance thresholds to judge the laser ranging signal, and determining the identification of a valid human target when both conditions are met simultaneously, can more accurately identify valid human targets. This provides a reliable basis for accurately triggering congestion warnings of different levels, thereby improving the accuracy of congestion warnings in key areas, reducing the occurrence rate of congestion in key areas, and reducing the delay of new task dispatching.

[0022] Preferably, the dynamic weight value W(i,j) of the traversable grid in the 3D environment topology map mentioned in step S1 is calculated using the following formula: W(i,j)=α·d occupancy (i,j)+β·r friction (i,j)+γ·c safety In the formula (i,j), d occupancy (i,j) represents the instantaneous obstacle density of the grid (i,j) based on lidar point cloud inversion, r friction (i,j) represents the historical travel friction coefficient of grid (i,j), c safety (i,j) represents the distance cost from grid (i,j) to the preset safety point, and α, β, and γ are calibrated weighting coefficients.

[0023] By adopting the above technical solution, a dynamic weight value calculation method for passable grids is constructed. It integrates factors such as instantaneous obstacle density, historical passage friction coefficient, and distance cost from the preset safety point. Combined with the steps of constructing multi-source state information and performing dynamic task allocation and load balancing, it can make task allocation more accurate, reduce the single-machine task overload rate, and improve the task acceptance rate of nearby idle robots.

[0024] A multimodal intelligent dynamic scheduling system for intelligent interactive stations includes several robots, a distributed infrared laser monitoring network deployed in key areas, and a central control unit communicating with the robots and the distributed infrared laser monitoring network. Each robot includes a multi-source state perception module, an environment modeling module, a reinforcement learning task allocation engine, a hierarchical state machine control module, and a path planning and execution module. The robot is configured to receive early warning signals from the distributed infrared laser monitoring network and perform corresponding path replanning or task suspension operations. The distributed infrared laser monitoring network includes several sensing nodes, and the robot is configured to receive early warning signals from the distributed infrared laser monitoring network and perform corresponding path replanning and task suspension operations.

[0025] By adopting the above technical solutions, the overload rate of single-machine tasks was reduced, the task acceptance rate of nearby idle robots was increased, the response time of mode switching was shortened, the error execution rate of instruction semantics was reduced, the congestion rate in key areas was reduced, the alarm accuracy of the central control console was improved, and the delay of new task dispatching circuit breakers was lower.

[0026] The beneficial effects of this invention are: 1. By integrating real-time robot battery level, positioning coordinates, task queue length, and dynamic weight values ​​of passable grids in the 3D environment topology map, dynamic task allocation and load balancing are achieved, reducing the single-machine task overload rate and increasing the task acceptance rate of nearby idle robots.

[0027] 2. A hierarchical state machine architecture is adopted to adaptively switch the robot's operating modes, which shortens the mode switching response time and reduces the error rate of instruction semantics.

[0028] 3. Deploy a distributed infrared laser monitoring network for grid-based monitoring and trigger congestion warnings of different levels, linking path replanning and task assignment circuit breakers to reduce congestion rates in key areas, improve the accuracy of alarms from the central control console, and reduce the delay of new task assignment circuit breakers. Attached Figure Description

[0029] Figure 1 This is a flowchart of the method of the present invention.

[0030] Figure 2 This is a flowchart illustrating the adaptive switching mechanism of this invention.

[0031] Figure 3 This is a flowchart illustrating the different levels of congestion warnings for this invention.

[0032] Figure 4 This is a diagram showing the connection relationships between the system modules of the present invention. Detailed Implementation

[0033] The technical solutions in the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. The described embodiments are only possible technical implementations of the present invention, but are not limited thereto. Other embodiments obtained by those skilled in the art in conjunction with the embodiments of the present invention without creative effort are also within the protection scope of the present invention.

[0034] Example 1: This application provides a multimodal intelligent dynamic scheduling method for intelligent interactive stations, comprising the following steps: S1. Construct multi-source state information for each robot. The multi-source state information includes real-time battery level, positioning coordinates, task queue length, and dynamic weight values ​​based on passable grids in the 3D environment topology map.

[0035] Real-time battery level can be obtained through the robot's built-in battery management unit, which uses a high-precision battery monitoring chip to report the remaining battery level in 5% increments. Alternatively, other battery monitoring devices with similar precision can be used.

[0036] The positioning coordinates are provided by high-frequency (≥100Hz) robot angular velocity and acceleration data from the IMU (Inertial Measurement Unit), and low-frequency (10-50Hz) tag ranging information from the UWB (Ultra-Wideband) base station network. Sensor fusion is then performed using Extended Kalman Filter (EKF) or Error State Kalman Filter (ESKF). The robot's position, velocity, and attitude are integrated using IMU data to obtain predicted values, a process that accumulates errors. When a ranging observation from the UWB is received, it is compared with the theoretical ranging value calculated based on the predicted pose to obtain the observation residual. The Kalman gain is used to feed this residual back to the predicted state, correcting the accumulated IMU error and outputting the final high-precision, high-frequency fused pose (x, y, z). An RMS error ≤ 8.7cm is a quantitative indicator of the fusion algorithm's performance.

[0037] Based on the dynamic weight value W(i,j) of the traversable grid in the 3D environment topology map, it is calculated using the following formula: W(i,j)=α·d occupancy (i,j)+β·r friction (i,j)+γ·c safety (i,j) In the formula, d occupancy(i,j) represents the instantaneous obstacle density of the grid (i,j) based on lidar point cloud inversion, r friction (i,j) represents the historical travel friction coefficient of grid (i,j), c safety (i,j) represents the distance cost from grid (i,j) to the preset safety point, and α, β, and γ are calibrated weighting coefficients. These parameters work together to comprehensively reflect the robot's state and environmental information, providing an accurate basis for subsequent task allocation.

[0038] The core of this step is to generate input data for subsequent decision-making, and the key lies in the calculation of dynamic weight W(i,j), which includes real-time LiDAR point cloud, historical access logs, and preset safety point coordinates.

[0039] First calculate the instantaneous obstacle density d occupancy (i,j): For the current frame point cloud P t Perform raster mapping. For a raster G(i,j), calculate the number of point clouds N inside it. occ Normalization: d occupancy (i,j)=N occ / N max In the formula N max Normalize the estimated maximum number of points that the grid can physically hold.

[0040] Then update the historical friction coefficient r. friction (i,j): Maintain a passage history matrix H. Whenever the robot successfully passes through a grid (i,j), record the increase in its motor current at that time, r. friction (i,j) represents the moving average or exponentially decaying average of the historical current increment data of this grid.

[0041] Then calculate the safety distance cost c safety (i,j): Calculate the distance from grid (i,j) to the nearest preset safe point (x). s , y s If the Euclidean distance D between c and c is given, then c safety (i,j)=1 / (D+ε), where ε is a very small constant to prevent division by zero errors. The closer the distance, the higher the cost, indicating that occupation should be avoided.

[0042] The final fused output is W(i,j)=α·d occupancy (i,j)+β·r friction (i,j)+γ·c safety (i,j), where α+β+γ=1, and the specific weights are calibrated through A / B testing to achieve the best balance between traffic efficiency and safety.

[0043] S2. Based on multi-source state information, a task allocation module deployed in the edge computing unit is used to dynamically allocate tasks and balance loads for several robots.

[0044] The task allocation module employs a distributed reinforcement learning task allocation engine, with a single robot acting as the agent. Its state space S includes its own battery level e. t Coordinates (x) t ,y t ,z t ), queue length l t and the set of other robot states within a neighborhood radius R = 8m {e k ,x k ,y k ,z k ,l k}, where the action space A is the task unloading target ID or keeping the original task, and the reward function R = -0.3·max(0,l) t -3)-0.5·dist(x t ,y t ,x target , y target )-0.2·W(x target ,y target This function evaluates the quality of actions by penalizing actions that are too long in their queues, those that are too far from the target point, and those that are located in high-cost areas. This guides the agent to learn a load-balanced, distance-optimal, and safe-passage strategy. A policy gradient-based reinforcement learning algorithm is used to periodically update the task allocation strategy on the edge computing unit using the state, action, and reward data sequences generated by the agent's interaction with the environment, thereby completing dynamic task redistribution decisions.

[0045] When any robot l t When the value is ≥3, the engine automatically traverses all l values ​​within the neighborhood R. k For robots with a W=0 value, the top three candidates are sorted in ascending order of Euclidean distance. Their target grid weighted distances (W values) are calculated, and the robot with the best overall score is selected to receive the excess task. Simultaneously, DWA local path planning parameters, including the start point, end point, and avoidance anchor points (from local minima of the W value), are simultaneously distributed. This dynamic task allocation method can rationally allocate tasks based on the robot's real-time status and environmental conditions, avoiding single-machine task overload and improving the utilization rate of nearby idle resources.

[0046] State encoding: At decision time t, for an agent, its own state [e t ,x t ,y t ,z t ,l tThe status of other robots within its communication range (R=8m) {[e k ,x k ,y k ,z k ,l k The concatenation of these elements forms a fixed-dimensional state vector S. t For robots outside the range, their state can be filled with zero or a specific value.

[0047] Policy network decision-making: The state vector S t Input a neural network (policy network π), the output of which is a probability distribution over all possible "actions", in this embodiment the actions include: {unload the current task to be assigned to robot R1, unload to R2, ..., unload to Rk, keep the task}.

[0048] Action sampling and execution: Based on the probability distribution output by the policy network, sample an action A. t For example, if the sampling result is "unload to robot Rj", then the engine sends task instructions and path anchor information to Rj.

[0049] Reward Calculation: After the action is performed, the immediate reward r is calculated based on the reward function. t : r t =-0.3 max(0,l t -3) #Punish queue for being too long.

[0050] -0.5 dist(x t ,y t ,x target ,y target #Punishment for being too far from the target.

[0051] -0.2 W(x target ,y target #The grid containing the penalty target point is difficult to traverse.

[0052] This reward function explicitly and quantitatively integrates three business metrics—load balancing, distance efficiency, and traffic safety—into the optimization objective, guiding the agent to learn the overall optimal allocation strategy, rather than a single nearest-principle approach.

[0053] Strategy Update: Every 200ms, utilize the collected (S) data during this period. t A t ,R tThe data sequence is used to calculate the "dominance function" to evaluate the quality of actions. The PPO algorithm updates the policy network parameters θ through a clipping optimization objective: L(θ) = E t [min(ratio t A t ,clip(ratio t ,1-ε,1+ε) A t )],in This process ensures that each policy update is not too drastic, resulting in more stable training. Ultimately, the policy network learns to select the action with the highest long-term cumulative reward in a given state.

[0054] Finally, a specific task assignment action instruction is output.

[0055] S3. Responding to external commands, the robot's operating mode is adaptively switched through a layered state machine architecture. The layered state machine architecture includes a bottom-level state machine, a middle-level scene recognition module, and a high-level semantic verification module. The state machine is defined as a station mode, a performance mode, and a free interaction mode. The station mode, performance mode, and free interaction mode are mutually exclusive.

[0056] The state transition table of the state machine is stored in the embedded Flash memory and supports OTA remote updates. State transitions are handled by the local MCU in response to external events via hardware interrupts.

[0057] State machine logic: State: Define a finite set of states, such as {FreeInteraction, Show, ChargingStation}.

[0058] Transition conditions: Define a Boolean trigger condition for each state transition.

[0059] FreeInteraction->ChargingStation: (BMS reported battery level < 15%) AND (Current status == FreeInteraction) Any status -> Show: (Received valid performance instructions) AND (Semantic validation passed) Show->FreeInteraction: (Performance task completed) OR (Received "Stop" command and verification passed) During state transition, the corresponding behavior control module is invoked.

[0060] Adaptive switching of robot operating modes using a hierarchical state machine architecture includes the following steps: S3.1 The scene recognition module processes the point cloud data of the LiDAR on the robot to identify the current scene features, and triggers scene tags when it identifies preset high-density crowd features.

[0061] The scene recognition module processes point cloud data from the robot's onboard LiDAR to identify current scene features, specifically including the following steps: S3.1.1 Perform voxel meshing on the current scan frame of the lidar.

[0062] S3.1.2 Count the number of point clouds within a preset radius centered on the robot, and generate a point cloud density heatmap.

[0063] S3.1.3 Calculate the average point cloud density in the central region based on the temporal variation of the point cloud density heatmap within the sliding time window.

[0064] S3.1.4 If the average point cloud density continuously exceeds the first density threshold, the "high-density crowd" scene label is triggered.

[0065] Mid-level scene recognition module logic: Input a 10Hz lidar point cloud frame.

[0066] Voxelization: The three-dimensional space is divided into a cubic grid with a side length of 0.3m, and each laser point is classified into the corresponding voxel.

[0067] Neighborhood statistics: Construct a spherical space with the robot's center as the center and a radius of 3m, and use spatial indexing to quickly retrieve all voxels falling within this sphere.

[0068] Density calculation: For each retrieved voxel, count the number of laser points it contains. Calculate the average point cloud density ρ of all voxels. current .

[0069] Maintain a first-in-first-out queue of length 5 to store the density values ​​[ρ] of the most recent 5 frames. t-4 ,...,ρ t ].

[0070] Judgment condition: (ρ t-1 >260)AND(ρ t >280), meaning that the density of two consecutive frames exceeds the threshold, the previous frame >260, the current frame >280.

[0071] If the condition is met, set the SCENE flag. CROWDED=1, and send an event message to higher levels. The technological advancement lies in the use of non-single-frame temporal continuous judgment and non-simple counting spatial density statistics, which improves the robustness of "crowd gathering" scene recognition and reduces false triggering caused by a single person's brief approach or sensor noise.

[0072] S3.2 The semantic verification module receives the speech-to-text instruction and determines whether to execute the instruction by calculating the similarity between the semantic vector of the instruction and the pre-stored instruction vector in the whitelist rule base corresponding to the current mode.

[0073] Step S3.2 specifically includes the following steps: S3.2.1 Input the voice command into the lightweight natural language processing model and output the command semantic vector.

[0074] Specifically, the text input is used to lightweight the BERT model, and the final layer output vector (768 dimensions) corresponding to the BERT [CLS] label is used as the semantic vector v of the instruction. cmd .

[0075] S3.2.2 Obtain at least one pre-stored rule semantic vector from the whitelist rule base corresponding to the robot's current mode.

[0076] Specifically, based on the robot's current mode, the corresponding whitelist instruction vector set {v} is loaded from the rule base. rule1 ,v rule2 ,...}.

[0077] S3.2.3 Calculate the cosine similarity sim between the instruction semantic vector and each rule semantic vector.

[0078] sim=(v cmd ·v rule ) / (||v cmd || ||v rule ||).

[0079] S3.2.4, If the maximum cosine similarity value sim max If the similarity is not less than the preset similarity threshold, the voice command will be executed; otherwise, execution will be refused and an error will be returned.

[0080] This hierarchical state machine architecture enables robots to accurately switch operating modes according to the scenario and instructions, improving the real-time response of the mode and the security of instruction execution.

[0081] S4. The target area is monitored in a grid pattern by a distributed infrared laser monitoring network deployed in key areas. Based on the monitoring results, different levels of congestion warnings are triggered, and corresponding robot response strategies are executed in conjunction with the monitoring. The response strategies include path replanning and task dispatch circuit breaker.

[0082] The distributed infrared laser monitoring network consists of several composite sensing nodes deployed in a three-dimensional grid in key areas. The composite sensing nodes include infrared pyroelectric sensors and laser ranging modules.

[0083] Triggering different levels of congestion warnings based on monitoring results includes the following steps: S4.1 Each composite sensing node performs fusion processing on the infrared pyroelectric signal and the laser ranging signal locally, and counts the number of valid human targets identified within the sliding window with a preset time length.

[0084] Step S4.1 specifically includes the following steps: S4.1.1 Filter the output signal of the infrared pyroelectric sensor to eliminate environmental temperature drift interference. S4.1.2 Determine whether the amplitude of the filtered pyroelectric signal exceeds a first voltage threshold and the duration exceeds a first time threshold.

[0085] S4.1.3 Determine that the distance value measured by the laser ranging module is within the preset effective human body distance range, and that the variance of the measured distance is less than the preset variance threshold.

[0086] S4.1.4 When the judgment conditions of the laser ranging signal in steps S4.1.2 and S4.1.3 are met simultaneously, it is determined that a valid human target has been identified.

[0087] Parallel signal processing: Infrared channel: The pyroelectric sensor ADC signal is subjected to Butterworth low-pass filtering (fc=0.5Hz) to remove high-frequency noise and rapid temperature fluctuations, retaining low-frequency signals that approximate human movement. The filtered signal V is then evaluated. filtered >1.2V and duration ≥0.8s.

[0088] Laser Channel: Read the laser ranging value d and calculate the variance σ of the most recent N measurements. 2 Determine if d ∈ [0.45, 0.55]m and σ 2 <0.002 (the target is relatively stationary or moving slowly).

[0089] Fusion Decision: Using AND logic, a "valid human target" is identified only when the judgment conditions of both the infrared channel and the laser channel are true. The counter count is incremented by 1. Through multimodal signal fusion and continuous judgment, environmental and small animal interference is effectively eliminated, and the accuracy of single node detection is greatly improved.

[0090] S4.2 When the number of valid human targets in the first consecutive sliding windows is not less than the first quantity threshold, a level one congestion warning is triggered.

[0091] S4.3 When a Level-1 congestion warning is triggered, if the number of valid human targets in the second consecutive number of sliding windows is not less than the second number threshold which is higher than the first number threshold, a Level-2 congestion warning is triggered.

[0092] If the robot receives a Level 1 congestion warning, it immediately invokes the path planning algorithm to replan the path to avoid the area corresponding to the warning grid coordinates. This includes the following steps: Parse the congestion grid center coordinates (x) from the message. g , y g ).

[0093] In the robot's costmap, (x) g , y g Set the cost of a circular area with a center and a radius of 1.5m to "Fatal Obstacle" (impassable).

[0094] Immediately call A The global planner replans the path from the current point to the target point based on the new costmap.

[0095] If the robot and the central control station receive a Level 2 congestion warning, the task assignment circuit breaker mechanism will be triggered, suspending the assignment of new tasks to the affected robots. The specific steps include: Execute all responses to the Level 1 congestion warning mentioned above.

[0096] Set a software lock flag in the robot's local task manager. block_dispatch =1.

[0097] When the Task Manager receives a new task from the central console, it first checks this lock. If it is 1, it directly replies "Busy, refuse to receive", thus implementing task circuit breaking.

[0098] The lock can be unlocked by the CLEAR_BLOCK command on the central control panel, or it can automatically unlock after a timeout of 180 seconds.

[0099] Window management: Each node maintains a sliding time window with a length of 30 seconds, and the current count value is added to the window's cumulative value N every second. window In the middle, remove the count value from the second 30 seconds ago.

[0100] Graded judgment: Level-1 warning conditions: N window The condition is met if the value is >=15 and there are 3 consecutive sliding windows of this value (i.e., for a duration of 90 seconds).

[0101] Level-2 warning conditions: Based on the activation of Level-1, Nwindow The condition is met if the value is greater than or equal to 25 and the next two consecutive sliding windows (i.e., for another 60 seconds) are also met.

[0102] When the conditions are met, a warning message (GRID) is broadcast via the CAN bus. OVERLOAD_ALERT ,grid id ,level).

[0103] This distributed monitoring network and hierarchical response mechanism can effectively provide early warnings of congestion, reduce the congestion rate in key areas, and improve the accuracy of alarms from the central control console.

[0104] The following example, using a smart service station in East China, illustrates the technical solution of this application. Ten service robots (numbered R1-R10) are deployed to perform tasks such as product display, interactive guidance, and lightweight item delivery. A 3D grid map of the environment has been constructed using SLAM, and a distributed infrared laser monitoring network has been deployed in key areas. The edge server is equipped with a reinforcement learning task allocation model based on a policy gradient algorithm, and initial training has been completed.

[0105] Initial state (time t0): Task: Create a new task T, which requires delivering a promotional gift from the "Service Desk" (coordinates (5, 10)) to "Seat A" (coordinates (50, 30)).

[0106] Relevant robot states (reported in real time by the multi-source state perception layer): R1: Battery level e=65%, coordinates (8, 12), task queue length l=2 (currently executing 2 short-range bootstrap tasks).

[0107] R2: Battery level e = 90%, coordinates (6, 11), task queue length l = 0 (idle).

[0108] R3: Battery level e=40%, coordinates (48, 28), task queue length l=1 (currently executing a task nearby).

[0109] Environmental dynamic weights: Calculated based on real-time point cloud and historical data, the weight W of the grid cell (50, 30) near the target point "Seating A". target =0.7, due to its proximity to the restroom passage, safety cost csafety The weight W of the current position grid (6,11) of R2 is relatively high. R2 =0.2 (open corridor).

[0110] When the edge server receives a new task T, the allocation engine is triggered to select the optimal robot to handle the task T.

[0111] Status Code: For a candidate robot, the engine constructs its state vector S.

[0112] For R1, its own state is [65, 8, 12, 2] (battery power, x, y, queue length). The states of other robots in its neighborhood, such as the state of nearby R2 [90, 6, 11, 0], are also included. These values, after normalization, together constitute the state feature vector S input to the reinforcement learning model. t (R1).

[0113] Similarly, construct the state feature vector S of R2. t (R2).

[0114] Model inference and action probability: S t (R1) and S t (R2) Input the trained policy network, and the network output is a "recommendation rating" or "selection probability" for each robot. Assume the model outputs the following after inference: P(R1)=0.25 P(R2)=0.70 P(other robots) = 0.05 The model tends to choose R2.

[0115] Basis for decision: The model's bias stems from its learned policy, which aims to maximize long-term cumulative rewards. We can understand why choosing R2 is better by calculating the immediate reward. Based on our defined reward function: R=-0.3 max(0,l t -3)-0.5 dist(...)-0.2 W(...) Calculate for R1: Load penalty: -0.3 max(0,2-3)=0 (The queue has not exceeded the threshold) Distance cost: -0.5 dist((8,12),(50,30))≈-0.5 42.2≈-21.1 (relatively far) Safety weight cost: -0.2 0.7 = -0.14 (Target point has high weight) The estimated instantaneous reward R(R1) ≈ -21.24 Calculate for R2: Load penalty: 0 (idle) Distance cost: -0.5 dist((6,11),(50,30))≈-0.5 43.6 ≈ -21.8 Safety weight cost: -0.2 0.7 = -0.14 (same target point) Estimated instantaneous reward R(R2)≈-21.94 Although R2 is slightly further away, its load is 0, and its overall evaluation is better than R1, which already has two tasks. More importantly, the reinforcement learning model learns a long-term policy that considers the global state during training. It may "know" that R1 is about to complete the current task and is more suitable for handling the upcoming atrium performance guidance task. Therefore, even if the immediate rewards are not much different, it makes a better global allocation decision.

[0116] The engine makes the final decision: assign task T to R2, and at the same time calculate the "anchor points" (10,15) (W=0.1) with lower dynamic weights on the path from the current position of R2 to the task target point, and generate local path planning parameters and send them to R2.

[0117] On the way to the destination, R2 passes through the central performance area.

[0118] Mid-level scene recognition: The R2 lidar continuously scans, and after voxelizing a frame of point cloud, it calculates the average point cloud density ρ of voxels within a radius of 3 meters.

[0119] Timing decision: The ρ values ​​of 5 consecutive frames are [265, 272, 285, 290, 288] pts / m³.

[0120] Logical judgment: (ρ[2]=285>280)&&(ρ[3]=290>280)&&(ρ[4]=288>280) condition is true.

[0121] The scene recognition module determines that it is a "high-density crowd" and sends a SCENE_CROWDED event to the high-level semantic verification module. At this time, the underlying state machine may still be in the "free interaction" mode.

[0122] At this moment, a tourist said to R2 Voice, "Robot, show us a dance!" Edge-based speech recognition (ASR) transcribed it into text: "Robot, show us a dance."

[0123] The text is input to a lightweight BERT model, which outputs a 768-dimensional semantic vector v. cmd .

[0124] The system queries the whitelist instruction vector set in the current mode, which includes vectors such as ["Hello", "What time is it", "Take the way"], and calculates v.cmd Cosine similarity to the "dance" instruction vector.

[0125] v cmd The maximum similarity sim with any vector in the "free interaction" whitelist max =0.45, semantic mismatch, validation fails because 0.45 < 0.85 (threshold).

[0126] The system refuses to execute the dance command, the robot's LED light flashes red slowly as a warning, and it may reply with a voice message: "Sorry, I am currently unable to perform." This effectively prevents the safety risks caused by accidentally executing performance commands in crowded passageways.

[0127] Almost simultaneously, congestion was detected by the distributed monitoring network in a key area at the entrance to "Section A".

[0128] A grid node (Grid) 15 The pyroelectric sensor detected an effective heat source signal (V) lasting 1.2 seconds. out >1.2V).

[0129] Its laser ranging module detected an obstacle 0.48 meters ahead, and the measurement variance σ 2 =0.0015 (stable).

[0130] Both conditions must be met simultaneously for Grid to function. 15 If a node is determined to be a valid human target, the local counter count is incremented by 1.

[0131] Statistics for all nodes in this area are compiled in 30-second windows. 15 The number of valid targets N counted by the node and its five adjacent nodes within the most recent 30-second window. window They are [18, 20, 15, 22, 19, 16].

[0132] Level-1 warning condition judgment: all 6 nodes N window All values ​​are ≥15, and this condition is met for three consecutive statistical periods (i.e., the past 90 seconds).

[0133] Trigger: When the condition is met, the monitoring network broadcasts a Level-1 warning to all robots in the area via the CAN FD bus: {type:GRID} OVERLOAD_ALERT ,level:1,grid id :[Grid 15 [Surrounding area], coordinates:(48, 29)}.

[0134] Robot response: R2, which is heading towards (50, 30), and R3, which is nearby, both received the warning.

[0135] R2's path planner immediately designated the area centered at the warning coordinates (48,29) with a radius of 1.5 meters as a temporary obstacle and called A. The algorithm replans the path, and the new path detours through the left-hand passage, adding a new path point (45, 28).

[0136] At this time, the warning is Level-1, so the task circuit breaker is not triggered and R2 continues to execute the task. If the warning is upgraded to Level-2 in the future, the central control station will send a "pause receiving new tasks" command to the robot in that area.

[0137] This embodiment provides comprehensive and accurate data support for dynamic task allocation by constructing multi-source state information. This allows task allocation to be dynamically adjusted based on the robot's real-time status and environmental conditions, thereby achieving load balancing, reducing single-machine task overload rate, and improving the utilization rate of nearby idle resources. The hierarchical state machine architecture's adaptive switching of operating modes, through the collaborative work of the state machine, scene recognition module, and semantic verification module, enables the robot to quickly and accurately switch operating modes according to the scene and instructions, improving the real-time nature of mode response and the security of instruction execution. The congestion warning and response mechanism of the distributed infrared laser monitoring network can monitor congestion in key areas in real time, promptly trigger different levels of warnings and take corresponding response strategies, effectively reducing the congestion rate in key areas, improving the accuracy of central control alarms, and overall enhancing the robot's task execution efficiency and reliability in the intelligent demonstration station scenario. This represents a significant improvement and enhancement compared to existing technologies.

[0138] Example 2: This application provides a multimodal intelligent dynamic scheduling system for intelligent performance stations, applicable to the multimodal intelligent dynamic scheduling method for intelligent performance stations in Embodiment 1. The system includes several robots, a distributed infrared laser monitoring network deployed in key areas, and a central control station communicating with the robots and the distributed infrared laser monitoring network. The robots, the distributed infrared laser monitoring network, and the central control station cooperate with each other. The robots receive early warning signals from the distributed infrared laser monitoring network and execute corresponding operations. The distributed infrared laser monitoring network monitors the situation in key areas in real time and issues early warnings. The central control station coordinates the work of the robots and the monitoring network, achieving the effects of improving robot task execution efficiency, reducing congestion rates, and improving command execution security.

[0139] The robot comprises a multi-source state perception module, an environment modeling module, a reinforcement learning task allocation engine, a hierarchical state machine control module, and a path planning and execution module. The multi-source state perception module acquires real-time battery power, localization coordinates, task queue length, and other multi-source state information, and has a built-in battery management unit to monitor real-time battery power. The environment modeling module constructs a 3D grid map based on SLAM and calculates the dynamic weight values ​​of passable grids. The reinforcement learning task allocation engine uses a distributed reinforcement learning algorithm to dynamically allocate tasks and balance load based on multi-source state information. The hierarchical state machine control module includes a state machine unit, a scene recognition unit, and a semantic verification unit. The state machine unit defines station mode, performance mode, and free interaction mode and performs state transitions. The scene recognition unit processes LiDAR point cloud data to identify scene features. The semantic verification unit calculates the similarity between the semantic vector of a voice command and pre-stored vectors in a whitelist rule base to determine whether to execute the command. The path planning and execution module plans the path and executes tasks based on task allocation and warning information.

[0140] The distributed infrared laser monitoring network comprises several sensing nodes, each consisting of a microcontroller, an infrared pyroelectric sensor, and a laser ranging module. The detection axes of the infrared pyroelectric sensor and the laser ranging module are arranged at a preset angle. The microcontroller is configured to run signal fusion firmware to locally perform the identification of valid human targets, counting of sliding windows, and determination of warning levels. For example, the microcontroller runs the FreeRTOS operating system, filters the infrared pyroelectric signal, combines it with the laser ranging signal to determine whether a valid human target has been identified, counts the number of valid human targets within the sliding window, and triggers different levels of congestion warnings based on the number.

[0141] The central control console communicates with multiple robots and a distributed infrared laser monitoring network, receives early warning information from the monitoring network, coordinates task allocation and path planning for the robots, and triggers a task dispatch circuit breaker mechanism when a level 2 congestion warning is received, suspending the dispatch of new tasks to the affected robots.

[0142] This embodiment provides a multimodal intelligent dynamic scheduling system that, through the collaborative work of its various components, achieves dynamic allocation of robot tasks, adaptive switching of operating modes, and effective early warning and response to congestion. The multi-source state perception module and environment modeling module provide accurate information for task allocation, and the reinforcement learning task allocation engine rationally allocates tasks based on this information, improving the robot's load balancing. The hierarchical state machine control module enables the robot to flexibly switch operating modes according to the scenario and instructions, improving the safety of instruction execution and the real-time performance of mode response. The distributed infrared laser monitoring network monitors congestion in key areas in real time, issuing timely warnings. The central control console coordinates the work of each component to ensure the system's efficient operation. Compared with existing technologies, this system significantly improves the task execution efficiency and reliability of robots in intelligent demonstration station scenarios.

[0143] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A multimodal intelligent dynamic scheduling method for intelligent performance stations, characterized in that, Includes the following steps: S1. Construct multi-source state information for each robot, including real-time battery level, positioning coordinates, task queue length, and dynamic weight values ​​of passable grids in a three-dimensional environment topology map. S2. Based on the multi-source state information, dynamic task allocation and load balancing are performed for several robots through the task allocation module deployed in the edge computing unit. S3. In response to external commands, the robot's operating mode is adaptively switched through a hierarchical state machine architecture. The hierarchical state machine architecture includes a state machine, a scene recognition module, and a semantic verification module. The state machine is defined as a station mode, a performance mode, and a free interaction mode. The station mode, performance mode, and free interaction mode are mutually exclusive. S4. The target area is monitored in a grid pattern by a distributed infrared laser monitoring network deployed in key areas. Based on the monitoring results, different levels of congestion warnings are triggered, and corresponding robot response strategies are executed in conjunction. The response strategies include path replanning and task dispatch circuit breaker.

2. The multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 1, characterized in that, In step S3, the adaptive switching of the robot's operating mode through a hierarchical state machine architecture specifically includes the following steps: S3.1 The scene recognition module processes the point cloud data of the lidar on the robot to identify the current scene features, and triggers scene tags when it identifies preset high-density crowd features; S3.2 The semantic verification module receives the speech-to-text instruction and determines whether to execute the instruction by calculating the similarity between the semantic vector of the instruction and the pre-stored instruction vector in the whitelist rule base corresponding to the current mode.

3. The multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 2, characterized in that, The scene recognition module identifies current scene features by processing point cloud data from the LiDAR on the robot, specifically including the following steps: S3.1.1 Perform voxel meshing on the current scan frame of the lidar; S3.1.2 Count the number of point clouds within voxels centered on the robot and within a preset radius, and generate a point cloud density heatmap; S3.1.3 Calculate the average point cloud density of the central region based on the temporal variation of the point cloud density heatmap within the sliding time window; S3.1.4 If the average point cloud density continues to exceed the first density threshold, the "high-density crowd" scene label is triggered.

4. The multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 2, characterized in that, Step S3.3 specifically includes the following steps: S3.2.1 Input the voice command into the lightweight natural language processing model and output the command semantic vector; S3.2.2 Obtain at least one pre-stored rule semantic vector from the whitelist rule base corresponding to the robot's current mode; S3.2.3 Calculate the cosine similarity between the instruction semantic vector and each of the rule semantic vectors; S3.2.4 If the maximum cosine similarity value is not less than the preset similarity threshold, then execute the voice command; otherwise, refuse to execute and return an error.

5. A multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 1, characterized in that, The distributed infrared laser monitoring network includes several composite sensing nodes deployed in a three-dimensional grid in key areas. Each composite sensing node includes an infrared pyroelectric sensor and a laser ranging module. The process of triggering different levels of congestion warnings based on the monitoring results specifically includes the following steps: S4.1 Each composite sensing node performs fusion processing on infrared pyroelectric signals and laser ranging signals locally, and counts the number of valid human targets identified within the sliding window with a preset time length. S4.2 When the number of valid human targets in the first consecutive sliding windows is not less than the first number threshold, a level one congestion warning is triggered. S4.3 When a Level 1 congestion warning is triggered, if the number of valid human targets in a second consecutive number of sliding windows is not less than a second number threshold that is higher than the first number threshold, a Level 2 congestion warning is triggered.

6. A multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 5, characterized in that, If the robot receives a Level 1 congestion warning, it immediately invokes a path planning algorithm to replan the path in order to avoid the area corresponding to the warning grid coordinates.

7. A multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 5, characterized in that, If the robot and the central control station receive a Level 2 congestion warning, the task assignment circuit breaker mechanism will be triggered to suspend the assignment of new tasks to the affected robot.

8. A multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 5, characterized in that, Step S4.1 specifically includes the following steps: S4.1.1 Filter the output signal of the infrared pyroelectric sensor to eliminate environmental temperature drift interference; S4.1.2 Determine that the amplitude of the filtered pyroelectric signal exceeds the first voltage threshold and the duration exceeds the first time threshold; S4.1.3 Determine that the distance value measured by the laser ranging module is within the preset effective human body distance range, and that the variance of the measured distance is less than the preset variance threshold. S4.1.4 When the judgment conditions of the laser ranging signal in steps S4.1.2 and S4.1.3 are met simultaneously, it is determined that a valid human target has been identified.

9. A multimodal intelligent dynamic scheduling method for intelligent performance stations according to claim 1, characterized in that, The dynamic weight value W(i,j) of the traversable grid in the 3D environment topology map mentioned in step S1 is calculated using the following formula: W(i,j)=α·d occupancy (i,j)+β·r friction (i,j)+γ·c safety (i,j) In the formula, d occupancy (i,j) represents the instantaneous obstacle density of the grid (i,j) based on lidar point cloud inversion, r friction (i,j) represents the historical travel friction coefficient of grid (i,j), c safety (i,j) represents the distance cost from grid (i,j) to the preset safety point, and α, β, and γ are calibrated weighting coefficients.

10. A multimodal intelligent dynamic scheduling system for intelligent performance stations, applicable to the multimodal intelligent dynamic scheduling method for intelligent performance stations as described in any one of claims 1-9, characterized in that, The system includes several robots, a distributed infrared laser monitoring network deployed in key areas, and a central control unit that communicates with the robots and the distributed infrared laser monitoring network. Each robot includes a multi-source state perception module, an environment modeling module, a reinforcement learning task allocation engine, a hierarchical state machine control module, and a path planning and execution module. The robot is configured to receive early warning signals from the distributed infrared laser monitoring network and perform corresponding path replanning and task circuit breaker operations.

Images