VLA dynamic multi-task real-time decision-making system of small service robot

Abstract

The invention discloses a small service robot VLA dynamic multi-task real-time decision-making system, which relates to the technical field of robots and comprises an acquisition end, a storage module, a feature extraction module, a scheduling module, a decision-making module, a precise action execution analysis module and an execution end. The VLA dynamic multi-task real-time decision-making system of the small service robot is based on event-driven and spiking neural level scheduling, and realizes personalized dynamic multi-task decision-making and execution with low power consumption, high real-time performance and high accuracy. The method is suitable for a small service robot which needs to quickly respond to a dynamic environment, efficiently process multi-instruction conflicts, stably operate with low power consumption and have personalized interaction capability.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a VLA dynamic multi-task real-time decision-making system for a small service robot. Background Technology

[0002] As service robot technology iterates towards lightweighting and intelligence, small service robots, due to their compact size and flexible operation, have become core carriers in scenarios such as daily home care and office assistance. The core interaction logic of these robots relies on the Visual Language Action (VLA) model, requiring a closed-loop collaboration of "visual environment perception - natural language understanding - precise action execution." The maturity of this technology directly determines the user experience. However, since its emergence in 2021, the VLA model, despite undergoing three development stages—basic integration (2022-2023), specialized entity reasoning (2024), and generalization and secure deployment (2025)—still faces multiple technical bottlenecks in its application to small service robot scenarios. Existing solutions struggle to balance real-time performance, reliability, low power consumption, and personalized requirements. Specific pain points are as follows: 1. The response to dynamic scenes is lagging, making it difficult to adapt to real-time interaction requirements. Traditional VLA models generally employ the Transformer architecture, which inherently suffers from "heavy parameters and high latency." For example, Google's early classic VLA model, RT-2-PaLI-X, had a parameter scale of 55B and an inference latency exceeding 300ms. Even with cloud GPU support, it could not meet the "millisecond-level response" interaction requirements of small service robots (such as users temporarily changing commands or avoiding sudden obstacles). Small service robots, limited by hardware, typically use low-power microcontroller units (MCUs) (such as the STM32H743), whose computing power is only one-thousandth that of high-end GPUs. This makes them unable to handle the batch computation logic of traditional VLAs, further amplifying response latency to ≥20ms in dynamic environments (such as object movement or sudden changes in user actions), easily leading to action execution errors or task interruptions. Although recent high-efficiency VLA models (such as RoboMamba) have compressed latency to 63ms through Mamba architecture optimization, they still haven't solved the problem of "instant response to sudden environmental changes." Furthermore, their core optimizations focus on industrial scenarios and lack adaptation mechanisms designed for the rapid decision-making needs of dynamic home scenarios (such as children running or pets crossing).

[0003] 2. The multi-task scheduling mechanism is rigid, resulting in frequent conflicts and low efficiency. In home / office scenarios, users often issue parallel commands (such as "get a file + adjust the air conditioner" or "pass a water glass + close the curtains"). However, existing VLA models have significant shortcomings in multitasking capabilities: on the one hand, they lack dynamic priority scheduling logic and execute commands only in the order they are received, which can easily lead to action conflicts (such as a robotic arm attempting to grasp and turn simultaneously); on the other hand, they do not consider task correlation and execution constraints, resulting in unbalanced resource allocation. Similar to the problem of "30% of devices running overload and 20% of devices being idle" in warehouse automated guided vehicles (AGV) clusters due to scheduling algorithm defects, a similar issue exists in small service robots—for example, when high-priority tasks (such as obstacle avoidance) conflict with low-priority tasks (such as tidying up a desktop), existing systems cannot quickly suppress the low-priority task, which can easily lead to safety hazards. In addition, mainstream VLA models (such as OpenVLA) rely on a fixed rule base for multitasking and cannot dynamically adjust task priorities according to user habits, further reducing multitasking efficiency.

[0004] 3. Poor anti-interference ability and poor adaptability to complex home environments. Home environments present multiple sources of interference, directly impacting the perception accuracy of VLA models: Optically, low light (illuminance <100 lux), high light (illuminance >10000 lux), and shadow occlusion can easily lead to distortion in visual feature extraction; acoustically, background noise from televisions / appliances (frequency 200-3000Hz) and interference from family conversations can cause errors in language command recognition; semantically, users' common dialects and ambiguous expressions (such as "hand over that red one") further increase the difficulty of understanding. Traditional VLA models use fixed-threshold feature extraction logic in their perception modules, lacking anti-interference designs specific to home environments, resulting in cross-modal feature extraction accuracy generally ≤85%. For example, in scenarios with both low light and background noise, traditional filtering algorithms and noise suppression models struggle to effectively distinguish between valid and interfering signals, significantly increasing the command misrecognition rate and directly affecting the accuracy of action execution.

[0005] 4. The power consumption is not matched with the computing power, and the battery life is insufficient to meet practical needs. Battery capacity for small service robots is typically limited to 1000-2000mAh, while the computational demands of traditional VLA models are vastly different from the hardware capabilities of small devices. On one hand, model training is extremely costly; full training of OpenVLA requires 21,500 A100-GPU hours, and even in edge deployment, a 7-parameter VLA model consumes ≥150mA on an MCU. On the other hand, traditional VLA employs a "continuous data processing" mode, continuously activating all computing units regardless of the presence of valid signals, resulting in over 40% of power being wasted. This directly leads to existing small service robots equipped with VLA generally having a battery life of ≤6 hours, making it difficult to meet the continuous service needs of home use throughout the day or office scenarios. Although efficient VLA has reduced power consumption by 30%-50% in recent years through techniques such as 4-bit quantization and layer pruning, a low-power solution covering the entire "perception-scheduling-execution" chain for small robots has not yet been established, and power consumption rebounds significantly in multi-tasking scenarios.

[0006] 5. The execution of actions lacks hardware compatibility, resulting in insufficient accuracy and safety. The mechanical structure of small service robots has inherent constraints: load capacity is typically ≤1kg, joint degrees of freedom are mostly 5-7DoF, repeatability is approximately ±0.5mm, and braking response has physical delay. Existing VLA models' motion generation modules are generally based on general robot designs, failing to fully consider the mechanical characteristics of small devices. For example, ignoring load limitations and directly outputting grasping commands may lead to overload and damage to the robotic arm; unoptimized joint motion trajectories result in a motion stuttering rate exceeding 30%; and the lack of load feedback and force adjustment mechanisms makes them prone to damage when grasping fragile items. Furthermore, traditional VLA's decision-making and execution operate in an "open-loop mode," unable to adjust motion parameters based on real-time environmental changes (such as object position shifts), further reducing the accuracy and safety of motion execution.

[0007] 6. Lack of personalized adaptation capabilities and severe homogenization of interactive experience. Significant differences exist in the usage habits of different user groups (elderly, young people, and children): the elderly have slower movement rhythms (raising their hands 50% slower than young people) and use concise commands; young people prefer quick responses and multitasking; children's commands are often semantically ambiguous and difficult to predict. Existing VLA models employ fixed execution logic and interaction rhythms, lacking user behavior learning mechanisms: for example, insufficient waiting time is not provided when performing actions on the elderly, resulting in the action being completed before the user is ready; ambiguous commands for children (such as "I want that fun thing") cannot be accurately mapped to specific actions. According to industry research, 68% of users reported that existing small service robots "don't understand me enough," and insufficient personalized adaptation has become one of the key factors restricting user acceptance. Summary of the Invention

[0008] The purpose of this invention is to design a VLA dynamic multi-task real-time decision-making system for small service robots in order to solve the above-mentioned problems.

[0009] The present invention achieves the above objectives through the following technical solutions: A VLA (Dynamic Multitasking) real-time decision-making system for small service robots includes: The acquisition end is used to collect task information and sensor information in real time. The task information includes visual information and voice information. Storage module; The storage module stores an interference feature library, a "fuzzy command-entity action" dynamic mapping library, and a "command-pulse intensity" mapping library; Feature extraction module; The feature extraction module extracts the pulse signal of the task information based on the interference feature library, analyzes the intensity level of the pulse signal, and outputs the pulse signal with intensity level; The scheduling module prioritizes pulse signals with intensity levels and sorts them with other current tasks to generate an execution order. Decision module; The decision module generates a basic execution path based on the execution order, and performs path pre-simulation verification to generate the execution instructions for the optimal execution path; Motion Precision Execution Analysis Module: This module dynamically adjusts motion parameters based on execution commands and sensor information to generate joint movement commands for the robot. The execution end; the execution end performs tasks according to joint movement commands.

[0010] The beneficial effects of this invention are as follows: The VLA dynamic multi-task real-time decision-making system for small service robots is based on event-driven and spiking neural hierarchical scheduling, which realizes low power consumption, high real-time performance, high accuracy, and personalized dynamic multi-task decision-making and execution. It is suitable for small service robots that need to respond quickly to dynamic environments, efficiently handle multiple instruction conflicts, operate stably with low power consumption, and have personalized interaction capabilities. Detailed Implementation

[0011] In the description of this invention, it should be understood that the terms "upper," "lower," "inner," "outer," "left," and "right," etc., indicate the orientation or positional relationship as commonly used when the product of this invention is in use, or the orientation or positional relationship commonly understood by those skilled in the art. They are only used to facilitate the description of this invention and to simplify the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0012] Furthermore, the terms "first," "second," etc., are used only to distinguish descriptions and should not be interpreted as indicating or implying relative importance.

[0013] In the description of this invention, it should also be noted that, unless otherwise explicitly specified and limited, terms such as "set" and "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium; it can be a connection within two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0014] The specific embodiments of the present invention will be described in detail below.

[0015] A VLA (Dynamic Multitasking) real-time decision-making system for small service robots includes: The acquisition end is used to acquire task information and sensor information in real time. The task information includes visual information and voice information. The acquisition end includes an event camera for capturing dynamic pixel events in the home scene and outputting discrete event streams, a miniature microphone, an integrated ambient light sensor for real-time detection of scene illumination, a force-sensitive load sensor for task load determination and force feedback, and a lightweight visual motion capture unit for capturing the rhythm of user movements. The event camera has a high dynamic range of ≥120dB and microsecond-level time resolution; the miniature microphone supports a wide frequency response (covering 200-3000Hz) and is compatible with dialects and fuzzy commands; the force-sensitive load sensor has a detection accuracy of ≥0.1N. Storage module; The storage module stores an interference feature library, a "fuzzy instruction-entity action" dynamic mapping library, and an "instruction-pulse intensity" mapping library; The interference feature library adopts a lightweight storage format (each feature vector occupies ≤64 bytes), which is adapted to the storage resource limitations of small service robots (≥4MB cache pool can fully support it); Feature extraction module; The feature extraction module extracts the pulse signal of the task information based on the interference feature library, analyzes the intensity level of the pulse signal, and outputs the pulse signal with intensity level; The scheduling module prioritizes pulse signals with intensity levels and sorts them with other current tasks to generate an execution order. Decision module; The decision module generates a basic execution path based on the execution order, and performs path pre-simulation verification to generate the execution instructions for the optimal execution path; The decision module is activated only when a new pulse signal is input (new instructions / environmental changes), and the decision module is an SNN decision network; Motion Precision Execution Analysis Module: This module dynamically adjusts motion parameters based on execution commands and sensor information to generate joint movement commands for the robot. The actuator executes tasks according to joint movement commands. The actuator includes a small, high-precision servo motor adapted to 5-7DoF joint movement, a motor drive module supporting direct drive of pulse commands, and a braking energy recovery module with back EMF recovery function. The repeatability of the small, high-precision servo motor is ≤±0.3mm. The braking energy recovery module can feed back the electrical energy generated during movement / braking to the power supply system. Closed-loop feedback module; The closed-loop feedback module is used to correct the joint motion commands of the execution end during task execution, specifically: A. Acquire sensor information in real time and convert it into feedback pulse signals. The sensor information includes visual, force, and tactile information. B. Determine the validity of the deviation based on the level of feedback pulse and locate the type of deviation (path / force / attitude). C. The closed-loop feedback module automatically adjusts the intensity and timing of joint motion commands using the pulse timing-dependent plasticity (STDP) rule to obtain the corrected joint motion commands. Specifically, if the corrected joint motion command is earlier than the feedback pulse (Δt≤20ms), the synaptic weight is enhanced (LTP), and vice versa (LTD). The weight change (Δw) is calculated according to a fixed formula. Intensity adjustment: Weight enhancement → pulse intensity increase (maximum 50%), weight reduction → intensity decrease (maximum 30%), adapting to force / speed deviation; Timing adjustment: weight enhancement → pulse interval shortening (speeding up the action), weight reduction → interval lengthening (slowing down the action), adapting to path / attitude deviations.

[0016] The interference feature library in the storage module is constructed based on SNN synaptic plasticity, specifically including: ① Collect interference data from home scene interference sources. Select typical optical interference sources in home / office scenarios, including fixed interference (TV screen flickering, LED light brightness fluctuations, projector light and shadow changes), environmental interference (light and shadow of swaying leaves outside the window, changes in wall reflection), and temporary interference (sudden changes in mobile phone screen brightness, handheld light source movement), covering three illuminance environments: low light (≤100 lux), normal light (100-1000 lux), and strong light (≥10000 lux). For each interference source, continuously collect 10-20 sets of valid data (each set lasting 30 seconds) using an event camera, simultaneously recording label information such as ambient illuminance, interference source type, and interference intensity to form the original interference dataset; ② The core pulse features of each interference source are extracted using the pulse time coding characteristics of the SNN feature extraction network. Reinforcement learning is then applied to these core pulse features using SNN synaptic plasticity (STDP rule). Stably recurring core pulse features are combined and encoded to generate interference feature vectors, with each interference source corresponding to a unique feature vector. Core pulse features include temporal, intensity, and spatial features. Temporal features include the pulse firing interval (e.g., a fixed pulse interval of 20ms for television flicker interference) and the pulse sequence period (e.g., a pulse period of 50ms for LED light fluctuations). The intensity features include the pulse peak intensity range (e.g., the pulse intensity of a sudden change in a mobile phone screen is 1.2-1.5V) and the pulse density change rate (e.g., the pulse density fluctuation of the shadow of swaying leaves is ±30% / s); the spatial features include the distribution of pixel regions activated by the pulse (e.g., the pulses of wall reflection are concentrated in the upper right 1 / 4 area of ​​the image). The architecture of the SNN feature extraction network is shown in Table 1. The SNN feature extraction network consists of an input layer, a pulse convolutional layer 1, a pulse pooling layer 1, a pulse convolutional layer 2, a pulse pooling layer 2, a feature fusion layer, and an output layer from input to output. Table 1: SNN Feature Extraction Network Architecture is as follows

[0017] ③ Construct an interference feature library according to a three-level structure of "interference type - illumination scene - feature vector", and optimize the interference feature library regularly. The three-level structure of the interference feature library is as follows: first-level classification, second-level classification, and third-level storage. The first-level classification is based on the type of interference, including fixed optical interference, ambient optical interference, and temporary optical interference. The second-level classification is based on the illumination scene, such as fixed optical interference - weak light and fixed optical interference - strong light. The third-level storage stores the corresponding interference feature vector, interference source label, and confidence threshold (default 85%, used for subsequent matching and judgment) for each category. ④ During system operation, when the feature extraction module detects an unmatched novel pulse feature (appearing 5 times consecutively, and the similarity between the newly appearing unmatched novel pulse features is <60%), interference learning is triggered to update the interference feature library. The interference learning specifically involves recording the pulse timing, intensity, and spatial distribution data of the novel pulse feature, and simultaneously collecting environmental labels. Then, after user confirmation, the novel pulse feature is included in the interference feature library and assigned a classification label.

[0018] The feature extraction module includes a visual extraction module, a language extraction module, and an encoding module. The visual extraction module is used to extract effective pulse signals from visual information, specifically: 1) Dynamically adjust the filtering threshold based on the ambient light intensity of the visual information, and use the filtering threshold to initially filter out invalid signals in the visual information; the dynamic adjustment of the filtering threshold is as follows: Low light scene (≤100 lux): the threshold is reduced by 30% to retain low-intensity effective pulses (such as slow user movements) and filter out extremely weak noise; Strong light scene (≥10000 lux): the threshold is increased by 50% to filter out saturation pulses caused by overexposure of strong light; Normal light scene (100-1000 lux): the default threshold is maintained to balance the retention of effective signals and noise filtering; 2) Use an SNN feature extraction network to extract real-time pulse feature vectors from the pre-filtered task information; 3) Analyze the similarity between the real-time pulse feature vector and all interference feature vectors in the interference feature library, and determine whether the real-time pulse feature vector is an interference signal based on the similarity threshold; specifically: the similarity calculation is based on SNN synaptic weight mapping: if the synaptic activation pattern matching degree between the real-time pulse feature vector and the interference feature vector in the interference feature library is ≥85% (confidence threshold), it is determined to be an interference signal; for scenarios with multiple interference sources superimposed (such as TV flicker + wall reflection), the "feature decomposition and one-to-one matching" strategy is adopted, and the parallel processing capability of the SNN decision network is used to simultaneously identify multiple interference features and label the corresponding pulse signals; the architecture of the SNN decision network is shown in Table 2. The SNN decision network includes an input layer, a feature decomposition layer, a synaptic weight mapping layer, a parallel matching layer, and a decision output layer from input to output; Table 2: Examples of SNN Decision Network Architectures

[0019] 4) Activate the SNN synaptic suppression mechanism to suppress the pulse transmission of interference signals by suppressing the transmission channel of the corresponding pulses. Specifically: for fixed interference (such as television flicker), the transmission channel of the corresponding pulse is suppressed to directly block its entry into the subsequent encoding process; for dynamic interference (such as the shadow of swaying leaves), a "dynamic suppression window" is used, which temporarily suppresses the pulse transmission in the corresponding area within the corresponding time window based on the periodic pulse release of the interference characteristics; the suppression process does not affect the effective pulse channel, ensuring the normal transmission of pulse signals for core events such as user actions and object movement. 5) Analyze whether the interference signal meets the characteristics of a valid event. If so, adjust the similarity threshold in reverse using the SNN synaptic plasticity rule, and include the interference signal in the valid pulse stream to obtain a valid pulse stream. The characteristics of a valid event are that the spatiotemporal correlation between the interference signal and user actions (such as raising a hand or pointing) or object movement (such as a cup being picked up) is ≥90%. The language extraction module extracts core command words (such as "take", "close", "adjust", and "avoid") from language information using a lightweight keyword recognition model. It then employs a first-peak time-of-flight (TTFS) coding strategy combined with a "command-pulse intensity" mapping library (e.g., emergency commands "avoid obstacles / stop" correspond to intensity level 5, core requirement commands "take / adjust" correspond to levels 3-4, and auxiliary commands "organize / deliver" correspond to levels 1-2) to obtain effective commands with pulse intensity. The TTFS strategy ensures that the greater the signal strength, the earlier the pulse trigger time. The language extraction module also addresses dialect / fuzzy commands. To improve the performance, the system learns from historical user interaction data using SNN synaptic plasticity (STDP rule: Δw=a+・exp(-Δt / τ+), enhancing the preceding and following actions; Δw=-a-・exp(Δt / τ-), weakening the following and preceding actions) to build a dynamic mapping library of "fuzzy commands - entity actions" (e.g., if the user repeatedly associates "that red one" with "red water cup", the mapping relationship is automatically updated); a new "background noise suppression module" is added, which analyzes the spectral characteristics of the speech signal through SNN to separate TV / home appliance background noise from valid commands, further improving the robustness of speech recognition.

[0020] The encoding module uses pulse signal compression encoding to encode valid instructions with pulse intensity and valid pulse streams, and outputs pulse signals with intensity levels.

[0021] The scheduling module assigns priorities to pulse signals with intensity levels according to priority mapping rules, and generates a logical order for all tasks based on the priorities of other tasks. Then, it uses SNN to fuse environmental constraints and predict the feasibility of executing the highest priority task. If it is feasible, the logical order is used as the execution order; otherwise, it checks if there is a similar alternative task. If so, the highest priority task is replaced with the alternative task, the logical order is updated as the execution order, and the reason for the replacement is fed back to the user. Otherwise, the highest priority task is paused, the logical order is updated as the execution order, and the reason for the pause is fed back to the user.

[0022] The priority mapping rule is based on pulse intensity, dividing the scheduling hierarchy into 5 levels (5>4>3>2>1), with level 5 (urgent instructions) > level 4 (core requirements + high-frequency user instructions) > level 3 (core requirements) > level 2 (auxiliary instructions) > level 1 (low-frequency auxiliary instructions). User instruction preferences are continuously learned through SNN synaptic plasticity (e.g., the high-frequency instruction "turn on the air conditioner" is dynamically upgraded from level 3 to level 4), automatically updating the "instruction-intensity" mapping relationship. A new task correlation analysis unit is added, which learns commonly used user instruction combinations (e.g., "get a file + turn on the computer" and "hand over a water cup + adjust the water temperature") through SNN, marking related tasks as "collaborative task groups." During scheduling, these tasks are executed sequentially according to logical order, avoiding efficiency losses caused by frequent switching.

[0023] When conflicts occur, a conflict resolution mechanism is used to build a pulse contention scheduling unit. The SNN synaptic inhibition principle is adopted. When multiple pulse signals are input in parallel, the high-intensity pulse signal will occupy computing resources first by suppressing the transmission channel of the low-intensity pulse signal. The low-intensity pulse signal is temporarily stored in the pulse buffer pool, which supports "dynamic update of buffer priority" - if the pulse signal of the low-priority task is not activated for a long time (exceeding the waiting threshold set by the user), the intensity will be automatically increased by 1 level to avoid long-term backlog of tasks.

[0024] The decision module generates the execution instructions for the optimal action execution path, which specifically include: (1) A basic execution path is generated based on the execution order using an SNN-based impulse timing-action mapping model; specifically: When the task is a single task (such as "passing a water cup"): The SNN-based pulse timing-action mapping model calls the pre-trained "instruction-action path" basic template, combines the robot's current pose with the target object coordinates, and generates a preliminary path of "starting point → intermediate node → target point" through pulse timing encoding (the intermediate node is a key point for obstacle avoidance, and one node is set every 10cm by default). When the task is multi-tasking (e.g., "adjusting the air conditioner + handing out a water cup"): For parallel tasks (e.g., "adjusting the air conditioner + handing out a water cup"), the SNN-based pulse timing-action mapping model adopts a "path reuse + dynamic switching" strategy. It prioritizes planning paths for high-priority tasks and reuses parts of the paths for low-priority tasks (e.g., both "adjusting the air conditioner" and "handing out a water cup" require moving to the living room, so the path segment "moving the aircraft to the living room" is reused), reducing path duplication and switching losses. n-path encoding: The initial path is transformed into a pulse-time signal that can be processed by the SNN decision network as the basic execution path (each node corresponds to a pulse trigger signal, and the distance between nodes corresponds to the pulse interval). (2) Optimize the basic execution path according to the constraints to obtain the optimized execution path; the constraints include mechanical constraints, environmental constraints and interaction constraints. Mechanical constraints include the robot's joint motion angle range, maximum load threshold and upper limit of motion acceleration; environmental constraints include obstacle safety distance, target object reachability range and space channel width; interaction constraints include user action rhythm coefficient and waiting tolerance time; the specific optimization method is as follows: Obstacle avoidance optimization for environmental constraints: The event camera provides real-time feedback on obstacle coordinates, and the SNN decision network optimizes the path through "pulse signal offset adjustment". If the distance between a node on the basic execution path and the obstacle is less than 5cm, the SNN synaptic weight adjustment is triggered, generating a "circumvention movement pulse". Intermediate nodes are added on both sides of the obstacle (e.g., the original path "A→B" is adjusted to "A→C→D→B", where C and D are circumvention nodes). The circumvention path maintains a safe distance of ≥5cm from the obstacle. For dynamic obstacles (such as a moving pet), the movement trajectory is predicted by the SNN pulse timing, and the path nodes are adjusted in advance to avoid real-time collisions.

[0025] Mechanical and load adaptation optimization for mechanical constraints: Based on load sensor data, if the target object weighs ≥0.8kg, the SNN decision network reduces the joint movement speed in the path (pulse interval increased by 50%) and adds intermediate node buffers (e.g., adding "approach → pre-grab → stable grasp" nodes for the grasping action); if the weight >1kg, the current path generation is terminated, and an alternative action path is triggered (e.g., "push to target position" instead of "grab and move"); the SNN decision network analyzes the movement angle of each joint in the path, and if it exceeds the constraint range (e.g., the robotic arm needs to extend to 120°, exceeding the ±90° limit), the spatial coordinates of the path nodes are automatically adjusted, and the action is split (e.g., "move the body forward 10cm → extend the robotic arm") to ensure that all joint movements are within the constraint range.

[0026] Interaction rhythm adaptation optimization for interaction constraints: Based on the user action rhythm coefficient, the SNN decision network adjusts the path execution speed: For the elderly scenario, the pulse interval is extended by 30%, and the waiting time between path nodes is increased by 0.5s (e.g., the "delivery action" is extended from 0.8s to 1.2s); for the young people scenario, the pulse interval is shortened by 20%, improving the path execution efficiency without additional waiting time.

[0027] (3) The SNN decision network performs path pre-testing and verification on the optimized execution path based on the current environment and hardware status. The pre-testing time is ≤2ms. If the verification is successful, proceed to (4); otherwise, the optimized execution path is used as the basic execution path and returned to (2). The maximum iteration is 3 times. If it still fails, the user is given voice feedback. The simulation is performed to verify the spatial relationship between the robot and obstacles and target objects in the detection path, and to determine whether there is a risk of collision; to verify whether the joint movement and load bearing meet the mechanical constraints, and whether the continuity of the action meets the standard (stuttering rate ≤5%); and to determine the probability of completing the target task (such as grasping and manipulation) after the simulated path is executed.

[0028] If the pre-rehearsal meets the requirements of "no collision + feasible actions + task completion rate ≥ 95%", it is considered to have passed the verification. (4) The optimized basic execution path is used as the optimal action execution path, and the execution instructions of the optimal action execution path are generated. The execution instructions are pulse control instructions that can be recognized by the robot execution end. Each path node of the pulse control instruction corresponds to a pulse signal, the pulse intensity corresponds to the action force, and the pulse interval corresponds to the action speed.

[0029] The decision-making module also has an emergency decision priority circuit breaker mechanism. Specifically, when the highest level emergency pulse is detected, the emergency decision priority circuit breaker mechanism is immediately activated, interrupting the path generation and execution of all current tasks, and calling the preset emergency protection path template. No rehearsal is required (saving response time). The emergency execution path is generated directly only considering the non-collision constraint, ensuring that the corresponding delay is ≤5ms. When the highest level emergency pulse is released, the decision-making module reconstructs the optimal action execution path of all tasks before the interruption.

[0030] The motion precision execution analysis module dynamically adjusts motion parameters based on the execution command, the robot's joint range of motion, and load sensor data to generate joint motion commands for the robot.

[0031] When the task involves handling fragile items, the motion precision execution analysis module employs a flexible motion control strategy to further adjust motion parameters. Motion acceleration is reduced by 40%, and force control precision is improved to 0.05N to prevent damage to the items. Specifically, this includes: a. Extract the core features of the joint motion commands of the executed task and transform the core features into joint motion commands corresponding to each pulse. The core features include pulse basic parameters and hardware constraint parameters. The joint motion commands are "pulse-action" mapping vectors, and the joint motion target (angle, force, time) corresponding to each pulse is clearly defined. The basic pulse parameters include pulse intensity (corresponding to the target value of the action force, such as 0.2N), pulse timing (the interval between adjacent pulses, corresponding to the action execution time window, such as 100ms / pulse), and pulse triggering nodes (each pulse corresponds to a joint motion target point, such as two pulses for the robotic arm from 30° to 60°, corresponding to the starting point of 30° and the ending point of 60°); the hardware constraint parameters include the current joint angle, the joint motion range (such as ±90°), and the motor response delay (≤2ms). b. Based on the target point and time window of the discrete pulse signal, plan intermediate interpolation nodes to fill the action gaps in the pulse interval and obtain the filled pulse signal; specifically: Node density setting: Based on the principle of "even time distribution + distance adaptation", N intermediate nodes are inserted between two adjacent pulse target points. The number of N is determined by the pulse interval and the joint movement distance. Time dimension: When the pulse interval is ≤50ms, insert 3 intermediate nodes (one node every 12.5ms on average); when the pulse interval is 50-100ms, insert 5 intermediate nodes (one node every 10-20ms on average) to ensure the continuity of action; Distance dimension: When the difference in joint movement angle is ≥30° or the straight-line distance is ≥5cm, two additional intermediate nodes are added to avoid jamming caused by large-span movements; Node constraint verification: The joint angles of all intermediate nodes must be within the mechanical constraint range (e.g., ±90°) and maintain a safe distance of ≥5cm from obstacles; otherwise, the node positions should be dynamically adjusted.

[0032] c. Perform smooth fitting on the filled pulses based on cubic spline interpolation; specifically: Construct a cubic spline interpolation function S(t) with time (t) as the x-axis and joint angle / force (θ / F) as the y-axis. The cubic spline interpolation function S(t) satisfies the following conditions: Interpolation conditions: S(t0) = θ0 (angle of the first pulse target point), S(t1) = θ1 (angle of the first intermediate node), ... (Next pulse target angle) to ensure the fitted curve accurately passes through all nodes; Smoothing conditions: The first derivative (velocity) and second derivative (acceleration) of adjacent interpolation segments are continuous, avoiding abrupt changes in action (such as in the process of moving from 30° to 60°, the velocity gradually increases from 0 to the peak value and then gradually decreases to 0, without sudden acceleration / deceleration). Force interpolation synchronization: While interpolating the angle, the force is interpolated using cubic spline based on the force target value corresponding to the pulse intensity. This ensures that the force smoothly transitions from the initial value (e.g., 0 N) to the target value (e.g., 0.2 N). Each 0.05 N is a force gradient to avoid sudden changes in force that could damage fragile items.

[0033] d. Incorporate the flexible motion constraint parameters into the smoothly fitted pulse curve to obtain the optimized joint motion command; specifically: Acceleration limitation: By adjusting the second derivative threshold of the cubic spline interpolation function, the motion acceleration is strictly controlled within 0.48 m / s² (0.8 m / s² × 60% for normal motion). The specific method is as follows: calculate the second derivative d²θ / dt² of the interpolation curve. If the acceleration of a certain segment of the curve exceeds the threshold, adjust the interpolation coefficient of that segment of the curve to reduce the rate of change of the curve slope and ensure that the acceleration is always within the constraint range. Force accuracy calibration: Based on the real-time feedback of the force-sensitive load sensor, the force interpolation curve is fine-tuned. The force value is calibrated every 5ms. When the deviation exceeds 0.02N, the force parameters of subsequent interpolation nodes are corrected in real time to ensure that the final force control accuracy is ≤0.05N. Action buffer optimization: At the beginning and end of the pulse target point (such as the "approach → contact → grasp" node of the grasping action), an additional "buffer node" is added to reduce the acceleration of the interpolation curve by 20% to achieve "gentle start and smooth stop".

[0034] The workflow of a multi-task real-time decision-making system is as follows: Perception triggering: The event camera, miniature microphone, and visual motion capture unit monitor dynamic events, voice commands, and user actions in real time, and generate pulse streams only when a valid signal is detected; Feature extraction: The feature extraction module performs anti-interference processing on the pulse stream through dynamic threshold adjustment, multi-source interference separation, and TTFS / time-intensity coding, and outputs a pulse signal with intensity level and association marker; Hierarchical scheduling: The scheduling module prioritizes tasks based on pulse intensity, resolves multi-task conflicts through synaptic inhibition, and predicts execution feasibility by combining task correlation and environmental constraints, and allocates computing resources. Dynamic decision-making: The decision-making module generates action instructions that adapt to the mechanical structure and user rhythm. After path pre-simulation and verification, the execution layer is driven to execute, and multi-sensor fusion feedback corrects deviations. Closed-loop optimization: The low-power module dynamically switches its operating mode based on the task scenario, computing power usage, and battery level, while the energy recovery module provides auxiliary power supply.

[0035] The technical effects of this system are as follows: Dual improvement in real-time performance and reliability: Event-driven mechanism + high-speed serial direct transmission + emergency decision circuit breaker, instruction response speed ≤10ms, environmental change response delay ≤5ms, more than 3 times better than traditional VLA; decision path pre-playing and multi-sensor closed-loop feedback, action execution accuracy ≥98%, task interruption rate reduced to below 5%, adapting to the real-time interaction needs of dynamic home scenes. Significantly optimized multitasking efficiency: Pulse intensity-based scheduling + task correlation analysis + conflict prediction and compromise mechanism improve the execution timeliness of high-priority tasks by 40% and the execution efficiency of collaborative task groups by 30%, completely resolving the problem of multi-instruction parallel conflict; Anti-interference capability meets practical standards for home scenarios: dynamic threshold adjustment + multi-source interference separation + background noise suppression, cross-modal feature extraction accuracy is improved to 99%, and the error rate of instruction recognition in complex environments such as low light, noise, and fuzzy instructions is reduced by 85%, adapting to various home interference scenarios; Breaking through hardware limitations to balance power consumption and battery life: Tiered sleep + dynamic computing power allocation + energy recovery + adaptive wake-up, the overall power consumption is reduced by more than 60% (neuromorphic chip solution power consumption ≤50mA), and the 1500mAh battery extends the battery life to 8-9 hours, which is 33%-50% higher than the traditional solution, meeting the needs of all-day use; Comprehensive assurance of motion accuracy and safety: Mechanical structure adaptation + flexible motion control + 0.1N precision load detection + multi-sensor feedback, reducing motion lag rate by 60%, reducing the damage rate of fragile items to 0, reducing the risk of damage due to overload to 0, and obstacle avoidance response speed to meet home safety needs. Personalized interactive experience achieves differentiated adaptation: user profile construction based on STDP rules + adaptive action rhythm + customized interactive feedback improves user interaction satisfaction by 60% for users of different ages and usage habits, solving the problem of homogeneous interaction in traditional VLA; Strong hardware adaptability and practicality: The lightweight SNN design is compatible with low-power main control chips and neuromorphic chips, eliminating the need for high-performance processors and reducing hardware costs; it can be directly applied to upgrade existing small service robots without significant changes to the hardware structure, demonstrating significant industrialization value.

[0036] Example 1 (a) Setting up the implementation scenario Taking a mixed home scenario (including office and daily care needs) as an example, a small service robot (size: 20cm×15cm×30cm, battery capacity 1500mAh, load ≤1kg, equipped with a neuromorphic chip that supports SNN low-power inference) performs the following task: an elderly user issues a dialect command "hand over that red cup + set the air conditioner to 26 degrees" (vague + dialect command), while a child suddenly runs past the robot quickly 1 meter in front of it (sudden change in environment), and a fragile glass cup is placed next to the target red cup (requiring flexible grasping).

[0037] (II) Specific Implementation Steps Cross-modal sensing and anti-interference processing: On the visual end: An event camera with a high dynamic range of 140dB detects "child running" (a dynamic event with rapid changes) and "red cup, glass water cup" (triggered by user dialect commands, turning static objects into dynamic tracking targets). An ambient light sensor detects the current scene illumination as 600 lux (normal light), and the pulse trigger threshold is set to the default value. Fixed interference features of TV screen flickering are identified, and corresponding invalid pulses are filtered out, retaining only valid dynamic events. Time-intensity mapping encoding is used, generating a level 5 pulse stream (frequency 10Hz) for "child running", a level 3 pulse stream (frequency 6Hz) for "red cup", and marking "glass water cup" as a fragile item. After compression encoding, the pulse signals are transmitted to the main control unit through a high-speed serial interface with a transmission delay of 0.7ms.

[0038] On the language side: A miniature microphone with a wide frequency response is used to collect the dialect commands of elderly users, such as "pass that red cup + adjust the air conditioner to 26 degrees". The background noise suppression module separates the TV background noise (frequency 500-800Hz). A lightweight keyword recognition model extracts the core words "pass" and "adjust". TTFS encoding is used to generate level 3 pulse sequences for both "pass" and "adjust" (because "adjust the air conditioner" is a high-frequency command in the user's history, it has been dynamically upgraded to level 4 through SNN synaptic plasticity). The fuzzy command mapping library accurately maps the fuzzy dialect expression to "red ceramic cup" through the user's historical data (the user has previously matched "that red cup" with "red ceramic cup" many times). After the pulse encoding is completed, it is transmitted synchronously with the visual pulse signal.

[0039] Multi-task scheduling: Pulse competition: Level 5 (child running → obstacle avoidance) pulses suppress Level 4 (adjusting air conditioning) and Level 3 (passing red cup) pulses through synaptic inhibition mechanism, giving priority to the computing power of the neuromorphic chip; Level 4 and Level 3 pulse signals are temporarily stored in an 8MB pulse buffer pool. The task correlation analysis unit recognizes that "passing the cup" and "adjusting air conditioning" are not directly related, and sorts them by priority (adjusting air conditioning Level 4 > passing the cup Level 3).

[0040] Conflict prediction: The SNN integrates environmental constraint detection and identifies a fragile glass cup next to the red cup through visual features (spatial constraint). It predicts that the "passing the cup" action needs to be enabled by flexible grasping. At the same time, the load sensor pre-detects that the weight of the red cup is about 0.3kg (≤1kg, no load constraint), confirming that the task can be executed. The low-priority pulses in the buffer pool have not exceeded the waiting threshold, so the original priority is maintained.

[0041] Dynamic decision-making and action execution: Emergency Decision: The SNN decision network triggers the "emergency decision priority circuit breaker mechanism," interrupting all non-emergency tasks and generating a pulse decision signal of "emergency pause + detour 40cm to the left." The decision path pre-simulation unit quickly simulates the detour path, confirms no collision risk (pre-simulation pass rate 98%), and outputs the execution command. The motor drive module receives the pulse signal, controls the robot to brake and detour to the left, with a response delay of 3ms. The braking energy recovery module is activated to recover electrical energy during deceleration and feed it back to the power supply system.

[0042] Subsequent Decision-Making and Flexible Grasp: After the detour is completed, the Level 5 pulse signal terminates, and the Level 4 (adjusting the air conditioner) pulse in the buffer pool is activated; the SNN generates the action command "move to the air conditioner remote control position → press the 26℃ button". During the execution, the event camera detects no dynamic interference, and the action speed is executed according to the default setting, taking 600ms; after the air conditioner adjustment task is completed, the Level 3 (handing over the red cup) pulse is activated. The mechanical structure adaptation unit combines the load sensor data (0.3kg) and the fragile item marker to enable the flexible motion control strategy; the pulse interpolation compensation algorithm converts discrete pulses into continuous joint motion commands. The robotic arm approaches the red cup with low acceleration (0.5m / s²), and the force is controlled at 0.2N (accuracy 0.05N) to avoid colliding with the glass cup next to it; during the action execution, the event camera tracks the position of the red cup in real time (without movement), and the feedback pulse signal keeps the path stable, with a 100% grasping success rate.

[0043] Motion rhythm adaptation: The visual motion capture unit detected that the user was an elderly person with a slower movement rhythm. When the robotic arm handed out the red cup, the speed was reduced by 30%, and it paused for 2 seconds when the user's hand approached the cup to wait for the user to take it, so as to avoid the user being unable to adapt to the movement too fast.

[0044] Low power consumption optimization: Operating mode switching: When performing obstacle avoidance and air conditioning adjustment tasks (multi-task + complex scene), the SNN runs in full mode, and the neuromorphic chip power consumption is 42mA; when performing the cup passing task (single task + simple scene), it switches to lightweight mode, activates 50% of neurons, and the power consumption drops to 23mA; when there is no task, the extended unit sleeps, and the core unit power consumption is 16mA.

[0045] Energy-saving adaptation: The power consumption dynamic monitoring unit provides real-time statistics. After 2 hours of task execution, the battery power consumption was 21%, and the estimated battery life was about 9.5 hours. During this period, the braking energy recovery module cumulatively recovered electrical energy, reducing battery consumption by 8%.

[0046] Interactive feedback: When avoiding obstacles, the miniature display flashes at a high frequency of 10Hz (priority level 5), and the voice broadcast module announces "Child detected running, detouring" at a volume increased by 20%. When the air conditioning adjustment task is executed, the display screen flashes at 8Hz (level 4) and announces "Adjusting the air conditioning to 26℃ for you"; When the cup-handling task is executed, the display screen flashes at 6Hz (level 3). After a successful grab, it announces "The red cup has been handed to you, please accept it" and displays "Completed" on the screen. User profiles are continuously updated throughout the task, the priority of the "adjust air conditioner" command is further solidified, and the mapping accuracy of ambiguous dialect expressions is improved.

[0047] (III) Verification of Implementation Results Response speed: Obstacle avoidance response latency is 3ms, the air conditioning adjustment task takes 600ms from command issuance to completion, and the cup handing task takes 1500ms, all of which meet the real-time requirements; Power consumption and battery life: 2 hours of task consumption consumed 21% of the power, and the estimated battery life is 9.5 hours, which exceeds the target value (8 hours) by 18.75%; Accuracy and safety: No error in command recognition (accurate mapping of fuzzy dialect commands), smooth execution of actions, successful obstacle avoidance, no collision with fragile items, and no task conflicts; Interactive experience: Elderly users were satisfied with the speed of action, voice volume, and waiting time. They could clearly understand the task priority and progress through the display screen and there were no misunderstandings in the interaction.

[0048] The technical solutions of the present invention are not limited to the specific embodiments described above. Any technical modifications made in accordance with the technical solutions of the present invention fall within the protection scope of the present invention.

Claims

1. A VLA (Dynamic Multi-Task Real-Time Decision-Making System for Small Service Robots, characterized in that: include: Data acquisition end; The acquisition unit is used to collect task information and sensor information in real time. The task information includes visual information and voice information. Storage module; The storage module stores an interference feature library, a "fuzzy command-entity action" dynamic mapping library, and a "command-pulse intensity" mapping library; Feature extraction module; The feature extraction module extracts the pulse signal of the task information based on the interference feature library, analyzes the intensity level of the pulse signal, and outputs the pulse signal with intensity level. Scheduling module; The scheduling module prioritizes pulse signals with intensity levels and sorts them with other current tasks to generate an execution order; Decision module; The decision module generates basic execution paths based on the execution order and performs path pre-simulation verification to generate the execution instructions for the optimal execution path; Action precision execution analysis module; The motion precision execution analysis module dynamically adjusts motion parameters based on execution commands and sensor information to generate joint motion commands for the robot. The execution end; the execution end performs tasks according to joint movement commands.

2. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 1, characterized in that, The interference feature library in the storage module is constructed using the following method: ① Collect interference data from interference sources in the home environment; ② The pulse time coding characteristics of the SNN feature extraction network are used to extract the core pulse features of each interference source, and the core pulse features are encoded to generate interference feature vectors. The core pulse features include temporal features, intensity features and spatial features. ③ Construct an interference feature library according to a three-level structure of "interference type - illumination scene - feature vector", and optimize the interference feature library regularly. The three-level structure of the interference feature library is as follows: first-level classification, second-level classification and third-level storage. The first-level classification is based on the type of interference, including fixed optical interference, environmental optical interference and temporary optical interference. The second-level classification is based on the illumination scene. The three-level storage stores the corresponding interference feature vector, interference source label, and confidence threshold for each category; ④ During system operation, when the feature extraction module detects an unmatched novel pulse feature, it triggers interference learning and updates the interference feature library. The interference learning process involves recording the pulse timing, intensity, and spatial distribution data of the novel pulse feature and simultaneously collecting environmental labels. After user confirmation, the novel pulse feature is included in the interference feature library and assigned a classification label.

3. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 1, characterized in that, The feature extraction module includes a visual extraction module, a language extraction module, and an encoding module. The visual extraction module is used to extract effective pulse signals from visual information, specifically: 1) Dynamically adjust the filtering threshold based on the ambient light intensity of the visual information, and use the filtering threshold to initially filter out invalid signals in the visual information; 2) Use an SNN feature extraction network to extract real-time pulse feature vectors from the pre-filtered task information; 3) Analyze the similarity between the real-time pulse feature vector and all interference feature vectors in the interference feature library, and determine whether the real-time pulse feature vector is an interference signal based on the similarity threshold; 4) Suppress the pulse transmission of the interference signal by using the transmission channel that suppresses the corresponding pulse; 5) Analyze whether the interference signal meets the characteristics of a valid event. If so, adjust the similarity threshold in reverse using the SNN synaptic plasticity rule, and include the interference signal into the valid pulse stream to obtain a valid pulse stream. The language extraction module extracts the core instruction words of language information through a lightweight keyword recognition model, and uses the first peak time coding strategy (TTFS) combined with the "instruction-pulse intensity" mapping library to obtain effective instructions with pulse intensity. The encoding module uses pulse signal compression encoding to encode valid instructions with pulse intensity and valid pulse streams, and outputs pulse signals with intensity levels.

4. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 1, characterized in that, The scheduling module divides the pulse signals with intensity levels into priorities according to the priority mapping rules, and generates the logical order of all tasks according to the priorities of other tasks. Then, it uses SNN to fuse environmental constraints to predict the feasibility of executing the highest priority task. If it is feasible, the logical order is used as the execution order. Conversely, if there is a similar alternative task, check if there is one. If so, replace the highest priority task with the alternative task, update the logical order as the execution order, and provide feedback to the user about the reason for the replacement. Otherwise, pause the highest priority task, update the logical order as the execution order, and provide feedback to the user about the reason for the pause.

5. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 1, characterized in that, The decision module generates the execution instructions for the optimal action execution path, which specifically include: (1) A basic execution path is generated based on the execution order using an SNN-based pulse timing-action mapping model; (2) Optimize the basic execution path according to the constraints to obtain the optimized execution path; (3) Perform path pre-test verification on the optimized execution path. If the verification passes, proceed to (4); otherwise, use the optimized execution path as the basic execution path and return to (2). (4) The optimized basic execution path is used as the optimal action execution path, and the execution instructions of the optimal action execution path are generated.

6. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 5, characterized in that, Constraints include mechanical constraints, environmental constraints, and interaction constraints. Mechanical constraints include the robot's joint motion angle range, maximum load threshold, and upper limit of motion acceleration. Environmental constraints include safe distances to obstacles, reachability range of target objects, and width of spatial passages. Interaction constraints include the user action rhythm coefficient and the waiting tolerance time.

7. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 5 or 6, characterized in that, The decision-making module also has an emergency decision priority circuit breaker mechanism. Specifically, when the highest level emergency pulse is detected, the path generation and execution of all current tasks are interrupted, and a preset emergency protection path template is called to directly generate an emergency execution path, considering only the non-collision constraint. When the highest level emergency pulse is released, the decision-making module reconstructs the optimal action execution path of all tasks before the interruption.

8. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 1, characterized in that, The motion precision execution analysis module dynamically adjusts motion parameters based on the execution command, the robot's joint range of motion, and load sensor data to generate joint motion commands for the robot.

9. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 8, characterized in that, When the task involves fragile items, the motion precision execution analysis module employs a flexible motion control strategy to further adjust the motion parameters, specifically including: a. Extract the core features of the joint motion commands of the task being executed, and convert the core features into joint motion commands corresponding to each pulse. The core features include pulse basic parameters and hardware constraint parameters. b. Based on the target point and time window of the discrete pulse signal, plan intermediate interpolation nodes to fill the action gaps in the pulse interval and obtain the filled pulse signal; c. Perform smooth fitting on the filled pulses based on cubic spline interpolation; d. Integrate the flexible motion constraint parameters into the smoothed pulse curve to obtain the optimized joint motion command.

10. The VLA dynamic multi-task real-time decision-making system for small service robots according to claim 1, characterized in that, The dynamic multi-task real-time decision-making system also includes a closed-loop feedback module, which is used to correct the joint motion commands of the execution end during task execution. Specifically: A. Acquire sensor information in real time and convert it into feedback pulse signals. The sensor information includes visual, force, and tactile information. B. Determine the deviation of the currently executed task based on the feedback pulse signal; C. The closed-loop feedback module automatically adjusts the intensity and timing of joint motion commands using the STDP rule based on pulse timing dependence plasticity to obtain the corrected joint motion commands.

Images