Network edge-to-edge collaboration method and system for deploying embodied intelligent fast and slow dual systems
By combining a lightweight generative policy model on the edge side with a visual language model on the edge side, and by utilizing saliency-aware compression and dynamic adjustment of inference configuration, the problems of limited network bandwidth and disconnection are solved, thereby improving the real-time performance and robustness of the embodied intelligence system.
Patent Information
- Application Number
- CN202611469085.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-09-21
- Publication Date
- 2026-11-17
AI Technical Summary
Existing embodied intelligence systems experience increased latency when network bandwidth is limited, making it impossible to make effective local decisions when the network is disconnected. This leads to a decrease in navigation efficiency and path accuracy, and the fixed inference configuration on the edge side causes latency bottlenecks.
By employing a lightweight generative strategy model on the end side in collaboration with a visual language model on the edge side, and through saliency-aware compression, dynamic adjustment of inference configuration, and local anti-network interruption inference, a closed-loop adaptive framework is formed to proactively adapt to network fluctuations and intelligently maintain connectivity during disconnection.
It effectively reduces communication latency and end-to-end latency, improves the real-time performance and robustness of the system in dynamic network environments, and ensures the continuity and stability of navigation.
Smart Images

Figure CN123124015A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of embodied intelligence and robot autonomous navigation technology, specifically to a network-edge collaboration method and system for deploying both fast and slow embodied intelligence systems. Background Technology
[0002] With the rapid development of embodied intelligence, an increasing number of robotic tasks require the reliance on large-scale vision-language models (VLMs) for environmental understanding, task planning, and decision-making reasoning, combined with lightweight edge control models for real-time interaction and execution. To balance high-level semantic reasoning capabilities with low-level reactive control efficiency, existing embodied intelligence systems widely adopt a fast-slow dual-system edge-edge collaborative architecture: a large-scale vision-language model running on an edge server, acting as System 2, is responsible for high-level semantic reasoning and task planning, while a lightweight generative policy model deployed on the robot, acting as System 1, is responsible for high-frequency motion control and action execution. Taking vision-language navigation (VLN) as an example, System 2 generates navigation sub-targets based on natural language instructions and environmental observations, while System 1 generates continuous motion trajectories based on these sub-targets, enabling autonomous robot navigation. This architecture is applicable to various embodied intelligence application scenarios such as mobile operation, intelligent inspection, warehousing and logistics, and home services.
[0003] With the rapid development of large-scale language models and vision-language models, Visual Language Models (VLMs) have demonstrated outstanding capabilities in cross-modal semantic understanding and high-level reasoning. In the field of embodied navigation, VLMs are widely used to understand natural language instructions, parse visual scenes, and generate navigation sub-targets. To balance reasoning capabilities and execution efficiency, recent VLN frameworks have generally adopted a fast-slow dual-system architecture: System 2 performs high-level semantic reasoning and sub-target planning based on VLM, while System 1 uses lightweight generative strategies (such as diffusion models or flow matching models) for high-frequency motion trajectory generation and control.
[0004] In terms of deployment, early centralized deployments placed both systems on edge servers, with end-side devices only collecting and uploading sensor data. However, this approach faces two major bottlenecks: first, the uplink data volume is too large; the additional modal data required by System 1, such as depth maps and multi-view images, increases the uplink transmission load by more than 2.4 times per step, and transmission latency rises sharply when bandwidth is limited; second, it relies entirely on a continuous network connection, and the robot cannot receive any control signals when the connection is lost, resulting in frozen movements. Another solution is pure end-side deployment, but due to limited computing power on the end side, System 1's inference latency is too long, making it difficult to meet real-time control requirements. For example, the end-to-end inference DualVLN latency on Jetson AGX Orin reaches 15 seconds.
[0005] To address the aforementioned issues, existing distributed edge deployment paradigms offload lightweight System 1 to the robot for local execution, while only uploading the high-level visual-text data required by System 2 to the edge server for periodic inference. This approach significantly reduces communication load and allows System 1 to maintain short-term operation using outdated sub-goals during network outages. However, existing edge deployments still have the following key limitations: 1. Communication bottlenecks are not proactively adapted. Existing methods mainly rely on passive approaches such as asynchronous decoupling or outdated observations to address communication delays, lacking proactive compression and adaptation for visual transmission. When network bandwidth fluctuates, fixed-resolution image transmission leads to uncontrollable communication delays, severely impacting end-to-end latency.
[0006] 2. Fixed inference overhead and lack of adaptive scheduling. In existing methods, the inference configuration of System 1 (such as the number of denoising steps and candidate trajectories in the diffusion model) is static and cannot be dynamically adjusted based on the remaining latency budget. Even when the network is in good condition, System 1 inference still takes about 1.2 seconds, becoming a significant bottleneck in end-to-end latency.
[0007] 3. Lack of effective local maintenance mechanisms during network disconnection. Existing solutions simply repeat the last received action or adopt a fixed strategy during disconnection, failing to fully utilize edge observations and historical action information for intelligent local decision-making, resulting in a significant decrease in navigation efficiency and path accuracy. Summary of the Invention
[0008] To overcome the shortcomings of the prior art, one of the objectives of this invention is to provide a network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems. This method actively adapts to wireless network fluctuations, dynamically adjusts end-side inference configuration, and performs intelligent local maintenance when the network is interrupted, thereby reducing communication overhead and end-to-end latency and improving the real-time performance and robustness of embodied intelligent systems in dynamic network environments.
[0009] To achieve one of the objectives of this invention, the following solution is adopted: A network-edge collaborative method for deploying embodied intelligent dual-system (fast and slow) is proposed, applied to edge robots and edge servers. The edge robot deploys a lightweight generative policy model, and the edge server deploys an edge visual language model. The method includes: S1: The edge device collects visual observation data and natural language commands in each control cycle and monitors the network status; S2: When the network is available, the terminal determines whether it is necessary to perform saliency-aware compression on the visual observation data based on the network status to obtain the observation data to be transmitted; and uploads the observation data to be transmitted and the natural language command to the edge server. S3: The edge-side visual language model of the edge server receives the observation data to be transmitted and the natural language instructions, performs high-level semantic reasoning and task planning, generates sub-targets, and returns the sub-targets to the edge side; S4: The terminal receives the sub-target and determines the remaining end-to-end delay budget based on the communication delay consumed in this control cycle, the edge-side visual language model inference delay, and the preset end-to-end delay budget; and dynamically determines the inference configuration of the terminal-side lightweight generative strategy model based on the remaining end-to-end delay budget. S5: The edge-side lightweight generative strategy model generates motion trajectories based on the sub-targets and the inference configuration, decodes the motion trajectories into action sequences for execution, and updates historical action information based on the executed actions; S6: If the network is unavailable, the terminal enables local anti-network interruption inference and determines the comprehensive risk based on the duration of the disconnection, the fluctuation of actions, and the visual offset. Based on the comprehensive risk, it selects to execute a safe stop action, generate actions based on local inference of cached sub-targets, or perform inertial actions based on historical action information to continuously output control signals. S7: When the network recovers, exit the local anti-network interruption reasoning, obtain the latest sub-target, and return to step S1 to enter the end-edge collaborative decision-making loop of the next control cycle.
[0010] Further, in step S2, determining whether saliency-perceptual compression of the visual observation data is needed based on the network status to obtain the observation data to be transmitted includes: The visual observation data is divided into multiple image blocks; A lightweight visual encoder deployed on the edge is used to extract the saliency scores of each image patch for high-level semantic reasoning; A sampling budget is determined based on the network state, whereby the sampling budget represents the proportion of image patches that retain the original resolution. Based on the saliency score and the sampling budget, image patches that retain their original resolution and image patches that are downsampled are selected and combined to generate the observation data to be transmitted.
[0011] Further, in step S4, the inference configuration of the end-side lightweight generative policy model is dynamically determined based on the remaining end-to-end latency budget, including: Multiple inference configurations of a lightweight generative policy model on the pre-configured edge are tested, and the actual inference latency corresponding to each inference configuration is measured to establish the correspondence between inference configuration and inference latency. Based on the remaining end-to-end latency budget, and under the constraint that the inference latency does not exceed the remaining end-to-end latency budget, the inference configuration that maximizes the inference utility is selected. The inference configuration includes the number of denoising iteration steps and the number of candidate trajectories, and the inference utility is positively correlated with the number of denoising iteration steps and the number of candidate trajectories.
[0012] Furthermore, in step S6, a comprehensive risk is determined based on the duration of the disconnection, the fluctuation of movement, and the visual deviation, including: During the disconnection period, the disconnection step count, action volatility, and visual offset are maintained, wherein the action volatility represents the normalized value of the action transition frequency in the historical action window, and the visual offset represents the feature difference between the current observation and the observation when the subtarget was most recently successfully received. The comprehensive risk is calculated by weighting the number of disconnection steps, the volatility of the action, and the visual offset. If the overall risk is greater than or equal to the safety threshold, then a safety stop action is performed; If the overall risk is less than the security threshold, then based on the semantic timeliness of the cached sub-target, either an action is generated by reasoning through the lightweight generative strategy model on the edge based on the current observation on the edge and the cached sub-target, or a modal action based on the historical action window is selected.
[0013] Furthermore, step S7 specifically includes: When the network is restored, the edge device exits the local anti-network interruption inference mode, resets the disconnection steps, action volatility, visual offset and comprehensive risk, obtains the latest sub-target returned by the edge server, and resumes the execution of the edge-end collaborative decision-making loop that dynamically determines the inference configuration based on the remaining end-to-end latency budget.
[0014] Furthermore, the method forms a closed-loop adaptive end-edge collaborative framework that includes network awareness, compressed transmission, dynamic scheduling, and connection interruption maintenance. The edge-side lightweight generative strategy model performs high-frequency motion control and action execution, while the edge-side visual language model performs high-level semantic reasoning and task planning. The edge-side saliency-aware compression, dynamic inference configuration, and local anti-network interruption inference are all implemented using lightweight computing and do not modify the model structure of the edge-side visual language model.
[0015] Furthermore, when the network is available, the endpoint predicts the original transmission delay based on the network status; If the original transmission delay exceeds the preset communication delay budget, then saliency-perceived compression of the visual observation data is triggered. If the original transmission delay does not exceed the preset communication delay budget, compression is not triggered, and the visual observation data is directly used as the observation data to be transmitted. The saliency-aware compression maximizes the retention of highly saliency information that is important for the sub-objective planning while satisfying communication delay constraints.
[0016] Furthermore, when the remaining end-to-end delay budget is sufficient, a larger number of denoising iteration steps and / or a larger number of candidate trajectories are selected to improve the trajectory generation quality; When the remaining end-to-end delay budget is tight, a smaller number of denoising iteration steps and / or a smaller number of candidate trajectories are selected to ensure real-time control; If the remaining end-to-end latency budget is insufficient to support the minimum inference configuration, the edge-side lightweight generative policy model falls back to the minimum latency configuration.
[0017] Furthermore, the sub-targets generated by the edge-side visual language model are navigation sub-targets, used to indicate the target position or direction of the robot in the next stage; The edge-side lightweight generative strategy model generates a continuous motion trajectory based on the navigation sub-target and decodes the continuous motion trajectory into a discrete action sequence for execution.
[0018] The second objective of this invention is to provide a network-side collaborative system for deploying embodied intelligent fast and slow dual systems. This system actively adapts to wireless network fluctuations, dynamically adjusts end-side inference configuration, and performs intelligent local maintenance when the network is interrupted, thereby reducing communication overhead and end-to-end latency and improving the real-time performance and robustness of embodied intelligent systems in dynamic network environments.
[0019] To achieve the second objective of this invention, the following solution is adopted: A network-side collaborative system for deploying embodied intelligent fast and slow dual systems, used to execute a network-side collaborative method for deploying embodied intelligent fast and slow dual systems as described in one of the objectives of this invention, including an end-side robot and an edge server; The edge robot deploys an edge-side lightweight generative policy model, which includes a saliency-aware compression module, an adaptive scaling inference module, and a local network interruption-resistant inference module. The edge server deploys an edge-side visual language model; The end-side robot is used to collect visual observation data and natural language commands in each control cycle, and to monitor network status; The saliency-aware compression module is used to predict the original transmission delay based on the network status when the network is available, and to perform saliency-aware compression on the visual observation data when the original transmission delay exceeds the preset communication delay budget, so as to obtain the observation data to be transmitted, and upload the observation data to be transmitted and the natural language command to the edge server. The edge-side visual language model is used to receive the observation data to be transmitted and the natural language instructions, perform high-level semantic reasoning and task planning, generate sub-targets, and return the sub-targets to the edge side; The adaptive scaling inference module is used to receive the sub-target, determine the remaining end-to-end delay budget based on the communication delay consumed in this control cycle, the inference delay of the edge-side visual language model, and the preset end-to-end delay budget, and dynamically determine the inference configuration of the end-side lightweight generative policy model based on the remaining end-to-end delay budget. The edge-side lightweight generative policy model is used to generate motion trajectories based on the sub-targets and the inference configuration, decode the motion trajectories into action sequences for execution, and update historical action information based on the executed actions. The local anti-network interruption inference module is used to enable local anti-network interruption inference when the network is unavailable. It determines the comprehensive risk based on the duration of the disconnection, the fluctuation of actions, and the visual offset. Based on the comprehensive risk, it selects to execute a safe stop action, generate actions based on local inference of cached sub-targets, or perform inertial actions based on historical action information to continuously output control signals. When the network is restored, it exits the local anti-network interruption inference, obtains the latest sub-targets, and enters the end-edge collaborative decision-making loop of the next control cycle.
[0020] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves proactive adaptation to wireless network fluctuations, effectively reducing communication latency and end-to-end latency. In each control cycle, this invention monitors the network status and, when the network is available, performs significant perceptual compression on the visual observation data based on the network status. This proactively adapts to bandwidth changes, reduces uplink transmission load, avoids transmission latency exceeding the end-to-end latency budget, and ensures the real-time performance of the robot control cycle.
[0021] 2. This invention achieves dynamic adaptive adjustment of edge-side inference overhead, maximizing trajectory generation quality under latency constraints. Based on the communication latency consumed in the current control cycle, the edge-side visual language model inference latency, and a preset end-to-end latency budget, this invention determines the remaining end-to-end latency budget and dynamically determines the inference configuration of the edge-side lightweight generative strategy model accordingly. This improves trajectory quality when latency is ample and reduces computational overhead when latency is tight, achieving an optimal trade-off between trajectory quality and real-time performance.
[0022] 3. This invention achieves intelligent local maintenance during network disconnection, significantly improving navigation robustness. When the network is unavailable, this invention enables local anti-network interruption inference, determines the comprehensive risk based on the duration of the disconnection, motion fluctuations, and visual offset, and accordingly selects a safe stop action, an action generated through local inference based on cached sub-targets, or an inertial action based on historical motion information. This allows for continuous output of control signals during the disconnection period, reducing path deviation and navigation failure.
[0023] 4. This invention achieves seamless reconnection after network recovery, forming a closed-loop adaptive end-edge collaborative framework. Upon network recovery, this invention exits local anti-network interruption inference, obtains the latest sub-target, and returns to the next control cycle, thereby quickly restoring end-edge collaborative decision-making and ensuring the system maintains navigation continuity and stability even after experiencing a momentary network outage.
[0024] 5. The present invention forms a closed-loop adaptive end-edge collaborative mechanism that integrates network perception, compressed transmission, dynamic scheduling, disconnection maintenance, and restoration connection. This enables the embodied intelligent fast and slow dual system to balance high-level semantic reasoning capabilities and low-level real-time control efficiency when running in a wireless network environment. It is suitable for various application scenarios such as mobile operation, intelligent inspection, warehousing and logistics, and home services. Attached Figure Description
[0025] Figure 1 This is a flowchart of a network end-edge collaboration method for deploying a dual-system fast and slow intelligent system in an embodiment of the present invention; Figure 2 This is a block diagram of a network-edge collaborative system for deploying a dual-system fast and slow intelligent system, as described in this embodiment of the invention. Figure 3 This is a schematic diagram of saliency-aware importance scoring and image compression based on ViT guidance in an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating a specific application of a network-edge collaborative system deployed for both fast and slow intelligent systems in an embodiment of the present invention. Detailed Implementation
[0026] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.
[0027] Example 1 like Figure 1 As shown, this embodiment of the invention provides a network-edge collaboration method for deploying embodied intelligent fast and slow dual systems, applied to an edge robot and an edge server. The edge robot deploys an edge-side lightweight generative policy model, and the edge server deploys an edge-side visual language model. The method includes the following steps: S1: The edge device collects visual observation data and natural language commands in each control cycle and monitors the network status.
[0028] S2: When the network is available, the terminal determines whether it is necessary to perform saliency-aware compression on the visual observation data based on the network status to obtain the observation data to be transmitted; and uploads the observation data to be transmitted and the natural language command to the edge server.
[0029] S3: The edge-side visual language model of the edge server receives the observation data to be transmitted and the natural language instructions, performs high-level semantic reasoning and task planning, generates sub-targets, and returns the sub-targets to the edge side.
[0030] S4: The terminal receives the sub-target and determines the remaining end-to-end delay budget based on the communication delay consumed in this control cycle, the edge-side visual language model inference delay, and the preset end-to-end delay budget; and dynamically determines the inference configuration of the terminal-side lightweight generative policy model based on the remaining end-to-end delay budget.
[0031] S5: The edge-side lightweight generative strategy model generates motion trajectories based on the sub-targets and the inference configuration, decodes the motion trajectories into action sequences for execution, and updates historical action information based on the executed actions.
[0032] S6: If the network is unavailable, the terminal enables local anti-network interruption inference and determines the comprehensive risk based on the duration of the disconnection, the fluctuation of actions, and the visual offset. Based on the comprehensive risk, it selects to execute a safe stop action, generate actions based on local inference of cached sub-targets, or perform inertial actions based on historical action information to continuously output control signals.
[0033] S7: When the network recovers, exit the local anti-network interruption reasoning, obtain the latest sub-target, and return to step S1 to enter the end-edge collaborative decision-making loop of the next control cycle.
[0034] Further, in step S2, determining whether saliency-perceptual compression of the visual observation data is needed based on the network status to obtain the observation data to be transmitted includes: The visual observation data is divided into multiple image blocks; A lightweight visual encoder deployed on the edge is used to extract the saliency scores of each image patch for high-level semantic reasoning; A sampling budget is determined based on the network state, whereby the sampling budget represents the proportion of image patches that retain the original resolution. Based on the saliency score and the sampling budget, image patches that retain their original resolution and image patches that are downsampled are selected and combined to generate the observation data to be transmitted.
[0035] Further, in step S4, the inference configuration of the end-side lightweight generative policy model is dynamically determined based on the remaining end-to-end latency budget, including: Multiple inference configurations of a lightweight generative policy model on the pre-configured edge are tested, and the actual inference latency corresponding to each inference configuration is measured to establish the correspondence between inference configuration and inference latency. Based on the remaining end-to-end latency budget, and under the constraint that the inference latency does not exceed the remaining end-to-end latency budget, the inference configuration that maximizes the inference utility is selected. The inference configuration includes the number of denoising iteration steps and the number of candidate trajectories, and the inference utility is positively correlated with the number of denoising iteration steps and the number of candidate trajectories.
[0036] Furthermore, in step S6, a comprehensive risk is determined based on the duration of the disconnection, the fluctuation of movement, and the visual deviation, including: During the disconnection period, the disconnection step count, action volatility, and visual offset are maintained, wherein the action volatility represents the normalized value of the action transition frequency in the historical action window, and the visual offset represents the feature difference between the current observation and the observation when the subtarget was most recently successfully received. The comprehensive risk is calculated by weighting the number of disconnection steps, the volatility of the action, and the visual offset. If the overall risk is greater than or equal to the safety threshold, then a safety stop action is performed; If the overall risk is less than the security threshold, then based on the semantic timeliness of the cached sub-target, either an action is generated by reasoning through the lightweight generative strategy model on the edge based on the current observation on the edge and the cached sub-target, or a modal action based on the historical action window is selected.
[0037] Furthermore, step S7 specifically includes: When the network is restored, the edge device exits the local anti-network interruption inference mode, resets the disconnection steps, action volatility, visual offset and comprehensive risk, obtains the latest sub-target returned by the edge server, and resumes the execution of the edge-end collaborative decision-making loop that dynamically determines the inference configuration based on the remaining end-to-end latency budget.
[0038] Furthermore, the method forms a closed-loop adaptive end-edge collaborative framework that includes network awareness, compressed transmission, dynamic scheduling, and connection interruption maintenance. The edge-side lightweight generative strategy model performs high-frequency motion control and action execution, while the edge-side visual language model performs high-level semantic reasoning and task planning. The edge-side saliency-aware compression, dynamic inference configuration, and local anti-network interruption inference are all implemented using lightweight computing and do not modify the model structure of the edge-side visual language model.
[0039] Furthermore, when the network is available, the endpoint predicts the original transmission delay based on the network status; If the original transmission delay exceeds the preset communication delay budget, then saliency-perceived compression of the visual observation data is triggered. If the original transmission delay does not exceed the preset communication delay budget, compression is not triggered, and the visual observation data is directly used as the observation data to be transmitted. The saliency-aware compression maximizes the retention of highly saliency information that is important for the sub-objective planning while satisfying communication delay constraints.
[0040] Furthermore, when the remaining end-to-end delay budget is sufficient, a larger number of denoising iteration steps and / or a larger number of candidate trajectories are selected to improve the trajectory generation quality; When the remaining end-to-end delay budget is tight, a smaller number of denoising iteration steps and / or a smaller number of candidate trajectories are selected to ensure real-time control; If the remaining end-to-end latency budget is insufficient to support the minimum inference configuration, the edge-side lightweight generative policy model falls back to the minimum latency configuration.
[0041] Furthermore, the sub-targets generated by the edge-side visual language model are navigation sub-targets, used to indicate the target position or direction of the robot in the next stage; The edge-side lightweight generative strategy model generates a continuous motion trajectory based on the navigation sub-target and decodes the continuous motion trajectory into a discrete action sequence for execution.
[0042] Example 2 like Figure 2-4 As shown, this embodiment of the invention provides a network-side collaborative system for deploying a dual-system fast and slow embodied intelligence, used to execute the network-side collaborative method for deploying a dual-system fast and slow embodied intelligence as described in Embodiment 1, including an end-side robot and an edge server; The edge robot deploys an edge-side lightweight generative policy model, which includes a saliency-aware compression module, an adaptive scaling inference module, and a local network interruption-resistant inference module. The edge server deploys an edge-side visual language model; The end-side robot is used to collect visual observation data and natural language commands in each control cycle, and to monitor network status; The saliency-aware compression module is used to predict the original transmission delay based on the network status when the network is available, and to perform saliency-aware compression on the visual observation data when the original transmission delay exceeds the preset communication delay budget, so as to obtain the observation data to be transmitted, and upload the observation data to be transmitted and the natural language command to the edge server. The edge-side visual language model is used to receive the observation data to be transmitted and the natural language instructions, perform high-level semantic reasoning and task planning, generate sub-targets, and return the sub-targets to the edge side; The adaptive scaling inference module is used to receive the sub-target, determine the remaining end-to-end delay budget based on the communication delay consumed in this control cycle, the inference delay of the edge-side visual language model, and the preset end-to-end delay budget, and dynamically determine the inference configuration of the end-side lightweight generative policy model based on the remaining end-to-end delay budget. The edge-side lightweight generative policy model is used to generate motion trajectories based on the sub-targets and the inference configuration, decode the motion trajectories into action sequences for execution, and update historical action information based on the executed actions. The local anti-network interruption inference module is used to enable local anti-network interruption inference when the network is unavailable. It determines the comprehensive risk based on the duration of the disconnection, the fluctuation of actions, and the visual offset. Based on the comprehensive risk, it selects to execute a safe stop action, generate actions based on local inference of cached sub-targets, or perform inertial actions based on historical action information to continuously output control signals. When the network is restored, it exits the local anti-network interruption inference, obtains the latest sub-targets, and enters the end-edge collaborative decision-making loop of the next control cycle.
[0043] The following is a more detailed description of the network-edge collaborative system for deploying both fast and slow intelligent dual systems according to an embodiment of the present invention.
[0044] This invention adopts a network-adaptive edge-to-edge collaborative approach for deploying embodied intelligent dual-system (fast and slow). This approach differs from traditional static edge-to-edge deployment methods, driven by real-time network awareness and supported by three core pillars: edge-side visual compression, dynamic inference configuration, and local anti-network-interruption inference, forming a closed-loop adaptive VLN framework. At the edge, the system monitors uplink bandwidth and estimated latency budget in real time, dynamically reducing the visual transmission load through a saliency-aware compression module. Upon receiving sub-targets returned by edge system 2 (S2, i.e., the edge-side visual language model), the system dynamically selects the optimal inference configuration for system 1 (S1, i.e., the edge-side lightweight generative strategy model) based on consumed communication and System 2 inference latency through an adaptive scaling module. This configuration includes the number of denoising steps and candidate trajectories, strictly adhering to the remaining end-to-end latency budget. When the network connection is lost, the system switches to the local anti-network-interruption inference module, utilizing edge-side observations, cached sub-targets, and historical actions for intelligent local decision-making, maintaining continuous motion until connection is restored or a stop action is executed.
[0045] This invention designs a saliency-aware adaptive visual compression method. It utilizes the attention map of the edge-side ViT encoder to extract saliency scores from image patches, establishes a closed-loop optimization model constrained by transmission load and communication delay, and dynamically determines the full-resolution preservation or downsampling state for each image patch, achieving a compression strategy that maximizes semantic information preservation under communication budget constraints. This invention also designs a System-1 inference configuration dynamic optimization method based on a bivariate delay model, using a pre-fitted denoising step count... and candidate trajectory number The bivariate linear model between inference latency and the time inference is dynamically solved at each decision cycle based on the remaining end-to-end latency budget. The optimal combination achieves the best trade-off between trajectory quality and real-time constraints. This invention designs a local inference maintenance method based on multi-index risk arbitration during network disconnection. During disconnection, risk is quantified by comprehensively considering three indicators: time drift, action volatility, and visual offset. When the risk is below a safety threshold, the method adaptively selects either trajectory generation based on local inference or inertial maintenance strategy based on the semantic timeliness of cached sub-targets, continuously outputting control signals until the connection is restored or a safe stop action is executed. This invention also designs an end-edge collaborative network adaptive navigation system architecture, integrating the above three mechanisms into the end-edge VLN navigation system to form a closed-loop adaptive framework of network awareness, compressed transmission, dynamic scheduling, and disconnection maintenance. This allows the navigation system to maintain low latency and high robustness in the face of wireless network fluctuations and momentary interruptions.
[0046] The network-edge collaborative system for deploying both fast and slow intelligent dual-systems according to this invention is distributed across edge robot devices and edge servers, and mainly includes three core functional modules, specifically: Saliency-Aware Compression (SAC): Located on the edge, it uses a lightweight ViT encoder on the edge to extract the saliency map of visual observation. Before transmission, it dynamically determines the sampling strategy for image block segmentation, prioritizing the retention of high-resolution information in high-saliency regions and compressing low-saliency regions, thereby minimizing the communication load while ensuring the accuracy of sub-target planning.
[0047] The Adaptive Iterative Scaling (AIS) module, located on the edge side, upon receiving the sub-target returned by edge system 2, dynamically solves for the optimal inference configuration of system 1 (including the number of denoising iteration steps) based on the current communication latency, system 2 inference latency, and a preset end-to-end latency budget. and candidate trajectory number ( ), to maximize the trajectory generation quality under the remaining time delay constraint.
[0048] Local Outage-Resilient Inference (LORI): Located on the edge, it takes over control when the network connection is interrupted. Based on real-time observations on the edge, cached sub-targets and historical action windows, it calculates a comprehensive risk index and adaptively decides to execute actions or safely stop to maintain navigation continuity.
[0049] like Figure 2 The diagram illustrates the overall workflow and data flow of this system. End-side devices acquire visual observations in real time. and natural language commands Simultaneously monitor uplink bandwidth In each control cycle, if the network connection is available, this system first determines whether to trigger SAC compression based on the predicted transmission delay, and uploads the compressed observations to edge system 2; after system 2 completes inference, it returns the sub-target. The edge-side AIS module calculates the remaining budget based on the accumulated latency and selects the optimal one. Configuration: System 1 generates a trajectory and decodes it into an action sequence for execution. If the network connection is lost, this system switches to the LORI module, using cached sub-targets and local information to generate actions.
[0050] In this embodiment, the saliency-aware compression module (SAC) contains the following components: The saliency-aware compression module aims to solve the problem of uncontrollable transmission delay caused by fluctuations in communication bandwidth. Its core design includes: Salience perception importance score: such as Figure 3 As shown, given the observed image The size is The system divides it into blocks according to the standard ViT partitioning method. indivual Image patches of various sizes. A lightweight ViT encoder (i.e., the visual encoder of the edge system 2 VLM, accelerated by TensorRT) deployed on the edge performs forward inference, extracting the [CLS] tokens from the last layer of multi-head self-attention to the attention weights of each image patch, and across all... Attention heads are aggregated to obtain each image patch. significance score The saliency score reflects the importance of the image patch for high-level semantic reasoning and sub-goal planning.
[0051] Dynamic sampling budget optimization: The system monitors uplink bandwidth in real time. And preset the communication latency budget. Before each transmission, the transmission delay is first predicted using the original image size. ,in The original image data volume, This refers to the amount of text instruction data. If... This triggers compression. The system sets the sampling budget. This indicates the proportion of the full-resolution image patch to be retained, with the remainder... proportional image blocks Downsampling by a factor of 1.5. The compressed image transmission load is... Additionally, transmission is required. Bit metadata is used to indicate the sampling status of each block. The optimization objective is to maximize the time taken while satisfying the transmission delay constraint. This means preserving the original image information to the greatest extent possible. The optimization has a closed-form solution: The time complexity of solving this problem is O(n). .
[0052] Adaptive Triggering and Compression Execution: The SAC module triggers only when the predicted transmission delay exceeds the budget, avoiding unnecessary calculations when the network is good. After triggering, the system... Before choosing The image patch with the highest significance score retains its original resolution, while the remaining patches are downsampled and combined to generate a compressed image. The data is then uploaded to an edge server. This design reduces communication load by up to approximately 83% while maintaining the accuracy of sub-objective planning.
[0053] In this embodiment, the specific contents of the Adaptive Scaling Inference (AIS) module are as follows: The adaptive inference scaling module aims to address the end-to-end latency exceeding limits caused by fixed inference settings on the edge system 1. Its core design includes: Bivariate inference delay modeling: System 1 employs a generative strategy based on diffusion models or flow matching for high-frequency motion trajectory generation. Its inference delay is mainly determined by the number of denoising iteration steps. and candidate trajectory number Decision. This system pre-configures a set of... By combining and measuring the actual inference latency, a bivariate linear model is fitted to represent the inference latency of System 1: ,in Simultaneously, the joint computational latency overhead of denoising depth and trajectory concurrency was characterized. For the fixed computational latency overhead of each denoising step, The model incorporates constant system overhead for feature extraction and kernel startup. Its coefficient of determination (COP) reaches 0.99, demonstrating the high accuracy of this system's 1-latency modeling.
[0054] Remaining latency budget calculation: In each decision cycle, after completing image transmission and System 2 inference, the system calculates the remaining latency budget: ,in For end-to-end latency budgeting, This represents the actual communication delay. For image processing functions, For System 2 inference latency, For the edge-side ViT encoding delay, For other system overhead.
[0055] Optimal configuration solution and dynamic scaling: The optimization objective of the system is to satisfy... Maximizing reasoning utility under constraints This means simultaneously maximizing the denoising depth and the coverage breadth of candidate trajectories. By... Substituting into the time delay model and solving it inversely, we obtain the result for a given time delay. The largest for The system traverses the discrete candidate set. , choose to Large configuration .like Insufficient to support minimum configuration If so, system 1 will revert to the lowest latency configuration. The time complexity of this optimization is O(n log n). and Since it is a small constant, the solution is... It meets the real-time requirements of this system.
[0056] In this embodiment, the specific contents of the Local Network Interruption Resistant Inference Module (LORI) are as follows: The local network outage-resistant inference module aims to solve the problems of navigation action interruption and path deviation during network outages. Its core design includes: Offline operation risk monitoring: During the disconnection period, the system continuously maintains three risk indicators: time drift. (Number of steps with continuous disconnection), motion volatility (Normalized value of action transition frequency in the sliding history window), visual offset (Characteristic differences between the current observation and the observation at the time of the most recent successful reception of System 2 sub-targets). This system models the comprehensive risk as... ,in Preset weights.
[0057] Adaptive action arbitration: when When the (safety threshold) is reached, the system considers the risk too high and will execute a STOP action to ensure safety. Otherwise, if the current disconnection duration does not exceed the preset semantic decay threshold, i.e., the time decay coefficient... and The system utilizes current observations from the end side. and the last received sub-target System 1 inference is performed to generate the trajectory and execute the initial action. If the semantic decay threshold has been exceeded, it indicates that the semantics of the cached sub-target are severely outdated. The system then executes the majority action from the historical action window to maintain basic motion inertia.
[0058] Seamless Reconnection Recovery: When the network connection is restored, the system immediately exits LORI mode, resets the risk counter, obtains the latest System 2 sub-target, and resumes the normal end-edge collaborative decision-making loop.
[0059] The complete flow of each control cycle of the network-edge collaborative system for deploying embodied intelligent fast and slow dual systems according to this embodiment of the invention is as follows: End-side acquisition of current observations and instructions Measure uplink bandwidth .
[0060] If the network is available: a. Predict the transmission delay of the original image; if it exceeds the budget, trigger SAC compression; otherwise, preserve the original image; b. Upload the compressed / original image and instructions to edge system 2, and receive the returned sub-target. c. Calculate the consumed delay and solve for the optimal AIS. Configuration; d. System 1 generates a trajectory, decodes it into an action sequence for execution.
[0061] If the network is unavailable: a. Update the disconnection step count. Calculate the volatility of the action and visual offset b. Calculate comprehensive risk If the threshold is exceeded, then STOP is executed; otherwise, the mode action based on the cached subtarget's inference or history is selected according to the time decay coefficient.
[0062] Perform the action and update the history window, then proceed to the next iteration of the loop.
[0063] The network-edge collaborative system for deploying both fast and slow intelligent dual systems according to embodiments of the present invention has the following advantages: 1. This invention designs a saliency-aware adaptive visual compression mechanism. Unlike existing methods that use fixed resolution or uniform downsampling, this invention, for the first time, utilizes the inherent attention mechanism of edge-side ViT to extract saliency maps in edge-side VLN navigation, and establishes a closed-form optimization solution for transmission load and communication delay constraints, achieving dynamic optimization reduction of communication load (up to approximately 83%), while fully preserving key regional information for sub-target planning, ensuring that navigation accuracy does not decrease.
[0064] 2. This invention designs a dynamic latency adaptive scaling mechanism for edge-side generative strategies. Unlike existing methods that use static system 1 inference configuration, this system establishes a high-precision bivariate latency model. In each decision cycle, it dynamically solves for the optimal number of denoising steps and candidate trajectories based on the real-time remaining latency budget. When latency is ample, it maximizes trajectory quality, and when latency is tight, it actively reduces computational overhead to ensure real-time control, effectively solving the edge-side computing power bottleneck problem.
[0065] 3. This invention designs a local anti-network interruption inference mechanism based on multi-index risk arbitration. Unlike existing methods that simply freeze or repeat actions during disconnection, this invention comprehensively evaluates the risk by considering three indicators: time drift, action fluctuation, and visual offset during end-edge disconnection. It also adaptively selects either inference generation or inertial retention strategies based on the semantic timeliness of cached sub-targets, maximizing navigation continuity within the safety boundary and significantly improving the system's robustness in wireless network interruption scenarios.
[0066] 4. This invention designs a lightweight edge-side system architecture. All edge-side modules of this invention are based on a lightweight computational design (the ViT encoder is accelerated by approximately 0.1 seconds using TensorRT, and the time complexity of the optimal solution for the SAC and AIS modules is [missing information - likely OCR error]). to (This means that the edge system 2 model does not need to be modified, has good engineering feasibility and portability, and can be easily integrated into the existing VLN framework.)
[0067] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.
Claims
1. A network edge-to-edge collaborative method for deploying embodied intelligent dual-system fast and slow systems, characterized in that, The method is applied to edge robots and edge servers, wherein the edge robot deploys an edge-side lightweight generative policy model, and the edge server deploys an edge-side visual language model; the method includes the following steps: S1: The edge device collects visual observation data and natural language commands in each control cycle and monitors the network status; S2: When the network is available, the terminal determines whether it is necessary to perform saliency-aware compression on the visual observation data based on the network status to obtain the observation data to be transmitted; and uploads the observation data to be transmitted and the natural language command to the edge server. S3: The edge-side visual language model of the edge server receives the observation data to be transmitted and the natural language instructions, performs high-level semantic reasoning and task planning, generates sub-targets, and returns the sub-targets to the edge side; S4: The terminal receives the sub-target and determines the remaining end-to-end delay budget based on the communication delay consumed in this control cycle, the edge-side visual language model inference delay, and the preset end-to-end delay budget; and dynamically determines the inference configuration of the terminal-side lightweight generative strategy model based on the remaining end-to-end delay budget. S5: The edge-side lightweight generative strategy model generates motion trajectories based on the sub-targets and the inference configuration, decodes the motion trajectories into action sequences for execution, and updates historical action information based on the executed actions; S6: If the network is unavailable, the terminal enables local anti-network interruption inference and determines the comprehensive risk based on the duration of the disconnection, the fluctuation of actions, and the visual offset. Based on the comprehensive risk, it selects to execute a safe stop action, generate actions based on local inference of cached sub-targets, or perform inertial actions based on historical action information to continuously output control signals. S7: When the network recovers, exit the local anti-network interruption reasoning, obtain the latest sub-target, and return to step S1 to enter the end-edge collaborative decision-making loop of the next control cycle.
2. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, In step S2, based on the network status, it is determined whether saliency-perceptual compression of the visual observation data is needed to obtain the observation data to be transmitted, including: The visual observation data is divided into multiple image blocks; A lightweight visual encoder deployed on the edge is used to extract the saliency scores of each image patch for high-level semantic reasoning; A sampling budget is determined based on the network state, whereby the sampling budget represents the proportion of image patches that retain the original resolution. Based on the saliency score and the sampling budget, image patches that retain their original resolution and image patches that are downsampled are selected and combined to generate the observation data to be transmitted.
3. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, In step S4, based on the remaining end-to-end latency budget, the inference configuration of the lightweight generative policy model on the end side is dynamically determined, including: Multiple inference configurations of a lightweight generative policy model on the pre-configured edge are tested, and the actual inference latency corresponding to each inference configuration is measured to establish the correspondence between inference configuration and inference latency. Based on the remaining end-to-end latency budget, and under the constraint that the inference latency does not exceed the remaining end-to-end latency budget, the inference configuration that maximizes the inference utility is selected. The inference configuration includes the number of denoising iteration steps and the number of candidate trajectories, and the inference utility is positively correlated with the number of denoising iteration steps and the number of candidate trajectories.
4. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, In step S6, the comprehensive risk is determined based on the duration of the disconnection, the fluctuation of movement, and the visual deviation, including: During the disconnection period, the disconnection step count, action volatility, and visual offset are maintained, wherein the action volatility represents the normalized value of the action transition frequency in the historical action window, and the visual offset represents the feature difference between the current observation and the observation when the subtarget was most recently successfully received. The comprehensive risk is calculated by weighting the number of disconnection steps, the volatility of the action, and the visual offset. If the overall risk is greater than or equal to the safety threshold, then a safety stop action is performed; If the overall risk is less than the security threshold, then based on the semantic timeliness of the cached sub-target, either an action is generated by reasoning through the lightweight generative strategy model on the edge based on the current observation on the edge and the cached sub-target, or a modal action based on the historical action window is selected.
5. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, Step S7 is as follows: When the network is restored, the edge device exits the local anti-network interruption inference mode, resets the disconnection steps, action volatility, visual offset and comprehensive risk, obtains the latest sub-target returned by the edge server, and resumes the execution of the edge-edge collaborative decision-making loop that dynamically determines the inference configuration based on the remaining end-to-end latency budget.
6. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, The method forms a closed-loop adaptive end-edge collaborative framework that includes network awareness, compressed transmission, dynamic scheduling, and connection interruption maintenance. The edge-side lightweight generative strategy model performs high-frequency motion control and action execution, while the edge-side visual language model performs high-level semantic reasoning and task planning. The edge-side saliency-aware compression, dynamic inference configuration, and local anti-network interruption inference are all implemented using lightweight computing and do not modify the model structure of the edge-side visual language model.
7. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, When the network is available, the endpoint predicts the original transmission delay based on the network status. If the original transmission delay exceeds the preset communication delay budget, then saliency-perceived compression of the visual observation data is triggered. If the original transmission delay does not exceed the preset communication delay budget, compression is not triggered, and the visual observation data is directly used as the observation data to be transmitted. The saliency-aware compression maximizes the retention of highly saliency information that is important for the sub-objective planning while satisfying communication delay constraints.
8. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 3, characterized in that, When the remaining end-to-end delay budget is sufficient, a larger number of denoising iteration steps and / or a larger number of candidate trajectories are selected to improve the trajectory generation quality; When the remaining end-to-end delay budget is tight, a smaller number of denoising iteration steps and / or a smaller number of candidate trajectories are selected to ensure real-time control; If the remaining end-to-end latency budget is insufficient to support the minimum inference configuration, the edge-side lightweight generative policy model falls back to the minimum latency configuration.
9. The network end-edge collaboration method for deploying embodied intelligent fast and slow dual systems according to claim 1, characterized in that, The sub-targets generated by the edge-side visual language model are navigation sub-targets, used to indicate the target position or direction of the robot in the next stage; The edge-side lightweight generative strategy model generates a continuous motion trajectory based on the navigation sub-target and decodes the continuous motion trajectory into a discrete action sequence for execution.
10. A network-end-edge collaborative system for deploying embodied intelligent fast and slow dual systems, used to execute the network-end-edge collaborative method for deploying embodied intelligent fast and slow dual systems as described in any one of claims 1-9, characterized in that, This includes edge robots and edge servers; The edge robot deploys an edge-side lightweight generative policy model, which includes a saliency-aware compression module, an adaptive scaling inference module, and a local network interruption-resistant inference module. The edge server deploys the edge-side visual language model; The end-side robot is used to collect visual observation data and natural language commands in each control cycle, and to monitor network status; The saliency-aware compression module is used to predict the original transmission delay based on the network status when the network is available, and to perform saliency-aware compression on the visual observation data when the original transmission delay exceeds the preset communication delay budget, so as to obtain the observation data to be transmitted, and upload the observation data to be transmitted and the natural language command to the edge server. The edge-side visual language model is used to receive the observation data to be transmitted and the natural language instructions, perform high-level semantic reasoning and task planning, generate sub-targets, and return the sub-targets to the edge side; The adaptive scaling inference module is used to receive the sub-target, determine the remaining end-to-end delay budget based on the communication delay consumed in this control cycle, the inference delay of the edge-side visual language model, and the preset end-to-end delay budget, and dynamically determine the inference configuration of the end-side lightweight generative policy model based on the remaining end-to-end delay budget. The edge-side lightweight generative policy model is used to generate motion trajectories based on the sub-targets and the inference configuration, decode the motion trajectories into action sequences for execution, and update historical action information based on the executed actions. The local anti-network interruption inference module is used to enable local anti-network interruption inference when the network is unavailable. It determines the comprehensive risk based on the duration of the disconnection, the fluctuation of actions, and the visual offset. Based on the comprehensive risk, it selects to execute a safe stop action, generate actions based on local inference of cached sub-targets, or perform inertial actions based on historical action information to continuously output control signals. When the network is restored, it exits the local anti-network interruption inference, obtains the latest sub-targets, and enters the end-edge collaborative decision-making loop of the next control cycle.
