Streaming Token dual-path transmission method, system and device and storage medium

Abstract

The invention relates to the technical field of data transmission, and particularly provides a streaming Token double-path transmission method, system and device and a storage medium, and the method comprises the steps: responding to a streaming request, completing the initialization of a system and a session, building a bypass channel from a model to a client, and creating a session context in a control flow engine; identifying the type of each Token generated by the model, and shunting according to the type: data Token routing to a bypass channel and control Token routing to a control flow path processed by a control flow engine; data bypass transmission and control flow service processing are executed in parallel, and the states of the data bypass transmission and the control flow service processing are ensured to be consistent through a state synchronization mechanism; and the dual-path operation state is continuously monitored, and dynamic path adjustment or abnormity recovery is carried out when abnormity is detected. According to the method, the data flow and the control flow are decoupled, so that the transmission delay of the streaming Token is remarkably reduced while the state consistency is ensured.

Description

Dual-path transmission method, system, device and storage medium for streaming tokens Technical Field

[0001] This invention belongs to the field of data transmission technology, specifically relating to a dual-path transmission method, system, device, and storage medium for streaming tokens. Background Technology

[0002] With the widespread application of large language models and multi-agent systems, streaming token transmission has become a key technology for achieving real-time interaction. Existing agent platforms typically employ a linear pipeline architecture, where tokens generated by the model are first sent to a streaming engine for processing and forwarding before reaching the client via multiple network nodes. This approach results in each token undergoing multiple network hops, and as concurrency increases and response text lengthens, cumulative latency rises significantly, impacting user experience. Although existing technologies have attempted improvements through network optimization or caching mechanisms, they haven't changed the fundamental path of tokens being forwarded through a central engine, making it difficult to achieve low-latency transmission while maintaining state consistency. Therefore, a streaming token transmission solution that balances transmission efficiency and control reliability is urgently needed. Summary of the Invention

[0003] In view of the above-mentioned shortcomings of the prior art, the present invention provides a dual-path transmission method, system, device and storage medium for streaming tokens to solve the above-mentioned technical problems.

[0004] In a first aspect, the present invention provides a dual-path transmission method for streaming tokens, comprising: responding to a streaming request, performing system and session initialization, including establishing a bypass channel for direct data transmission between the model and the client, and creating a corresponding session context in the control flow engine; identifying the type of each token generated by the model, and routing it according to the type: if it is a data token, routing it to the bypass channel; if it is a control token, routing it to a control flow path that needs to be processed by the control flow engine; executing data transmission and business control processes in parallel, and ensuring state consistency between the bypass channel and the control flow engine through a state synchronization mechanism; wherein, the data transmission includes directly pushing the data token to the client through the bypass channel; the business control process includes the control flow engine processing the control token and updating the session state; monitoring the operating status of the bypass channel and the control flow engine, and dynamically adjusting the path or recovering from the anomaly when an anomaly is detected.

[0005] In an optional implementation, in response to a streaming request, system and session initialization is performed, including: establishing the bypass channel, specifically including pre-establishing a persistent connection between the model and the client using the WebSocket or QUIC protocol, and enabling connection multiplexing for multiple streaming requests from the same client; predicting the initial cache capacity based on historical streaming data characteristics, and setting a dynamic buffer for the bypass channel based on the initial cache capacity; creating a session context corresponding to the streaming request in the control flow engine, the session context containing at least a unique session identifier; and negotiating and determining an initial dual-path transmission strategy based on the characteristics of the streaming request, the characteristics including the estimated response length and / or the real-time requirements of the business logic.

[0006] In an optional implementation, the negotiation of the initial dual-path transmission strategy may also be based on the data type of the streaming token, adjusting the corresponding transmission parameters and chunk size for text, audio, or video stream types.

[0007] In an optional implementation, the type of each token generated by the identification model is determined by: judging whether the token is a control instruction or pure data content based on a preset classification rule; wherein, control instructions include tokens used for process control, tool invocation, or business logic processing, and pure data content includes streaming output content of text, audio, or video; and adding streaming metadata to the identified pure data content tokens, wherein the metadata includes at least: stream identifier, chunk sequence number, whether it is an end block, and high-precision timestamp.

[0008] In an optional implementation, the classification rules include: keyword matching rule: if the content of the token matches a predefined set of control instruction keywords, it is determined to be a token of the control instruction type; sequence position rule: if the token is in a predefined specific sequence position, including the session start position, session end position, or a synchronization checkpoint position set at fixed intervals, it is determined to be a token of the metadata type used for state synchronization; content exclusion rule: if the token does not belong to the control instruction type and the metadata type, it is determined to be a token of the pure data type; dynamic adaptive rule: dynamically adjust the token type determination or transmission path selection based on the current system context information, wherein the system context information includes at least one of network conditions, system load, or session stage.

[0009] In one optional implementation, data transmission and business control processes are executed in parallel, and a state synchronization mechanism ensures state consistency between the bypass channel and the control flow engine. This includes: directly pushing the identified data token to the client for real-time rendering via the bypass channel; sending the identified control token to the control flow engine, which then executes the corresponding business process and updates the session state; and ensuring, through a lightweight state synchronization mechanism, that the client's progress in receiving data via the bypass channel remains consistent with the session state maintained by the control flow engine. The lightweight state synchronization mechanism includes: the control flow engine periodically sending data to the client... A heartbeat signal is sent, which at least includes the latest processed token sequence number recorded by the control flow engine. The client compares the latest token sequence number that it has successfully received through the bypass channel with the sequence number in the heartbeat signal. When the comparison finds that the sequence number difference exceeds a preset threshold, a state repair process is triggered. The state repair process includes at least one of the following methods: if the client's receiving progress is ahead of the control flow engine's progress, the client buffers and waits for the received ahead tokens; if the client's receiving progress is behind the control flow engine's progress or a token loss is detected, a retransmission mechanism for the missing token is triggered.

[0010] In an optional implementation, monitoring the operating status of the bypass channel and the control flow engine, and dynamically adjusting the path or recovering from the anomaly when an anomaly is detected, includes: real-time monitoring of the transmission quality indicators of the bypass channel and the processing status of the control flow engine, wherein the transmission quality indicators include at least one of transmission delay, packet loss rate, and transmission rate fluctuation; when it is determined that the bypass channel is abnormal according to the transmission quality indicators, dynamically adjusting the transmission path, switching the transmission of at least a portion of the subsequent data tokens from the bypass channel to the control flow path forwarded by the control flow engine; for token loss that occurs during the anomaly, selecting and executing a corresponding retransmission mechanism from a variety of preset retransmission strategies based on the real-time network conditions, wherein the various retransmission strategies include immediate retransmission, batch retransmission, or progressive retransmission.

[0011] Secondly, this invention provides a dual-path transmission system for streaming tokens, comprising: an initialization module, used to initialize the system and session in response to streaming requests, including establishing a bypass channel for direct data transmission between the model and the client, and creating a corresponding session context in the control flow engine; a path routing module, used to identify the type of each token generated by the model and route it according to the type: if it is a data token, it is routed to the bypass channel; if it is a control token, it is routed to the control flow path that needs to be processed by the control flow engine; a state synchronization module, used to execute data transmission and business control processes in parallel, and to ensure the state consistency between the bypass channel and the control flow engine through a state synchronization mechanism; wherein, the data transmission includes pushing the data token directly to the client through the bypass channel; the business control process includes the control token being processed by the control flow engine and the session state being updated; and an exception handling module, used to monitor the operating status of the bypass channel and the control flow engine, and to dynamically adjust the path or perform exception recovery when an exception is detected.

[0012] Thirdly, an apparatus is provided, comprising: a memory for storing a dual-path transmission program for streaming tokens; and a processor for executing the dual-path transmission program for streaming tokens to implement the steps of the dual-path transmission method for streaming tokens as provided in the first aspect.

[0013] Fourthly, a computer-readable storage medium is provided, on which a dual-path transmission program for streaming tokens is stored, wherein when the dual-path transmission program for streaming tokens is executed by a processor, the program implements the steps of the dual-path transmission method for streaming tokens as provided in the first aspect.

[0014] The beneficial effects of this invention are that the dual-path transmission method, system, device, and storage medium for streaming tokens provided by this invention significantly reduce the transmission latency of streaming tokens while ensuring system state consistency by decoupling the data flow and control flow. Data tokens reach the client directly via a bypass channel, minimizing the number of jumps and enabling users to perceive responses earlier, thus improving real-time interaction. The control flow engine can focus on processing business logic, improving overall system throughput and resource utilization. Built-in state synchronization and anomaly adaptive mechanisms ensure the robustness and eventual consistency of the transmission process. The solution design balances performance and compatibility, supporting smooth integration into existing intelligent agent platforms, and has high practical and promotional value. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 is a schematic flowchart of a method according to an embodiment of the present invention.

[0017] Figure 2 is a schematic diagram of a method according to an embodiment of the present invention.

[0018] Figure 3 is a schematic block diagram of a system according to an embodiment of the present invention.

[0019] Figure 4 is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions of this invention, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this invention.

[0021] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0022] The dual-path transmission method for streaming tokens provided in this embodiment of the invention is executed by a computer device, and correspondingly, the dual-path transmission system for streaming tokens runs in the computer device.

[0023] Figure 1 is a schematic flowchart of a method according to an embodiment of the present invention. The entity executing the method in Figure 1 can be a dual-path transmission system for streaming tokens. Depending on different requirements, the order of the steps in this flowchart can be changed, and some steps can be omitted.

[0024] As shown in Figure 1, the method includes: S1. Responding to a streaming request, performing system and session initialization, including establishing a bypass channel for direct data transmission between the model and the client, and creating a corresponding session context in the control flow engine; S2. Identifying the type of each token generated by the model and routing it according to the type: if it is a data token, it is routed to the bypass channel; if it is a control token, it is routed to the control flow path that needs to be processed by the control flow engine; S3. Executing data transmission and business control processes in parallel, and ensuring state consistency between the bypass channel and the control flow engine through a state synchronization mechanism; wherein, the data transmission includes directly pushing the data token to the client through the bypass channel; the business control process includes the control flow engine processing the control token and updating the session state; S4. Monitoring the running status of the bypass channel and the control flow engine, and dynamically adjusting the path or performing abnormal recovery when an anomaly is detected.

[0025] Referring to Figure 2, in the traditional architecture, the token needs to go through N hops (N ≥ 2) from model output to client presentation. The specific path is: Model → Streaming Engine → (possible middleware) → Client. While this design is beneficial for unified control flow management, as concurrent requests increase and the length of the streaming output grows, transmission latency becomes a bottleneck affecting user experience. Existing technologies attempt to alleviate this problem by optimizing network topology or improving hardware performance, but they do not fundamentally change the token transmission path. For example, some solutions use preloading or caching mechanisms, but they still cannot avoid token processing latency in the streaming engine. Other solutions, such as dual-route path generation methods, provide path redundancy, but they are not optimized for the characteristics of token transmission.

[0026] The overall architecture of this invention adopts a dual-path design, as shown in Figure 2. The dual-path routing module is the core component of the system, responsible for making immediate decisions on the transmission path after the model generates a token. This module is located between the model output and the streaming engine and includes the following sub-functions: Path Decision Maker: Dynamically selects the optimal transmission path based on the token type, streaming session characteristics, and system load. Important control commands are still processed by the streaming engine, while pure data tokens are sent directly through a bypass channel.

[0027] Metadata tagger: Adds lightweight metadata to each token, including session ID, sequence number and status stamp, providing basic support for subsequent state synchronization.

[0028] Traffic controller: Monitors the load status of both paths, implements load balancing in high-concurrency scenarios, and prevents single path overload.

[0029] The Token Bypass Channel is a key innovation of this invention, providing a direct data transmission path from the model to the client. This channel has the following characteristics: Low-latency protocol: Employing an optimized network transmission protocol, it eliminates unnecessary handshakes and authentication steps, focusing on efficient data transmission.

[0030] Streaming Adaptation: Designed for streaming token features, it supports continuous, chunked transmission and can start client rendering without waiting for a complete response.

[0031] Fault tolerance mechanism: Includes automatic retransmission and sequence verification mechanisms to ensure that tokens are delivered to the client in the correct order even in unstable network environments.

[0032] The Flow Engine continues to handle control flow logic, but no longer assumes the responsibility of forwarding all tokens. Its functions have been adjusted to: Session Management: Maintaining the state machine of streaming sessions and tracking the progress and context of each session.

[0033] Business logic execution: Handling complex business processes, such as multi-agent collaboration, tool invocation, and conditional branching.

[0034] Lightweight state tracking: Maintains awareness of token transmission progress by interacting with the bypass channel, without actually processing the token data.

[0035] The state synchronization module ensures the consistency of state between the two paths, which is a key guarantee for the correct operation of the system: Heartbeat mechanism: Periodically exchange lightweight state information between the model, the streaming engine and the client to confirm that the progress of each node is consistent.

[0036] Difference Detection and Repair: Monitors the differences between the bypass channel and the streaming engine status, and triggers an automatic repair process when inconsistencies are detected.

[0037] Checkpoint mechanism: Establishing state checkpoints at specific intervals facilitates rapid recovery and synchronization.

[0038] In one embodiment of the present invention, based on step S1, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0039] S101. Establish the bypass channel, specifically by using the WebSocket or QUIC protocol to pre-establish a persistent connection between the model and the client, and enabling a connection multiplexing mechanism for multiple streaming requests from the same client.

[0040] Specifically, the system uses the WebSocket or QUIC protocol to pre-establish a low-latency, full-duplex persistent connection between the AI ​​model (or model server endpoint) and the client. To achieve efficient resource utilization, the system enables a connection reuse mechanism for multiple streaming requests from the same client, sharing the persistent connection and thus reducing the overhead of repeatedly establishing connections.

[0041] S102. Predict the initial buffer capacity based on historical streaming data characteristics, and set a dynamic buffer for the bypass channel based on the initial buffer capacity.

[0042] Based on the data characteristics of historical streaming sessions (such as average token size and traffic patterns), the initial buffer capacity required for the current session is predicted, and a dynamic buffer is configured for the bypass channel based on this prediction. The size of this buffer can be dynamically adjusted according to the actual traffic during transmission to balance memory usage and transmission smoothness.

[0043] S103. Create a session context in the control flow engine corresponding to the streaming request, wherein the session context contains at least a unique session identifier.

[0044] For each received streaming request, the control flow engine creates a unique session ID and initializes a lightweight session context object. This object is used to track the overall progress of the session, the associated business logic state, and resource binding information in subsequent processes.

[0045] S104. Based on the characteristics of the streaming request, negotiate and determine an initial dual-path transmission strategy, including the estimated response length and / or the real-time requirements of the business logic. The negotiation of the initial dual-path transmission strategy also includes the data type of the streaming token, adjusting the corresponding transmission parameters and chunk size for text, audio, or video stream types.

[0046] The system analyzes the characteristics of current streaming requests, including but not limited to the estimated response data length, the business logic's sensitivity to real-time performance, and the data type of the token (such as plain text, audio stream, or video stream). Based on these characteristics, the system dynamically determines the initial dual-path allocation strategy. For example, in real-time dialogue scenarios, the strategy prioritizes the bandwidth and low latency of the bypass channel; for large video streams, it negotiates larger chunk sizes and adjusts corresponding encoding and transmission parameters to optimize throughput. This negotiation result will serve as the initial basis for subsequent token path decisions.

[0047] In one embodiment of the present invention, based on step S2, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0048] S201. Based on preset classification rules, determine whether the Token is a control instruction or pure data content; wherein, control instructions include Tokens used for process control, tool invocation or business logic processing, and pure data content includes streaming output content of text, audio or video.

[0049] The token type identification is implemented by a dual-path routing module deployed at the model output, which incorporates a real-time classification engine based on multi-rule fusion. This engine processes each streaming output token according to the following logic: First, keyword matching rules are applied. The system maintains a predefined set of control instruction keywords (e.g., {"<tool_call> ”, “<end_of_turn> ”, “<switch_agent> The classification engine compares the content of the token with the set. If it matches exactly or conforms to a specific prefix pattern (such as starting with "func_"), the token is immediately identified as a control instruction type.

[0050] Secondly, the sequence position rules are applied. The engine accesses the sequence number (sequence_number) carried by the token and the synchronization interval (sync_interval) defined in the session context. If the token's sequence number is 0 (at the start of the session), is marked as a final token (is_final=True), or its sequence number satisfies the condition "sequence_number % sync_interval == 0" (i.e., synchronization checkpoints appearing at fixed intervals), then the token is determined to be a metadata type used for state synchronization.

[0051] If a token does not conform to either of the above two rules, the content exclusion rule is triggered, meaning it is classified as a plain data type by default. This type typically contains text characters or encoded audio / video data blocks to be presented to the user.

[0052] In addition, the system runs a dynamic adaptive rule as an overlay layer. This rule collects system context information in real time, including but not limited to network latency and packet loss rate (network conditions), CPU / memory usage (system load), and the current session's stage (startup, stabilization, or termination). Based on this information, the rule can dynamically correct the aforementioned basic classification results. For example, during periods of severe network jitter, some low-priority pure data tokens can be temporarily reclassified as control types and transmitted via a more reliable control flow path; or at the initial stage of a session, the first few data tokens can be temporarily processed as metadata to prioritize connection stability.

[0053] All rules can be executed in parallel or in order of priority, and the final output is the definite type and suggested transmission path of each token, which is then handed over to the subsequent traffic splitting and routing logic for processing.

[0054] S202. Attach streaming metadata to the token of the identified plain data content, the metadata including at least: stream identifier, block sequence number, whether it is the end block, and high-precision timestamp.

[0055] Lightweight but critical streaming metadata is attached to all tokens categorized as "pure data". This metadata includes: a globally unique stream_id, used to associate all data blocks within the same session; a chunk_sequence, used to ensure clients assemble the data in order; an is_final_chunk flag, used to notify clients of the termination of the streaming; and a high-precision timestamp, providing an accurate time base for subsequent latency analysis, state synchronization, and debugging.

[0056] S203. Distribute the data according to its type: if it is a data token, route it to the bypass channel; if it is a control token, route it to the control flow path that needs to be processed by the control flow engine.

[0057] Tokens marked as "control instructions" and containing some "metadata" are routed to the control flow path because they carry critical information that drives business processes and maintains state consistency. These tokens are then delivered to the control flow engine for logical processing and state updates. Tokens marked as "pure data," on the other hand, are immediately routed to an established bypass channel, carrying their metadata. This direct link allows for minimal latency push to the client for real-time rendering. Dynamic adaptive rules may also directly intervene in path selection at this stage, enabling online adjustments to the transmission strategy.

[0058] In one embodiment of the present invention, based on step S3, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0059] S301. The identified data token is directly pushed to the client for real-time rendering through the bypass channel.

[0060] Plain data tokens routed to the bypass channel immediately enter an optimized sending queue. The system processes these tokens using zero-copy or incremental encoding techniques to minimize processing overhead. Subsequently, these tokens are pushed directly to the client in contiguous chunks via a pre-established persistent connection based on the WebSocket or QUIC protocol. The client can initiate real-time rendering (such as displaying text word by word or decoding and playing audio and video) upon receiving the first data chunk, allowing the user to perceive partial response content in near real-time without waiting for the entire task to complete.

[0061] S302. The identified control token is sent to the control flow engine, which then executes the corresponding business process and updates the session state.

[0062] Control tokens (including instructions and key metadata) routed to the control flow path are received and processed by the control flow engine. The engine executes predefined business process logic based on the semantics carried by the token (such as invoking specific tools, switching agents, or marking step boundaries). After processing each control token, the engine atomically updates the session state record corresponding to that session ID, for example, by advancing the state machine, recording completed logical steps, or updating context variables. This process ensures that complex business workflows proceed correctly.

[0063] S303. A lightweight state synchronization mechanism ensures that the progress of the client receiving data through the bypass channel remains consistent with the session state maintained by the control flow engine. This lightweight state synchronization mechanism is implemented by the system's state synchronization module, and its core objective is to ensure and maintain eventual consistency between the client's data reception progress in the bypass channel and the session logic state maintained internally by the control flow engine. This mechanism is implemented through periodic heartbeat interactions and intelligent repair processes, specifically as follows: First, the control flow engine, as the authoritative source of state, records the latest global token sequence number corresponding to the state while processing control tokens and updating the session state. The engine-side state synchronization component actively generates and sends a lightweight heartbeat signal to the client at a preset period (e.g., every 10 control tokens processed or every 200 milliseconds). This signal is a simplified data packet, whose core payload is the aforementioned "latest processed token sequence number," and optionally may also include metadata such as timestamps and session IDs for verification.

[0064] On the client side, a local progress tracker is maintained, continuously recording the sequence number of the last data token successfully received, verified, and confirmed from the bypass channel. When the client receives a heartbeat signal from the control flow engine, it immediately extracts the sequence number from the engine side from its payload and compares it with the locally recorded sequence number in real time.

[0065] The system presets a threshold for determining the health of a status (e.g., a sequence number difference greater than 3). The comparison results and processing logic are as follows: Progress advance handling: If the client's sequence number is significantly greater than the engine's sequence number (i.e., the client receives data faster), it indicates that the heartbeat signal may be delayed due to network latency. In this case, the client will not submit the received advanced data to the upper-layer application for final processing. Instead, it will temporarily store it in an ordered buffer queue until a subsequent heartbeat signal indicates that the engine's progress has caught up. Only then will the buffered data be submitted sequentially, thus ensuring the causal correctness of the business logic.

[0066] Handling of Progress Delays or Packet Losses: If the client's sequence number is less than the engine's sequence number, or if a check of local records reveals discontinuous intervals in the sequence number, it is determined that a data token has been lost or there is a severe delay. The client will immediately send a retransmission request to the control flow engine or a designated coordination service, explicitly carrying the range of missing token sequence numbers. Upon receiving the request, the system triggers the retransmission mechanism, retransmitting the token data packet with the specified sequence number from the model's output buffer or through the control flow path, based on the current network conditions and system policies, until the client confirms complete reception and the progress difference is corrected.

[0067] The entire synchronization process continues until the streaming session ends. This mechanism effectively solves the state drift problem that may be caused by dual-path parallel transmission with extremely low communication and computational overhead, ensuring that the entire system has a reliable guarantee of strong eventual consistency while pursuing high transmission performance.

[0068] In one embodiment of the present invention, based on step S4, the following will provide a possible embodiment and describe its specific implementation in a non-limiting manner.

[0069] S401. Monitor the transmission quality indicators of the bypass channel and the processing status of the control flow engine in real time, wherein the transmission quality indicators include at least one of transmission delay, packet loss rate and transmission rate fluctuation.

[0070] Probes are deployed on both the model and client sides of the bypass channel to collect and calculate key transmission quality metrics in real time: 1) Transmission latency: measuring the end-to-end time from the model outputting a token to the client confirming receipt; 2) Packet loss rate: statistically analyzing the percentage of packets lost within a specific time window through sequence number continuity checks; 3) Transmission rate fluctuation: monitoring the rate of change in the number of successfully transmitted tokens per unit time. Simultaneously, the module also monitors key states of the control flow engine, such as its processing queue depth, average response time, and error rate. These metrics are aggregated into a real-time dashboard and fed into a lightweight anomaly detector.

[0071] S402. When it is determined that the bypass channel is abnormal according to the transmission quality index, the transmission path is dynamically adjusted to switch the transmission of at least part of the subsequent data tokens from the bypass channel to the control flow path forwarded by the control flow engine.

[0072] The anomaly detector sets dynamic thresholds for various metrics (e.g., latency consistently exceeding 200ms, packet loss rate greater than 5%). Once an anomaly is detected in the bypass channel (e.g., severe network jitter or persistent high packet loss), the system immediately triggers path switching logic. Switching decisions can be based on session granularity or token priority: for the entire affected session, the transmission path of all subsequent data tokens may be downgraded from the "bypass channel" to the control flow path forwarded by the control flow engine; for finer-grained switching, only low-priority or non-real-time data tokens may be switched, while high-priority tokens continue to attempt transmission through the bypass channel. The switching process should be smooth, minimizing the perceived interruption to client rendering.

[0073] S403. In response to token loss during an abnormal period, select and execute a corresponding retransmission mechanism from a variety of preset retransmission strategies based on real-time network conditions. The various retransmission strategies include immediate retransmission, batch retransmission, or gradual retransmission.

[0074] For tokens confirmed to be lost during an anomaly or handover, the system initiates intelligent retransmission. The retransmission manager selects the optimal strategy from a pre-defined strategy library based on real-time network conditions (such as RTT and bandwidth estimation): 1) Immediate retransmission: When network conditions are good, a separate retransmission request is immediately initiated for each lost sequence number, aiming for the fastest possible recovery; 2) Batch retransmission: When network congestion occurs, all lost sequence numbers within a time window are packaged and retransmitted all at once, reducing request overhead; 3) Gradual retransmission: In uncertain network environments, the most critical part of the token (such as the beginning of a sentence) is retransmitted first, and then the remaining tokens are gradually completed based on subsequent conditions. All retransmitted tokens are sent via a more reliable control flow path by default during the recovery phase. When monitoring shows that the bypass channel quality has returned to stable and has remained stable for a period of time, the system can automatically switch the path strategy back to optimized mode and re-enable the bypass channel for high-speed transmission.

[0075] In some embodiments, the dual-path transmission system for streaming tokens may include multiple functional modules composed of computer program segments. The computer programs of each program segment in the dual-path transmission system for streaming tokens may be stored in the memory of a computer device and executed by at least one processor to perform the function of dual-path transmission of streaming tokens (see Figure 1 for details).

[0076] In this embodiment, the dual-path transmission system for streaming tokens can be divided into multiple functional modules according to the functions it performs, as shown in Figure 3. A module, as referred to in this invention, is a series of computer program segments that can be executed by at least one processor and perform a fixed function, stored in memory. In this embodiment, the functions of each module will be detailed in subsequent embodiments.

[0077] An initialization module is used to initialize the system and session in response to streaming requests, including establishing a bypass channel for direct data transmission between the model and the client, and creating a corresponding session context in the control flow engine. A path routing module is used to identify the type of each token generated by the model and route it according to its type: if it is a data token, it is routed to the bypass channel; if it is a control token, it is routed to the control flow path that needs to be processed by the control flow engine. A state synchronization module is used to execute data transmission and business control processes in parallel, and ensures state consistency between the bypass channel and the control flow engine through a state synchronization mechanism. The data transmission includes directly pushing the data token to the client through the bypass channel; the business control process includes the control flow engine processing the control token and updating the session state. An exception handling module is used to monitor the operating status of the bypass channel and the control flow engine, and dynamically adjust the path or perform exception recovery when an exception is detected.

[0078] Figure 4 illustrates a dual-path transmission method for streaming tokens provided in this embodiment, applicable to a device. Those skilled in the art will understand that the device structure described in this embodiment does not constitute a limitation on the device. A device may include more or fewer components than illustrated, or combine certain components, or have different component arrangements. In this embodiment, the device includes, but is not limited to, laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments described and / or claimed herein.

[0079] The device 400 may include a processor 410, a memory 420, and a communication unit 430. These components communicate via one or more buses. Those skilled in the art will understand that the server structure shown in the figure does not constitute a limitation of the present invention. It may be a bus topology or a star topology, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0080] The memory 420 can be used to store execution instructions of the processor 410. The memory 420 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. When the execution instructions in the memory 420 are executed by the processor 410, the device 400 is able to perform some or all of the steps in the above method embodiments.

[0081] The processor 410 serves as the control center of the storage device, connecting various parts of the electronic device via various interfaces and lines. It executes software programs and / or modules stored in the memory 420, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of integrated circuits (ICs), such as a single packaged IC or multiple packaged ICs with the same or different functions connected together. For example, the processor 410 may consist only of a central processing unit (CPU). In this embodiment of the invention, the CPU may have a single processing core or include multiple processing cores.

[0082] The communication unit 430 is used to establish a communication channel, enabling the storage device to communicate with other devices. It can receive user data sent by other devices or send user data to other devices.

[0083] The present invention also provides a computer storage medium, wherein the computer storage medium may store a program, which, when executed, may include some or all of the steps provided in the embodiments of the present invention. The storage medium may be a magnetic disk, an optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0084] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, or any other medium capable of storing program code. It includes several instructions to cause a computer device (which may be a personal computer, a server, or a second device, network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0085] The same or similar parts between the various embodiments in this specification can be referred to mutually. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple, and the relevant parts can be referred to the description in the method embodiments.

[0086] In the embodiments provided by this invention, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or modules may be electrical, mechanical, or other forms.

[0087] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0088] In addition, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0089] Although the present invention has been described in detail with reference to the accompanying drawings and preferred embodiments, the present invention is not limited thereto. Various equivalent modifications or substitutions can be made to the embodiments of the present invention by those skilled in the art without departing from the spirit and essence of the invention, and such modifications or substitutions should all be within the scope of the present invention. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should also be covered within the protection scope of the present invention.

Claims

1. A dual-path transmission method for streaming tokens, characterized in that, include: In response to streaming requests, system and session initialization is performed, including establishing a bypass channel for direct data transmission between the model and the client, and creating the corresponding session context in the control flow engine; Identify the type of each token generated by the model and route it according to the type: if it is a data token, route it to the bypass channel; If it is a control token, it is routed to the control flow path that needs to be processed by the control flow engine; data transmission and business control processes are executed in parallel, and the state consistency between the bypass channel and the control flow engine is ensured through a state synchronization mechanism; wherein, the data transmission includes pushing the data token directly to the client through the bypass channel; The business control process includes the control flow engine processing the control token and updating the session state; monitoring the running status of the bypass channel and the control flow engine, and dynamically adjusting the path or performing abnormal recovery when an anomaly is detected.

2. The method according to claim 1, characterized in that, In response to a streaming request, system and session initialization is performed, including: establishing the bypass channel, specifically by pre-establishing a persistent connection between the model and the client using the WebSocket or QUIC protocol, and enabling connection reuse for multiple streaming requests from the same client; predicting the initial cache capacity based on historical streaming data characteristics, and setting a dynamic buffer for the bypass channel based on the initial cache capacity; creating a session context corresponding to the streaming request in the control flow engine, wherein the session context contains at least a unique session identifier; and negotiating and determining an initial dual-path transmission strategy based on the characteristics of the streaming request, wherein the characteristics include the estimated response length and / or the real-time requirements of the business logic.

3. The method according to claim 2, characterized in that, The negotiation basis for the initial dual-path transmission strategy also includes the data type of the streaming token, and adjusting the corresponding transmission parameters and chunk size for text, audio or video stream types.

4. The method according to claim 1, characterized in that, The identification model identifies the type of each token generated, including: determining whether the token is a control instruction or pure data content based on preset classification rules; wherein, control instructions include tokens used for process control, tool invocation, or business logic processing, and pure data content includes streaming output content of text, audio, or video; and attaching streaming metadata to the identified pure data content tokens, wherein the metadata includes at least: stream identifier, block sequence number, whether it is an end block, and high-precision timestamp.

5. The method according to claim 4, characterized in that, The classification rules include: Keyword matching rule: If the content of the token matches a predefined set of control instruction keywords, it is determined to be a token of the control instruction type; Sequence position rule: If the token is in a predefined specific sequence position, including the session start position, session end position, or synchronization checkpoint position set at fixed intervals, it is determined to be a token of the metadata type used for state synchronization; Content exclusion rule: If the token does not belong to the control instruction type or the metadata type, it is determined to be a token of the pure data type; Dynamic adaptive rule: The token type determination or transmission path selection is dynamically adjusted based on the current system context information, which includes at least one of network conditions, system load, or session stage.

6. The method according to claim 1, characterized in that, The system executes data transmission and business control processes in parallel, and ensures state consistency between the bypass channel and the control flow engine through a state synchronization mechanism. This includes: directly pushing the identified data token to the client for real-time rendering via the bypass channel; sending the identified control token to the control flow engine, which then executes the corresponding business process and updates the session state; and ensuring, through a lightweight state synchronization mechanism, that the client's progress in receiving data via the bypass channel remains consistent with the session state maintained by the control flow engine. This lightweight state synchronization mechanism includes the control flow engine periodically sending heartbeat signals to the client. The heartbeat signal contains at least the latest processed token sequence number recorded by the control flow engine. The client compares the latest token sequence number it has successfully received through the bypass channel with the sequence number in the heartbeat signal. When the comparison finds that the sequence number difference exceeds a preset threshold, a state repair process is triggered. The state repair process includes at least one of the following methods: if the client's receiving progress is ahead of the control flow engine's progress, the client buffers and waits for the received ahead tokens; if the client's receiving progress is behind the control flow engine's progress or a token loss is detected, a retransmission mechanism for the missing token is triggered.

7. The method according to claim 1, characterized in that, The system monitors the operational status of the bypass channel and the control flow engine, and dynamically adjusts the path or performs abnormal recovery when an anomaly is detected. This includes: real-time monitoring of the transmission quality indicators of the bypass channel and the processing status of the control flow engine, wherein the transmission quality indicators include at least one of transmission latency, packet loss rate, and transmission rate fluctuation; when an anomaly is determined to have occurred in the bypass channel based on the transmission quality indicators, the system dynamically adjusts the transmission path, switching the transmission of at least a portion of the subsequent data tokens from the bypass channel to the control flow path forwarded by the control flow engine; for token loss occurring during the anomaly, the system selects and executes a corresponding retransmission mechanism from multiple preset retransmission strategies based on real-time network conditions, wherein the multiple retransmission strategies include immediate retransmission, batch retransmission, or gradual retransmission.

8. A dual-path transmission system for streaming tokens, characterized in that, include: The initialization module is used to respond to streaming requests and perform system and session initialization, including establishing a bypass channel for direct data transmission between the model and the client, and creating the corresponding session context in the control flow engine; The path routing module is used to identify the type of each token generated by the model and to route it according to the type: if it is a data token, it is routed to the bypass channel; If it is a control token, it is routed to a control flow path that needs to be processed by the control flow engine. A state synchronization module is used to execute data transmission and business control processes in parallel, and to ensure state consistency between the bypass channel and the control flow engine through a state synchronization mechanism. The data transmission includes directly pushing the data token to the client through the bypass channel. The business control process includes the control flow engine processing the control token and updating the session state. An exception handling module is used to monitor the operating status of the bypass channel and the control flow engine, and to dynamically adjust the path or perform exception recovery when an exception is detected.

9. A dual-path transmission device for streaming tokens, characterized in that, include: Memory, used to store the dual-path transmission program for streaming tokens; A processor, configured to implement the steps of the dual-path transmission method for streaming tokens as described in any one of claims 1-7 when executing the dual-path transmission procedure for the streaming token.

10. A computer-readable storage medium storing a computer program, characterized in that, The readable storage medium stores a dual-path transmission program for streaming tokens, which, when executed by a processor, implements the steps of the dual-path transmission method for streaming tokens as described in any one of claims 1-7.

Images