A voice session continuity guaranteeing method based on dual-protocol cooperation

Abstract

The application discloses a voice conversation continuity guarantee method based on double-protocol cooperation, which is applied to a client and comprises the following steps: receiving voice data input by a user; based on a real-time judgment result of a current network state, splitting the voice data into core data and non-core data according to preset data priority rules; transmitting the core data through a first protocol communication channel; and transmitting the non-core data through a second protocol communication channel; wherein the first protocol is a lightweight publish / subscribe protocol, and the second protocol is a security protocol based on full-duplex communication. Dynamic optimization of the transmission strategy is realized on the client side, thereby providing the voice conversation with the ability of double guarantee of basic connectivity and high-quality experience in a complex and changeable network environment, and effectively solving the contradiction that a single protocol cannot adapt to network dynamic changes and business diverse demands.

Description

Technical Field

[0001] This invention belongs to the field of voice session assurance technology, specifically relating to a method for ensuring voice session continuity based on dual-protocol collaboration. Background Technology

[0002] With the deep integration of IoT and AI technologies, hardware voice interaction devices, such as smart speakers, in-vehicle voice assistants, wearable devices, and industrial voice terminals, have been widely used in daily life and production scenarios. The essence of these devices is to enable real-time, continuous, and reliable voice dialogue between users and cloud services. Their core experience highly depends on the performance of the communication link between the device and the server, especially in complex network environments such as mobile and outdoor environments, where three key requirements must be met: adaptability to weak network conditions, high-quality transmission, and rapid response.

[0003] The industry generally uses application-layer protocols based on TCP / IP to build communication links. Among them, WebSocket Secure and MQTT are two mainstream but very different technical approaches.

[0004] As a security-enhanced version of WebSocket, the WSS protocol provides a full-duplex communication channel through TLS / SSL encryption. Its advantages include high transmission speed, strong security, and support for continuous transmission of large amounts of data, making it ideal for carrying high-fidelity high-definition voice streams and detailed business data containing rich information. However, establishing a connection via the WSS protocol requires a TCP three-way handshake and a TLS four-way handshake, which takes a relatively long time (typically hundreds of milliseconds). Furthermore, maintaining the encrypted link requires continuous bandwidth overhead, making it extremely sensitive to network fluctuations in weak network environments. Insufficient bandwidth or excessive latency can easily lead to connection interruptions, causing session stuttering or termination.

[0005] In contrast, MQTT is an extremely lightweight publish / subscribe messaging protocol. Its header requires as little as 2 bytes, connection establishment is simple and fast (typically within milliseconds), and it supports QoS (Quality of Service) levels to ensure reliable message delivery. These characteristics make it excellent in resource-constrained embedded devices and weak network environments, making it ideal for transmitting small amounts of high-priority core information such as session control commands, critical status notifications, and basic responses. However, due to its design limitations, MQTT has limited data throughput and cannot efficiently handle continuous media streams requiring high bandwidth (such as raw high-definition audio). Forcing its use in this scenario would lead to a significant increase in transmission latency and a degraded user experience.

[0006] In the face of complex real-world network environments, it is impossible to simultaneously meet the conflicting demands of weak network connectivity and high-quality network performance. If only the WSS protocol is used, its advantage of high-quality transmission becomes a burden when network quality is poor, leading to frequent session interruptions. If only the MQTT protocol is used, although it can guarantee basic connectivity under weak network conditions, it cannot support high-quality voice interaction, limiting the upper limit of product functionality and user experience.

[0007] Although there have been some attempts to integrate dual protocols, their design ideas are mostly limited to a primary / backup switchover mode, where the two protocols operate in a mutually exclusive state, and only passively switch to the backup protocol (MQTT) after the primary protocol (usually WSS) completely fails. Such solutions cannot guarantee the smooth flow of core commands through lightweight protocols in the early stages of network fluctuations, nor can they coordinate the two protocols to maximize transmission efficiency in a strong network environment. Summary of the Invention

[0008] This invention provides a method for ensuring voice session continuity based on dual-protocol collaboration. It can dynamically collaborate and intelligently switch between WSS and MQTT protocols according to the real-time network status, so as to realize or solve the problem that weak network sessions are easily interrupted and strong network high-quality transmission cannot be guaranteed at the same time due to the limitations of single protocol characteristics.

[0009] The technical solution adopted in this invention is as follows: A method for ensuring voice session continuity based on dual-protocol collaboration, applied to a client, the method comprising: Receive voice data input by the user; Based on the real-time determination of the current network status, and according to the preset data priority rules, the voice data is divided into core data and non-core data. The core data is transmitted through the first protocol communication channel; The non-core data is transmitted through the second protocol communication channel; The first protocol is a lightweight publish-subscribe protocol, and the second protocol is a security protocol based on full-duplex communication.

[0010] Based on the real-time assessment of the current network status, and according to preset data priority rules, the voice data is divided into core data and non-core data, specifically including: When the determination result is a weak network state, instruction features and / or text features are extracted from the speech data as the core data, and the original speech data stream is used as the non-core data.

[0011] Prior to the step of receiving user-input voice data, a connection pre-enabling step is also included: Simultaneously initiate connection requests to establish the first protocol communication channel and the second protocol communication channel with the server; If the second protocol communication channel has not yet been successfully established, but the first protocol communication channel has been established, the user's voice input data is received and transmitted through the first protocol communication channel.

[0012] The method also includes a session migration step: After detecting that the second protocol communication channel has been successfully established, a migration notification containing the current session state is generated; The migration notification is sent to the server through the second protocol communication channel, and the default transmission channel for subsequent non-core data is switched to the second protocol communication channel.

[0013] The default transmission channel for subsequent non-core data will be switched to the second protocol communication channel, specifically as follows: Based on the silence intervals of user-input voice data, the default transmission channel for non-core data is switched.

[0014] The non-core data is transmitted through the second protocol communication channel, including: When the transmission of the non-core data fails in the second protocol communication channel, the non-core data is cached in the local data cache module; After the network status is detected to be restored, the cached non-core data will be retransmitted through the second protocol communication channel.

[0015] This invention also provides a method for ensuring voice session continuity based on dual-protocol collaboration, applied to the server side, the method comprising: Simultaneously receive connection requests from the client to establish a first protocol communication channel and a second protocol communication channel, wherein the first protocol is a lightweight publish-subscribe protocol and the second protocol is a security protocol based on full-duplex communication; If the second protocol communication channel has not yet been successfully established, but the first protocol communication channel has been established, the voice data transmitted through the first protocol communication channel is received. The voice data includes core data and non-core data. After detecting that the second protocol communication channel has been successfully established, a migration notification containing the current session state is received through the second protocol communication channel, and the default transmission channel for subsequent non-core data is switched to the second protocol communication channel.

[0016] Before the second protocol communication channel is successfully established, including: A service failure notification is sent to the client through the first protocol communication channel, and the service failure notification is used to indicate that the second protocol service is abnormal. In response to the service failure notification, the voice data transmission channel is switched to the first protocol communication channel via the client.

[0017] The method also includes: After switching to the first protocol communication channel, receive the session takeover confirmation information sent through the first protocol communication channel; Continuously probe whether the second protocol service on the server side has been restored; After the recovery is confirmed, a switchback process from the first protocol communication channel to the second protocol communication channel is triggered.

[0018] This invention provides another voice conversation system. This includes both client-side and server-side components to execute the method. The client and the server interact with each other through the first protocol communication channel and the second protocol communication channel.

[0019] Due to the adoption of the above technical solution, the beneficial effects achieved by this invention are as follows: 1. This invention introduces a decision-making process based on real-time network status assessment. When a weak network condition (e.g., low bandwidth, high packet loss) is detected, the voice data is forcibly split into core and non-core parts. The lightweight, low-overhead, and highly reliable characteristics of a first protocol (e.g., MQTT) are used to specifically transmit core data (e.g., commands, keywords), ensuring that the most basic control flow and interaction logic of the session remain connected even under adverse network conditions. This reduces the risk of overall session interruption due to network fluctuations to a problem of ensuring the transmission of core data streams.

[0020] Furthermore, a dynamic and clearly defined transmission system is constructed through a two-step process of data priority splitting and protocol feature matching. A lightweight first protocol ensures the reliable arrival of core data, maintaining session continuity. Non-core data is allowed to be transmitted via a more powerful second protocol. This means that, when network conditions permit, the high bandwidth and low latency advantages of the second protocol can be automatically utilized to improve the user experience.

[0021] Moreover, in weak network scenarios, limited network resources are especially valuable. By layering data, non-core high-volume data is prevented from crowding out the already strained bandwidth, thus freeing up resource channels for the transmission of core commands and reducing latency in core interactions. This resource allocation strategy ensures that, under the same network conditions, priority is given to ensuring the most sensitive user operation feedback, improving perceived responsiveness. From the perspective of overall system efficiency, the most suitable transmission channels are dynamically allocated based on data value and application scenarios, reducing the ineffective consumption of network resources.

[0022] In summary, dynamic optimization of transmission strategies was implemented on the client side, thereby providing voice conversations with both basic connectivity and high-quality experience in complex and ever-changing network environments. This effectively resolves the contradiction that a single protocol cannot adapt to dynamic network changes and diverse business needs. Attached Figure Description

[0023] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a flowchart illustrating the voice session continuity guarantee method based on dual-protocol collaboration according to one embodiment of the present invention, wherein the method is applied to the client. Detailed Implementation

[0024] To more clearly illustrate the overall concept of the present invention, a detailed description will be provided below with reference to the accompanying drawings and examples.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0026] like Figure 1 As shown, a method for ensuring voice session continuity based on dual-protocol collaboration is applied to a client. The method includes: S100: Receives voice data input by the user; S200: Based on the real-time determination of the current network status, the voice data is divided into core data and non-core data according to the preset data priority rules; S300: Transmit the core data through the first protocol communication channel; The non-core data is transmitted through the second protocol communication channel; The first protocol is a lightweight publish-subscribe protocol, and the second protocol is a security protocol based on full-duplex communication.

[0027] The main purpose of this method is to resolve the contradiction between the existing single-protocol solutions and the inability to simultaneously ensure basic connectivity and a high-quality experience for voice conversations in dynamic network environments. Its core design idea is to abandon reliance on a single protocol and introduce a real-time, policy-based intelligent decision-making layer. Based on the current network conditions, this layer dynamically distributes voice data streams to two complementary communication protocols, thereby achieving optimal matching between network resources and service requirements.

[0028] First, the system receives voice data input from the user. This step marks the beginning of the interaction. The client acquires the user's raw analog voice signal through audio input devices such as microphones, and converts it into a processable digital format (such as PCM, Opus, etc.) via an audio codec. To enable session tracking and management, a unique session ID is typically generated at this stage for this interaction. This ID will be used throughout all subsequent data processing and transmission. The main purpose of this step is to transform physical-world voice interaction into a standardized, identifiable digital information stream, providing raw materials for subsequent intelligent analysis and scheduling.

[0029] Secondly, based on the real-time assessment of the current network status, and according to preset data priority rules, the voice data is divided into core data and non-core data. This is a key decision-making step for achieving intelligent collaboration, and it involves two parallel logic steps.

[0030] Firstly, real-time network status assessment: The client's built-in network status monitoring module continuously operates, quantitatively evaluating key indicators such as bandwidth, latency, and stability of the current link in real time by sending probe packets, analyzing the round-trip time (RTT) and packet loss rate of protocol heartbeat packets, or statistically analyzing recent data transmission rates. Its core strategy is to set dynamic or static thresholds (e.g., bandwidth below 1Mbps and packet loss rate above 20%). When monitored indicators exceed the threshold, the network is determined to be in a weak state.

[0031] Secondly, intelligent data priority allocation: a pre-built data priority rule base is provided. The core definition of this rule is that core data is the minimum dataset necessary to ensure the continuity of conversation logic, typically including instruction features extracted by local lightweight speech recognition (VAD, keyword wake-up), control command text, conversation state synchronization signaling, etc. This type of data is small in volume, but has extremely high requirements for transmission reliability and real-time performance; loss or delay will lead to conversation interruption or logical errors.

[0032] Non-core data mainly consists of raw or high-definition encoded voice streams to improve the quality of the experience. It has a large data volume and allows for a certain amount of transmission delay or occasional packet loss. This can be partially compensated for through retransmission or error correction mechanisms without affecting the main logic of the session.

[0033] When a weak network is identified, this step will extract the core instruction features from the original voice data packets as core data according to this rule, while marking the high-volume audio stream portion as non-core data.

[0034] This breaks through the traditional, crude transmission model. By stratifying data by value, it ensures the smooth transmission of high-value, low-volume core commands when network resources are strained, guaranteeing uninterrupted sessions. Simultaneously, it provides precise input for subsequent differentiated transmission utilizing the characteristics of different protocols.

[0035] Furthermore, it should be noted that the core data is transmitted through the first protocol communication channel. Specifically, the first protocol is MQTT (Message Queuing Telemetry Transport) or its equivalent lightweight publish-subscribe protocol. This leverages the inherent characteristics of MQTT: minimal header overhead, fast connection establishment, and support for QoS (Quality of Service) levels. Especially after a weak network detection is triggered, the extracted core data (instructions, keywords) is encapsulated into MQTT messages and typically published at QoS 1 (at least once) or a higher level.

[0036] Even under poor network conditions, this critical information can be transmitted with the highest probability and the lowest additional bandwidth consumption, ensuring that the server can understand the user's intent and make a basic response, thereby maintaining the most basic interactive continuity of the session.

[0037] The non-core data is transmitted through a second protocol communication channel. Preferably, the second protocol is WSS (WebSocket Secure) or an equivalent security protocol based on full-duplex communication. The high throughput, low latency, and streaming support of the WSS protocol are utilized to carry large volumes of data. In weak network conditions, transmission through this channel may be hindered, but the core session is guaranteed by the MQTT channel; therefore, the overall session does not depend on the real-time performance of this channel.

[0038] Once network conditions improve, this channel can immediately and efficiently retransmit or resume high-quality voice data, enabling the server to perform more accurate voice recognition or provide richer service responses, thereby ensuring connectivity and a superior user experience. In a strong network environment, this channel can serve as the backbone for high-quality data transmission.

[0039] This invention constructs a closed-loop, adaptive transmission control system on the client side. It enables WSS and MQTT protocols to work together and complement each other, dynamically balancing session robustness and high-quality experience, ultimately achieving the goal of ensuring uninterrupted voice sessions in weak networks and high-quality voice sessions in strong networks.

[0040] In a preferred embodiment of the present invention, based on the real-time determination of the current network status, the voice data is divided into core data and non-core data according to a preset data priority rule, specifically including: When the determination result is a weak network state, instruction features and / or text features are extracted from the speech data as the core data, and the original speech data stream is used as the non-core data.

[0041] When the network monitoring module detects that metrics (such as bandwidth and packet loss rate) continuously exceed preset thresholds, it will issue a clear weak network command to the protocol scheduling center. This command is the sole switch that triggers this refined segmentation logic, ensuring that this operation, which may incur additional processing overhead, is not executed in a strong network environment.

[0042] In this mode, it does not attempt to transmit complete, unprocessed raw speech over a network with unstable quality. Instead, it utilizes the client's local computing resources to perform lightweight real-time processing.

[0043] Command Feature Extraction: Through integrated Voice Activity Detection (VAD) and keyword recognition modules, effective speech segments are detected in real time from a continuous speech stream, and preset key command words (such as "play", "pause", "search") are identified. These recognition results are converted into structured command codes or very short text tags.

[0044] Text Feature Extraction: For more complex voice requests, a lightweight on-device speech recognition engine can be enabled to convert user speech into a concise text stream in real time. This process is optimized, for example, by recognizing and outputting only core entity words and intent keywords, while omitting modifiers, to generate a text summary with a very small amount of data but sufficient to express the core intent.

[0045] The instruction codes or key text generated by the above processing, together with the unique identifier of this session (Session ID), constitute the core data. Its data size is usually only tens to hundreds of bytes, which is negligible in terms of transmission burden compared to the tens of KB to several MB of the original audio stream.

[0046] The raw voice data stream is explicitly defined as non-core data. This refers to the complete, raw coded voice packets that have not undergone the aforementioned feature extraction processing. Even if network conditions permit, in weak network conditions, the transmission of this data may be strategically delayed or attempted, and its success or failure will not affect the progress of the core interaction.

[0047] By transforming the transmission of complete voice streams into the transmission of minimally categorized characteristic data, this method reduces the amount of data that must be delivered by several orders of magnitude in weak network conditions. This ensures that even in environments with extremely limited bandwidth and severe packet loss, core data has a very high probability of reaching the server quickly and reliably via lightweight protocols such as MQTT, fundamentally solving the problem of transmitting large volumes of data in weak network conditions.

[0048] It should be noted that although features are transmitted rather than the original audio, the core user intent has been accurately extracted and conveyed. Upon receiving the instruction code or key text, the server can effectively drive the business logic to respond (e.g., start playing the specified song, return basic information of the query results), thereby maintaining the continuity and functionality of the session.

[0049] In a preferred embodiment of the present invention, a connection pre-activation step is further included before the step of receiving user-input voice data: Simultaneously initiate connection requests to establish the first protocol communication channel and the second protocol communication channel with the server; If the second protocol communication channel has not yet been successfully established, but the first protocol communication channel has been established, the user's voice input data is received and transmitted through the first protocol communication channel.

[0050] In this embodiment, the connection pre-enabling step, performed before the step of receiving user-input voice data, aims to completely eliminate the session response gap caused by waiting for a high-quality secure connection (such as WSS) to be established in traditional solutions, achieving zero-wait instant response to user voice requests after device wake-up or application startup. The core strategy of this step is to reconstruct the serial "connection-service" process into a parallel "connection|service" process, utilizing the time difference in establishing dual-protocol connections to preemptively provide service capabilities using the first-ready lightweight channel.

[0051] When the client starts up or the voice session function is initialized, its core control module (protocol scheduling center) synchronously and independently initiates two connection requests to the server: one is a communication channel connection request based on a first protocol (such as MQTT), and the other is a communication channel connection request based on a second protocol (such as WSS). This is an active, parallel connection establishment process, rather than a traditional master-slave sequential connection.

[0052] Internally, a connection state machine is maintained to continuously monitor the connection establishment progress of the two channels. Due to the fundamental differences in protocol design, the first protocol (MQTT) typically only requires a simple TCP connection and protocol layer handshake, and can quickly enter the ready state within a few hundred milliseconds; while the second protocol (WSS) requires multiple rounds of interaction, including TCP handshake, TLS negotiation, and WebSocket handshake, and the establishment time is usually longer, with a clear time window.

[0053] Once a state combination is detected where the first protocol communication channel has been successfully established but the second protocol communication channel has not yet been successfully established, the pre-activation mode is immediately triggered. In this mode, the protocol scheduling center will logically prioritize routing the incoming or already received initial user voice data to the ready first protocol channel for transmission and processing. At this time, the connection establishment process of the second protocol continues in the background without affecting the activation of foreground services.

[0054] In traditional solutions, users must wait for the time-consuming WSS connection to be fully established before they can begin a conversation, resulting in a poor user experience of no response after wake-up. This approach leverages the fast nature of MQTT connections to advance the service activation point to the millisecond level, allowing users to receive initial interaction feedback almost imperceptibly, significantly improving the responsiveness and perceived usability of voice products.

[0055] As a preferred embodiment of this implementation, the method further includes a session migration step: After detecting that the second protocol communication channel has been successfully established, a migration notification containing the current session state is generated; The migration notification is sent to the server through the second protocol communication channel, and the default transmission channel for subsequent non-core data is switched to the second protocol communication channel.

[0056] In this embodiment, the session migration step, introduced after the connection pre-enabling step successfully achieves an immediate response, primarily aims to transition from a fast-response channel to a high-quality transmission channel. It ensures that once a high-performance second protocol (such as WSS) channel is ready, the data transmission backbone is switched from the first protocol (such as MQTT) to the second protocol.

[0057] The client continuously monitors the status of the second protocol communication module. When it detects that the channel has not only completed the underlying handshake but also confirmed its connection stability and low latency through several consecutive heartbeats, it determines that the channel has been successfully established and has reached a service state capable of supporting business operations. At this point, the system generates a specific internal event, triggering the migration process.

[0058] The client core module (protocol scheduling center) extracts the key state context of the currently ongoing session, which includes at least: a unique session identifier (Session ID), the current interaction round, the history of user requests processed through the first protocol channel and their corresponding response summaries, and any business states that need to be continued (such as playback progress, query context). This information is encapsulated into a structured migration notification message. Specifically, this notification is sent to the server through the newly established second protocol communication channel. This serves a dual purpose: first, to verify the availability of the new channel; and second, to use the new channel to synchronize the state with the server, ensuring that subsequent requests through this channel can be correctly associated and processed.

[0059] Simultaneously with or shortly after issuing the migration notification (e.g., upon receiving confirmation from the server), the client protocol scheduling center internally updates the routing rules, redirecting the default output path for subsequent non-core data (typically high-quality raw or encoded audio streams) from the first protocol communication module to the second protocol communication module. This means that subsequent spoken words will have their high-quality audio data transmitted directly through the higher-performance WSS channel.

[0060] Specifically, the default transmission channel for subsequent non-core data will be switched to the second protocol communication channel, as follows: Based on the silence intervals of user-input voice data, the default transmission channel for non-core data is switched.

[0061] The primary objective of this embodiment is to achieve the highest degree of seamless handover. This strategy aims to completely avoid the voice stuttering, data packet timing disorder, or loss that inevitably occurs when a user is continuously speaking and voice data packets are being transmitted continuously. This ensures that the migration process is completely transparent to the user and does not interfere with their natural voice interaction flow.

[0062] The client integrates a highly sensitive voice activity detection module. This module continuously analyzes the input audio stream, accurately identifying not only the start and end of speech but also precisely determining short pauses (i.e., silence intervals) between user statements. This is typically achieved by analyzing features such as audio energy and zero-crossing rate, combined with preset time thresholds (e.g., silence lasting more than 200 milliseconds). When a silence interval meeting the criteria is detected, the VAD module sends a clear silence window signal to the protocol scheduling center.

[0063] The protocol scheduling center maintains a state machine internally. When it receives a trigger signal indicating that the second protocol channel is ready and plans to execute the migration, it does not act immediately. Instead, it waits and listens for a silence window signal from the VAD module. Only when both the channel is ready and a silence gap is detected are the scheduling center determined that the current time is a safe handover opportunity.

[0064] Once the timing is deemed suitable, the system will rapidly execute a series of atomic operations within this brief silence window: immediately suspend the transmission queue for newly acquired audio data; update the internal routing table, redirecting the default exit point for non-core data to the second protocol module; send a migration notification via the second protocol channel (if it has not yet been sent); and restore the transmission queue, allowing subsequent audio data to flow out via the new channel. All of these operations must be completed before the silence interval ends and the next voice sentence begins.

[0065] It's worth noting that a dynamic buffering and prediction mechanism could be considered. For example, during the waiting period between silences, newly acquired audio data could be temporarily stored in a very short circular buffer. Simultaneously, the system could combine historical interaction patterns to learn from the user's speaking rhythm, more intelligently predicting potentially longer silences and triggering a switch at the most appropriate time, further reducing the risk of minor delays introduced by waiting for silence.

[0066] Voice communication demands extremely high temporal continuity. Switching network channels mid-transmission of a voice stream can easily lead to audible pops, stutters, or syllable loss at the receiving end. By strictly constraining the switching operation to the intervals between the user's natural pauses, the segmentation of the continuous voice stream is perfectly avoided, ensuring that the voice response heard by the user is consistently smooth from beginning to end, thus achieving seamless migration. Moreover, this prevents the audio frames of the same sentence from being split and transmitted across two different protocol channels, thereby completely eliminating the risk of data packet reordering, loss, or server-side processing logic confusion that might result, ensuring the correctness of business processing.

[0067] In a preferred embodiment of the present invention, the non-core data is transmitted through a second protocol communication channel, including: When the transmission of the non-core data fails in the second protocol communication channel, the non-core data is cached in the local data cache module; After the network status is detected to be restored, the cached non-core data will be retransmitted through the second protocol communication channel.

[0068] The main purpose of this implementation is to address the issue of momentary or continuous transmission failures that may occur in the high-quality transmission channel (second protocol) during periods of weak network or network fluctuations. By introducing local caching and delay compensation mechanisms, the final accessibility and integrity of non-core data (high-quality voice stream) are guaranteed to the maximum extent while ensuring the continuity of core sessions, thus achieving a complete high-quality experience.

[0069] When sending non-core data packets, the client's second protocol communication module implements a reliable transmission confirmation mechanism. If no application-layer confirmation is received from the server within the preset timeout period after transmission, or if the underlying protocol (such as TCP) indicates a transmission failure, the transmission is considered to have failed.

[0070] At this point, instead of simply discarding the data packet or requesting the upper layer to re-record the voice, the caching process is immediately triggered. The protocol scheduling center writes the complete data packet that failed to be sent (including its sequence number, timestamp, and associated session ID) into the local data cache module. This module is typically designed as a circular buffer or queue with a sequential structure to manage limited device storage space.

[0071] The network status monitoring module continues to operate. Network recovery is not determined simply by a single successful probe, but rather by a more robust strategy, such as: the average bandwidth of multiple consecutive probes recovers to above a set threshold, and the packet loss rate decreases to an acceptable range and remains stable for a period of time. This determination aims to distinguish between brief network fluctuations and genuine quality recovery, avoiding premature retransmissions in an still unstable network that could lead to secondary failures or exacerbate congestion.

[0072] Once the network status is confirmed to have recovered and the second protocol channel is available again, the protocol scheduling center will instruct the data caching module to resubmit the cached non-core data to the second protocol communication module for retransmission, following the original time sequence of the data packets. The retransmission strategy can be intelligently optimized, for example, by initially using a lower rate to test the waters and then gradually increasing the speed, to avoid overwhelming the newly recovered network with a large amount of data retransmission at any given moment.

[0073] It should be noted that before caching, deduplication can be performed based on the unique identifier of the data packet to avoid the same data packet being cached repeatedly due to retry mechanisms. Additionally, on resource-constrained devices, non-core data in the cache can be lightweight compressed to save storage space and extend the cacheable data duration.

[0074] In weak network conditions, while the transmission of non-core data (raw high-definition voice stream) can be temporarily sacrificed to ensure the smooth operation of core data, the complete voice information it carries is crucial for the server to perform accurate speech recognition, sentiment analysis, or generate detailed responses. Caching and retransmission mechanisms ensure that this data is eventually delivered, enabling the server to provide more accurate and richer services based on complete information once network conditions improve, thus achieving a superior conversational experience.

[0075] This invention also provides a method for ensuring voice session continuity based on dual-protocol collaboration, applied to the server side, the method comprising: Simultaneously receive connection requests from the client to establish a first protocol communication channel and a second protocol communication channel, wherein the first protocol is a lightweight publish-subscribe protocol and the second protocol is a security protocol based on full-duplex communication; If the second protocol communication channel has not yet been successfully established, but the first protocol communication channel has been established, the voice data transmitted through the first protocol communication channel is received. The voice data includes core data and non-core data. After detecting that the second protocol communication channel has been successfully established, a migration notification containing the current session state is received through the second protocol communication channel, and the default transmission channel for subsequent non-core data is switched to the second protocol communication channel.

[0076] This invention provides a method for ensuring voice session continuity based on dual-protocol collaboration applied to the server side. The main purpose of this method is to build a server architecture that can intelligently identify, adapt to, and coordinate with the client's dual-protocol strategy, ensuring that the server can correctly receive, route, and process data from different protocol channels regardless of whether the client is in a pre-enabled, migrated, or normal working state, maintaining unified session logic and context, thereby achieving a seamless end-to-end experience together with the client.

[0077] During system initialization, the server deploys and starts a first protocol service (such as MQTT Broker) and a second protocol service (such as WSS Server) in parallel. These two services independently listen on their respective network ports, forming a parallel access layer. When a client initiates a connection request, the server processes each request independently. The core strategy is that the server's protocol coordination center records the dual-channel connection status from the same client, but does not mandate that the second channel (WSS) must be the primary or sole data entry point. This design prepares the architecture for a pre-enabled client adoption strategy.

[0078] During this phase, the server's state mirrors that of the client. When the protocol coordination center detects that the first protocol channel from a client is already connected and ready, while the second protocol channel is not yet connected or is in the process of handshaking, it recognizes that the client has entered pre-enabled mode.

[0079] At this point, all voice data delivered through the first protocol channel (in the case of weak network conditions, this data is actually core data that has been split and encapsulated by the client, possibly accompanied by simplified non-core data) will be received by the protocol coordination center. The coordination center will parse the session ID in the data packet and create or associate it with a unified session context, and then forward the session data (whether it is core commands or non-core audio that is attempted to be transmitted) to the backend business processing module for immediate processing and response. This means that the server-side business logic has already started working before the second channel is established.

[0080] Continuously monitor for connection establishment events on the second protocol channel. Once a client successfully establishes a second protocol channel, the server-side protocol coordination center will anticipate and prepare to receive migration notifications.

[0081] Once the migration notification containing the session ID and state context is received through the newly established second protocol channel, the protocol coordination center will perform key operations: First, it will locate the corresponding ongoing session instance based on the session ID; second, it will update the main data receiving channel identifier of that session instance from the first protocol to the second protocol and synchronize the state context from the notification to the session to ensure business processing continuity; finally, in the internal routing rules, it will direct the default receiving path for non-core data (high-definition voice stream) of this session to the second protocol service module. From then on, the server will receive subsequent high-quality voice data through the higher-performance second protocol channel, completing the coordinated handover with the client.

[0082] The server does not passively receive data, but actively cooperates with the client's pre-activation and migration strategies through state awareness and logical switching. Only when the server understands and adapts to the client's dual-protocol working state can it ensure that data is received correctly and that the session context is not fragmented, thus enabling the client's intelligent collaboration strategy to take effect throughout the entire communication link.

[0083] In a preferred embodiment of the present invention, when the second protocol communication channel has not yet been successfully established, the following is included: A service failure notification is sent to the client through the first protocol communication channel, and the service failure notification is used to indicate that the second protocol service is abnormal. In response to the service failure notification, the voice data transmission channel is switched to the first protocol communication channel via the client.

[0084] The primary objective of this implementation is to construct a proactive, rapid, and reliable fault detection and emergency command mechanism on the server side. This mechanism aims to proactively notify all clients that depend on it when the server-side secondary protocol service itself malfunctions, guiding them to switch to a healthy backup channel. This prevents widespread voice session interruptions caused by single-point service failures, mitigating the impact of the fault from service unavailability to transmission channel switching, thereby significantly improving the availability and robustness of the entire system.

[0085] On the server side, independent of the business services, a protocol status monitor is deployed. This monitor proactively probes the health of the secondary protocol service at a high frequency (e.g., every second), with metrics including but not limited to: whether the service process is alive, whether the listening port is responding, whether the internal queue is overflowing, and the success rate and average response latency of the current connection. Its core strategy is to set explicit, configurable fault detection thresholds. For example, when "the number of consecutive failed new connections exceeds the threshold" or "the service process heartbeat is lost," the monitor immediately sends a clear "secondary protocol service failure" signal to the protocol coordination center. Compared to passive timeout detection on the client side, this achieves millisecond-level fault detection.

[0086] Upon receiving a fault signal, the protocol coordination center will not attempt to communicate through the failed second protocol service. Instead, it immediately invokes the fault notification module to generate a standardized fault notification message. This message crucially includes: a fault event identifier, a suggested action (such as "switch to the first protocol"), and a timestamp. The key point is that this notification is delivered through a completely independent and functioning first protocol service.

[0087] Typically, the system will pre-set a broadcast topic for all clients to subscribe to. The fault notification module publishes messages to this topic through the first protocol service, ensuring that the notification can be delivered to all online clients in a timely manner through a reliable and lightweight channel.

[0088] The server-side approach fulfills its command and control role at this step. It proactively instructs or suggests that clients execute emergency procedures by issuing fault notifications. Upon receiving and verifying the notification, the client switches its voice data transmission channel from the (unavailable) second protocol to the first protocol. After sending the notification, the server-side protocol coordination center updates its internal state, marks the affected client's session as being in a fault emergency state, and prepares to receive and process all subsequent voice data through the first protocol channel, thus coordinating and responding to the client's switchover action.

[0089] In traditional architectures, a server-side service crash often directly leads to client connection interruption and session failure. This approach constructs an independent fault command chain through proactive monitoring and backup channel notification, ensuring that sessions can still be preserved in an orderly manner even when the main service fails, significantly reducing service downtime.

[0090] Moreover, because the server can instantly detect its own failure and broadcast notifications via a pre-established, lightweight MQTT channel, all clients can receive the instruction and begin switching almost simultaneously. This is much faster and more synchronous than relying on independent timeout detection for each client. This rapid and unified response prevents request backlog and resource exhaustion caused by clients constantly reconnecting to the failed service, which is beneficial for the isolation and recovery of the failed service.

[0091] In a preferred embodiment of this implementation, after switching to the first protocol communication channel, a session takeover confirmation message sent through the first protocol communication channel is received; Continuously probe whether the second protocol service on the server side has been restored; After the recovery is confirmed, a switchback process from the first protocol communication channel to the second protocol communication channel is triggered.

[0092] The main purpose of this embodiment is to achieve automated, smooth and reliable recovery of the system from a fault emergency state to an optimal working state, ensuring that when the second protocol service is repaired, the user session can be seamlessly migrated back to the high-quality channel, thereby restoring the optimal experience and releasing the emergency backup resources of the first protocol channel.

[0093] After the client responds to the fault notification and completes the handover, the server receives the session takeover confirmation information sent back by the client through the first protocol channel. This confirmation information not only indicates that the client has successfully switched over, but may also contain the session breakpoint context at the time of the handover (such as the last processed voice packet sequence number). Upon receiving the confirmation, the server-side protocol coordination center officially marks the corresponding session as "fault takeover status," records the relevant context, completes bilateral synchronization in emergency situations, and prepares for possible handover back.

[0094] After detecting a failure, the server-side protocol status monitor does not stop working but instead enters recovery probing mode. It continuously performs health checks on the failed secondary protocol service (such as the restarted WSS service process). The key strategy here is to set more stringent recovery criteria than the failure detection criteria; for example, requiring multiple consecutive successful probes, a 100% success rate for newly established test connections, and normal service load. This aims to avoid triggering premature rollbacks due to incomplete service stability.

[0095] Once the monitor confirms that the second protocol service has been stably restored, it sends a "service restored" signal to the protocol coordination center. Subsequently, the server can actively or passively trigger the switchback process.

[0096] Preferably, the server sends a recovery-ready notification to the client via the first protocol channel. The client initiates a switchback at a safe time (e.g., when a voice silence interval is detected again), a process similar to the initial migration: a migration notification carrying the latest status is sent via the newly restored second protocol channel. Upon receiving this, the server updates the session's primary channel identifier and synchronizes the status. Finally, both parties switch back to the second protocol as the default transmission channel for subsequent non-core data. The first protocol channel then returns to standby status.

[0097] The system features a closed-loop automation mechanism for proactive monitoring, switching, and recovery. This not only significantly shortens the overall fault recovery time but also completely avoids delays and errors that may result from manual intervention, giving the entire system a high degree of self-healing capability.

[0098] Furthermore, it ensures that once high-quality service is restored, the system can automatically and seamlessly guide users back, promptly restoring the high-definition, low-latency voice interaction experience and preventing long-term experience degradation caused by failure to switch back in time after fault repair.

[0099] This invention provides another voice conversation system. This includes both client-side and server-side components to execute the method. The client and the server interact with each other through the first protocol communication channel and the second protocol communication channel.

[0100] Therefore, it is possible to achieve any effect in the voice session continuity guarantee method based on dual-protocol collaboration, which will not be elaborated here.

[0101] For any parts not mentioned in this invention, existing technologies can be used or referenced.

[0102] The various embodiments in this specification are described in a progressive manner. The same or similar parts between the various embodiments can be referred to each other. Each embodiment focuses on describing the differences from other embodiments.

[0103] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for ensuring voice session continuity based on dual-protocol collaboration, characterized in that, Applied to a client, the method includes: Receive voice data input by the user; Based on the real-time determination of the current network status, and according to the preset data priority rules, the voice data is divided into core data and non-core data. The core data is transmitted through the first protocol communication channel; The non-core data is transmitted through the second protocol communication channel; The first protocol is a lightweight publish-subscribe protocol, and the second protocol is a security protocol based on full-duplex communication.

2. The method according to claim 1, characterized in that, Based on the real-time assessment of the current network status, and according to preset data priority rules, the voice data is divided into core data and non-core data, specifically including: When the determination result is a weak network state, instruction features and / or text features are extracted from the speech data as the core data, and the original speech data stream is used as the non-core data.

3. The method according to claim 1 or 2, characterized in that, Prior to the step of receiving user-input voice data, a connection pre-enabling step is also included: Simultaneously initiate connection requests to establish the first protocol communication channel and the second protocol communication channel with the server; If the second protocol communication channel has not yet been successfully established, but the first protocol communication channel has been established, the user's voice input data is received and transmitted through the first protocol communication channel.

4. The method according to claim 3, characterized in that, It also includes a session migration step: After detecting that the second protocol communication channel has been successfully established, a migration notification containing the current session state is generated; The migration notification is sent to the server through the second protocol communication channel, and the default transmission channel for subsequent non-core data is switched to the second protocol communication channel.

5. The method according to claim 4, characterized in that, The default transmission channel for subsequent non-core data will be switched to the second protocol communication channel, specifically as follows: Based on the silence intervals of user-input voice data, the default transmission channel for non-core data is switched.

6. The method according to claim 1, characterized in that, The non-core data is transmitted through the second protocol communication channel, including: When the transmission of the non-core data fails in the second protocol communication channel, the non-core data is cached in the local data cache module; After the network status is detected to be restored, the cached non-core data will be retransmitted through the second protocol communication channel.

7. A method for ensuring voice session continuity based on dual-protocol collaboration, characterized in that, Applied to the server side, the method includes: Simultaneously receive connection requests from the client to establish a first protocol communication channel and a second protocol communication channel, wherein the first protocol is a lightweight publish-subscribe protocol and the second protocol is a security protocol based on full-duplex communication; If the second protocol communication channel has not yet been successfully established, but the first protocol communication channel has been established, the voice data transmitted through the first protocol communication channel is received. The voice data includes core data and non-core data. After detecting that the second protocol communication channel has been successfully established, a migration notification containing the current session state is received through the second protocol communication channel, and the default transmission channel for subsequent non-core data is switched to the second protocol communication channel.

8. The method according to claim 7, characterized in that, Before the second protocol communication channel is successfully established, including: A service failure notification is sent to the client through the first protocol communication channel, and the service failure notification is used to indicate that the second protocol service is abnormal. In response to the service failure notification, the voice data transmission channel is switched to the first protocol communication channel via the client.

9. The method according to claim 8, characterized in that, Also includes: After switching to the first protocol communication channel, receive the session takeover confirmation information sent through the first protocol communication channel; Continuously probe whether the second protocol service on the server side has been restored; After the recovery is confirmed, a switchback process from the first protocol communication channel to the second protocol communication channel is triggered.

10. A voice conversation system, characterized in that, Includes a client to perform the method according to any one of claims 1 to 6. And on the server side, to perform the method according to any one of claims 7 to 9, The client and the server interact with each other through the first protocol communication channel and the second protocol communication channel.

Images