Voice service connection control method, voice service gateway device, and computer program product

Abstract

The application is suitable for the technical field of voice service gateway, and provides a voice service connection control method, a voice service gateway device and a computer program product. The method comprises the following steps: receiving an uplink message sent by a client, determining a target voice engine from at least two voice engines according to a capability type identifier carried by the uplink message; establishing or multiplexing a downstream connection between the target voice engine, forwarding the load content of the uplink message to the target voice engine through the downstream connection; receiving an end identifier sent by the client; starting timing according to a grace period corresponding to the target voice engine; closing the downstream connection when the grace period expires and no final result returned by the target voice engine is received; wherein the grace periods corresponding to different voice engines are different. The embodiment of the application enables the voice engine with slow response to obtain a final result waiting time adapted to its processing characteristics, and the downstream connection corresponding to the voice engine with fast response can be released in time.

Description

Technical Field

[0001] This application belongs to the field of voice service gateway technology, and in particular relates to a voice service connection control method, a voice service gateway device, and a computer program product. Background Technology

[0002] Voice-based applications typically require integration with various voice processing capabilities, such as automatic speech recognition, speech synthesis, simultaneous interpretation, conference transcription, and streaming translation. These different voice processing capabilities can be implemented by voice engines from different service providers, and these engines differ in connection protocols, response speeds, termination handling methods, and disconnection characteristics. When a client integrates multiple voice engines, it usually requires maintaining multiple sets of interfaces and connections. Adding or replacing a voice engine may also involve adaptation modifications on the client side.

[0003] In scenarios involving multiple voice engines, existing connection management methods may apply the same or a fixed connection recycling duration to multiple voice engines. Since the time required for each voice engine to return the final result varies, if the recycling duration is too short, some connections may be closed before the final result is returned; if the recycling duration is too long, connections that are no longer transmitting business data may continue to occupy connection and storage resources.

[0004] Furthermore, in scenarios where clients access multiple speech engines and undertake multiple speech processing tasks through a unified entry point, the connection status and processing progress of each task may differ, and a fixed connection management strategy may struggle to balance task continuity with connection resource utilization efficiency. Summary of the Invention

[0005] In view of this, embodiments of this application provide a voice service connection control method, a voice service gateway device, and a computer program product, which are used to set a grace period for the final result for different voice engines when a client accesses multiple voice engines through a unified entry point, so as to improve the accuracy of the voice service gateway in managing downstream connections.

[0006] The first aspect of this application provides a voice service connection control method, including: Receive uplink messages sent by the client, and determine the target speech engine from at least two speech engines based on the capability type identifier carried in the uplink messages; Establish or reuse a downstream connection with the target speech engine, and forward the payload content of the uplink message to the target speech engine via the downstream connection; Receive the end identifier sent by the client; The timing begins according to the grace period corresponding to the target speech engine; When the grace period expires and no final result is received from the target speech engine, the downstream connection is closed. The grace period varies depending on the voice engine.

[0007] The method described above determines the target speech engine based on the capability type identifier and forwards the payload content to the target speech engine via the corresponding downstream connection. After the termination identifier is triggered, the system waits for the final result according to the grace period of the target speech engine, thus ensuring that connection routing and connection reclamation are adapted to the processing capabilities and response characteristics of the target speech engine. Therefore, speech engines with slower responses can receive a final result waiting time adapted to their processing characteristics, while downstream connections for speech engines with faster responses can be released promptly, thereby reducing the occurrence of premature connection closure or untimely release.

[0008] In some embodiments, the step of receiving an uplink message sent by a client and determining a target speech engine from at least two speech engines based on a capability type identifier carried in the uplink message includes: Receive multiple uplink messages sent by the client through the same client connection; Read the capability type identifier carried in each of the aforementioned uplink messages; Based on the speech engine processing branch corresponding to each of the capability type identifiers, the target speech engine corresponding to each of the uplink messages is determined.

[0009] By enabling the same client connection to carry multiple uplink messages with different capability type identifiers, the need for clients to establish separate entry connections for different voice engines can be reduced.

[0010] In some embodiments, establishing or reusing the downstream connection with the target speech engine includes: Obtain the user identifier that sent the uplink message; Search the connection registration table for available connections corresponding to the user identifier and the target speech engine; When an available connection is found, that available connection is used as the downstream connection. If no available connection is found, the downstream connection is established, and after successful establishment, the downstream connection is registered in the connection registration table along with the user identifier and the target speech engine.

[0011] By querying and reusing registered downstream connections, or by establishing and registering new downstream connections when no connection is found, the occurrence of duplicate connection establishment and missing connection records can be reduced.

[0012] In some embodiments, after obtaining the user identifier that sent the uplink message, the method further includes: Obtain the session identifier carried in the uplink message; The combination of the user identifier, the capability type identifier, and the session identifier is used as a composite index for querying and registering connections in the connection registration table, so as to isolate downstream connections corresponding to different voice processing capabilities of the same user from each other, and to isolate downstream connections corresponding to different sessions initiated by the same user for the same voice processing capability from each other.

[0013] By using a combination of user identifier, capability type identifier, and session identifier as an index, different capabilities or sessions of the same user can be associated with different connections, reducing the possibility of misusing connection records between different tasks.

[0014] In some embodiments, establishing or reusing the downstream connection with the target speech engine further includes: If no available connection is found, the composite index is marked as being built. When the downstream connection is successfully established, the downstream connection is registered as active. When the downstream connection fails to be established, the record corresponding to the establishment status is deleted to prevent failed connections from remaining in the connection registration table.

[0015] By integrating inspection and establishment into an atomic operation, the composite index is marked as being established when no available connection corresponding to the user identifier and target speech engine is found. After the connection is successfully established, it is registered as being active. When the connection fails to be established, the record corresponding to the being established state is deleted. This can reduce duplicate connection establishment caused by concurrent first packets and the residual of failed connections.

[0016] In some embodiments, before receiving the end identifier sent by the client, the method further includes: Monitor the elapsed time since the last time uplink audio data was received from the client; When the elapsed time reaches the keep-alive interval and the target speech engine needs to be kept alive, keep-alive data is sent to the target speech engine through the downstream connection to keep the downstream connection active. The keep-alive data indicates that the current voice processing task has not ended and is not used to trigger the timing of the grace period.

[0017] By sending keep-alive data when the keep-alive conditions are met, downstream connections that need to be kept alive can be kept active during the task, reducing the possibility of connection loss due to temporary interruptions in business load.

[0018] In some embodiments, after receiving the end identifier sent by the client, the method further includes: Stop sending the keep-alive data for the current session and update the downstream connection to a state awaiting final result; Upon receiving the final result within the grace period, the final result is returned to the client and the downstream connection is closed. The keep-alive process during task execution and the grace period recycling process triggered by the end flag are triggered by separate conditions.

[0019] By using separate triggering conditions for the keep-alive processing during task execution and the grace period recycling processing triggered by the end flag, it is possible to avoid misjudging the task as completed when no uplink audio data is received during task execution, and to continue waiting for the final result after receiving the end flag, thus taking into account both the active maintenance of downstream connections and timely recycling.

[0020] In some embodiments, the grace period is determined according to any of the following methods: A fixed grace period is pre-configured based on the response characteristics of the target speech engine; or, The system obtains at least one of the following: the protocol-specified duration of the target voice engine, the statistical value of the time interval between two adjacent transmissions of service data by the downstream connection within a historical period, and the error rate of the downstream connection within the historical period. Based on the obtained information, the system determines the grace period and constrains the determined grace period to a range from the lower limit to the upper limit.

[0021] By combining at least one of the following to determine the grace period: the duration specified in the agreement, the statistical value of historical business data intervals, and the historical error rate, and by using upper and lower limit constraints to determine the result, the possibility of abnormal waiting time due to short-term state fluctuations can be reduced.

[0022] In some embodiments, the voice service connection control method further includes: Receive an update instruction for the grace period and save the updated grace period indicated by the update instruction; For downstream connections that have already started measuring the grace period, continue using the grace period corresponding to the already started grace period, and start measuring according to the updated grace period from the next time the end flag is received.

[0023] By making the updated grace period take effect from the next end flag, tasks that have already started the grace period can continue to wait according to the parameters obtained at startup, reducing the impact of parameter switching on ongoing tasks.

[0024] In some embodiments, the at least two voice engines are respectively connected via corresponding adapters; Each adapter is used to provide a unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping, and performs the conversion between the unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping and the private protocol of the corresponding speech engine within the adapter.

[0025] By having each adapter provide a unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping, and by performing the conversion between the unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping and the corresponding private protocol of the voice engine within the adapter, the voice service gateway can access different voice engines in a unified manner and reduce the impact of changing voice engines on the client message protocol.

[0026] In some embodiments, closing the downstream connection includes: Stop receiving new payload for the current session and stop timing for the downstream connection; Invoke the adapter corresponding to the target speech engine and disconnect the downstream connection according to the protocol of the target speech engine; Cancel the corresponding record from the connection registration table used to register the downstream connection, and record the reason for the recycling of the downstream connection.

[0027] By stopping the reception of new load for the current session and the timing corresponding to the downstream connection when closing the downstream connection, calling the adapter corresponding to the target speech engine to disconnect the downstream connection according to the target speech engine's protocol, and deregistering the corresponding record and the reason for record reclamation from the connection registration table, the connection closing process can be kept consistent with the connection registration status, and it is easy to distinguish between normal termination, grace timeout, keep-alive failure, or connection abnormality and other reclamation situations.

[0028] In some embodiments, the voice service connection control method further includes: When the downstream connection is abnormally disconnected, the user identifier and the session identifier remain unchanged; Re-establish the connection with the target speech engine, and register the re-established connection and the corresponding composite index as the current downstream connection.

[0029] By keeping the user identifier and session identifier unchanged when the downstream connection is abnormally disconnected, re-establishing the connection with the target speech engine, and registering the re-established connection as the current downstream connection with a composite index consisting of the user identifier, capability type identifier, and session identifier, the user and session context of the task can be maintained, reducing the impact of abnormal connection loss on task continuity.

[0030] In some embodiments, the voice service connection control method further includes: In the absence of re-establishing the connection with the target speech engine, a backup speech engine is determined from the at least two speech engines that provides the same speech processing capabilities as the target speech engine; A connection is established through the adapter corresponding to the backup voice engine, and the established connection is registered as the current downstream connection with the composite index. During the switch to the backup voice engine, the message protocol used by the client and the session identifier remain unchanged.

[0031] By identifying a backup voice engine with the same voice processing capabilities from at least two voice engines when the connection with the target voice engine is not re-established, a connection can be established via the adapter corresponding to the backup voice engine, while keeping the message protocol and session identifier used by the client unchanged, thereby improving the continuity of task connections.

[0032] A second aspect of this application provides a voice service gateway device, including a processor and a memory. The memory stores a computer program, and when the computer program is executed by the processor, the voice service gateway device performs the voice service connection control method described in the first aspect of this application.

[0033] A third aspect of this application provides a computer program product, including a computer program, which, when run, causes the voice service connection control method described in the first aspect of this application to be executed.

[0034] It is understood that the beneficial effects of the second and third aspects mentioned above can be found in the relevant descriptions in the first aspect above, and will not be repeated here. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram of the overall architecture of the voice service gateway provided in the embodiments of this application; Figure 2 This is a flowchart illustrating the voice service connection control method provided in an embodiment of this application; Figure 3 This is a flowchart illustrating the connection registration and composite index processing procedure provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the keep-alive, grace period determination, and update process provided in the embodiments of this application; Figure 5 This is a schematic diagram of the state transitions in the downstream connection lifecycle provided in an embodiment of this application; Figure 6 This is a flowchart illustrating the abnormal reconnection and backup voice engine switching provided in the embodiments of this application; Figure 7 This is a schematic diagram of the structure of the voice service connection control device provided in the embodiments of this application; Figure 8 This is a schematic diagram of the structure of the voice service gateway device provided in the embodiments of this application; Figure label: 10-Client, 20-Voice Engine; 100-Voice service gateway, 101-Client entry point, 102-Capability routing unit, 103-Connection registration unit, 104-Lifecycle management unit, 105-Adapter; 200-Voice service connection control device, 201-Message receiving module, 202-Engine determination module, 203-Connection processing module, 204-Lifecycle management module, 205-Adaptation module; 300 - Voice service gateway device, 301 - Processor, 302 - Memory, 303 - Computer program. Detailed Implementation

[0037] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0038] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0039] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0040] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0041] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0042] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0043] The term "speech engine" as used in this application refers to a software service capable of providing at least one of the following speech processing capabilities: speech synthesis, real-time speech recognition, simultaneous interpretation, conference transcription, or streaming translation. Different speech engines may be deployed by different service providers, or they may be deployed separately by the same service provider for different speech processing capabilities.

[0044] The voice service gateway device referred to in this application is a device that carries out client entry, capability routing, connection registration, lifecycle management, and adapter functions. The voice service gateway is a logical functional entity running on the voice service gateway device. A client connection is the connection between a client and the voice service gateway; a downstream connection is the connection established between the voice service gateway on behalf of a user or session and the voice engine. A downstream connection is the connection established by the voice service gateway for a specific user, voice processing capability, and / or session with the voice engine. The differentiated lifecycle management in this application mainly targets downstream connections.

[0045] In applications, the capability type identifier is used to indicate the voice processing capability requested by the uplink message and the corresponding processing branch. One capability type identifier can correspond to a specific combination of voice engine and voice processing capability; when the same voice processing capability is connected to multiple voice engines, different capability type identifiers can be set for each combination.

[0046] The connection registry table referred to in this application is a data structure used to record the correspondence between connection indexes, downstream connections, and their current states. The connection registry table can be set up separately according to the combination of speech engine and speech processing capabilities, or it can be a single table with a composite index to distinguish each downstream connection.

[0047] The composite index referred to in this application is a connection query and registration key formed by a user identifier, a capability type identifier, and a session identifier. This composite index is used to distinguish between different users, different speech processing capabilities of the same user, and different sessions initiated by the same user for the same speech processing capability.

[0048] The grace period referred to in this application is the allowed duration for the voice service gateway to wait for the target voice engine to return the final result after receiving an end marker sent by the client for the current voice processing task. The grace period is different from the keep-alive interval used to determine whether to send keep-alive data during task execution.

[0049] The keep-alive interval, as referred to in this application, is the time interval during task execution from the last time uplink audio data was received from the client, used to determine whether to send keep-alive data. Keep-alive data is data sent to meet the connection maintenance requirements of the target speech engine, such as a silence audio frame. Keep-alive data indicates that the current speech processing task has not ended, that is, it does not indicate that the task has ended, and it is not used to trigger the timing of the grace period, nor is it used as a basis for reclaiming downstream connections.

[0050] The adapter referred to in this application is a protocol adaptation function entity set between the voice service gateway and the corresponding voice engine. It is used to map connection establishment, data transmission, data reception, connection closure and error codes to a unified interface or unified state inside the gateway, and to complete the conversion between the unified interface and the private protocol of the corresponding voice engine.

[0051] The term "final result" as used in this application refers to the final output result returned by the target speech engine for the current speech processing task, indicating that the task has been completed, such as the final recognized text, the final translation, or the synthesized audio output; the final result is distinct from the intermediate results returned during the task processing.

[0052] like Figure 1As shown, the voice service gateway 100, running on the voice service gateway device, is located between the client 10 and at least two voice engines 20. The voice service gateway 100 includes a client entry point 101, a capability routing unit 102, a connection registration unit 103, a lifecycle management unit 104, and multiple adapters 105. The client entry point 101 receives uplink messages; the capability routing unit 102 determines the target voice engine based on the capability type identifier; the connection registration unit 103 stores the correspondence between downstream connections and users, capabilities, and sessions; the lifecycle management unit 104 performs keep-alive, grace period timing, and connection recycling; each adapter 105 converts the connection operations and states of different voice engines 20 into a unified state within the gateway.

[0053] In the application, client 10 can send multiple uplink messages via the same WebSocket connection. Uplink messages may include at least one of the following: capability type identifier, payload content, language, speaker, device identifier, model identifier, and session identifier. The payload content can be audio data or control commands such as start and end. When client 10 hands with voice service gateway 100, it can complete user and tenant authentication based on a token; subsequent messages determine the user identifier based on the identity bound to the client 10 connection.

[0054] In the application, the voice service gateway 100 can set up adapters 105 for different voice engines 20. Each adapter 105 provides a unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping, and performs the conversion between the protocol and the corresponding voice engine 20 internally. When adding or replacing a voice engine 20, the corresponding adapter 105 can be added or replaced without changing the message protocol used by the client 10 connection.

[0055] like Figure 2 As shown, the voice service connection control method provided in this application embodiment may include steps S101 to S105 executed by the processor of the voice service gateway device.

[0056] Step S101: Receive the uplink message sent by the client, and determine the target voice engine from at least two voice engines based on the capability type identifier carried in the uplink message.

[0057] In the application, step S101 is executed collaboratively by the client entry point 101 and the capability routing unit 102: the client entry point 101 receives the uplink message sent by the client 10 and delivers the uplink message to the capability routing unit 102; the capability routing unit 102 reads the capability type identifier in the uplink message and queries the pre-configured capability routing relationship. The capability routing relationship records the correspondence between the capability type identifier and the processing branch of the speech engine 20. After finding the correspondence, the speech engine 20 used by that processing branch is determined as the target speech engine.

[0058] In some embodiments, step S101 includes the following steps S111 to S113: Step S111: Receive multiple uplink messages sent by the client through the same client connection; Step S112: Read the capability type identifier carried in each uplink message; Step S113: Determine the target speech engine for each uplink message according to the speech engine processing branch corresponding to each capability type identifier.

[0059] In the application, the same client 10 connection can carry uplink messages corresponding to different voice processing capabilities. The client 10 does not need to establish separate entry connections for each voice engine 20. Each uplink message still enters the corresponding processing branch according to its capability type identifier, and the downstream connections corresponding to different capabilities are independent of each other.

[0060] Step S102: Establish or reuse a downstream connection with the target speech engine, and forward the payload content of the uplink message to the target speech engine through the downstream connection.

[0061] In the application, in step S102, the voice service gateway 100 obtains the user identifier that sent the uplink message and queries the connection registration table corresponding to the target voice engine. If an available connection corresponding to the user identifier and the target voice engine is already registered in the connection registration table, the connection is reused; if no corresponding available connection is registered, a new downstream connection is established, and the new downstream connection is registered in the connection registration table after successful establishment.

[0062] In applications, connections can be established on demand using a request-triggered approach. For real-time speech recognition, simultaneous interpretation, or streaming translation services that require maintaining session state, a connection is established when a user first requests the corresponding capability on the current gateway node; subsequent requests use the already registered connection when reuse conditions are met. For stateless, one-off speech synthesis services, a connection can also be temporarily established for each request and closed after the request is completed, without long-term registration.

[0063] In some embodiments, querying the connection registration table for available connections corresponding to the user identifier and target speech engine, and establishing a downstream connection when no available connection is found, is accomplished through an integrated atomic operation of checking and establishing. Specifically, when performing this atomic operation, if no available connection is found, the composite index consisting of the user identifier, capability type identifier, and session identifier is marked as being established; after the connection is successfully established, the downstream connection corresponding to the composite index is registered as being active; if the connection establishment fails, the record corresponding to the being established state is deleted.

[0064] like Figure 3 As shown, in one implementation, each voice engine 20 and voice processing capability combination corresponds to a connection registration table, which is indexed by the user identifier. In another implementation, a single registration table can be used, with a composite index consisting of a user identifier, capability type identifier, and session identifier, to isolate different capability connections of the same user and different session connections under the same capability. The connection registration table can be stored in the memory of the gateway node; in a multi-node deployment scenario, session stickiness can be used to ensure that requests from the same user reach the same node, or the connection state can be stored in shared storage.

[0065] like Figure 3 As shown, in some embodiments, step S102 includes the following steps S301 to S306: Step S301: Obtain the user identifier that sent the uplink message, and read the capability type identifier and session identifier from the uplink message; Step S302: Use the combination of user identifier, capability type identifier, and session identifier as the connection index; Step S303: Query the connection registration table according to the connection index to determine whether there is an available connection; if yes, proceed to step S304; if no, proceed to step S305. Step S304: Use the available connections found in the query as downstream connections; Step S305: Establish a downstream connection with the target speech engine; Step S306: Register the newly established downstream connection and connection index to the connection registration table.

[0066] In applications, the connection registry allows established and available connections to be queried and reused by subsequent requests, reducing the need to repeatedly establish downstream connections; the composite index further distinguishes different capabilities and sessions of the same user, reducing the possibility of one voice processing task misusing the connection record of another task.

[0067] Step S103: Receive the end identifier sent by the client; Step S104: Start timing according to the grace period corresponding to the target speech engine; Step S105: When the grace period expires and no final result is received from the target speech engine, close the downstream connection.

[0068] In the application, steps S103 to S105 are executed collaboratively by the client entry point 101, the lifecycle management unit 104, and the adapter 105: The client entry point 101 receives the end identifier sent by the client 10 and delivers it to the lifecycle management unit 104; the lifecycle management unit 104 sets the downstream connection to a waiting state for the final result and starts a grace period timer corresponding to the target speech engine in step S104; in step S105, if the final result is not received after the grace period expires, the adapter 105 is invoked to close the downstream connection. If the final result is received within the grace period, the lifecycle management unit 104 returns the final result to the client 10 and closes the downstream connection.

[0069] In applications, the grace periods for different speech engines 20 are independent of each other. For example, a real-time speech recognition engine can be set to a grace period of approximately 500 milliseconds, a simultaneous interpretation engine can be set to a grace period of approximately 800 milliseconds, and a streaming translation engine with longer processing latency can be set to a grace period of approximately 8000 milliseconds. The above values ​​are only examples illustrating that different speech engines 20 can adopt different levels of grace periods; the actual values ​​can be configured according to the protocol and response characteristics of the corresponding speech engine 20.

[0070] Table 1. Examples of grace periods for different speech engines (20).

[0071] like Figure 4 As shown, in some embodiments, the service keep-alive process includes steps S401 to S403, and the grace period determination and update includes steps S404 to S408: Step S401: Monitor the elapsed time since the last time the client's uplink audio data was received; Step S402: Determine whether the elapsed time has reached the keep-alive interval corresponding to the target speech engine and whether the target speech engine needs to be kept alive; if yes, proceed to step S403; if no, return to step S401 to continue monitoring. Step S403: Send keep-alive data to the target voice engine via downstream connection; Step S404: Determine the grace period based on at least one of the following: the duration specified in the target speech engine's protocol, the historical business data interval statistics, and the historical error rate, and constrain it within the range of the lower limit duration to the upper limit duration. Step S405: Receive the update instruction for the grace period and save the updated grace period; Step S406: Determine whether the current downstream connection has started the grace period timing; if yes, proceed to step S407; if no, proceed to step S408. Step S407: The current grace period continues to use the grace period obtained at the start of the timing. Step S408: Use the updated grace period from the next time the end marker is received.

[0072] In some embodiments, the grace period may be a fixed value pre-configured for each type of voice engine 20. In other embodiments, the grace period may be determined based on at least one of the following: the protocol-specified duration of the target voice engine, a statistical value of the time interval between two adjacent transmissions of service data within a historical period, and a historical error rate. The statistical value may be a quantile of the historical time interval, and the determined result is constrained between a lower limit and an upper limit to avoid the grace period being too small or too large due to short-term fluctuations.

[0073] In applications, the grace period can also be configured as a runtime updatable parameter. After receiving an update command, the voice service gateway 100 saves the new grace period corresponding to the target voice engine; for grace period timing that has already started, the grace period obtained at the start of this timing is still used; the updated grace period is used from the next time an end flag is received and a new grace period timing is started, thereby reducing the impact of parameter updates on ongoing tasks.

[0074] In application, by determining the grace period based on the target speech engine's protocol and historical transmission status, and using upper and lower limit constraints to determine the result, the possibility of abnormal waiting time caused by short-term state fluctuations can be reduced; runtime updates take effect from the next grace period, allowing tasks that have already started to continue waiting according to the parameters obtained at startup, reducing the impact of parameter switching on ongoing tasks.

[0075] like Figure 4 As shown, the service keep-alive and termination / recycling use separate triggering conditions. During the task, the voice service gateway 100 records the time when it last received uplink audio data from the client 10. If no new audio data is received after the keep-alive interval from that time, and the target voice engine needs to keep alive, the voice service gateway 100 sends a small amount of keep-alive data to the target voice engine. The keep-alive data can be a silent audio frame that meets the target voice engine's protocol requirements, indicating that the current voice processing task has not ended. It is used to maintain the downstream connection in an active state and is not used to trigger the grace period timing.

[0076] In applications, for example, for a streaming translation engine that requires periodic data to maintain an active state, the keep-alive interval can be set to approximately 5 seconds; for a voice engine 20 that does not require business keep-alive, the step of sending keep-alive data can be omitted. The keep-alive interval and the grace period after the end are two different parameters; the former is used during the task, while the latter is triggered by the end marker.

[0077] In the application, keep-alive data is sent only when the keep-alive interval is reached and the target voice engine needs to keep-alive during the task, which can maintain the active state of the downstream connection; the keep-alive trigger condition and the grace period recycling trigger condition after the end flag are independent of each other, reducing the possibility of misjudging short pauses in business load as the end of the task.

[0078] In some embodiments, upon receiving an end flag from client 10, voice service gateway 100 stops sending keep-alive data for the current session and updates the downstream connection to a state awaiting final result; upon receiving the final result within the grace period, it returns the final result to client 10 and closes the downstream connection. Thus, the keep-alive processing during task execution and the grace period recycling processing triggered by the end flag employ separate triggering conditions.

[0079] In some embodiments, closing the downstream connection in step S105 includes the following steps S131 to S134: Step S131: Stop receiving new payloads for the current session and stop the corresponding keep-alive timer and grace period timer; Step S132: Call the unified shutdown interface of the adapter corresponding to the target speech engine; Step S133: The adapter sends a close message and disconnects the downstream connection according to the target speech engine's protocol; Step S134: Map the closed state or error code returned by the target voice engine to the unified state inside the gateway, and cancel the corresponding record in the connection registration table.

[0080] In the application, the connection shutdown and error code mapping of each voice engine 20 are completed by the corresponding adapter 105, so that the lifecycle management unit 104 can perform subsequent deregistration and error handling according to the unified state; when the voice engine 20 is replaced, the client 10 message protocol and lifecycle management process do not need to be changed with the private shutdown protocol of the voice engine 20.

[0081] In the application, when closing a downstream connection, you can first stop receiving new payloads for the session, stop keep-alive and grace timers, send a normal close message according to the target speech engine's protocol, disconnect the downstream connection, and deregister the corresponding record from the connection registry. You can also record reasons for termination such as normal termination, grace timeout, keep-alive failure, or connection abnormality.

[0082] In the application, grace period timing can be uniformly managed by the shared scheduler in the voice service gateway 100. The shared scheduler saves the expiration time for each downstream connection and submits the corresponding shutdown task when the expiration time is reached, without having to create an independent timing thread for each connection. In another implementation, a time wheel can be used to manage grace period timing and keep-alive timing.

[0083] like Figure 5 As shown, in some embodiments, the lifecycle of the downstream connection includes the following state transition steps S501 to S505: Step S501: When the first request arrives, update the corresponding connection index from the not established state to the establishing state; Step S502: After the downstream connection is successfully established, update it to an active processing state; Step S503: After receiving the end flag, update the downstream connection to the waiting for final result state and start the grace period timer; Step S504: After receiving the final result or after the grace period expires, close the downstream connection; Step S505: Cancel the corresponding record from the connection registration table and update it to the closed and canceled state.

[0084] like Figure 5 As shown, downstream connections can sequentially be in the states of Unestablished, Establishing, Active Processing, Waiting for Final Result, and Closed and Deregistered. The first request triggers the transition from the Unestablished state to the Establishing state; after successful connection establishment, it enters the Active Processing state; after receiving the end flag, it enters the Waiting for Final Result state; after receiving the final result or the grace period expires, it enters the Closed and Deregistered state; it can also enter the Closed and Deregistered state if the connection fails to establish or is abnormally disconnected.

[0085] In the application, a single client 10 connection can carry uplink messages corresponding to multiple voice processing capabilities. The voice service gateway 100 routes each uplink message to the corresponding voice engine 20 based on the capability type identifier in each uplink message, and performs connection registration, keep-alive, and grace period reclamation for each downstream connection. A change in the state of one downstream connection does not affect other downstream connections corresponding to different capabilities.

[0086] like Figure 6 As shown, when a downstream connection is abnormally disconnected, the voice service gateway 100 can maintain the user identifier and session identifier unchanged and re-establish the connection with the original target voice engine. When a backup voice engine 20 providing the same voice processing capabilities is configured, a connection with the backup voice engine 20 can also be established according to a preset policy, and this connection will be registered as a downstream connection of the current session. During the handover process, client 10 continues to use the original client 10 connection and session identifier.

[0087] like Figure 6As shown, in some embodiments, the processing after an abnormal disconnection of the downstream connection includes the following steps S601 to S606: Step S601: Detect whether the downstream connection is abnormally disconnected; if so, proceed to step S602. Step S602: Keep the user ID and session ID of the current task unchanged; Step S603: Determine whether to reconnect to the original target voice engine according to the preset strategy; if yes, proceed to step S604; if no, proceed to step S605. Step S604: Re-establish the connection with the original target speech engine; Step S605: Determine a backup voice engine from among voice engines that provide the same voice processing capabilities, and establish a corresponding connection; Step S606: Register the newly established connection and the original connection index as the current downstream connection, wherein the original connection index is a composite index composed of user identifier, capability type identifier and session identifier.

[0088] In the application, the user identifier and session identifier remain unchanged during exception handling, so that client 10 can continue to use the original client 10 connection and session context; when the original target speech engine is unavailable, it switches to the backup speech engine 20 that provides the same processing capabilities, which can reduce the situation where a single downstream connection failure causes the current task to be unable to continue.

[0089] In some embodiments, during the switch to the backup speech engine 20, the client 10 message protocol and session identifier remain unchanged, so that the client 10 does not need to change the message format or re-establish the session context due to the switch of the target speech engine.

[0090] In the application, switching to the backup voice engine 20 can be an optional implementation. Before switching, the voice service gateway 100 can confirm that the backup voice engine 20 provides the same voice processing capabilities as the target voice engine; after switching, the adapter 105 corresponding to the backup voice engine 20 performs connection establishment, data transmission and reception, closure, and error code mapping. When the original downstream connection becomes available again, the backup voice engine 20 can continue to be used according to the policy, or the original target voice engine can be reselected in subsequent new sessions.

[0091] In the application, when a connection establishment failure occurs, the voice service gateway 100 does not retain the failed connection in the connection registry and returns an error to the client 10 that has been normalized via adapter 105. When the voice engine 20 returns an error, fails to keep-alive, or the network disconnects, the voice service gateway 100 deregisters the corresponding record and, based on its configuration, decides to terminate the task, re-establish the connection, or switch to the backup voice engine 20. When the client 10 disconnects, all downstream connections corresponding to that client 10 can be released. When a gateway node restart causes the connection registry in memory to be cleared, subsequent requests from the client 10 will trigger on-demand connection establishment again.

[0092] In this application, the following example illustrates differentiated grace period recovery using three types of speech engines 20 simultaneously connected to the voice service gateway 100. A user sends real-time speech recognition, simultaneous interpretation, and streaming translation messages sequentially through the same client 10. The voice service gateway 100 selects the corresponding speech engine 20 based on the capability type identifier and establishes or reuses downstream connections for each of the three tasks. After the client 10 sends a completion flag, the three downstream connections wait for the final result according to their respective grace periods; the timing and closure of any one connection do not affect the other connections.

[0093] In application, to verify the technical effectiveness of differentiated grace period reclamation compared to uniform grace period reclamation, the two strategies can be run separately under the same concurrency, the same proportion of capability types, and the same network conditions. The number of erroneous reclamations of slow tasks, average number of connections in transit, memory usage, initial request latency, and number of reconnections should be observed. Test results should be recorded based on actual measurement data; before obtaining actual measurement data, only the directions listed in the table below can be considered as expected trends to be verified.

[0094] Table 2 Observational Indicators of Differentiated Grace Period Recovery Comparison Test

[0095] It should be understood that the step numbers in the above embodiments do not imply that they must be executed in numerical order. The execution order of each step should be determined based on its function and internal logic. Where there is no conflict, the technical features in the above embodiments can be combined with each other.

[0096] This application also provides a voice service connection control device for executing the steps in the above-described voice service connection control method embodiments. This voice service connection control device can be a virtual device within a voice service gateway device, run by the processor of the voice service gateway device, or it can be implemented collaboratively by multiple processors.

[0097] like Figure 7 As shown, the voice service connection control device 200 provided in this application embodiment includes: The message receiving module 201 is used to receive the uplink message sent by the client 10 and output the uplink message to the engine determination module 202; Engine determination module 202 is used to determine a target speech engine from at least two speech engines 20 based on the capability type identifier carried in the uplink message; The connection processing module 203 is used to establish or reuse a downstream connection with the target speech engine, and forward the payload content of the uplink message to the target speech engine through the downstream connection. The lifecycle management module 204 is used to receive the end identifier forwarded by the message receiving module 201, start timing according to the grace period corresponding to the target speech engine, and close the downstream connection when the grace period expires and no final result is received. The grace period corresponding to different speech engines 20 is different. The adapter module 205 is used to map the connection establishment, data transmission and reception, connection closure and error codes of each voice engine 20 to a unified state within the voice service connection control device 200.

[0098] In the application, the message receiving module 201 can be implemented by the processing circuit or program instructions corresponding to the client entry 101 in the voice service gateway device; the engine determination module 202 can be implemented by the processing circuit or program instructions corresponding to the capability routing unit 102; the connection processing module 203 can be implemented by the processing circuit or program instructions corresponding to the connection registration unit 103 and the adapter 105; the lifecycle management module 204 can be implemented by the processing circuit or program instructions corresponding to the lifecycle management unit 104; and the adaptation module 205 can be implemented by the processing circuit or program instructions corresponding to the adapter 105. The message receiving module 201 delivers the capability type identifier, payload content, and session identifier to the engine determination module 202; the engine determination module 202 outputs the target voice engine; the connection processing module 203 establishes or reuses the downstream connection and forwards the payload content to the target voice engine; the lifecycle management module 204 controls the closure of the downstream connection based on the end identifier and grace period; and the adaptation module 205 performs protocol conversion and unified state mapping.

[0099] In applications, the modules in the voice service connection control device 200 can be software program modules or implemented through different logic circuits in the processor. The above module division is used to describe the corresponding functions and does not limit each module to be implemented by independent hardware; two or more modules can be integrated into one functional module, and a module can also be divided into multiple functional units.

[0100] like Figure 8As shown, this application embodiment also provides a voice service gateway device 300, including: a processor 301 and a memory 302. The memory 302 stores a computer program 303. When the computer program 303 is executed by the processor 301, it causes the voice service gateway device 300 to perform the steps in any of the above-described voice service connection control method embodiments.

[0101] In applications, the processor 301 of the voice service gateway device 300 can be a central processing unit, a digital signal processor, an application-specific integrated circuit, a field-programmable gate array, or other programmable logic devices. The memory 302 can include internal storage units, external storage devices, or a combination of both, for storing the operating system, computer programs, and connection registration data.

[0102] In applications, the voice service gateway device 300 may include, but is not limited to, a processor 301 and a memory 302. Those skilled in the art will understand that... Figure 8 This is merely an example of a voice service gateway device 300 and does not constitute a limitation on the voice service gateway device 300. It may include more or fewer components than shown in the figure, or combine certain components, or different components. For example, it may also include a client entry 101, a capability routing unit 102, a connection registration unit 103, a lifecycle management unit 104, and multiple adapters 105, input / output devices, network access devices, etc.

[0103] In applications, processor 301 can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or processor 301 can be any conventional processor.

[0104] In applications, memory 302 may be an internal storage unit of the voice service gateway device 300 in some embodiments, such as a hard disk or memory of the voice service gateway device 300. In other embodiments, memory 302 may be an external storage device of the voice service gateway device 300, such as a pluggable hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the voice service gateway device 300. Furthermore, memory 302 may include both internal storage units and external storage devices of the voice service gateway device 300. Memory 302 is used to store operating systems, applications, boot loaders, data, and other programs, such as program code for computer programs. Memory 302 can also be used to temporarily store data that has been output or will be output.

[0105] It should be noted that the information interaction and execution process between the above-mentioned devices / equipment / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0106] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units is merely an example. In practical applications, the above functions can be assigned to different functional units or modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0107] This application also provides a computer-readable storage medium storing a computer program, which, when executed by processor 301, implements the steps in any of the above-described voice service connection control method embodiments.

[0108] This application also provides a computer program product, including a computer program. When the computer program is run, it causes the voice service gateway device 300 to perform the steps in any of the above-described voice service connection control method embodiments.

[0109] In applications, computer programs may include source code, object code, executable files, or intermediate code.

[0110] This application embodiment may also provide a computer-readable storage medium storing a computer program, which, when executed by processor 301, implements the steps in any of the above-described voice service connection control method embodiments.

[0111] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed by the processor 301, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / app, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0112] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0113] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0114] In the embodiments provided in this application, it should be understood that the disclosed apparatus / device and method can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A voice service connection control method, characterized in that, include: Receive uplink messages sent by the client, and determine the target speech engine from at least two speech engines based on the capability type identifier carried in the uplink messages; Establish or reuse a downstream connection with the target speech engine, and forward the payload content of the uplink message to the target speech engine via the downstream connection; Monitor the elapsed time since the last time uplink audio data was received from the client; When the elapsed time reaches the keep-alive interval and the target speech engine needs to be kept alive, keep-alive data is sent to the target speech engine through the downstream connection to keep the downstream connection active. Receive the end identifier sent by the client; The timing begins according to the grace period corresponding to the target speech engine; When the grace period expires and no final result is received from the target speech engine, the downstream connection is closed. The grace period varies depending on the voice engine. The keep-alive data indicates that the current voice processing task has not ended and is not used to trigger the timing of the grace period.

2. The voice service connection control method as described in claim 1, characterized in that, The receiving client sends an uplink message, and based on the capability type identifier carried in the uplink message, determines the target speech engine from at least two speech engines, including: Receive multiple uplink messages sent by the client through the same client connection; Read the capability type identifier carried in each of the aforementioned uplink messages; Based on the speech engine processing branch corresponding to each of the capability type identifiers, the target speech engine corresponding to each of the uplink messages is determined.

3. The voice service connection control method as described in claim 1, characterized in that, The establishment or reuse of the downstream connection with the target speech engine includes: Obtain the user identifier that sent the uplink message; Search the connection registration table for available connections corresponding to the user identifier and the target speech engine; When an available connection is found, that available connection is used as the downstream connection. If no available connection is found, the downstream connection is established, and after successful establishment, the downstream connection is registered in the connection registration table along with the user identifier and the target speech engine.

4. The voice service connection control method as described in claim 3, characterized in that, After obtaining the user identifier that sent the uplink message, the method further includes: Obtain the session identifier carried in the uplink message; The combination of the user identifier, the capability type identifier, and the session identifier is used as a composite index for querying and registering connections in the connection registration table, so as to isolate downstream connections corresponding to different voice processing capabilities of the same user from each other, and to isolate downstream connections corresponding to different sessions initiated by the same user for the same voice processing capability from each other.

5. The voice service connection control method as described in claim 4, characterized in that, The establishment or reuse of the downstream connection with the target speech engine also includes: If no available connection is found, the composite index is marked as being built. When the downstream connection is successfully established, the downstream connection is registered as active. When the downstream connection fails to be established, the record corresponding to the establishment status is deleted to prevent failed connections from remaining in the connection registration table.

6. The voice service connection control method as described in claim 1, characterized in that, After receiving the end identifier sent by the client, the method further includes: Stop sending the keep-alive data for the current session and update the downstream connection to a state awaiting final result; Upon receiving the final result within the grace period, the final result is returned to the client and the downstream connection is closed. The keep-alive process during task execution and the grace period recycling process triggered by the end flag are triggered by separate conditions.

7. The voice service connection control method as described in claim 1, characterized in that, The grace period is determined according to any of the following methods: A fixed grace period is pre-configured based on the response characteristics of the target speech engine; or, The system obtains at least one of the following: the protocol-specified duration of the target voice engine, the statistical value of the time interval between two adjacent transmissions of service data by the downstream connection within a historical period, and the error rate of the downstream connection within the historical period. Based on the obtained information, the system determines the grace period and constrains the determined grace period to a range from the lower limit to the upper limit.

8. The voice service connection control method as described in claim 7, characterized in that, Also includes: Receive an update instruction for the grace period and save the updated grace period indicated by the update instruction; For downstream connections that have already started measuring the grace period, continue using the grace period corresponding to the already started grace period, and start measuring according to the updated grace period from the next time the end flag is received.

9. The voice service connection control method as described in claim 1, characterized in that, The at least two voice engines are connected via corresponding adapters; Each adapter provides a unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping, and performs the conversion between the unified interface for connection establishment, data transmission, data reception, connection closure, and error code mapping and the private protocol of the corresponding speech engine within the adapter.

10. The voice service connection control method as described in claim 1, characterized in that, The closure of the downstream connection includes: Stop receiving new payload for the current session and stop timing for the downstream connection; Invoke the adapter corresponding to the target speech engine and disconnect the downstream connection according to the protocol of the target speech engine; Cancel the corresponding record from the connection registration table used to register the downstream connection, and record the reason for the recycling of the downstream connection.

11. The voice service connection control method as described in claim 4, characterized in that, Also includes: When the downstream connection is abnormally disconnected, the user identifier and the session identifier remain unchanged; Re-establish the connection with the target speech engine, and register the re-established connection as the current downstream connection corresponding to the composite index.

12. The voice service connection control method as described in claim 11, characterized in that, Also includes: In the absence of re-establishing the connection with the target speech engine, a backup speech engine is determined from the at least two speech engines that provides the same speech processing capabilities as the target speech engine; A connection is established through the adapter corresponding to the backup voice engine, and the established connection is registered as the current downstream connection with the composite index. During the switch to the backup voice engine, the message protocol used by the client and the session identifier remain unchanged.

13. A voice service gateway device, characterized in that, It includes a processor and a memory, the memory storing a computer program, which, when executed by the processor, causes the voice service gateway device to perform the voice service connection control method according to any one of claims 1 to 12.

14. A computer program product, characterized in that, It includes a computer program, which, when run, causes the voice service connection control method of any one of claims 1 to 12 to be executed.

Images