Model online hot replacement deployment method and device and electronic equipment

Abstract

The application is suitable for the technical field of model deployment, and provides a model online hot replacement deployment method and device and electronic equipment. The method comprises the following steps: in the process of traffic migration from an old version model to a new version model, for a streaming voice conversation still processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then the speech processing hidden state of the streaming voice conversation is subjected to pre-migration feasibility verification according to the state compatibility contract; if the speech processing hidden state passes the pre-migration feasibility verification, then the speech processing hidden state is migrated from the old version model to the new version model via a unified intermediate representation; and after the migration is successful, subsequent speech service requests of the streaming voice conversation are routed to the new version model. Through the application, the context continuity of the streaming voice conversation can be maintained during version switching.

Description

Technical Field

[0001] This application belongs to the field of model deployment technology, and in particular relates to a method, apparatus and electronic device for online hot-swap deployment of models. Background Technology

[0002] Artificial intelligence (AI) models used for voice services typically require periodic upgrades to improve accuracy, naturalness, or scalability. Related technologies often involve downtime updates or coarse-grained switching with overall traffic after dual deployment. Downtime updates can easily lead to minute-level service interruptions, and switching traffic after dual deployment can result in inconsistent request destinations and loss of session context during the switching process. These issues with related technologies can disrupt the contextual continuity of streaming voice sessions when AI models are being switched. Summary of the Invention

[0003] This application provides a method, apparatus, and electronic device for online hot-swap deployment of models, which can maintain the contextual continuity of streaming voice sessions during version switching.

[0004] In a first aspect, embodiments of this application provide a method for online hot-swap deployment of a model, including: During the traffic migration process from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then the voice processing hidden state of the streaming voice session is checked for feasibility before migration based on the state compatibility contract; the state compatibility contract refers to the compatibility agreement between the old version model and the new version model regarding the hidden state structure. If the speech processing hidden state passes the pre-migration feasibility check, then the speech processing hidden state is migrated from the old version model to the new version model via a unified intermediate representation. After a successful migration, subsequent voice service requests for the streaming voice session will be routed to the new version model.

[0005] In this embodiment, the state compatibility contract refers to the compatibility agreement between the old version model and the new version model regarding the hidden state structure. Based on this, during the traffic migration process from the old version model to the new version model, for streaming voice sessions still processed by the old version model, it is determined whether the old version model and the new version model satisfy the state compatibility contract. This allows for the determination of whether the hidden state structure of the old version model is portable during version switching. Only when the state compatibility contract is satisfied, a pre-migration feasibility check is performed on the voice processing hidden state of the streaming voice session according to the state compatibility contract. This ensures that, if the hidden state structure is portable, the voice processing hidden state has the conditions for safe cross-version migration at the current switching time. When the voice processing hidden state passes the pre-migration feasibility check, the voice processing hidden state is migrated from the old version model to the new version model via a unified intermediate representation. This ensures that the voice processing hidden state is safely migrated to the new version model, provided that the voice processing hidden state has the conditions for safe cross-version migration at the current switching time. After a successful migration, subsequent voice service requests for the streaming voice session are routed to the new version model. This allows the context that the streaming voice session has already processed on the old version model to be fully inherited by the new version model through implicit state migration, thereby achieving seamless continuation of the streaming voice session during traffic migration and maintaining the context continuity of the streaming voice session without service interruption.

[0006] In some embodiments of the first aspect, the pre-transfer feasibility check of the voice processing hidden state of the streaming voice session based on the state-compatible contract includes: Based on the state compatibility contract, the expected value of each verification item in the new version model is determined; each verification item includes at least one of the following: state structure version, state tensor dimension, encoder cache time step, and length of speech synthesis attention history key value sequence. If the actual values ​​of each verification item in the hidden state of speech processing match the corresponding expected values ​​in the new version model, then the hidden state of speech processing is determined to have passed the pre-migration feasibility verification. If any of the verification items contains a verification item in which the actual value in the hidden state of speech processing does not match the corresponding expected value in the new version model, then it is determined that the hidden state of speech processing has failed the pre-migration feasibility verification.

[0007] In some embodiments of the first aspect, after performing a pre-transfer feasibility check on the speech processing hidden state according to the state-compatible contract, the method further includes: If the hidden state of the voice processing fails the feasibility check before migration, then a conservative mode is enabled for the streaming voice session. In the conservative mode, the streaming voice session is maintained in the old version model until the streaming voice session ends.

[0008] In some embodiments of the first aspect, after migrating the speech processing hidden state from the old version model to the new version model via a unified intermediate representation, the method further includes: The streaming voice session is input into the new version model after the last N voice frames processed by the old version model, so as to compare the output difference between the old version model and the new version model at the switching point; N is a positive integer, and the switching point refers to the boundary between the last voice frame processed by the old version model and the first voice frame followed by the new version model. If the output difference is not within the preset difference range, the migration is determined to have failed, and a conservative mode is enabled for the streaming voice session. In the conservative mode, the streaming voice session is maintained in the old version model until the streaming voice session ends; If the output difference is within the preset difference range, the migration is considered successful.

[0009] In some embodiments of the first aspect, the method further includes: When performing traffic migration from the old version model to the new version model according to multiple incremental traffic phases, in the target traffic phase, comparable samples are obtained based on the received voice service requests; the target traffic phase is the traffic phase in which the traffic carried by the new version model is less than the preset traffic. The sample layer to which each comparable sample belongs is determined according to the scene type, device type, and language to which each comparable sample belongs; at least one of the scene type, device type, and language to which each comparable sample in each sample layer belongs is the same; For the i-th key sample layer, based on each comparable sample in the i-th key sample layer, the degradation result of the i-th key sample layer is determined, and based on the degradation results of all key sample layers, a decision is made to deploy the new version model; the i-th key sample layer is any key sample layer among all sample layers, and the degradation result of the i-th key sample layer characterizes whether the new version model has degraded relative to the old version model.

[0010] In some embodiments of the first aspect, the degradation result of the i-th critical sample layer includes the online degradation result; the step of determining the degradation result of the i-th critical sample layer based on each comparable sample in the i-th critical sample layer, and making a decision on the deployment of the new version model based on the degradation results of all critical sample layers, includes: Based on each comparable sample in the i-th key sample layer, the proxy quality indicators of the new version model and the old version model in the i-th key sample layer are collected in real time. Based on the proxy quality indicators of the new version model and the old version model respectively in the i-th key sample layer, the online degradation result of the i-th key sample layer is determined; If any of the key sample layers has an online degradation result indicating that the new version model has degraded relative to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the online degradation results of each key sample layer indicate that the new version model has not degraded relative to the old version model, then the offline evaluation indicators of the new version model and the old version model in the i-th key sample layer are obtained respectively. The decision to launch the new version model is made based on the offline evaluation metrics of the new version model and the old version model at the i-th key sample layer.

[0011] In some embodiments of the first aspect, the degradation result of the i-th key sample layer includes offline degradation results; the decision to deploy the new version model based on the offline evaluation metrics of the new version model and the old version model respectively in the i-th key sample layer includes: Based on the offline evaluation metrics of the new version model and the old version model respectively in the i-th key sample layer, the offline degradation result of the i-th key sample layer is determined; If any of the key sample layers contains a sample layer with offline degradation results indicating that the new version model has degraded relative to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the offline degradation results of each key sample layer indicate that the new version model has not degraded relative to the old version model, the new version model is controlled to enter the next traffic stage of the target traffic stage.

[0012] In some embodiments of the first aspect, prior to performing the pre-transfer feasibility check on the speech processing hidden state of the streaming voice session according to the state-compatible contract, the method further includes: Keep the old version of the model online voice service; Load and warm up the new version of the model in an independent operating environment; After the new version model has been warmed up, the new version model is registered to the traffic switcher; The traffic switcher gradually increases the proportion of voice service requests routed to the new version model and correspondingly decreases the proportion of voice service requests routed to the old version model according to multiple incremental traffic phases, so as to perform traffic migration from the old version model to the new version model. For a target session, all voice service requests of the target session in the same traffic phase are routed to the same version model; the target session includes a new session, or a streaming voice session that is still being processed in the old version model or the new version model during the traffic migration from the old version model to the new version model in conservative mode. During the traffic migration process from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then a pre-migration feasibility check is performed on the hidden state of voice processing of the streaming voice session based on the state compatibility contract, including: In migration mode, during the process of migrating traffic from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then the voice processing hidden state is checked for feasibility before migration based on the state compatibility contract.

[0013] Secondly, embodiments of this application provide a model online hot-swap deployment device, comprising: The migration verification module is used to perform a pre-migration feasibility verification of the voice processing hidden state of the streaming voice session during the traffic migration process from the old version model to the new version model. If the old version model and the new version model satisfy a state compatibility contract, the module will perform a pre-migration feasibility verification of the hidden state of the streaming voice session based on the state compatibility contract. The state compatibility contract refers to the compatibility agreement between the old version model and the new version model regarding the hidden state structure. The state transition module is used to migrate the speech processing hidden state from the old version model to the new version model via a unified intermediate representation if the speech processing hidden state passes the pre-migration feasibility check. The request routing module is used to route subsequent voice service requests of the streaming voice session to the new version model after the migration is successful.

[0014] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the electronic device enables the online hot-swap deployment method for the model as described in any one of the first aspects above.

[0015] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computer, implements the online hot-swap deployment method for the model as described in any one of the first aspects above.

[0016] Fifthly, embodiments of this application provide a computer program product, including a computer program, which, when run, causes the method as described in any one of the first aspects above to be performed.

[0017] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the online hot-swap deployment method for models provided in this application embodiment; Figure 2 This is another flowchart illustrating the online hot-swap deployment method for models provided in this application embodiment; Figure 3 This is a schematic diagram of the architecture of session-level consistent routing provided in an embodiment of this application; Figure 4 This is another flowchart illustrating the online hot-swap deployment method for models provided in this application embodiment; Figure 5 This is another flowchart illustrating the online hot-swap deployment method for models provided in this application embodiment; Figure 6 This is another flowchart illustrating the online hot-swap deployment method for models provided in this application embodiment; Figure 7 This is an example diagram of the architecture for progressive traffic switching and degradation processing provided in the embodiments of this application; Figure 8 This is a schematic diagram of the online hot-swap deployment device for models provided in this application embodiment; Figure 9 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0020] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0021] It should be understood that, when used in this application specification, the term "comprising" indicates the presence of the described feature, integral, step, operation, element, and / or component, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0022] Furthermore, in the description of this application, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0024] AI models used for voice services typically require periodic upgrades to improve accuracy, naturalness, or scalability. Related technologies often involve downtime updates or coarse-grained switching with overall traffic after dual deployment. Downtime updates can easily lead to service interruptions lasting minutes, and switching traffic after dual deployment can result in inconsistent request destinations and loss of session context during the switching process. Voice services have high requirements for context continuity; user-perceptible interruptions, jitter in recognition style, or issues with synthesized timbre and rhythmic abrupt changes can significantly impact user experience.

[0025] The relevant technologies typically have the following limitations when applied to voice services: (1) Upgrades involving shutdown or restart can easily lead to service interruptions lasting minutes, failing to meet the continuity requirement of "imperceptible interruption" for voice services; (2) After dual deployment, the overall traffic switching is mostly a coarse-grained one-time switch, lacking gradual grayscale and session-level consistency. At the moment of switching, different requests in the same session may fall into different versions, causing jitter in recognition style, changes in synthesized timbre or prosody, and contextual breaks; (3) There is a lack of real-time quality backstop during the deployment process. If the new version model has quality regression, it is often only discovered after offline statistics, making it difficult to detect in a small area. When the rollback occurs, the impact is amplified; and the word error rate and subjective quality score of the voice service often require the annotation of true values ​​or manual scoring, which are difficult to obtain online in real time, further increasing the difficulty of real-time quality assurance; (4) the hidden state of the streaming voice conversation in transit at the moment of switching (such as the encoder cache of streaming recognition and the historical key value of synthesized attention) is difficult to continue across versions, resulting in the interruption of the context of long sentences and long conversations, and the relevant technologies do not stipulate the compatibility conditions and failure rollback strategies between versions; (5) the lack of structured audit of the entire upgrade process makes it difficult to trace the traffic ratio, quality indicator comparison and online / rollback decision of each stage afterward.

[0026] Among the relevant solutions, some focus on model training and compression, while others emphasize the canary release and traffic scheduling of general microservices. These solutions typically fail to simultaneously address the following set of issues: zero-disruption dual-model buffering for voice services, consistent request routing by session, maintaining context continuity for ongoing streaming voice sessions during version switching through conservative emptying or contractual migration, and real-time quality gating with online-available proxy quality metrics on low-traffic conditions, with automatic traffic reallocation in case of anomalies. Therefore, there is an urgent need for a zero-disruption online hot-swap solution for AI models used in voice services that minimizes user-perceptible interruptions.

[0027] Based on this, embodiments of this application provide a model online hot-swap deployment method, apparatus, electronic device, and product. This model online hot-swap deployment method is a zero-disruption AI model online hot-swap deployment method for voice services, including a dual-model buffer pool and session-level consistent routing scheme that enables zero-disruption parallel hosting of new and old version models; a conservative natural drain and contractual migration dual-mode scheme that enables continuity of streaming voice session switching; and a proxy quality indicator fallback scheme that enables real-time quality gating on low traffic volumes and automatic traffic reallocation in case of anomalies.

[0028] The dual-model buffer pool and session-level consistent routing scheme: the old version model continues to serve online; the new version model is loaded and warmed up in an independent runtime environment (e.g., an independent container) (e.g., loading weights, establishing a memory cache, and performing several idle inferences to bring it to a stable state), and then registered with the traffic switcher after warming up; the traffic switcher migrates traffic gradually in multiple configurable incremental traffic stages, rather than switching the entire system at once; at the same time, session-level consistent routing ensures that all requests in the same session stably fall on the same version model, reducing the risk of cross-version jitter within a session. Thus, the new and old versions of the model run in parallel during the upgrade, transitioning smoothly and significantly reducing the risk of user-perceptible interruptions.

[0029] A dual-mode solution of conservative natural draining and contractual migration: For streaming voice sessions still being processed in the old version model during version switching, this application embodiment maintains context continuity using a session switching continuity strategy and provides two selectable modes: conservative mode and migration mode. In conservative mode, the streaming voice session remains in its current version model until the session ends; only newly initiated sessions enter the new version model according to the current traffic stage (i.e., the old version model is naturally drained at the session level). In migration mode, the streaming voice session is allowed to cross versions and migrate its voice processing hidden state only when the new version model and the old version model satisfy a state compatibility contract. Migration verification is performed according to the state compatibility contract; if the verification fails, it reverts to conservative mode. Thus, by default, conservative natural draining ensures consistency, and migration only occurs when the state compatibility contract is satisfied. The logic is closed and controllable, ensuring that session consistency routing and in-transit state migration no longer contradict each other. Here, natural draining of the old version model at the session level can mean no longer allocating new sessions to the old version model, allowing existing sessions on the old version model to naturally continue until their end, and the old version model can be taken offline after the number of sessions it carries gradually decreases to zero.

[0030] Proxy quality metric fallback solution: The new version model is initially tested on a small percentage of traffic, with real-time collection of available online proxy quality metrics. These metrics are then compared to the old version, which carries the control group traffic. Comparable samples can be obtained using methods such as random bucketing, shadow traffic mirroring, and dual-run evaluation of the same request on both versions of the model. Comparisons can be stratified by scenario type, device type, or language. For metrics requiring ground truth labeling or manual scoring, non-online evaluation metrics, such as sampling annotation, shadow traffic, or offline replay, are used in conjunction with proxy quality metrics for deployment decisions. When the degradation of any non-online evaluation metric or any core proxy quality metric relative to the control group exceeds a configurable threshold, the traffic allocated to the new version model is automatically redistributed to the old version model, and the current traffic ramp-up is stopped; otherwise, the traffic carried by the new version model is gradually increased in stages until full capacity is reached.

[0031] The online hot-swap deployment method for a model provided in this application embodiment may further include a full-process audit scheme for hot-swap. The audit module can record structured information for each hot-swap: the version number of the new model, the version number of the old model, the start and end times and traffic ratios of each traffic phase, the session switching continuity mode (conservative mode / migration mode) used and the migration verification results, the quality and performance indicators of each traffic phase and their comparison results with the control group, the statistical comparison conclusions of each traffic phase, the final decision (going live or reallocating traffic), and the triggering indicators and reallocation time in the case of traffic reallocation. The above audit records can be used for post-event tracing of the upgrade process, reviewing decision-making basis, and supporting compliance reviews.

[0032] The online hot-swap deployment method for models provided in this application includes the above-mentioned schemes, which together constitute a zero-disruption hot-swap closed loop for voice services. That is, it replaces downtime or overall switching with dual-model buffer pools and progressive gray-scale deployment; it replaces coarse-grained routing with session-level consistency; it replaces immediate switching interruption with conservative natural draining or contractual migration; and it replaces post-event offline discovery with real-time gating of proxy quality indicators and automatic traffic reallocation.

[0033] The online hot-swap deployment method for models provided in this application can be applied to electronic devices such as audio devices (e.g., speakers), wearable devices, mobile phones, tablets, in-vehicle systems, augmented reality (AR) / virtual reality (VR) devices, desktop computers, servers, laptops, ultra-mobile personal computers (UMPCs), netbooks, and personal digital assistants (PDAs). This application does not impose any restrictions on the specific type of electronic device.

[0034] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0035] Please see Figure 1 , Figure 1 The diagram illustrates a flowchart of an online hot-swap deployment method for a model provided in an embodiment of this application. This is an example, not a limitation, and can be applied to electronic devices. The online hot-swap deployment method includes the following steps: Step 101: During the traffic migration process from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy the state compatibility contract, then the feasibility of the voice processing hidden state of the streaming voice session before migration is verified according to the state compatibility contract.

[0036] The traffic migration from the old version model to the new version model can refer to the complete process of switching from the old version model to the new version model. Based on this, during the traffic migration process from the old version model to the new version model, electronic devices can determine whether the old version model and the new version model satisfy a state compatibility contract for streaming voice sessions still being processed in the old version model during the version switch. If the old version model and the new version model satisfy the state compatibility contract, then a pre-migration feasibility check is performed on the hidden state of voice processing for that streaming voice session. If the old version model and the new version do not satisfy the state compatibility contract, then the streaming voice session continues to be maintained in the old version model until the session ends, without needing to perform a migration feasibility check on the hidden state of voice processing.

[0037] A streaming voice conversation that is still being processed by the old version model during a version switch can refer to a conversation that has not yet ended at the point when the rollout is accelerated and the routing shifts from the old version model to the new version model.

[0038] Streaming voice conversations can refer to ongoing voice interaction processes that have not yet ended. For example, an unfinished long sentence in a streaming Automatic Speech Recognition (ASR) session or an unfinished text-to-speech (TTS) synthesis session are both examples of streaming voice conversations.

[0039] A state compatibility contract can refer to a compatibility agreement between an old version model and a new version model regarding the hidden state structure. It can be a set of pre-agreed compatibility conditions and adaptation rules for determining whether the hidden states of speech processing in the old version model can be safely continued in the new version.

[0040] The hidden state structure of the old version model can refer to the static specifications of the data organization used to carry and pass context within the old version model. It defines the hidden state tensors included in the old version model, as well as the shape, meaning, and layout of each hidden state tensor.

[0041] The hidden state structure of the new version model can refer to the static specifications of the data organization used to carry and pass context within the new version model. It defines the hidden state tensors included in the new version model, as well as the shape, meaning, and layout of each hidden state tensor.

[0042] It should be understood that all sessions of the same version of the model share the same hidden state structure, while different versions of the model may have different hidden state structures.

[0043] The latent state of speech processing in the streaming speech session processed by the old version model can refer to the dynamic data instances generated and maintained in real time by the old version model during the processing of the streaming speech session. It is the specific filling of the latent state structure of the old version model on the streaming speech session.

[0044] It should be understood that the speech processing latent state is dynamic and real-time, changing as the conversation progresses. Each session has an independent speech processing latent state, which contains the entire context of the session from its beginning to the current moment.

[0045] The aforementioned hidden state structure may include at least one of a state structure version, the dimension of the hidden state, and the number of levels. Based on this, the state-compatible contract may include at least one of the following: state structure (schema) version consistency, the mapping relationship between the dimension and level of the hidden state, and a state adapter when this mapping relationship is a non-identical mapping. The state structure version may refer to a version of the hidden state structure.

[0046] State structure version consistency refers to the consistency between the state structure version of the old model and the state structure version of the new model. That is, whether the hidden state structure of the old model and the hidden state structure of the new model are the same version.

[0047] The dimension of the hidden state can refer to the shape dimension of the hidden state tensor (e.g., feature dimension, time step, etc.). The number of hidden state layers can refer to the number of network layers in the corresponding model that carry the hidden state (e.g., multiple layers of encoders / decoders each holding a cache / key value).

[0048] The mapping relationship between the dimensions and layers of the hidden states can refer to a mapping relationship between the dimensions and layers of the hidden states in the old version model and the dimensions and layers of the hidden states in the new version model. This mapping relationship can be an identity mapping or a non-identical mapping. An identity mapping means that the number of layers of the hidden states in the old version model is the same as the number of layers of the hidden states in the new version model, and the dimension of the hidden states in each layer of the old version model is the same as the dimension of the hidden states in the corresponding layer of the new version model. A non-identical mapping means that the number of layers of the hidden states in the old version model is different from the number of layers of the hidden states in the new version model, or that the dimension of the hidden states in a certain layer of the old version model is different from the dimension of the hidden states in the corresponding layer of the new version model.

[0049] A state adapter with a non-identical mapping relationship can refer to a state adapter that can adapt the number of hidden states in the old version model to the number of hidden states in the new version model, and adapt the dimension of the hidden states in each layer of the old version model to the dimension of the hidden states in the corresponding layer of the new version model, when the mapping relationship between the dimension of the hidden states and the number of layers of the hidden states in the old version model is not an identity mapping.

[0050] In this embodiment, by determining whether the old version model and the new version model satisfy the state compatibility contract, it is possible to determine whether the hidden state structure of the old version model has transferability when switching versions. This reduces the risk of subsequent speech processing hidden states being incorrectly consumed due to the lack of transferability of the hidden state structure of the old version model, thereby reducing the risk of recognition style jitter and synthesized timbre and prosody jumps caused by the incorrect consumption of speech processing hidden states.

[0051] Step 102: If the speech processing hidden state passes the pre-migration feasibility check, then the speech processing hidden state is migrated from the old version model to the new version model via a unified intermediate representation.

[0052] The pre-migration feasibility check is used to confirm whether the hidden state of voice processing has the conditions for safe cross-version migration at the current switching time (i.e., the version switching time).

[0053] A unified intermediate representation can refer to a set of standardized hidden state structures and their encoding / decoding / serialization rules that are independent of specific versions. These rules define field layouts, dimension / layer semantics, and serialization formats, enabling different version models to be encoded and decoded according to the same contract.

[0054] In this embodiment, when the speech processing hidden state passes the pre-migration feasibility check, the speech processing hidden state is migrated from the old version model to the new version model via a unified intermediate representation. This ensures that the speech processing hidden state is safely migrated to the new version model, provided that it is confirmed that the speech processing hidden state has safe cross-version migration capabilities at the current switching time. Furthermore, the direct conversion from the old version model to the new version model is converged into a conversion from the old version model to the intermediate representation and a conversion from the intermediate representation to the new version model. This reduces the coupling between the old and new version models and facilitates the expansion of the new version model.

[0055] Step 103: After the migration is successful, route subsequent voice service requests for the streaming voice session to the new version model.

[0056] Here, subsequent voice service requests can refer to voice service requests for streaming voice sessions received after the migration is successful.

[0057] It should be understood that before the migration is successful, voice service requests for the streaming voice session are routed to the old model. After the migration is successful, voice service requests for the streaming voice session are routed from the old model to the new model.

[0058] In this embodiment, by combining state-compatible contracts and pre-migration feasibility checks during the online hot replacement of the model to migrate the hidden state of voice processing of the in-transit streaming voice session, it is possible to ensure that the hidden state of voice processing is safely migrated to the new version model. On this basis, after the migration is successful, the subsequent voice service requests of the streaming voice session are routed to the new version model, which enables the context that the streaming voice session has processed on the old version model to be fully inherited by the new version model through the migration of hidden states. This achieves seamless continuity of the streaming voice session during the traffic migration process and maintains the context continuity of the streaming voice session without service interruption.

[0059] like Figure 2 As shown, in some embodiments of this application, before performing a pre-migration feasibility check on the voice processing hidden state of the streaming voice session according to the state-compatible contract, steps 201 to 206 are also included.

[0060] Step 201: Keep the old version of the model online with voice service.

[0061] The voice service includes, but is not limited to, at least one of speech recognition, speech synthesis, and natural language understanding.

[0062] In this embodiment, keeping the old version model online for voice service allows it to continue processing voice service requests received by electronic devices, ensuring uninterrupted service.

[0063] Step 202: Load and warm up the new version of the model in a standalone runtime environment.

[0064] Alternatively, the standalone runtime environment can be a standalone container or a standalone process.

[0065] Loading the new model version includes, but is not limited to, loading weights and establishing a memory cache. Warming up the new model version includes, but is not limited to, performing several idle inferences to bring it to a stable state. Optionally, the number of idle inferences can be configured according to actual needs or empirical values.

[0066] Step 203: After the new version model has been warmed up, register the new version model to the traffic switcher.

[0067] Among them, the traffic switcher can refer to the routing control module deployed at the request entry point, which is used to dynamically adjust the request allocation ratio to each version model according to the preset traffic stage between the old version model and the new version model.

[0068] In this embodiment, loading and preheating the new version model in an independent operating environment allows the computation graph and cache of the new version model to enter a stable state. Registering the new version model after preheating to the traffic switch allows it to enter a ready state that can be gradually scaled up. Based on this, the new version model and the old version model reside in parallel, forming a dual-model buffer pool.

[0069] Step 204: The traffic switcher gradually increases the proportion of voice service requests routed to the new version model and correspondingly reduces the proportion of voice service requests routed to the old version model according to multiple incremental traffic phases, so as to perform traffic migration from the old version model to the new version model.

[0070] Optionally, multiple incremental traffic phases can be pre-configured based on actual needs or experience, and the dwell time of each traffic phase can also be pre-configured based on actual needs or experience. For example, multiple incremental traffic phases may include: 1% → 5% → 25% → 100%.

[0071] In this embodiment, the traffic switcher performs traffic migration from the old version model to the new version model according to multiple incremental traffic stages. This enables the gradual migration of voice service requests from the old version model to the new version model, achieving online model upgrades. Moreover, the upgrade process does not require downtime or any overall switching interruption, thus achieving uninterrupted online upgrades and meeting the continuity requirements of voice service without imperceptible interruptions.

[0072] In this embodiment, the gradual release with small initial volumes and phased increases can limit the potential risks of each volume phase to a controllable range, making it easier to detect and handle anomalies early during the volume increase process.

[0073] Step 205: For the target session, route all voice service requests of the target session in the same traffic phase to the same version model; the target session includes new sessions, or streaming voice sessions that are still being processed in the old version model or the new version model during the traffic migration from the old version model to the new version model in conservative mode.

[0074] Conservative mode can refer to a session processing method that keeps the streaming voice session on the current version model for processing until the streaming voice session ends naturally.

[0075] It should be understood that in conservative mode, if the streaming voice session is currently being processed by the old version model, the old version model can be determined as the current version model; if the streaming voice session is currently being processed by the new version model, the new version model can be determined as the current version model.

[0076] In this embodiment, for the target session, all voice service requests in the same traffic phase of the target session are routed to the same version model, which can achieve session-level consistent routing, reduce the risk of recognition style jitter and synthesized timbre and rhythm jumps, and thus achieve intra-session consistency and no cross-version jitter.

[0077] In some embodiments, the traffic switch can perform session-level consistent routing based on a session affinity key. The session affinity key can be a session identifier used to uniquely identify a session. For example... Figure 3 As shown, for any traffic phase, the routing table includes the traffic allocation ratio (phase ratio) for each version model. For a target session, the traffic switcher can perform a hash calculation on the session affinity key of the target session, and determine the version model corresponding to the target session based on the proportion range of the obtained hash value falling into the routing table. All voice service requests of the target session within this traffic phase are then routed to the determined version model to reduce the risk of cross-version jitter within the session.

[0078] In one application scenario, this embodiment can achieve session-level consistent routing during TTS model upgrades. In synthesis services, switching between versions midway through the same synthesis session can easily cause timbre or rhythmic abrupt changes. This embodiment calculates a stable session affinity key for each synthesis session and maps it to the version according to the routing table, ensuring that all synthesis requests for the same session consistently fall under the same version during the scaling-up phase; only newly initiated sessions enter the new version according to the current traffic phase, thereby reducing the risk of timbre and rhythmic abrupt changes within the session while gradually scaling up.

[0079] Step 206: In migration mode, during the process of migrating traffic from the old version model to the new version model, for streaming voice conversations still being processed by the old version model, if the old version model and the new version model satisfy the state compatibility contract, then the feasibility of the voice processing hidden state before migration is verified according to the state compatibility contract.

[0080] Migration mode can refer to migrating the implicit state of voice processing in a streaming voice session to a new version model. After successful migration, subsequent voice service requests for the streaming voice session are routed to the session processing mode of the new version model. In some embodiments, migration mode can be enabled when the deployment requires quickly phasing out the old version model to achieve rapid deployment of the new version model.

[0081] In this embodiment, in migration mode, for streaming voice conversations that are still in the old version model when switching versions, contractual state migration is enabled only when the old version model and the new version model satisfy the state compatibility contract. This enables the context of streaming voice conversations such as long sentences and long conversations to remain unbroken before and after the switching point, and the state migration does not contradict the consistent routing, thus achieving conversation continuity and logical self-consistency at the moment of switching.

[0082] In this embodiment, through the collaborative mechanism of progressive migration, session-level consistent routing, and conservative / migration dual modes, it is ensured that newly initiated sessions (i.e., new sessions) are accessed to the new version model in an orderly manner according to traffic stages, and the context continuity of the streaming voice session in transit is ensured when switching versions. Thus, the service is not interrupted, requests are not scattered, and context is not lost during the online hot replacement of AI models used for voice services, which significantly improves user experience and system availability.

[0083] like Figure 4 As shown, in some embodiments of this application, performing a pre-migration feasibility check on the hidden state of voice processing of a streaming voice session based on a state-compatible contract may include steps 401 to 403.

[0084] Step 401: Based on the state-compatible contract, determine the expected value of each verification item in the new version model.

[0085] Each validation item may include at least one of the following: state structure version, state tensor dimension, encoder buffer time step, and length of the speech synthesis attention history key-value sequence. The state tensor dimension may include the shape dimension and the layer dimension of a single hidden state tensor.

[0086] The expected value of a validation item in the new version model can refer to the target value or allowable range set by the new version model for the validation item. These values ​​can be predefined in the state-compatible contract.

[0087] Step 402: If the actual values ​​of each verification item in the speech processing hidden state match the corresponding expected values ​​in the new version model, then the speech processing hidden state is determined to have passed the pre-migration feasibility verification.

[0088] The actual value of the verification item in the implicit state of speech processing can refer to the true value or attribute of the verification item in the implicit state of speech processing.

[0089] Step 403: If there is a verification item in each verification item whose actual value in the speech processing hidden state does not match the corresponding expected value in the new version model, then it is determined that the speech processing hidden state has not passed the feasibility verification before migration.

[0090] State-compatible contracts can also include mapping relationships when state structure versions are inconsistent. Based on this, when each verification item includes a state structure version, the state structure version of the new model can be determined according to the mapping relationships in the state-compatible contract when state structures are consistent or inconsistent. If the state structure version in the speech processing hidden state (usually the state structure version of the old model) is consistent with the state structure version of the new model, or if the state structure version in the speech processing hidden state is inconsistent with the state structure version of the new model but a mapping relationship exists between them, then the state structure version in the speech processing hidden state is determined to match the state structure version of the new model. If the state structure version in the speech processing hidden state is inconsistent with the state structure version of the new model and no mapping relationship exists between them, then the state structure version in the speech processing hidden state is determined to not match the state structure version of the new model.

[0091] When each verification item includes the state tensor dimension, the expected state tensor dimension (i.e., the expected value of the state tensor dimension) corresponding to the state tensor dimension of the speech processing hidden state in the new version model can be determined based on the mapping relationship between the hidden state dimension and the layer number in the state compatibility contract or the state adapter. If the state tensor dimension of the speech processing hidden state is consistent with the expected state tensor dimension, or inconsistent and a state adapter exists, then the state tensor dimension of the speech processing hidden state is determined to match the expected state tensor dimension. If the state tensor dimension of the speech processing hidden state is inconsistent with the expected state tensor dimension and a state adapter does not exist, then the state tensor dimension of the speech processing hidden state is determined to not match the expected state tensor dimension.

[0092] The state-compatible contract may also include the expected encoder cache time step of the new version model (i.e., the expected value of the encoder cache time step). Based on this, when each verification item includes the encoder cache time step, the expected encoder cache time step of the new version model can be determined according to the state-compatible contract; if the encoder cache time step of the speech processing hidden state is less than or equal to the expected encoder cache time step, then the encoder cache time step of the speech processing hidden state is determined to match the expected encoder cache time step; if the encoder cache time step of the speech processing hidden state is greater than the expected encoder cache time step, then the encoder cache time step of the speech processing hidden state is determined to not match the expected encoder cache time step.

[0093] The state compatibility contract may also include the expected length of the speech synthesis attention history key-value sequence of the new version model (i.e., the expected value of the speech synthesis attention history key-value sequence length). When each validation term includes the length of the speech synthesis attention history key-value sequence, the expected length of the speech synthesis attention history key-value sequence of the new version model is determined according to the state compatibility contract. If the length of the speech synthesis attention history key-value sequence of the hidden state of speech processing is less than or equal to the expected length, then the length of the speech synthesis attention history key-value sequence of the hidden state of speech processing matches the length of the expected length. If the length of the speech synthesis attention history key-value sequence of the hidden state of speech processing is greater than the length of the expected length, then the length of the speech synthesis attention history key-value sequence of the hidden state of speech processing does not match the length of the expected length.

[0094] In this embodiment, the feasibility verification before migration described above can effectively reduce the risk of migration failure or context corruption caused by the hidden state of speech processing exceeding the carrying capacity of the new version model (e.g., excessive time step, excessive sequence length, dimension mismatch and no state adapter). Thus, under the premise of ensuring migration security, it provides accurate and reliable cross-version continuity guarantee for streaming speech sessions that meet the conditions, and further improves the robustness of session continuity during model hot replacement.

[0095] In some embodiments of this application, after performing a pre-transition feasibility check on the latent state of speech processing based on the state-compatible contract, the method further includes: If the hidden state of voice processing fails the feasibility check before migration, then the conservative mode is enabled for the streaming voice session. In conservative mode, the streaming voice session remains in the old model until the streaming voice session ends.

[0096] In this embodiment, after performing a pre-migration feasibility check on the hidden state of voice processing in a streaming voice session based on the state compatibility contract, if the check fails, it indicates that the hidden state of voice processing does not meet the conditions for safe migration to the new version model at the current switching time. In this case, since the streaming voice session was originally processed on the old version model, a conservative mode is directly enabled, allowing it to continue processing on the old version model until the session ends naturally. This mechanism reduces the risks of state corruption, context loss, or service anomalies that may result from forced migration, ensuring that such streaming voice sessions are not interrupted or abrupt during version switching, fully preserving the user's entire context from the start to the end of the session, guaranteeing the continuity of the streaming voice session, and further improving the robustness of the model hot-swap scheme and user experience.

[0097] In some embodiments of this application, after migrating the speech processing hidden state from the old version model to the new version model via a unified intermediate representation, the method further includes: The last N speech frames processed by the old version model are input into the new version model to compare the output differences between the old and new version models at the switching point; N is a positive integer, and the switching point is the boundary between the last speech frame processed by the old version model and the first speech frame followed by the new version model. If the output difference is not within the preset difference range, the migration is determined to have failed, and conservative mode is enabled for the streaming voice session. In conservative mode, the streaming voice session remains in the old model until the streaming voice session ends; If the output difference is within the preset difference range, the migration is considered successful.

[0098] Optionally, N and a preset difference range can be set according to actual needs or experience.

[0099] Voice service requests typically consist of multiple consecutive voice frames. A streaming voice session may have already processed at least one voice service request on the older model, and each voice service request contains at least one voice frame. The last N voice frames mentioned above are the last N frames out of all voice frames processed by the streaming voice session on the older model.

[0100] It should be understood that the method of measuring output differences is related to the voice service scenario, in order to ensure that the substantial differences between the new version of the model and the old version can be sensitively captured through output differences. For example, for ASR scenarios, output differences can be measured by unnormalized prediction scores (logits) or the hidden states of a certain layer; for TTS scenarios, acoustic frames can be used to measure output differences.

[0101] In this embodiment, after migrating the hidden state of voice processing from the old version model to the new version model, the output differences between the new and old versions are compared at the switching point. If the output difference is within a preset range, the migration is confirmed as successful, and subsequent voice service requests for the streaming voice session are routed to the new version model, achieving seamless continuation. If the output difference is not within the preset range, the migration is confirmed as failed, and a conservative mode can be enabled to retain the streaming voice session in the old version model until it ends. This mechanism, based on the feasibility verification before migration, adds post-migration consistency verification, improving the security and reliability of cross-version migration of streaming voice sessions.

[0102] like Figure 5 As shown, in some embodiments of this application, the electronic device may also perform steps 501 to 503 to make a decision on the launch of the new version model.

[0103] Step 501: When performing traffic migration from the old version model to the new version model according to multiple incremental traffic stages, in the target traffic stage, obtain comparable samples based on the received voice service requests; the target traffic stage is the traffic stage in which the traffic carried by the new version model is less than the preset traffic.

[0104] Optionally, a preset flow rate can be set based on actual needs or experience. For example, the preset flow rate could be 5%.

[0105] In this embodiment, during the target traffic phase, for each voice service request received, comparable samples can be obtained by methods such as random bucketing, shadow traffic mirroring, and dual-run evaluation of the same request on two version models. It can be determined whether the comparable sample is a comparable sample of the old version model or a comparable sample of the new version model.

[0106] For example, using random bucketing, a received voice service request can be routed to either the old or new version model based on a hash or random number. If routed to the old version model, the voice service request is considered a comparable sample of the old version model; if routed to the new version model, it is considered a comparable sample of the new version model. Using shadow traffic mirroring, a received voice service request can be duplicated (i.e., two identical voice service requests) and sent simultaneously to both the old and new version models, thus identifying these two voice service requests as comparable samples of the old and new version models, respectively. Using a dual-run evaluation method on both version models for the same request, a received voice service request is sent to both the old and new version models simultaneously, thus identifying this voice service request as a comparable sample of both versions.

[0107] It should be noted that the shadow traffic mirroring method and the dual-run evaluation method for the same request on two version models are only used for inference evaluation and do not trigger real business actions (such as dialogue action execution or external write operation); user-facing actions with side effects are only executed by the results of the version carrying the real traffic, so that the quality evaluation has no side effects on online business.

[0108] Step 502: Determine the sample layer to which each comparable sample belongs based on the scene type, device type, and language of each comparable sample; each sample layer includes comparable samples that share at least one of the following: scene type, device type, and language.

[0109] The scene type to which the comparable samples belong can refer to the acoustic scene type or interactive scene type to which the comparable samples belong (e.g., quiet scene, noisy scene, in-vehicle scene, meeting scene, etc.).

[0110] The device type to which the comparable sample belongs can refer to the type of device that carries the comparable sample (e.g., mobile phone, speaker, in-vehicle system, wearable device, etc.).

[0111] The language of the comparable samples can refer to the language used by the comparable samples (e.g., Mandarin, Cantonese, English, etc.).

[0112] In this embodiment, by stratifying comparable samples according to three dimensions—scene type, device type, and language—fine-grained degradation detection of the new version model can be performed from three core dimensions: acoustic environment, hardware platform, and language characteristics. This can effectively reduce the risk that the normal performance of indicators in one dimension may mask the degradation of indicators in other dimensions.

[0113] Step 503: For the i-th key sample layer, based on each comparable sample in the i-th key sample layer, determine the degradation result of the i-th key sample layer, and based on the degradation results of all key sample layers, make a decision on the deployment of the new version model.

[0114] Here, the i-th key sample layer is any key sample layer among all sample layers, and the degradation result of the i-th key sample layer represents whether the new version model has degraded compared to the old version model.

[0115] It should be understood that all key sample layers can refer to the sample layers selected from all sample layers obtained by stratifying according to scenario type, device type, and language, used to make decisions on the deployment of the new version model. All key sample layers can be all sample layers, or a selection of sample layers from all sample layers. They can be manually configured based on business priorities (e.g., determining the sample layers corresponding to in-vehicle scenarios, mobile phones, and Mandarin, as well as the sample layers corresponding to noisy scenarios, speakers, and English as key sample layers), or automatically selected based on historical data (e.g., automatically identifying sample layers that have experienced degradation or large fluctuations in indicators based on historical volume data or online monitoring data, and using them as key sample layers), or dynamically adjusted based on sample size (e.g., sample layers with too few comparable sample sizes are unreliable and should not be used as key sample layers; when the comparable sample size of a sample layer reaches a preset threshold, it is used as a key sample layer).

[0116] The decision to go live can refer to the process of advancing the new version of the model from the target traffic stage to the next traffic stage until the full volume during the gradual rollout, or the decision to reallocate the traffic allocated to the new version of the model to the old version of the model and stop the current rollout.

[0117] In this embodiment, by obtaining comparable samples based on received voice service requests during the target traffic phase, and dividing the comparable samples into multiple sample layers according to at least one of scenario type, device type, and language, determining the degradation results for all key sample layers, and making a decision on the online deployment of the new version model based on the degradation results of all key sample layers, potential problems of the new version model in specific scenarios, specific devices, or specific languages ​​can be discovered in the early stages of deployment, and traffic redistribution or deployment can be triggered in a timely manner, reducing the risk of degradation spreading to a large number of users, and significantly improving the safety, accuracy, and controllability of online hot replacement of the model.

[0118] like Figure 6 As shown, in some embodiments of this application, the degradation result of the i-th key sample layer includes the online degradation result; based on each comparable sample in the i-th key sample layer, the degradation result of the i-th key sample layer is determined, and based on the degradation results of all key sample layers, a decision is made to deploy the new version model, which may include steps 601 to 605.

[0119] Step 601: Based on the comparable samples in the i-th key sample layer, collect the proxy quality indicators of the new version model and the old version model in the i-th key sample layer in real time.

[0120] Agent quality metrics can refer to quantifiable indicators that reflect the quality of model services and are obtained in real time online during the processing of comparable samples.

[0121] Optionally, the agent quality metrics include, but are not limited to, at least one of the following: speech recognition confidence, retry rate, user interruption rate, empty recognition rate, endpoint detection failure rate, speech synthesis failure rate, latency quantile, and anomaly rate.

[0122] It should be understood that the i-th key sample layer includes at least one comparable sample of the old version model and one comparable sample of the new version model. The proxy quality index of the old version model in the i-th key sample layer is collected in real time based on all comparable samples of the old version model (for example, after inputting all comparable samples of the old version model into the old version model, the speech recognition confidence scores corresponding to all comparable samples of the old version model are obtained, and the average of the speech recognition confidence scores corresponding to all comparable samples of the old version model is determined as the speech recognition confidence score of the i-th key sample layer). The proxy quality index of the new version model in the i-th key sample layer is collected in real time based on all comparable samples of the new version model.

[0123] Step 602: Based on the proxy quality indicators of the new version model and the old version model respectively in the i-th key sample layer, determine the online degradation result of the i-th key sample layer.

[0124] In this embodiment, for each proxy quality index, the relative degradation of the i-th key sample layer at the i-th key sample layer can be calculated based on the proxy quality index of the new version model and the old version model at the i-th key sample layer. Based on the relative degradation of the i-th key sample layer across all proxy quality indices, the online degradation result of the i-th key sample layer is determined. The relative degradation of the i-th key sample layer at a proxy quality index measures the degree of performance change of the new version model relative to the old version model at that proxy quality index.

[0125] Since different proxy quality metrics may exhibit different degradation directions, a degradation direction metric can be considered when calculating the relative degradation of the i-th key sample layer in relation to the proxy quality metric. For example, the proxy quality metric of the new version model in the i-th key sample layer can be subtracted from the proxy quality metric of the old version model in the i-th key sample layer. This result can be multiplied by the degradation direction coefficient of that proxy quality metric and divided by the proxy quality metric of the old version model in the i-th key sample layer to obtain the relative degradation of the i-th key sample layer in relation to that proxy quality metric. For positive metrics (higher metric indicating better performance), the degradation direction coefficient is -1; for negative metrics (lower metric indicating better performance), the degradation direction coefficient is 1.

[0126] It should be understood that each agent quality indicator has a corresponding gating threshold, and the gating thresholds for different agent quality indicators can be the same or different. Optionally, the gating threshold for each agent quality indicator can be set according to actual needs or experience.

[0127] The determination of the online degradation result of the i-th key sample layer based on the relative degradation of each agent quality indicator can include: if the relative degradation of the i-th key sample layer in any core agent quality indicator exceeds the corresponding threshold, then the online degradation result of the i-th key sample layer indicates that the new version model has degraded compared to the old version model; if the relative degradation of the i-th key sample layer in all core agent quality indicators does not exceed the corresponding threshold, then the online degradation result of the i-th key sample layer indicates that the new version model has not degraded compared to the old version model. Alternatively, when the relative degradation of the i-th key sample layer in any core agent quality indicator exceeds the corresponding threshold, a significance test can be performed on the relative degradation to determine whether the relative degradation is statistically significant; if the relative degradation is statistically significant, then the online degradation result of the i-th key sample layer indicates that the new version model has degraded compared to the old version model; otherwise, it is determined that no degradation has occurred, which can reduce the risk of misjudgment caused by small sample noise. It should be understood that the significance level in the significance test is configurable.

[0128] It should be understood that core agent quality indicators can refer to the agent quality indicators selected from all agent quality indicators for degradation assessment. Core agent quality indicators can be all agent quality indicators, or a subset of them, and can be selected based on actual needs.

[0129] In this embodiment, if the relative degradation of the i-th key sample layer in all core agent quality indicators does not exceed the corresponding gating threshold, the relative degradation of the i-th key sample layer in all core agent quality indicators can be weighted and fused to obtain the first comprehensive risk score of the i-th key sample layer. If the first comprehensive risk score of the i-th key sample layer exceeds the first preset score threshold, it is determined that the online degradation result of the i-th key sample layer indicates that the new version model has degraded compared to the old version model. If the first comprehensive risk score of the i-th key sample layer does not exceed the first preset score threshold, it is determined that the online degradation result of the i-th key sample layer indicates that the new version model has not degraded compared to the old version model. Optionally, the first preset score threshold can be set according to actual needs or empirical values.

[0130] Step 603: If there are sample layers in each key sample layer that have online degradation results indicating that the new version model has degraded compared to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped.

[0131] For example, before scaling up, the old version model has 100% traffic; during the scaling up phase (e.g., the target traffic phase is 5%), the new version model has 5% traffic and the old version model has 95% traffic; after degradation is detected, the traffic is redistributed, and after redistribution, the new version model has 0% traffic and the old version model has 100% traffic.

[0132] Stopping this rollout can mean terminating the current launch attempt and not proceeding to the next traffic phase.

[0133] In some embodiments, after the traffic allocated to the new version model is reallocated to the old version model and the current traffic ramping is stopped, the development / operations side can correct the issues of the new version model that triggered the traffic reallocation and related configurations (such as weights, preprocessing, postprocessing, thresholds, resources, etc. that caused the degradation). After the correction, the new version model is warmed up again and the gradual traffic migration is performed again.

[0134] Step 604: If the online degradation results of each key sample layer all indicate that the new version model has not degraded compared to the old version model, then obtain the offline evaluation index of the new version model and the old version model at the i-th key sample layer.

[0135] Offline evaluation metrics can refer to quality metrics that cannot be obtained in real time during the processing of comparable samples and require subsequent processing (such as sampling labeling, shadow traffic, offline playback, etc.).

[0136] Offline evaluation metrics can include at least one of near-real-time evaluation metrics and offline evaluation metrics. For example, offline evaluation metrics can include word error rate, subjective quality score, and intent accuracy.

[0137] Step 605: Based on the offline evaluation metrics of the new version model and the old version model at the i-th key sample layer, make a decision on whether to launch the new version model.

[0138] In this embodiment, by using online degradation results for rapid gating, when degradation is indicated by the online degradation results of any key sample layer, the traffic allocated to the new version model can be immediately redistributed to the old version model and the current rollout can be stopped. This prevents degradation from spreading to a larger number of users in a timely manner. Only when the online degradation results of all key sample layers do not indicate degradation, the offline evaluation indicators of the new and old versions of the model at each key sample layer are further obtained. A second rollout decision is made based on these indicators that are closer to the true value. This two-level decision-making system, which uses online proxy indicators for rapid gating and offline true value indicators for precise verification, leverages the real-time advantage of proxy indicators to achieve second-level degradation response and the high accuracy of offline indicators to reduce the risk of misjudgment. It ensures rollout security while taking into account rollout efficiency, significantly improving the sensitivity and reliability of quality gating during online hot replacement of the model.

[0139] In some embodiments of this application, the degradation result of the i-th critical sample layer includes the offline degradation result; based on the offline evaluation metrics of the new version model and the old version model respectively in the i-th critical sample layer, a decision is made on the online deployment of the new version model, including: Based on the offline evaluation metrics of the new version model and the old version model at the i-th key sample layer, the offline degradation result of the i-th key sample layer is determined. If any of the key sample layers contains offline degradation results indicating that the new version model has degraded compared to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the offline degradation results of each key sample layer indicate that the new version model has not degraded compared to the old version model, then the new version model is controlled to enter the next flow stage of the target flow stage.

[0140] In this embodiment, for each offline evaluation metric, the relative degradation of the i-th key sample layer at the i-th key sample layer can be calculated based on the offline evaluation metrics of the new version model and the old version model at the i-th key sample layer. Based on the relative degradation of the i-th key sample layer at each offline evaluation metric, the offline degradation result of the i-th key sample layer is determined. The relative degradation of the i-th key sample layer at a given offline evaluation metric measures the degree of performance change of the new version model relative to the old version model at that offline evaluation metric.

[0141] Since different offline evaluation metrics may exhibit different degradation directions, a degradation direction metric can be considered when calculating the relative degradation of the i-th key sample layer in relation to offline evaluation metrics. For example, the offline evaluation metric of the new version model in the i-th key sample layer can be subtracted from the offline evaluation metric of the old version model in the i-th key sample layer. This result can be multiplied by the degradation direction coefficient of the offline evaluation metric and divided by the offline evaluation metric of the old version model in the i-th key sample layer to obtain the relative degradation of the i-th key sample layer in relation to that offline evaluation metric. For positive metrics (higher metric indicating better performance), the degradation direction coefficient is -1; for negative metrics (lower metric indicating better performance), the degradation direction coefficient is 1.

[0142] It should be understood that each offline evaluation indicator has a corresponding threshold, and the thresholds for different offline evaluation indicators can be the same or different. Optionally, the threshold for each offline evaluation indicator can be set according to actual needs or experience.

[0143] The determination of the offline degradation result of the i-th key sample layer based on the relative degradation of each offline evaluation index can include: if the relative degradation of the i-th key sample layer in any core offline evaluation index exceeds the corresponding index threshold, then the offline degradation result of the i-th key sample layer indicates that the new version model has degraded compared to the old version model; if the relative degradation of the i-th key sample layer in all core offline evaluation indexes does not exceed the corresponding index threshold, then the offline degradation result of the i-th key sample layer indicates that the new version model has not degraded compared to the old version model. Alternatively, when the relative degradation of the i-th key sample layer in any core offline evaluation index exceeds the corresponding index threshold, a significance test can be performed on the relative degradation to determine whether the relative degradation is statistically significant; if the relative degradation is statistically significant, then the offline degradation result of the i-th key sample layer indicates that the new version model has degraded compared to the old version model; otherwise, it is determined that no degradation has occurred. This can reduce the risk of misjudgment caused by small sample noise.

[0144] It should be understood that core offline assessment metrics can refer to the offline assessment metrics selected from all offline assessment metrics for degradation assessment. Core offline assessment metrics can be all offline assessment metrics, or a subset of them, and can be selected based on actual needs.

[0145] In this embodiment, if the relative degradation of the i-th key sample layer does not exceed the corresponding indicator threshold in all core offline evaluation indicators, the relative degradation of the i-th key sample layer in all core offline evaluation indicators can be weighted and fused to obtain the second comprehensive risk score of the i-th key sample layer. If the second comprehensive risk score of the i-th key sample layer exceeds the second preset score threshold, it is determined that the offline degradation result of the i-th key sample layer indicates that the new version model has degraded compared to the old version model. If the second comprehensive risk score of the i-th key sample layer does not exceed the second preset score threshold, it is determined that the offline degradation result of the i-th key sample layer indicates that the new version model has not degraded compared to the old version model. Optionally, the second preset score threshold can be set according to actual needs or empirical values. The second preset score threshold can be the same as or different from the first preset score threshold.

[0146] In this embodiment, after initially eliminating obvious degradation through online gating, the model performance is verified a second time using offline evaluation metrics. This effectively compensates for the potential risks of insufficient sensitivity or misjudgment by online proxy metrics, ensuring that mass production is only promoted after both online and offline verification confirms that there is no degradation. This maximizes mass production efficiency while ensuring the safety of online deployment, and significantly improves the accuracy and reliability of quality gating during the online hot replacement of the model.

[0147] It should be noted that during the stage of obtaining comparable samples, comparable samples can be obtained based on the principles of being of the same caliber and being able to be paired, so as to ensure that the indicator data (such as agent quality indicators and offline evaluation indicators) of the new version model and the old version model are comparable, and reduce the deviation caused by the difference in traffic composition.

[0148] like Figure 7 The diagram shown is an example of the architecture for progressive traffic switching and degradation processing provided in an embodiment of this application. Figure 7As shown, after the electronic device has completed the preheating and readiness of the new version model, it can gradually migrate traffic from the old version model to the new version model in multiple traffic stages such as 1%→5%→25%→100%. In each traffic stage, session-level consistent routing can be performed, and quality gating judgment can be performed in the low traffic stage to determine whether the new version model has degraded relative to the old version model. If degraded, the traffic allocated to the new version model will be reallocated to the old version model, and the current traffic ramping will be stopped. If no degraded, it will proceed to the next traffic stage until the full traffic ramp is completed.

[0149] In some embodiments, electronic devices can structurally record the version number of the new version model, the version number of the old version model, the start and end times and traffic ratios of each traffic phase, the session switching continuity mode (conservative mode / migration mode) and migration verification results, the quality and performance indicators of each traffic phase and their comparison results with the control group, the statistical comparison conclusions of each traffic phase, the final decision (going live or reallocating traffic), and the trigger indicators and reallocation time in the case of reallocating traffic, to support post-event review and compliance audit, so that the upgrade process is traceable and the decision is auditable.

[0150] In one application scenario, the embodiments of this application can achieve zero-disruption hot replacement of the ASR acoustic model from a smaller specification to a larger specification. The old version of the ASR acoustic model remains online. The new version of the ASR acoustic model is loaded in an independent container, cached, and after several idle inferences until it stabilizes, it is registered to the traffic switch. The warm-up can be completed in a short time. After entering the low-volume phase (e.g., 1%), the quality gating system can collect real-time proxy metrics (e.g., speech recognition confidence, latency quantiles, anomaly rate) available online for the new version of the ASR acoustic model. These metrics are then compared with comparable samples from the old version of the ASR acoustic model carrying the control group's traffic. Simultaneously, word error rates are collected offline using methods such as shadow traffic, offline playback, and sampling annotation. In this embodiment, if the proxy quality metrics and offline word error rate of the new version of the ASR acoustic model are not inferior to those of the old version, and the increase in latency quantiles is within a configurable gating threshold, then the traffic is gradually increased in stages according to the gating decision (e.g., 1% → 5% → 25% → 100%, with configurable dwell time for each stage) until full traffic is achieved. The entire process is seamless, without any system downtime or overall interruption during switching. After deployment, the audit module records the comparison of metrics at each stage and the final decision for subsequent review.

[0151] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0152] Corresponding to the online hot-swap deployment method of the model in the above embodiment, Figure 8 A schematic diagram of the online hot-swap deployment device for models provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0153] Reference Figure 8 The aforementioned device includes: The migration verification module 801 is used to perform a pre-migration feasibility verification on the hidden state of the voice processing of the streaming voice session in the process of migrating traffic from the old version model to the new version model. If the old version model and the new version model satisfy the state compatibility contract, the state compatibility contract refers to the compatibility agreement between the old version model and the new version model regarding the hidden state structure. The state transition module 802 is used to migrate the speech processing hidden state from the old version model to the new version model via a unified intermediate representation if the speech processing hidden state passes the feasibility check before migration. The request routing module 803 is used to route subsequent voice service requests for the streaming voice session to the new version model after the migration is successful.

[0154] In some embodiments, the migration verification module 801 is specifically used for: Based on the state compatibility contract, the expected value of each verification item in the new version model is determined; each verification item includes at least one of the following: state structure version, state tensor dimension, encoder cache time step, and length of speech synthesis attention history key value sequence. If the actual values ​​of each verification item in the latent state of speech processing match the corresponding expected values ​​in the new version model, then the latent state of speech processing is determined to have passed the feasibility verification before migration. If any of the verification items contains a value in the speech processing hidden state that does not match the corresponding expected value in the new version model, then the speech processing hidden state is determined to have failed the feasibility verification before migration.

[0155] In some embodiments, the migration verification module 801 is further configured to: If the hidden state of voice processing fails the feasibility check before migration, then the conservative mode is enabled for the streaming voice session. In conservative mode, the streaming voice session remains in the old model until the streaming voice session ends.

[0156] In some embodiments, the migration verification module 801 is further configured to: The last N speech frames processed by the old version model are input into the new version model to compare the output differences between the old and new version models at the switching point; N is a positive integer, and the switching point is the boundary between the last speech frame processed by the old version model and the first speech frame followed by the new version model. If the output difference is not within the preset difference range, the migration is determined to have failed, and conservative mode is enabled for the streaming voice session. In conservative mode, the streaming voice session remains in the old model until the streaming voice session ends; If the output difference is within the preset difference range, the migration is considered successful.

[0157] In some embodiments, the above-described apparatus further includes: The sample acquisition module is used to acquire comparable samples based on received voice service requests in the target traffic stage when the traffic migration from the old version model to the new version model is performed in multiple incremental traffic stages; the target traffic stage is the traffic stage in which the traffic carried by the new version model is less than the preset traffic. The sample stratification module is used to determine the sample stratum to which each comparable sample belongs, based on the scene type, device type, and language of each comparable sample; each sample stratum includes comparable samples that share at least one of the following: scene type, device type, and language. The online decision module is used to determine the degradation result of the i-th key sample layer based on each comparable sample in the i-th key sample layer, and to make an online decision for the new version model based on the degradation results of all key sample layers. The i-th key sample layer is any key sample layer among all sample layers, and the degradation result of the i-th key sample layer represents whether the new version model has degraded compared to the old version model.

[0158] In some embodiments, the degradation result of the i-th key sample layer includes the online degradation result; the above-mentioned online decision module is specifically used for: Based on the comparable samples in the i-th key sample layer, the agent quality indicators of the new version model and the old version model in the i-th key sample layer are collected in real time. Based on the agent quality indicators of the new version model and the old version model at the i-th key sample layer, the online degradation result of the i-th key sample layer is determined. If any of the key sample layers has online degradation results indicating that the new version model has degraded compared to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the online degradation results of each key sample layer all indicate that the new version model has not degraded compared to the old version model, then obtain the offline evaluation index of the new version model and the old version model in the i-th key sample layer respectively. The decision to deploy the new version model is made based on the offline evaluation metrics of the new and old versions of the model at the i-th key sample layer.

[0159] In some embodiments, the degradation result of the i-th key sample layer includes the offline degradation result; the above-mentioned online decision module is specifically used for: Based on the offline evaluation metrics of the new version model and the old version model at the i-th key sample layer, the offline degradation result of the i-th key sample layer is determined. If any of the key sample layers contains offline degradation results indicating that the new version model has degraded compared to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the offline degradation results of each key sample layer indicate that the new version model has not degraded compared to the old version model, then the new version model is controlled to enter the next flow stage of the target flow stage.

[0160] In some embodiments, the above-described apparatus further includes: The service retention module is used to keep the old version of the model online for voice service; The model processing module is used to load and warm up new versions of models in a standalone runtime environment; The model registration module is used to register the new version model to the traffic switcher after the new version model has been warmed up. The traffic migration module is used by the traffic switcher to gradually increase the proportion of voice service requests routed to the new version model and correspondingly decrease the proportion of voice service requests routed to the old version model in multiple incremental traffic phases, so as to perform traffic migration from the old version model to the new version model. The session routing module is used to route voice service requests in the same traffic phase of a target session to the same version model. The target session includes new sessions or streaming voice sessions that are still being processed in the old or new version model during the traffic migration from the old version model to the new version model in conservative mode. The aforementioned migration verification module 801 is specifically used for: In migration mode, during the process of migrating traffic from the old version model to the new version model, for streaming voice conversations still being processed by the old version model, if the old version model and the new version model satisfy the state compatibility contract, then the feasibility of the voice processing hidden state before migration is verified according to the state compatibility contract.

[0161] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0162] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 9 of this embodiment includes: at least one processor 90 ( Figure 9 (Only one is shown in the diagram), memory 91, and computer program 92 stored in memory 91 and executable on at least one processor 90, wherein processor 90 executes computer program 92 to implement the steps in any of the above method embodiments.

[0163] The electronic device may include, but is not limited to, a processor 90 and a memory 91. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0164] The processor 90 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0165] In some embodiments, the memory 91 may be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. In other embodiments, the memory 91 may be an external storage device of the electronic device 9, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 9. Furthermore, the memory 91 may include both internal and external storage units of the electronic device 9. The memory 91 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0166] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0167] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0168] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0169] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0170] In the embodiments provided in this application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device or electronic device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0171] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0172] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for online hot-swap deployment of a model, characterized in that, include: During the traffic migration process from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then the voice processing hidden state of the streaming voice session is checked for feasibility before migration based on the state compatibility contract. The state compatibility contract refers to the compatibility agreement between the old version model and the new version model regarding the hidden state structure; If the speech processing hidden state passes the pre-migration feasibility check, then the speech processing hidden state is migrated from the old version model to the new version model via a unified intermediate representation. After a successful migration, subsequent voice service requests for the streaming voice session will be routed to the new version model.

2. The online hot-swap deployment method for models according to claim 1, characterized in that, The feasibility check before migrating the hidden state of the streaming voice session based on the state-compatible contract includes: Based on the state compatibility contract, the expected value of each verification item in the new version model is determined; each verification item includes at least one of the following: state structure version, state tensor dimension, encoder cache time step, and length of speech synthesis attention history key value sequence. If the actual values ​​of each verification item in the hidden state of speech processing match the corresponding expected values ​​in the new version model, then the hidden state of speech processing is determined to have passed the pre-migration feasibility verification. If any of the verification items contains a verification item in which the actual value in the hidden state of speech processing does not match the corresponding expected value in the new version model, then it is determined that the hidden state of speech processing has failed the pre-migration feasibility verification.

3. The online hot-swap deployment method for models according to claim 1, characterized in that, After performing the pre-transfer feasibility check on the speech processing hidden state according to the state compatibility contract, the method further includes: If the hidden state of the voice processing fails the feasibility check before migration, then a conservative mode is enabled for the streaming voice session. In the conservative mode, the streaming voice session is maintained in the old version model until the streaming voice session ends.

4. The online hot-swap deployment method for models according to claim 1, characterized in that, After migrating the speech processing hidden state from the old version model to the new version model via a unified intermediate representation, the method further includes: The streaming voice session is input into the new version model after the last N voice frames processed by the old version model, so as to compare the output difference between the old version model and the new version model at the switching point; N is a positive integer, and the switching point refers to the boundary between the last voice frame processed by the old version model and the first voice frame followed by the new version model. If the output difference is not within the preset difference range, the migration is determined to have failed, and a conservative mode is enabled for the streaming voice session. In the conservative mode, the streaming voice session is maintained in the old version model until the streaming voice session ends; If the output difference is within the preset difference range, the migration is considered successful.

5. The online hot-swap deployment method for models according to any one of claims 1 to 4, characterized in that, The method further includes: When performing traffic migration from the old version model to the new version model according to multiple incremental traffic phases, in the target traffic phase, comparable samples are obtained based on the received voice service requests; the target traffic phase is the traffic phase in which the traffic carried by the new version model is less than the preset traffic. The sample layer to which each comparable sample belongs is determined according to the scene type, device type, and language to which each comparable sample belongs; at least one of the scene type, device type, and language to which each comparable sample in each sample layer belongs is the same; For the i-th key sample layer, based on each comparable sample in the i-th key sample layer, the degradation result of the i-th key sample layer is determined, and based on the degradation results of all key sample layers, a decision is made to deploy the new version model; the i-th key sample layer is any key sample layer among all sample layers, and the degradation result of the i-th key sample layer characterizes whether the new version model has degraded relative to the old version model.

6. The online hot-swap deployment method for models according to claim 5, characterized in that, The degradation result of the i-th key sample layer includes the online degradation result; the process of determining the degradation result of the i-th key sample layer based on each comparable sample in the i-th key sample layer, and making a decision on the deployment of the new version model based on the degradation results of all key sample layers, includes: Based on each comparable sample in the i-th key sample layer, the proxy quality indicators of the new version model and the old version model in the i-th key sample layer are collected in real time. Based on the proxy quality indicators of the new version model and the old version model respectively in the i-th key sample layer, the online degradation result of the i-th key sample layer is determined; If any of the key sample layers has an online degradation result indicating that the new version model has degraded relative to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the online degradation results of each key sample layer indicate that the new version model has not degraded relative to the old version model, then the offline evaluation indicators of the new version model and the old version model in the i-th key sample layer are obtained respectively. The decision to launch the new version model is made based on the offline evaluation metrics of the new version model and the old version model at the i-th key sample layer.

7. The online hot-swap deployment method for models according to claim 6, characterized in that, The degradation results of the i-th key sample layer include offline degradation results; the decision to deploy the new version model based on the offline evaluation metrics of the new version model and the old version model respectively in the i-th key sample layer includes: Based on the offline evaluation metrics of the new version model and the old version model respectively in the i-th key sample layer, the offline degradation result of the i-th key sample layer is determined; If any of the key sample layers contains a sample layer with offline degradation results indicating that the new version model has degraded relative to the old version model, then the traffic allocated to the new version model will be reallocated to the old version model, and this traffic increase will be stopped. If the offline degradation results of each key sample layer indicate that the new version model has not degraded relative to the old version model, the new version model is controlled to enter the next traffic stage of the target traffic stage.

8. The online hot-swap deployment method for models according to any one of claims 1 to 4, characterized in that, Before performing the feasibility check on the voice processing hidden state of the streaming voice session according to the state-compatible contract, the method further includes: Keep the old version of the model online voice service; Load and warm up the new version of the model in an independent operating environment; After the new version model has been warmed up, the new version model is registered to the traffic switcher; The traffic switcher gradually increases the proportion of voice service requests routed to the new version model and correspondingly decreases the proportion of voice service requests routed to the old version model according to multiple incremental traffic phases, so as to perform traffic migration from the old version model to the new version model. For a target session, all voice service requests of the target session in the same traffic phase are routed to the same version model; the target session includes a new session, or a streaming voice session that is still being processed in the old version model or the new version model during the traffic migration from the old version model to the new version model in conservative mode. During the traffic migration process from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then a pre-migration feasibility check is performed on the hidden state of voice processing of the streaming voice session based on the state compatibility contract, including: In migration mode, during the process of migrating traffic from the old version model to the new version model, for streaming voice sessions still being processed by the old version model, if the old version model and the new version model satisfy a state compatibility contract, then the voice processing hidden state is checked for feasibility before migration based on the state compatibility contract.

9. A model online hot-swapping deployment device, characterized in that, include: The migration verification module is used to perform a pre-migration feasibility verification of the voice processing hidden state of the streaming voice session in the process of migrating traffic from the old version model to the new version model. If the old version model and the new version model satisfy the state compatibility contract, the module will perform a pre-migration feasibility verification of the streaming voice session's voice processing hidden state in accordance with the state compatibility contract. The state compatibility contract refers to the compatibility agreement between the old version model and the new version model regarding the hidden state structure; The state transition module is used to migrate the speech processing hidden state from the old version model to the new version model via a unified intermediate representation if the speech processing hidden state passes the pre-migration feasibility check. The request routing module is used to route subsequent voice service requests of the streaming voice session to the new version model after the migration is successful.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it causes the electronic device to implement the online hot-swap deployment method for the model as described in any one of claims 1 to 8.

Images