Data processing method, apparatus, and terminal

Abstract

The present disclosure relates to a data processing method and device and terminal, and belongs to the field of artificial intelligence technology. In the method, the terminal runs an AI model in an active state through two inference engine instances, a first version and a second version in an inactive state, records the second version bound session through a binding relationship entry, and determines whether the session has bound the second version through the session identifier of the session in the binding relationship entry for the inference request initiated in the session. If the session has bound the second version, the inference engine instance of the second version is run to process the inference request. Since the inference engine instance holds the complete session context of the session, the inference engine instance can process the inference request based on the complete session context of the session to ensure the accuracy of the inference result, thereby avoiding the inaccurate inference result caused by the missing context basis of the current inference.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to a data processing method, apparatus, and terminal. Background Technology

[0002] With the development of Artificial Intelligence (AI) technology, model management platforms can deploy AI models on terminals. Multiple versions of the same AI model can be deployed on the same terminal. The version that continuously loads and runs, accepts user inference requests, and provides inference services is called the active version of the AI ​​model. The active version of the AI ​​model is in an active state, while the inactive version is in an inactive state. As business needs change, the model management platform can switch the active version of the AI ​​model on the terminal; for example, it can hot-update the currently running active version to the latest version of the AI ​​model.

[0003] End users can establish sessions with AI models. During the duration of a single session, multiple rounds of human-computer interaction are supported. The active version of the inference engine instance running on the terminal will independently maintain the session context of the session and process the inference requests initiated by the user in the session based on the session context.

[0004] The inference engine instance running the active version before the switch is called the old inference engine instance, and the inference engine instance running the active version after the switch is called the new inference engine instance. Currently, when switching the active version of an AI model, the terminal creates a new inference engine instance to run the active version after the switch. When a user initiates a new inference request in a session established before the switch, the new inference engine instance will handle the inference request. However, because the resources of the old and new inference engine instances are completely isolated, the new inference engine instance does not hold the complete session context of the session. Ultimately, due to the lack of contextual basis for this round of inference, the inference result is inaccurate. Summary of the Invention

[0005] This disclosure provides a data processing method, apparatus, and terminal to at least solve the problem in related technologies where inaccurate reasoning results are caused by the lack of contextual basis in the current round of reasoning. The technical solution of this disclosure is as follows.

[0006] According to one aspect of the embodiments of this disclosure, a data processing method is provided, applied to a terminal, including: A first version of the AI ​​model is run through a first inference engine instance, and a second version of the AI ​​model is run through a second inference engine instance, wherein the AI ​​model has a first version in an active state and a second version in an inactive state; Receive a first inference request, the first inference request carrying a session identifier of a first session, the first inference request indicating to perform an inference task based on the AI ​​model; Query the session identifier of the first session in at least one binding relationship entry of the second version, wherein the binding relationship entry includes the version identifier of the second version and the session identifier of any session; If the session identifier of the first session is found in any binding relationship entry of the second version, the first inference request is processed by running the second inference engine instance of the second version.

[0007] Optionally, the method further includes: If the session identifier of the first session is not found in the at least one binding relationship entry, the first inference request is processed through the first inference engine instance.

[0008] Optionally, if the session identifier of the first session is not found in the at least one binding relationship entry, processing the first inference request through the first inference engine instance includes: If the session identifier of the first session is not found in the at least one binding relationship entry and the first session is not bound to the first version, then bind the first session to the first version; The first inference request is processed using the first inference engine instance.

[0009] Optionally, after running the first version of the AI ​​model through the first inference engine instance, the method further includes: Receive a second inference request that does not carry a session identifier, the second inference request instructing an inference task to be performed based on the AI ​​model; The second inference request is processed by running the first inference engine instance.

[0010] Optionally, at least one binding relationship entry of the second version is stored in the memory of the terminal; the method further includes: If any binding relationship entry in the second version meets the deletion condition, delete the binding relationship entry in memory.

[0011] Optionally, the deletion conditions include any of the following: The session indicated by the session identifier in the binding relationship entry is explicitly closed; The waiting time for querying the binding relationship entry has reached the preset time.

[0012] Optionally, the method further includes: The preset duration is adjusted based on the remaining storage capacity of the memory.

[0013] Optionally, the method further includes: Encrypt at least one binding relationship entry in the second version to obtain at least one encrypted binding relationship entry; At least one encrypted binding relationship entry is persistently stored.

[0014] Optionally, after running the first version of the AI ​​model through the first inference engine instance, the method further includes: If the second version is not bound to any session and the second inference engine instance is not processing any stateless requests, unload the second inference engine instance and release the memory space occupied by the second inference engine instance and the second version. The stateless request refers to an inference request that does not carry a session identifier.

[0015] Optionally, before running the first version of the AI ​​model through the first inference engine instance, the method further includes: In response to a version switching command for the AI ​​model, the state of the second version is switched from the active state to the inactive state, and the state of the first version is set to the active state.

[0016] Optionally, the method further includes: Receive a third inference request, which instructs the execution of an inference task based on the AI ​​model; If the state of the first version is switching, the third inference request is processed through the second inference engine instance.

[0017] According to another aspect of the embodiments of this disclosure, a data processing apparatus is provided, comprising: The running unit is configured to execute a first version of the AI ​​model running through a first inference engine instance and a second version of the AI ​​model running through a second inference engine instance, wherein the AI ​​model has a first version in an active state and a second version in an inactive state; The receiving unit is configured to receive a first inference request, the first inference request carrying a session identifier of a first session, the first inference request indicating that an inference task be performed based on the AI ​​model; The query unit is configured to query the session identifier of the first session in at least one binding relationship entry of the second version, the binding relationship entry including the version identifier of the second version and the session identifier of any session; The first processing unit is configured to query the session identifier of the first session in any binding relationship entry of the second version, and process the first inference request by running the second inference engine instance of the second version.

[0018] Optionally, the data processing apparatus further includes: The second processing unit is configured to process the first inference request through the first inference engine instance if the session identifier of the first session is not found in the at least one binding relationship entry.

[0019] Optionally, the second processing unit is configured to execute: If the session identifier of the first session is not found in the at least one binding relationship entry and the first session is not bound to the first version, then bind the first session to the first version; The first inference request is processed using the first inference engine instance.

[0020] Optionally, the data processing device further includes a second processing unit; The receiving unit is further configured to execute a second inference request that does not carry a session identifier, the second inference request indicating that an inference task be performed based on the AI ​​model; The second processing unit is configured to process the second inference request by running the first inference engine instance.

[0021] Optionally, at least one binding relationship entry of the second version is stored in the memory of the terminal; the data processing device further includes: The deletion unit is configured to delete the binding relationship entry in memory if any binding relationship entry of the second version meets the deletion condition.

[0022] Optionally, the deletion conditions include any of the following: The session indicated by the session identifier in the binding relationship entry is explicitly closed; The waiting time for querying the binding relationship entry has reached the preset time.

[0023] Optionally, the data processing apparatus further includes: The adjustment unit is configured to adjust the preset duration based on the remaining storage capacity of the memory.

[0024] Optionally, the data processing apparatus further includes: The encryption unit is configured to perform encryption on at least one binding relationship entry of the second version to obtain at least one encrypted binding relationship entry; The third processing unit is configured to perform persistent storage of at least one of the encrypted binding relationship entries.

[0025] Optionally, the data processing apparatus further includes: The unloading unit is configured to unload the second inference engine instance and release the memory space occupied by the second inference engine instance and the second version if the second version is not bound to any session and the second inference engine instance has no stateless requests being processed. The stateless requests refer to inference requests that do not carry session identifiers.

[0026] Optionally, the data processing apparatus further includes: The switching unit is configured to execute a version switching command in response to the AI ​​model, switching the state of the second version from the active state to the inactive state, and setting the state of the first version to the active state.

[0027] Optionally, the receiving unit is further configured to receive a third inference request, the third inference request indicating that an inference task be performed based on the AI ​​model; The first processing unit is configured to process the third inference request through the second inference engine instance if the state of the first version is switching.

[0028] According to another aspect of the embodiments of this disclosure, a terminal is provided, comprising: One or more processors; One or more memories for storing the one or more processor-executable instructions; The one or more processors are configured to perform a data processing method in any of the possible implementations of the above-described aspects.

[0029] According to another aspect of the present disclosure, a computer-readable storage medium is provided that, when at least one instruction in the computer-readable storage medium is executed by one or more processors of a terminal, enables the terminal to perform a data processing method in any possible implementation of the above aspect.

[0030] According to another aspect of the present disclosure, a computer program product is provided, including one or more instructions that can be executed by one or more processors of a terminal, enabling the terminal to perform a data processing method in any possible implementation of the above aspect.

[0031] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects: The terminal runs the AI ​​model in parallel using two inference engine instances: a first version in an active state and a second version in an inactive state. It records the sessions bound to the second version through binding relationship entries. For inference requests initiated within a session, it queries the session identifier in the binding relationship entries to determine if the session is already bound to the second version. If the session is bound to the second version, the inference request is processed by running the second version's inference engine instance. Since the inference engine instance holds the complete session context, it can process the inference request based on this context, ensuring accurate inference results and avoiding inaccurate results due to a lack of contextual basis in the current inference round.

[0032] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0033] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0034] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method according to an exemplary embodiment; Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment; Figure 3 This is a flowchart illustrating another data processing method according to an exemplary embodiment; Figure 4 This is a flowchart illustrating the allocation of a reasoning request according to an exemplary embodiment; Figure 5 This is a schematic diagram illustrating an unloading mechanism for an inference engine instance according to an exemplary embodiment; Figure 6 This is a schematic diagram illustrating the restoration of a binding relationship entry according to an exemplary embodiment; Figure 7 This is a logical structure block diagram of a data processing apparatus according to an exemplary embodiment; Figure 8 This is a structural block diagram of a terminal according to an exemplary embodiment. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0036] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] The information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this disclosure are all authorized by the user or fully authorized by the parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions. For example, the content carried by the inference requests involved in this disclosure was obtained under full authorization.

[0038] The present disclosure relates to a response to conditions or states on which the performed operation depends. When the conditions or states on which the operation depends are met, the one or more operations performed may be performed in real time or may have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0039] In some embodiments, the meaning of A and / or B includes three cases: A and B, and A and B.

[0040] Figure 1 This is a schematic diagram illustrating the implementation environment of a data processing method according to an exemplary embodiment. See also: Figure 1 The implementation environment includes at least one terminal 101 and a server 102. The terminal 101 is directly or indirectly connected to the server 102 through a wired network or a wireless network. The connection method between the terminal 101 and the server 102 is not limited in this embodiment of the disclosure.

[0041] Terminal 101 is a user equipment, which may be at least one of a smartphone, tablet computer, e-book reader, music player, personal computer, laptop computer, desktop computer, extended reality (XR) device, and wearable device. This disclosure does not limit the device type of terminal 101. Figure 1 Three terminals 101 are shown, but the number of terminals 101 is not limited to three. It can be less than three or more than three. Here, the present disclosure embodiment does not limit the number of terminals 101 in the implementation environment.

[0042] Server 102 runs a model management platform. The model management platform takes AI model assets as its core and provides integrated capabilities for model registration, version control, distribution and deployment, version switching, operation observation and permission auditing, realizing a standardized management platform for models from R&D to production and iterative updates.

[0043] The model management platform can deploy AI models on terminal 101. Multiple versions of the same AI model can be deployed on the same terminal 101; the higher the version of the AI ​​model, the better its performance. Among the multiple versions of the AI ​​model, the version that continuously loads and runs, accepts user inference requests, and provides inference services is called the active version of the AI ​​model. Versions other than the active version are called inactive versions. The active version of the AI ​​model is in an active state, and the inactive version is in an inactive state. In other words, when any version of the AI ​​model is in an active state, that version is considered the active version of the AI ​​model; when any version of the AI ​​model is in an inactive state, that version is considered the inactive version of the AI ​​model.

[0044] As business needs evolve, model performance is continuously optimized, and potential defects are fixed, the model management platform can switch the active version of the AI ​​model in terminal 101. Taking an AI model version update as an example, if a new version of the AI ​​model adds new features, fixes defects from previous versions, or optimizes display effects compared to the active version in terminal 101, the model management platform can issue a version update command to terminal 101 for that AI model. Terminal 101 responds to the update command by switching the active version of the AI ​​model to the new version. Taking an AI model version rollback as another example, assuming the latest version of the AI ​​model is the current active version, if the latest version of the AI ​​model experiences at least one problem such as malfunction, performance degradation, slow speed, excessive memory usage, poor interface compatibility, security vulnerabilities, or unsatisfactory online performance, the model management platform can issue a version rollback command to terminal 101 for that AI model. Terminal 101 responds to the rollback command by switching the active version of the AI ​​model from the latest version to a previous version.

[0045] In Terminal 101, different versions of the AI ​​model have their own dedicated memory space. That is, different versions of the AI ​​model have different memory spaces and run through different inference engine instances. Each inference engine instance is allocated independent running resources, including memory resources and computing resources.

[0046] Whenever any version of an AI model becomes active, terminal 101 can create a new inference engine instance for that version and run the active version of the AI ​​model in the newly created inference engine instance. When any version of the AI ​​model switches from active to inactive, terminal 101 unloads the inference engine instance running that version of the AI ​​model and releases the runtime resources occupied by the inference engine instance and the memory space occupied by the AI ​​model running by the inference engine instance.

[0047] Users of terminal 101 can establish a conversation with the AI ​​model in terminal 101. The conversation can be a single-turn conversation or a multi-turn conversation. A single-turn conversation consists of only one round of human-computer interaction, while a multi-turn conversation can consist of multiple rounds of human-computer interaction. Each round of human-computer interaction is a dialogue between the user and the AI ​​model. In one round of dialogue, the user can initiate an inference request to the AI ​​model. Terminal 101 processes the inference request by running an instance of the inference engine of a certain version of the AI ​​model, obtains the inference result, and returns the inference result to the user.

[0048] In the case of a multi-turn session, terminal 101 binds the session to the active version of the AI ​​model. The inference engine instance running this active version of the AI ​​model independently maintains the session context. Whenever a user initiates any inference request within the multi-turn session, terminal 101 processes the request through the inference engine instance. When processing the inference request, the inference engine instance uses the current session context to ensure accurate inference results. After processing the request and obtaining the inference result, the inference engine instance updates the current session context based on the result, thus obtaining the latest session context. When processing the next inference request within the session, the inference engine instance uses the latest session context, and so on, until the session is closed. The session context is a data set used to maintain the contextual continuity of the same session across multiple turns. For example, the session context is an intermediate data set generated by the inference engine instance during the processing of inference requests in a multi-turn session. The session context includes at least one of semantic context and inference runtime cache context, with the semantic context including the user embedding vector. The inference runtime cache context includes key-value cache (KV Cache) data, which is intermediate feature data generated during multiple rounds of inference by the AI ​​model.

[0049] Session context, also known as session state, is defined as follows: multi-turn sessions have session state, while single-turn sessions do not. Inference requests initiated within a multi-turn session are called stateful requests, while those initiated within a single-turn session are called stateless requests. The inference result of a stateful request depends on the session state of the multi-turn session, while the inference result of a stateless request does not. Multi-turn sessions have a session identifier that indicates the multi-turn session, while single-turn sessions do not. Stateful requests initiated within a multi-turn session carry the session identifier, while stateful requests initiated within a single-turn session do not. Therefore, by checking whether a inference request carries a session identifier, it can be determined whether the request is stateful or stateless. That is, a inference request carrying a session identifier is a stateful request, initiated within the session indicated by that identifier; a inference request not carrying a session identifier is a stateless request, initiated within a separate session.

[0050] As described above, when the AI ​​model is updated or rolled back, the terminal 101 switches the active version of the AI ​​model. For ease of description, the active version of the AI ​​model before the switch is called the second version, and the active version of the AI ​​model after the switch is called the first version. The inference engine instance running the second version of the AI ​​model is called the second inference engine instance (i.e., the old inference engine instance), and the inference engine instance running the first version of the AI ​​model is called the first inference engine instance (i.e., the new inference engine instance).

[0051] Before the active version of the AI ​​model switches from version two to version one, version two of the AI ​​model is bound to at least one session. When the active version of the AI ​​model switches from version two to version one, terminal 101 does not need to migrate the session context of each session bound to version two from the second inference engine instance to the first inference engine instance, avoiding the overhead of migrating session context, thus achieving a lossless version update of the AI ​​model. Since the second engine instance holds the session context of at least one session bound to version two, after the active version of the AI ​​model switches from version two to version one, if a user initiates an inference request in any session bound to version two, terminal 101 does not process the inference request through the first inference engine instance, but through the second inference engine instance. When processing the inference request, the second inference engine instance can process the inference request based on the session context of that session to ensure the accuracy of the inference result. After obtaining the inference result corresponding to the inference request, the second inference engine instance can also update the session context of the session based on the inference result. Thus, during the lifecycle of the session, the second inference engine instance can independently maintain the session context of the session, thereby ensuring the consistency and integrity of the session context, avoiding the loss of the session context, and thus avoiding the problem of inaccurate inference results due to the loss of the session context.

[0052] Server 102 provides backend services for the AI ​​model running on terminal 101. Server 102 includes at least one of a single server, multiple servers, a cloud computing platform, or a virtualization center. Optionally, server 102 undertakes the primary computing task, and terminal 101 undertakes the secondary computing task; or, server 102 undertakes the secondary computing task, and terminal 101 undertakes the primary computing task; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0053] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment, see [link to flowchart]. Figure 2 The data processing method is applied to a terminal, which may be the aforementioned terminal 101. The data processing method includes the following steps 201 to 204.

[0054] In step 201, the terminal runs a first version of the AI ​​model through a first inference engine instance and a second version of the AI ​​model through a second inference engine instance. The AI ​​model has a first version in an active state and a second version in an inactive state.

[0055] The AI ​​model can be any AI model deployed on the terminal by the model management platform. The AI ​​model includes, but is not limited to, Large Language Model (LLM), text-to-text multimodal dialogue model, voice dialogue large model, task-oriented dialogue model, special vertical interaction model or visual interaction continuous reasoning model, etc. Here, the type of AI model is not limited in this embodiment.

[0056] The terminal runs a target application, which is a conversational AI application that provides the ability to converse with an AI model. The target application has an AI conversational interface that the terminal can display. Users can initiate a conversation with the AI ​​model within this interface, performing at least one round of dialogue. The terminal processes the inference requests initiated by the user in each round of dialogue by running an instance of the AI ​​model's inference engine, and provides the corresponding inference results to the user, such as displaying the inference results on the AI ​​conversational interface.

[0057] The target application can be a conversational AI application running on a desktop using the Electron framework, or a web application running in a browser. This embodiment does not limit the type of target application, as long as it is installed and running on a terminal and can provide conversational inference services through the AI ​​model on the terminal. The model management platform provides background services for the target application.

[0058] The terminal stores multiple versions of the AI ​​model, including a first version and a second version. Before step 201, the second version is the active version of the AI ​​model and is in an active state. After the terminal switches the active version of the AI ​​model from the second version to the first version, the second version becomes the inactive version of the AI ​​model and is in an inactive state, while the first version becomes the active version of the AI ​​model and is in an active state. Therefore, the first version is the active version of the AI ​​model after the switch, and the second version is the active version of the AI ​​model before the switch. Step 201 is the step after switching the active version of the AI ​​model from the second version to the first version. Therefore, in step 201, the first version is in an active state, and the second version is in an inactive state; that is, the first version of the AI ​​model is the active version, and the second version is the inactive version.

[0059] Before switching the active version of the AI ​​model from version two to version one, the terminal runs the second version of the AI ​​model through the second inference engine instance. When switching the active version of the AI ​​model from version two to version one, or before switching, the terminal creates an inference engine instance for version one; this instance is referred to as the first inference engine instance. After switching the active version of the AI ​​model from version two to version one, the terminal runs the first version of the AI ​​model through the first inference engine instance and continues to run the second version of the AI ​​model through the second inference engine instance. Therefore, in step 201, the first and second versions are run through different inference engine instances. For example, the terminal runs the first version of the AI ​​model through the first inference engine instance and the second version of the AI ​​model through the second inference engine instance. At this time, the AI ​​model is in a dual-version coexistence state on the terminal. In this dual-version coexistence state, the terminal can provide inference services to users through both versions of the AI ​​model. For example, the first inference engine instance processes inference requests initiated in sessions bound to the first version, while the second inference engine instance processes inference requests initiated in sessions not bound to the first version.

[0060] In step 202, the terminal receives a first inference request, which carries a session identifier of a first session and indicates that an inference task will be performed based on the AI ​​model.

[0061] The first inference request is received after the active version of the AI ​​model is switched from version two to version one. The first inference request is any inference request initiated in the first session, which is any multi-turn session initiated by the user. Therefore, the first inference request carries the session identifier of the first session. Because it carries the session identifier, the first inference request is a stateful request, and the inference result corresponding to the first inference request depends on the session context of the first session.

[0062] The first session may have been bound to the second version of the AI ​​model, or it may have been bound to the first version of the AI ​​model, or it may not have been bound to either the second or the first version.

[0063] When the first session is bound to the second version, the second inference engine instance independently maintains the session context of the first session, and the first inference engine instance holds the session context of the first session. Therefore, the terminal sends the first inference request to the second inference engine instance, and the second inference engine instance processes the first inference request, as shown in step 204 below.

[0064] When the first session is bound to the first version, the first inference engine instance independently maintains the session context of the first session, and the first inference engine instance holds the session context of the first session. Therefore, the terminal sends the first inference request to the first inference engine instance, and the first inference request is processed by the first inference engine instance.

[0065] If the first session is not bound to either the second or first version, it is considered a new session. Once all sessions bound to the second version are closed (either explicitly or implicitly), the terminal no longer uses the second version of the AI ​​model to provide inference services to the user. It unloads the second inference engine instance and continues to provide inference services to the user using the first version of the AI ​​model. Therefore, when the first session is a new session, the terminal binds the first session to the first version of the AI ​​model, sends the first inference request to the first inference engine instance, and the first inference engine instance processes the first inference request. This allows the first inference engine instance to not only maintain the session context of the first session but also process the inference request based on that session context when handling inference requests initiated in the first session.

[0066] The terminal can first check whether the first session has been bound to the second version. If the first session has been bound to the second version, the terminal processes the first inference request by running the second inference engine instance of the second version. For example, the process shown in steps 203 and 204 below.

[0067] In step 203, the terminal queries the session identifier of the first session in at least one binding relationship entry of the second version. The binding relationship entry includes the version identifier of the second version and the session identifier of any session.

[0068] In some embodiments, the terminal stores a session binding relationship set, which records sessions bound to a first version and a second version. Before switching the active version of the AI ​​model from the second version to the first version, the session binding relationship set includes at least one binding relationship entry for the second version. Each binding relationship entry for the second version indicates a session bound to the second version, and the binding relationship entry includes a version identifier (ID) of the second version and a session ID for any session. Optionally, the binding relationship entry also includes an instance ID of the second inference engine instance to indicate that the AI ​​model of the second version is running in the second inference engine. Optionally, the binding relationship entry also includes either the creation time of the binding relationship entry or the last access time, wherein the last access time can also be understood as the last query time of the binding relationship entry, which is the time when the terminal last queried the binding relationship entry (i.e., the time when the terminal most recently queried the binding relationship entry).

[0069] Taking the version ID of the second version as P_A, the second version as bound to sessions S1 and S2, and the instance ID of the second inference engine instance as M_A as an example, the session binding relationship set is shown in Table 1 below. The second and third rows of Table 1 are respectively a binding relationship entry of the second version.

[0070] Table 1

[0071] Table 1 illustrates an example of the second version binding two sessions. In some embodiments, the second version can bind one session or more than two sessions. In this embodiment, the number of sessions bound to the second version is not limited. The creation time and last access time of each binding relationship entry in Table 1 are examples, and the creation time and last access time of the binding relationship entry may be other times.

[0072] The system checks whether a first session is bound to the second version by querying the session identifier of the first session in at least one binding relationship entry in the second version. For example, the terminal obtains the session identifier of the first session from a first inference request. The terminal then queries the session identifier of the first session in at least one binding relationship entry in the second version. If the session identifier of the first session is found in any binding relationship entry in the second version, it indicates that the first session is bound to the second version. If the session identifier of the first session is not found in any binding relationship entry in the second version, it indicates that the first session is not bound to the second version. The at least one binding relationship entry in the second version includes all binding relationship entries in the second version of the session binding relationship set.

[0073] The above-mentioned query of binding relationship entries in the session binding relationship set can be completed in microseconds. Therefore, by querying the binding relationship entries of the second version, it is possible to quickly determine whether the first session has been bound to the second version. This allows the inference engine instance used to process the first inference request to be identified from the first and second inference engine instances. Consequently, the first inference request can be quickly routed to that inference engine instance for processing, thereby improving the processing efficiency of the inference request.

[0074] In step 204, if the session identifier of the first session is found in any binding relationship entry of the second version, the terminal processes the first inference request by running the second inference engine instance of the second version.

[0075] In this embodiment of the disclosure, the terminal allocates independent memory space for each inference engine instance from its own memory. The memory space allocated to each inference engine instance is referred to as the memory space of each inference engine instance. Each inference engine instance can store the session context of its own maintained sessions, as well as its configuration information and execution status, in its own memory space. It should be understood that the memory spaces of different inference engine instances are different, and each inference engine instance has access to its own memory space but not access to the memory spaces of other inference engine instances.

[0076] For ease of description, the memory space of the first inference engine instance is referred to as the first memory space, and the memory space of the second inference engine instance is referred to as the second memory space. The first memory space stores the configuration information, execution status, and session contexts of the sessions bound to the first version of the AI ​​model for the first inference engine instance. The second memory space stores the configuration information, execution status, and session contexts of the sessions bound to the second version of the AI ​​model for the second inference engine instance.

[0077] In step 204, if the session identifier of the first session is found in any binding relationship entry of the second version, the terminal sends a first inference request to the second inference engine instance. The second inference engine instance receives the first inference request, retrieves the session context of the first session from the second memory space based on the session identifier of the first session in the first inference request, processes the first inference request based on the session context, obtains the inference result corresponding to the first inference request, and sends the inference result corresponding to the first inference request to the terminal. The second inference engine instance also updates the session context of the first session in the second memory space based on the inference result corresponding to the first inference request to ensure the integrity and consistency of the session context of the first session. This ensures that the context continuity of existing sessions can reach 100% during switching, without the user's awareness, achieving lossless update of the model version.

[0078] The terminal receives the inference result corresponding to the first inference request and provides the inference result to the user. Because the session context of the first session is taken into account when processing the first inference request, the accuracy of the inference result can be ensured, thereby avoiding inaccurate inference results caused by the lack of contextual basis in this round of inference.

[0079] It should be understood that if the session identifier of the first session is found in any binding relationship entry of the second version, it means that the first session has been bound to the second version. This step 203 is that if the first session has been bound to the second version, the first inference request is processed by running the second inference engine instance of the second version.

[0080] In the method provided in this embodiment, when the first version of the AI ​​model is active and the second version of the AI ​​model is inactive, the terminal runs the active first version and the inactive second version of the AI ​​model in parallel through two inference engine instances. For example, the first version of the AI ​​model is run through the first inference engine instance, and the second version of the AI ​​model is run through the second inference engine instance. The terminal records the sessions bound to the second version through binding relationship entries. For an inference request initiated in a session, the terminal determines whether the session is already bound to the second version by querying the session identifier in the binding relationship entry. If the session is already bound to the inactive second version (i.e., the second version before the switch), the inference request is processed by the second inference engine instance (i.e., the inference engine instance running the second version). Since the second inference engine instance holds the complete session context of the session, it can process the inference request based on the complete session context to ensure the accuracy of the inference result, thereby avoiding inaccurate inference results due to the lack of contextual basis in this round of inference.

[0081] Below, through Figure 3 The data processing method provided in the embodiments of this disclosure will be described in detail.

[0082] Figure 3 This is a flowchart illustrating another data processing method according to an exemplary embodiment, see [link to flowchart]. Figure 3 The data processing method is applied to a terminal, which may be the aforementioned terminal 101. The data processing method includes the following steps 301 to 307.

[0083] In step 301, in response to the version switching command for the AI ​​model in the terminal, the terminal switches the state of the second version of the AI ​​model from active to inactive, and sets the state of the first version of the AI ​​model to active.

[0084] The AI ​​model has multiple versions, one of which is the latest version. All other versions are historical versions of the AI ​​model. The second version is the active version of the AI ​​model before the version switch command was received; prior to step 301, the second version was active.

[0085] The version switching command originates from the model management platform. It instructs the active version of the AI ​​model to switch to the first version. The command includes the version identifier of the first version and the storage address of the corresponding model package. The version identifier of any version of the AI ​​model indicates that version. The first version could be a historical version or the latest version of the AI ​​model. For example, in a version rollback scenario, the version switching command is a version rollback command, the second version is the latest version of the AI ​​model, and the first version is a historical version. In this case, the first version could be the version preceding the latest version of the AI ​​model, or a historical version other than the version preceding the latest version. In a version update scenario, the version switching command is a version update command, the first version is the latest version of the AI ​​model, and the second version is a historical version of the AI ​​model. In this case, the second version is the version preceding the first version.

[0086] Before receiving the version switching command, the second version is the active version of the AI ​​model. The terminal runs the second version of the AI ​​model through the second inference engine instance. The second version is bound to at least one session, and the at least one session is a multi-turn session.

[0087] In some embodiments, the terminal stores a session binding relationship set, which includes at least one binding relationship entry of the second version. Upon receiving a version switching command, the terminal responds by loading the first version of the AI ​​model, switching the active version of the AI ​​model from the second version to the first version, thereby switching the state of the second version of the AI ​​model from an active state to an inactive state, and setting the state of the first version of the AI ​​model to an active state.

[0088] In this context, the terminal can load the first version of the AI ​​model in response to a version switching command in the following ways: The model management platform pushes a version switching command to the terminal. Upon receiving the command, the terminal obtains the storage address of the model package corresponding to the first version from the command and loads the model package from that address. This model package may be a differential package or a complete package of the first version. The differential package includes the differences between the first and second versions, while the complete package includes the AI ​​model of the first version. After loading the model package, the terminal performs an integrity check on the package, such as using SHA-256 verification or digital signature verification. This embodiment does not limit the integrity check method used.

[0089] If the model package corresponding to the first version passes the integrity verification, the terminal obtains the first version of the AI ​​model based on the model package. The terminal loads the first version of the AI ​​model into a separate memory area. For example, the terminal allocates a third memory space from memory for the first version of the AI ​​model and stores the model parameters (such as weight tensors) of the first version of the AI ​​model into the third memory space. The terminal also creates a first inference engine instance for the first version of the AI ​​model and allocates a first memory space from memory for the first inference engine instance so that the first inference engine instance can subsequently store its configuration information, execution status, and the session context of each session bound to the first version in the first memory space. After the terminal loads the first version of the AI ​​model into the separate memory area, the terminal preloads the first version of the AI ​​model and marks the status of the first version of the AI ​​model as READY.

[0090] The step of verifying the integrity of the model package corresponding to the first version is optional. In some embodiments, the terminal may not verify the integrity of the model package corresponding to the first version. After obtaining the model package corresponding to the first version, it can directly obtain the AI ​​model of the first version based on the model package corresponding to the first version.

[0091] In some embodiments, the terminal may also load the model package corresponding to the first version from the storage address based on a loading scheduling strategy. For example, the terminal may load the model package corresponding to the first version from the storage address based on at least one of the terminal's battery level and network conditions. For instance, when the terminal's battery level is lower than a preset battery level (e.g., 20% of the terminal's total battery), loading the model package corresponding to the first version is temporarily suspended; when the terminal's battery level is equal to or exceeds the preset battery level, the model package corresponding to the first version is loaded. When the terminal's network condition (e.g., a Wi-Fi environment) is poor, the download priority of the model package corresponding to the first version is reduced, and loading the model package corresponding to the first version is temporarily suspended; when the terminal's network condition (e.g., a Wi-Fi environment) is good, the download priority of the model package corresponding to the first version is increased, and the model package corresponding to the first version is loaded, to avoid affecting the terminal's foreground services. The preset battery level of 20% of the terminal's total battery is merely one example of a preset battery level; the preset battery level can be set according to specific implementation scenarios. This disclosure does not limit the preset battery level.

[0092] In some embodiments, after receiving a version switching command, the terminal can also preload a second version of the AI ​​model. Taking a version update scenario as an example, after obtaining the first version (new version) of the AI ​​model, the model management platform sends a version update notification message to the terminal. This message notifies the terminal that the AI ​​model has a new version, which is the first version. The notification message includes the version identifier of the first version and the storage address of the model package corresponding to that version. Upon receiving the notification message, the terminal can load the model package corresponding to the first version from the storage address as described above, and obtain the first version's AI model based on this package, thus completing the preloading of the first version. Afterward, the model management platform sends a version switching command to the terminal. This command includes the identifier of the first version, indicating that the active version of the AI ​​model should be switched to the first version. The terminal receives the version switching command and, based on it, switches the active version of the AI ​​model from the second version to the first version. In this case, the version switching command may not include the storage address of the model package corresponding to the first version.

[0093] The above-mentioned switching of the active version of the AI ​​model from the second version to the first version includes: hot-updating the active version of the AI ​​model to the second version, so that while the target application is providing inference services through the first version, the terminal can also switch the active version of the AI ​​model from the second version to the first version without affecting the inference service.

[0094] In some embodiments, the terminal uses an atomic switching identifier to indicate the active version of the AI ​​model. For example, the atomic switching identifier may contain information related to the active version, including the storage address of the active version of the AI ​​model or the version ID of the active version. Switching the active version of the AI ​​model from the second version to the first version involves the terminal updating the atomic switching identifier from information related to the second version to information related to the first version. Since the updated atomic switching identifier indicates the first version as the active version, not the second version, the first version switches from an inactive state to an active state, while the second version switches from an active state to an inactive state.

[0095] In some embodiments, prior to step 301, the terminal establishes a first mapping relationship between the active state and the version identifier of the second version of the AI ​​model, wherein the first mapping relationship indicates that the second version of the AI ​​model is in an active state. Prior to step 301, the version identifier of the active version in the first mapping relationship is the version identifier of the second version. Switching the active version of the AI ​​model from the second version to the first version includes: updating the version identifier of the second version in the first mapping relationship to the version identifier of the first version, such that the updated first mapping relationship indicates that the first version is in an active state; and establishing a second mapping relationship between the version identifier of the second version and the inactive state, wherein the second mapping relationship indicates that the second version is in an inactive state.

[0096] After switching the active version of the AI ​​model from version 2 to version 1, the terminal will not uninstall the second inference engine instance. Once all sessions bound to version 2 are closed (including explicit or implicit closure), the terminal will no longer use the second version of the AI ​​model to provide inference services to users, and will then uninstall the second inference engine instance.

[0097] After switching the active version of the AI ​​model from version 2 to version 1, the terminal runs the first version of the AI ​​model through the first inference engine instance so that it can subsequently provide inference services to users through the first version of the AI ​​model.

[0098] In some embodiments, before step 301 or during the process of switching the active version of the AI ​​model from the second version to the first version, the terminal receives a third inference request, which instructs the execution of an inference task based on the AI ​​model. Upon receiving the third inference request, the terminal determines whether the state of the first version is currently in the process of switching (i.e., switching the active version of the AI ​​model from the second version to the first version). If the state of the first version is in the process of switching, the third inference request is processed through the second inference engine instance. If the state of the first version has already switched to the active state (i.e., the active version of the AI ​​model has already been switched from the second version to the first version), the terminal selects one inference engine instance from the first and second inference engine instances to process the third inference request.

[0099] For example, the terminal sets a critical window and a switching status flag. The critical window is the time window for switching the active version of the AI ​​model from version two to version one. The critical window is a protective period set to force all new sessions to switch to the old model during the nanosecond-level instant of active version switching. The duration of the critical window can be in the range of nanoseconds to microseconds. For example, the duration of the critical window is not limited to 100 nanoseconds. Here, the embodiments of this disclosure do not limit the duration of the critical window. The switching status flag is used to indicate the state of the critical window. Taking the switching status flag as an atomic boolean type flag as an example, assuming the switching status flag is is_switching, when is_switching is true, it indicates that the critical window is in the open state, indicating that the active version switching is currently in progress; when is_switching is false, it indicates that the critical window is in the closed state, indicating that the active version switching has been completed.

[0100] Based on this, once the first version of the AI ​​model is loaded, the terminal sets the switching status flag to true to open the critical window. Then, it updates the atomic switching flag from the information of the second version to the information of the first version. After the update, it sets the switching status flag to false to close the critical window. Upon receiving a third inference request, the terminal queries the switching status flag. If the switching status flag is true, it indicates that the first version is currently in the process of switching, and the terminal processes the third inference request through the second inference engine instance. If the switching status flag is false, it indicates that the first version has already switched to an active state, and the terminal selects one inference engine instance from the first and second inference engine instances to process the third inference request. This ensures that inference requests arriving at the critical window are uniformly bound to the old version (i.e., the second version) and uniformly processed by the old inference engine instance, avoiding consistency disruption caused by some new sessions being incorrectly bound to the switched version due to race conditions. The race condition refers to different threads reading different atomic switching flags at the same instant of switching. The critical window eliminates this timing uncertainty by forcing all new sessions to use the old model during this period.

[0101] The process of selecting a reasoning engine instance from the first and second reasoning engine instances to process the third reasoning request is the same as the process of selecting a reasoning engine instance from the first and second reasoning engine instances to process the first reasoning request, and will not be repeated here.

[0102] If the terminal processes a third inference request through the second inference engine instance, before processing the third inference request again through the second inference engine instance, the terminal determines whether the third inference request carries a session identifier. If the third inference request does not carry a session identifier, the terminal processes the third inference request through the second inference engine instance. If the third inference request carries a session identifier, the session indicated by the session identifier carried by the third inference request is called the third session, and the terminal queries whether the third session has been bound to the second version. If the third session is not bound to the second version, it means that the third session is a new session, and the terminal binds the third session to the second version and processes the third inference request through the second inference engine instance. If the third session is already bound to the second version, it means that the third session was established before the active version switch, and the terminal processes the third inference request through the second inference engine instance.

[0103] The process of a terminal querying whether a third session is already bound to the second version includes: the terminal querying the session identifier of the third session in at least one binding relationship entry of the second version. If the session identifier of the third session is found in any binding relationship entry of the second version, it indicates that the third session is already bound to the second version. If the session identifier of the third session is not found in any binding relationship entry of the second version, it indicates that the third session is not bound to the second version. Therefore, if the session identifier of the third session is found in any binding relationship entry of the second version, the terminal processes the third inference request through the second inference engine instance; if the session identifier of the third session is not found in any binding relationship entry of the second version, the terminal binds the third session to the second version and processes the third inference request through the second inference engine instance. The at least one binding relationship entry of the second version includes all binding relationship entries of the second version in the session binding relationship set.

[0104] The process of binding the third session with the second version includes: adding a second binding relationship entry to the session binding relationship set. The second binding relationship entry includes the version identifier of the second version and the session identifier of the third session. Optionally, the second binding relationship entry also includes any one of the instance identifier of the second inference engine instance, the creation time of the second binding relationship entry, and the last access time. In this case, the last access time of the second binding relationship entry can be the same as the creation time of the second binding relationship entry. Subsequently, whenever an inference request carrying the session identifier of the third session is received, the terminal queries the session binding relationship set for the identifier of the third session. This allows the terminal to find the identifier of the third session in the second binding relationship entry, thereby determining that the third session is bound to the second version, and then sending the inference request to the second inference engine instance for processing. Each time the identifier of the third session is found in the second binding relationship entry, the terminal also modifies the last access time in the second binding relationship entry to the time of this query.

[0105] It should be understood that the first inference engine instance is the new inference engine instance, and the second inference engine instance is the old inference engine instance. In the absence of a fully updated active version of the AI ​​model, using the old inference engine instance to handle new inference requests (such as the third inference request) avoids the new inference request waiting to be created and processed by the new inference engine instance, reducing the waiting time for new inference requests and improving the processing efficiency of inference requests during the active version switch.

[0106] In this embodiment of the disclosure, the process by which the terminal processes any inference request through any inference engine instance is similar to the process in step 204 above where the terminal processes the first inference request by running the second version of the second inference engine instance.

[0107] In step 302, the terminal runs the first version of the AI ​​model through the first inference engine instance and the second version of the AI ​​model through the second inference engine instance.

[0108] After switching the active version of the AI ​​model from version 2 to version 1, the terminal runs the version 1 AI model through the first inference engine instance. If a session is bound to version 2 or an inference request is being processed in the second inference engine instance, the terminal does not uninstall the second inference engine instance; therefore, the terminal continues to run the version 2 AI model through the second inference engine instance.

[0109] In step 303, the terminal receives an inference request, which instructs the execution of an inference task based on the AI ​​model.

[0110] The inference request is any inference request received after the active version of the AI ​​model is switched from version 2 to version 1. The inference request may or may not carry a session identifier.

[0111] When the inference request carries a session identifier, the session indicated by the session identifier is called the first session, and the inference request is called the first inference request. This step 303 is that the terminal receives the first inference request, and the first inference request carries the session identifier of the first session.

[0112] If the inference request does not carry a session identifier, the inference request is called a second inference request. This step 303 is that the terminal receives the second inference request without carrying a session identifier.

[0113] In step 304, if the inference request is a second inference request without a session identifier, the terminal processes the second inference request through the first inference engine instance.

[0114] In the case where the inference request is a second inference request without carrying a session identifier, the terminal reads the atomic switch identifier. At this time, the atomic switch identifier contains information related to the first version. Therefore, based on the atomic switch identifier, the terminal processes the second inference request through the first inference engine instance.

[0115] It should be understood that any inference request in this disclosure indicates the execution of an inference task based on the AI ​​model. For any inference request received by the terminal, if the inference request does not carry a session identifier, it indicates that the inference request is a stateless request, and the inference result corresponding to the inference request does not depend on the session context. In this case, the terminal reads the atomic switching identifier and, based on the atomic switching identifier, processes the inference request through the inference engine instance running the active version of the AI ​​model. The atomic switching identifier indicates information related to the active version. For example, if the atomic switching identifier is information related to the first version, the inference request is processed through the first inference engine instance; if the atomic switching identifier is information related to the second version, the inference request is processed through the second inference engine instance. This allows stateless requests to be handled by the inference engine instance running the active version (i.e., the new inference engine instance).

[0116] By handing over these stateless requests to the new inference engine instance, the number of inference requests handled by the old inference engine instance can be reduced, allowing the old inference engine instance to reach the unloaded state as soon as possible, thereby unloading the old inference engine instance and releasing the memory space occupied by the old inference engine instance.

[0117] If the inference request is a first inference request, the terminal selects one inference engine instance from the first inference engine instance and the second inference engine instance to process the first inference request. For example, the process shown in steps 305 to 307 below.

[0118] In step 305, if the inference request is a first inference request, the terminal queries whether the first session has been bound to the second version.

[0119] For example, the terminal queries the session identifier of the first session in at least one binding relationship entry in the second version. If the session identifier of the first session is found in any binding relationship entry in the second version, it means that the first session has been bound to the second version. If the session identifier of the first session is not found in any binding relationship entry in the second version, it means that the first session has not been bound to the second version. The at least one binding relationship entry in the second version includes all binding relationship entries in the second version within the session binding relationship set.

[0120] In step 306, if the first session has been bound to the second version, the terminal processes the first inference request through the second inference engine instance.

[0121] For example, if the session identifier of the first session is found in any binding relationship entry in the second version, the terminal processes the first inference request through the second inference engine instance. The process of processing the first inference request through the second inference engine instance can be referred to step 204 above.

[0122] In step 307, if the first session is not bound to the second version, the terminal processes the first inference request through the first inference engine instance.

[0123] For example, if the session identifier of the first session is not found in at least one binding relationship entry in the second version (i.e., the session identifier of the first session is not found in any of the binding relationship entries in the second version), the terminal processes the first inference request through the first inference engine instance.

[0124] If the first session is not bound to the second version, indicating that the first session is a new session after the active version switch, the terminal will handle the first inference request through the first inference engine instance. This allows the first inference engine instance to independently maintain the complete session context of the first session, preventing the loss of the session context and maintaining the integrity and consistency of the session context of the first session.

[0125] If the first session is not bound to the second version, the first session may or may not be bound to the first version. If the first session is not bound to the first version, then the first session needs to be bound to the first version.

[0126] Therefore, in some embodiments, if the first session is not bound to the second version, the terminal queries whether the first session has been bound to the first version. For example, the terminal queries whether there is a binding relationship entry for the first version in the session binding relationship set. If there is no binding relationship entry for the first version in the session binding relationship set, it means that the first session is not bound to the first version. If there is at least one binding relationship entry for the first version in the binding relationship set, the terminal queries the session identifier of the first session in at least one binding relationship entry of the first version. If the session identifier of the first session is found in any binding relationship entry of the first version, it means that the first session has been bound to the first version. If the session identifier of the first session is not found in any binding relationship entry of the first version, it means that the first session is not bound to the first version.

[0127] Each binding entry for the first version includes a version identifier of the first version and a session identifier of any session, indicating that the first version is bound to that session. Optionally, the binding entry may also include any one of the instance identifier of the first inference engine instance, the creation time of the binding entry, and the last access time.

[0128] In some embodiments, if the first session is not bound to either the first version or the second version, the terminal binds the first session to the first version and processes the first inference request through the first inference engine instance. The first session not being bound to either the first version or the second version can manifest as follows: the session identifier of the first session is not found in at least one binding relationship entry of the second version, and the first session is not bound to the first version. That is, if the session identifier of the first session is not found in at least one binding relationship entry of the second version, and the first session is not bound to the first version, the terminal binds the first session to the first version and processes the first inference request through the first inference engine instance.

[0129] For example, if the session identifier of the first session is not found in any binding relationship entry of the second version and the session identifier of the first session is not found in any binding relationship entry of the first version, the terminal binds the first session to the first version and processes the first inference request through the first inference engine instance.

[0130] Binding the first session to the first version includes adding a first binding relationship entry to the session binding relationship set. The first binding relationship entry includes the version identifier of the first version and the session identifier of the first session. Optionally, the first binding relationship entry also includes any one of the instance identifier of the first inference engine instance, the creation time of the first binding relationship entry, and the last access time. The first binding relationship entry is the binding relationship entry for the first version.

[0131] Taking the session binding relationship set as shown in Table 1 above as an example, assuming that the version ID of the first version is P_B, the instance ID of the first inference engine instance is M_B, and the first session is session S3, the terminal adds the first binding relationship entry {P_B; S3; M_B; 10:04:30; 10:04:30} to the fourth row of Table 1, resulting in Table 2.

[0132] Table 2

[0133] Since the first binding relationship entry is a newly added entry and has not been accessed before, its last access time can be the same as its creation time, as shown in Table 2, where both creation and last access times are 10:04:30. Subsequently, each time a reasoning request carrying the session identifier of the first session is received, the terminal queries the session binding relationship set for the identifier of the first session. This allows the terminal to find the identifier of the first session in the first binding relationship entry, thus determining that the first session is bound to the first version, and then sending the reasoning request to the first inference engine instance for processing. Each time the identifier of the first session is found in the first binding relationship entry, the terminal also modifies the last access time in the first binding relationship entry to the time of this query. For example, for a reasoning request containing the session identifier of session S3, assuming the terminal finds the session identifier of session S3 in the first binding relationship entry in Table 1 at 10:06:20, and sends the reasoning request to the inference engine instance M_B for processing, the terminal updates the last access time in the binding relationship entry to 10:06:20, resulting in Table 3.

[0134] Table 3

[0135] If the new session is not bound to the active version after the switch (such as the first version), when an inference request (such as the first inference request) initiated in the new session (such as the first session) is received, the new session will be bound to the active version after the switch. This will allow subsequent inference requests initiated in the new session to be quickly routed to the new inference engine instance for processing based on the binding relationship between the new session and the active version after the switch, thereby speeding up the processing speed of inference requests and improving the processing efficiency of requests.

[0136] In some embodiments, if the first session is not bound to the second version but is already bound to the first version, the terminal processes the first inference request through the first inference engine instance. For example, if the session identifier of the first session is found in any binding relationship entry of the second version (such as the first binding relationship entry), the terminal processes the first inference request through the first inference engine instance.

[0137] In this disclosure, the various method steps executed by the terminal can be performed by the terminal's Software Development Kit (SDK). The following example, using an LLM AI model, illustrates the above method flow.

[0138] Assume the target application is an Electron desktop application, the AI ​​model is an LLM (Liquid Modeling Model), and the Electron desktop application integrates the LLM. Assume the current active version of the LLM is version M_A (e.g., version 1.0), and version M_A is bound to 342 sessions (i.e., active sessions). Assume the average number of conversation rounds per user per session is 20, and the average session lifespan is 30 minutes. The active version of the LLM needs to be hot-updated from version M_A to version M_B. The model parameters of versions M_A and M_B are slightly different, but the inference interface is identical. The SDK in the terminal downloads the model package (e.g., a differential package) corresponding to version M_B in a background thread. The SDK performs SHA-256 verification on the model package. After successful verification, it obtains the LLM for version M_B based on the model package, loads the LLM for version M_B into a dedicated memory area (e.g., allocates memory for version M_B to store its model parameters), creates an inference engine instance P_B for version M_B, and allocates memory space for the inference engine instance P_B. Loading takes approximately 8 seconds, and the main thread user interface (UI) runs smoothly without lag. The administrator sends a version switching command to the terminal via the model management platform. In response, the SDK sets the switching status flag `is_switching` to `true`, updates the atomic switching flag `g_active_model` from information related to version M_A to information related to version M_B, and then updates `is_switching` back to `false`. Assuming that session S123 in the 342 sessions initiates an inference request during the 6th round of dialogue, the SDK queries the session binding relationship set for a binding relationship entry containing the session flag S123. Let's assume this binding relationship entry is {P_A; S123; M_A; xxxx; xxxx}. Based on this binding relationship entry, the inference request is sent to the inference engine instance P_A running the LLM of version M_A. The inference engine instance P_A processes the inference request based on the session context of session S123 to generate a response, ensuring the continuity of the dialogue within session S123. Assuming a user establishes a new session, let's take session S456 as an example. If the user initiates an inference request in session S456, the SDK searches the session binding relationship set for a binding relationship entry containing the session identifier S456. If no binding relationship entry containing the session identifier S456 is found, is_switching is false. Then, version M_B indicated by g_active_model is used as the target version, and session S123 is bound to version M_B in the session binding relationship set. The inference request is then sent to the inference engine instance P_B, which processes the inference request.Suppose a user initiates a stateless request. The SDK, based on the version M_B indicated by g_active_model, sends the stateless request to the inference engine instance P_B, which then processes the stateless request.

[0139] In the method provided in this disclosure embodiment, requests received during the active version switching process are uniformly sent to the old inference engine instance for processing. New inference requests initiated in sessions established before the switch are uniformly sent to the old inference engine instance for processing, and new inference requests initiated in sessions established after the switch are uniformly sent to the new inference engine instance for processing. This ensures that all inference requests in the same multi-turn session are always processed by the same inference engine instance. The same inference engine instance maintains the complete session context of the same multi-turn session, so that the complete session context of the same multi-turn session is naturally retained in the same inference engine instance. This ensures the integrity and consistency of the session context of the same multi-turn session, avoids inaccurate inference results due to the lack of session context, and eliminates the need to migrate or serialize the session context, thereby achieving lossless version updates.

[0140] The above description illustrates the process of first checking if the first session is bound to the second version after the active state switch is completed, and then checking if the first session is bound to the first version if the first session is not bound to the second version. In other embodiments, after the active state switch, for a received inference request carrying a session identifier, the version (i.e., the target version) of the session indicated by the session identifier can be determined by querying each binding relationship entry in the session binding relationship set. The inference request is then processed by running the inference engine instance of the target version (i.e., the target inference engine instance). The target inference engine instance can also be understood as the inference engine instance bound to the session indicated by the session identifier.

[0141] by Figure 4For example, the terminal receives an inference request carrying a session identifier Sx. It queries each binding relationship entry in the session binding relationship set for the session identifier Sx (i.e., checks if the session identifier Sx exists in the session binding relationship set). If the session identifier Sx is found in any binding relationship entry, the version indicated by the version identifier in that binding relationship entry (first version or second version) is the target version, and the inference engine instance indicated by the instance identifier in the binding relationship entry (first inference engine instance or second inference engine instance) is the target inference engine instance. The terminal sends the inference request to the target inference engine instance and updates the last access time in the binding relationship entry to the time of this query. The target inference engine instance receives and processes the inference request. If the session identifier Sx is not found in any of the binding relationship entries in the session binding relationship set, it indicates that the session indicated by the session identifier is a new session. The terminal determines whether the active state is in the process of switching. If the active state is in the process of switching, the second version is used as the target version, and the inference request can be the third inference request mentioned above. If the active state is not in the process of switching (i.e., the active state has been switched), the first version is used as the target version, and the inference request can be the first inference request mentioned above. For example, the terminal determines whether the switching state identifier is_switching is true. If the switching state identifier is_switching is true, the second version is used as the target version, and the second inference engine instance is used as the target inference engine instance. If the switching state identifier is_switching is not true, the first version is used as the target version, and the first inference engine instance is used as the target inference engine instance. The terminal adds a target binding relationship entry to the session binding relationship set and sends the inference request to the target inference engine instance. The target inference engine instance receives and processes the inference request. The target binding relationship entry includes the version identifier of the target version, the session identifier Sx, the instance identifier of the target inference engine instance, the creation time and the last access time of the target binding relationship entry.

[0142] The method provided in this disclosure switches the active version of the AI ​​model from the second version to the first version, making the first version of the AI ​​model active and the second version of the AI ​​model inactive. The terminal runs the first version of the AI ​​model through the first inference engine instance and runs the second version of the AI ​​model through the second inference engine instance. For inference requests initiated in a session, if the session is already bound to the second version before the switch, the inference request is processed by the second inference engine instance. Since the second inference engine instance holds the complete session context of the session, it can process the inference request based on the complete session context to ensure the accuracy of the inference result, thereby avoiding inaccurate inference results due to the lack of contextual basis in this round of inference.

[0143] During the process of the terminal providing inference services through the second version, the aforementioned session binding relationship set is stored in the terminal's memory. Therefore, at least one binding relationship entry of the second version in the session binding relationship set is stored in the terminal's memory. After the active version of the AI ​​model switches from the second version to the first version, as the number of new sessions increases, the terminal adds more and more binding relationship entries to the session binding relationship set, and the information stored in the memory of the new inference engine instance increases. If the old binding relationship entries (such as the binding relationship entries of the second version) are not cleaned up in time, the old inference engine instance is not uninstalled in time, or the process of the old inference engine instance is not executed perfectly, the active version before the switch (i.e., the second version) and the old inference engine instance will remain in memory for a long time, which may cause memory resource waste or memory overflow (Out of Memory, OOM), resulting in memory leak.

[0144] Therefore, in some embodiments, after the active version of the AI ​​model switches from version two to version one, the terminal can gradually delete at least one binding relationship entry of version two from the internal session binding relationship set, thereby releasing the memory space occupied by at least one binding relationship entry of version two in memory and avoiding memory leaks. Once all binding relationship entries of version two have been deleted and all inference requests sent to the second inference engine instance have been processed, the second inference engine instance can be unloaded, releasing the second inference engine instance and the memory space occupied by it, further preventing memory leaks. This implementation method is described below.

[0145] In some embodiments, after the active version of the AI ​​model switches from the second version to the first version, the above data processing method further includes: if any binding relationship entry in the second version meets the deletion condition, the terminal deletes the binding relationship entry in memory.

[0146] The deletion condition is the condition that must be met to delete a binding relationship entry. For example, the deletion condition is that the session indicated by the session identifier in the binding relationship entry is closed.

[0147] In this context, closing any session can be either explicit or implicit. Explicit closing includes receiving a session close command or a session close notification command. A session close command instructs the user to close the session; this command is triggered by the user, and the terminal receives it when the user actively closes the session. A close notification command notifies the user of the session closure; for example, when a session closes, the target application generates a close notification command, which the terminal can receive. Implicit closing includes the session's binding relationship entries reaching a preset query duration. If the waiting query duration for any session's binding relationship entries reaches the preset duration, it indicates that the terminal has not received any new inference requests initiated in that session for an extended period, and the user has not initiated any new inference requests in that session for a long time, thus the session is considered implicitly closed. The preset duration can be 30 minutes, but is not limited to 30 minutes; it can be set according to the actual application scenario. This embodiment of the present disclosure does not limit the preset duration.

[0148] Therefore, the deletion criteria for any binding relationship entry include any of the following: The session indicated by the session identifier in the binding relationship entry is displayed as closed; The waiting time for the bound relationship entry to be queried has reached the preset time.

[0149] For example, for any binding relationship entry in the second version and the session indicated by the session identifier in the binding relationship entry, if the terminal receives a session close command or session close notification command for the session, it means that the session is explicitly closed and the binding relationship entry meets the corresponding deletion conditions, then the terminal deletes the binding relationship entry in memory.

[0150] Alternatively, for any binding relationship entry in the second version, the terminal uses the time interval between the current time and the last access time of the binding relationship entry as the waiting query duration for that binding relationship entry. If the waiting query duration of the binding relationship entry reaches a preset duration (e.g., the waiting query duration is greater than or equal to the preset duration), the binding relationship entry meets the corresponding deletion condition, and the terminal deletes the binding relationship entry from memory. For example, the terminal periodically checks whether the waiting query duration of each binding relationship entry in the second version reaches the preset duration. If the waiting query duration of any binding relationship entry reaches the preset duration, the terminal deletes the binding relationship entry from memory. The detection period includes, but is not limited to, 30 seconds. The detection period can be set according to the actual application scenario; here, this embodiment does not limit the detection period.

[0151] For example, the second version is AI model version M_A (e.g., version 1.0), and the first version is AI model version M_B (e.g., version 2.0). Before switching the active version of the AI ​​model from version M_A to version M_B, version M_A is bound to 342 sessions. The session binding relationship set contains 342 binding relationship entries for version M_A, and each binding relationship entry contains the session identifier of an active session bound to version M_A. Assuming a detection period of 30 seconds and a preset duration of 30 minutes, the terminal scans these 342 binding relationship entries in the session binding relationship set every 30 seconds, clearing binding relationship entries with a waiting query time exceeding 30 minutes. As the number of sessions closed by users or sessions with timeouts among these 342 active sessions increases, the number of binding relationship entries for version M_A gradually decreases. After approximately 30 minutes, the number of binding relationship entries for version M_A and the number of active sessions on version M_A drop to 0.

[0152] It should be understood that deleting a binding relationship entry in memory is equivalent to deleting that binding relationship entry from the session binding relationship set in memory. For any binding session relationship entry in the second version, the binding relationship entry in memory is immediately deleted by turning off the corresponding session display or by waiting for a preset query duration, thus preventing the binding relationship entry from occupying memory space for a long time and reducing memory leaks.

[0153] In some embodiments, the preset duration can be adjusted. For example, the data processing method further includes: the terminal adjusting the preset duration based on the remaining storage capacity of memory. The remaining storage capacity refers to the capacity of unused storage space remaining in memory. This is to ensure that the preset duration matches the actual memory environment, allowing for a suitable reduction in the preset duration when memory resources are scarce, enabling the second inference instance engine to reach an unloadable state as quickly as possible and improving memory utilization.

[0154] For example, multiple different candidate preset durations are set, each candidate preset duration corresponding to a storage capacity range. The storage capacity range includes multiple storage capacities of the storage space. The larger the candidate preset duration, the larger the storage capacity within the corresponding storage capacity range; conversely, the larger the candidate preset duration, the smaller the storage capacity within the corresponding storage capacity range. When the remaining storage capacity of the memory is within the storage capacity range corresponding to any candidate preset duration, the terminal adjusts the preset duration to that candidate preset duration.

[0155] Alternatively, when the remaining storage capacity of the memory is lower than the preset capacity, the terminal issues a low memory alarm message, which indicates that the remaining storage capacity of the memory is too low, and the terminal reduces the preset duration.

[0156] The aforementioned candidate preset durations and low memory alarm triggering schemes can also be combined. For example, assuming the terminal's total memory storage capacity is 8GB, the second version is AI model version M_A, and the first version is AI model version M_B. When versions M_A and M_B coexist, the AI ​​models of versions M_A and M_B require a total of 12GB, exceeding the terminal's physical memory. The SDK in the terminal starts swapping space, and the SDK issues a low memory alarm message. The SDK captures the low memory warning message and adjusts the preset duration from 30 minutes to 2 minutes. The SDK scans the session binding relationship set and cleans up binding relationship entries for version M_A with a wait time exceeding 2 minutes. Assuming version M_A has 342 binding relationship entries, of which 300 binding relationship entries have not been queried for more than 2 minutes, the SDK deletes these 300 binding relationship entries (equivalent to implicitly closing the 300 sessions corresponding to the 300 binding relationship entries). The number of binding relationship entries for version M_A decreases from 342 to 42. If memory pressure persists and the remaining memory storage capacity decreases to a certain storage capacity range among multiple storage capacity ranges, and the corresponding candidate preset duration for that storage capacity range is 30 seconds, then the SDK will reduce the preset duration from 2 minutes to 30 seconds. The SDK will scan the session binding relationship set and clean up binding relationship entries of version M_A with a wait query time exceeding 30 seconds.

[0157] By deleting a binding relationship entry when its query wait time reaches a preset duration, the session corresponding to that binding relationship entry can be implicitly closed. By continuously reducing the preset duration, the cleanup of second-version binding relationship entries can be accelerated, allowing the number of second-version binding relationship entries to quickly reach zero. This enables the second inference engine instance to quickly reach an unloadable state, thereby indirectly achieving memory reclamation.

[0158] In some embodiments, when the remaining storage capacity of the memory is lower than the target preset capacity or the preset duration is lower than or equal to the target preset duration, the terminal can also send a session closure warning message to the target application. The session closure warning message indicates that the session bound to the second version is about to be closed, so that the user can save the session content in a timely manner. For example, when the preset duration is equal to 30 seconds (the target preset duration), the SDK in the terminal sends a session closure warning message to the target application. The target preset duration includes, but is not limited to, 30 seconds. The target preset capacity and the target preset duration can be set according to the actual application scenario and are not limited thereto.

[0159] The method described above for deleting second-version binding relationship entries also applies to first-version binding relationship entries. Terminals can also delete first-version binding relationship entries in the session binding relationship set in a similar manner, which will not be elaborated further here. First-version binding relationship entries contain either the version identifier of the first version or the instance identifier of the first inference engine instance.

[0160] In some embodiments, after switching the active version of the AI ​​model from the second version to the first version, the above data processing method further includes: The terminal checks whether the second version is bound to any sessions and whether the second inference engine instance is processing any stateless requests. If the second version is not bound to any sessions and the second inference engine instance is not processing any stateless requests, the second inference engine instance reaches an unloadable state. The terminal unloads the second inference engine instance, releasing the memory space occupied by the second inference engine instance and the second version. Otherwise, the second inference engine instance is not unloaded, and the second version continues to provide inference services. This is to avoid affecting the processing of inference requests sent to the second inference engine instance after the switch.

[0161] The process of detecting whether the second version has a bound session includes: checking if a binding relationship entry for the second version exists in the session binding relationship set. If it exists, the second version has a bound session; otherwise, the second version is not bound to any session. A binding relationship entry for the second version refers to a binding relationship entry that includes either the version identifier of the second version or the instance identifier of the second inference engine instance.

[0162] Alternatively, the terminal can use a first counter to count the number of sessions bound to the second version. The count in the first counter represents the number of sessions (i.e., active sessions) bound to the second version. Initially, the count in the first counter is 0. Each time a second version binding entry is added to the binding relationship set, the terminal increments the count in the first counter by 1. Each time a second version binding entry is deleted from the binding relationship set, the terminal decrements the count in the first counter by 1. The above detection of whether the second version has bound sessions includes: checking if the count in the first counter is 0. If the count in the first counter is 0, the second version has bound sessions; if the count in the first counter is not 0, the second version has no bound sessions.

[0163] In some embodiments, the terminal counts the number of sessions bound to the second version using a second counter. The count value in the first counter is the number of sessions (i.e., active sessions) bound to the second version. Initially, the count value in the first counter is 0. Each time a second version binding entry is added to the binding relationship set, the terminal increments the count value in the first counter by 1. Each time a second version binding entry is deleted from the binding relationship set, the terminal decrements the count value in the first counter by 1. The above-mentioned detection of whether the second version has bound sessions includes: detecting whether the count value in the first counter is 0. If the count value in the first counter is 0, the second version has bound sessions; if the count value in the first counter is not 0, the second version has no bound sessions.

[0164] In some embodiments, the terminal uses a second counter to count the number of stateless requests processed by the second inference engine instance. The count value in the second counter is the number of stateless requests processed by the second inference engine instance. Initially, the count value in the second counter is 0. The terminal increments the count value in the second counter by 1 each time it sends a stateless request to the second inference engine instance, and decrements the count value in the second counter by 1 each time the second inference engine instance finishes processing a stateless request.

[0165] Whether the second inference engine instance has a stateless request in progress includes: checking whether the count value in the second counter is 0. If the count value in the second counter is 0, the second inference engine instance has no inference request in progress; otherwise, the second inference engine instance has an inference request in progress.

[0166] In some embodiments, after switching the active version of the AI ​​model from the second version to the first version, if the second version is not bound to any session and the second inference engine instance has no inference requests in progress, the second inference engine instance is not uninstalled temporarily. After a second period of time, if the second version is not bound to a new session, the terminal uninstalls the second inference engine instance, releasing the memory space occupied by the second inference engine instance and the second version; if the second version is not bound to a new session, the terminal waits until the new session is closed before uninstalling the second inference engine instance, releasing the memory space occupied by the second inference engine instance and the second version. The second period of time is the drainage period of the second inference engine instance. In this way, it is possible to avoid the second inference engine instance not being processed due to unexpected new inference requests being sent to it during the drainage period. The second period of time can be set according to the actual application scenario. Here, this embodiment of the disclosure does not limit the second period of time.

[0167] by Figure 5For example, assuming the second duration is 5 minutes, if the number of active sessions in the second version (i.e., the number of sessions bound to the second version) is 0 and the second inference engine is not processing any stateless requests, the second inference engine enters a drain state, starts a timer, and counts for 5 minutes (i.e., the second duration). When the timer expires and the second version is not bound to a new session, the second inference engine instance is unloaded, releasing the memory space occupied by the second inference engine instance and the second version. When the timer does not expire or the second version is bound to a new session, the drain state of the second inference engine is canceled.

[0168] The memory space occupied by the second inference engine instance is designated as the second memory space, while the memory space occupied by the second version is designated as the fourth memory space. The second memory space stores data such as the session context, configuration information of the second inference engine instance, and execution status. The fourth memory space stores the model parameters (such as weight tensors) of the AI ​​model for the second version. Releasing the memory space occupied by the second inference engine instance and the second version includes clearing the data stored in both the second and fourth memory spaces, thereby reducing the memory usage of the second inference engine instance and the second version.

[0169] The session binding relationship set is stored in the terminal's memory. In some embodiments, to prevent the loss of binding relationship entries in the session binding relationship set (e.g., the termination of the target application process will cause the binding relationship entries in the binding relationship set to be lost), the terminal can persist the session binding relationship set. To prevent the session binding relationship set from being maliciously read during persistent storage, the terminal encrypts the session binding relationship set to obtain an encrypted session binding relationship set. The encrypted session binding relationship set is then persisted, thereby ensuring the security of the data in the session binding relationship set and improving robustness.

[0170] The encryption method for the session binding relationship set can be as follows: the terminal performs symmetric encryption on the session binding relationship set based on its derived key. The derived key can be generated based on the terminal ID and the user's login credentials for the target application. Of course, other encryption methods can also be used for the session binding relationship set; however, this disclosure does not limit the encryption method for the session binding relationship set.

[0171] When the aforementioned session binding relationship set includes at least one binding relationship entry from the first version, the encrypted session binding relationship set includes the encrypted at least one binding relationship entry. The encryption of the session binding relationship set to obtain the encrypted session binding relationship set includes encrypting at least one binding relationship entry from the second version to obtain the encrypted at least one binding relationship entry. The persistent storage of the encrypted session binding relationship set includes persistent storage of the encrypted at least one binding relationship entry. This prevents the at least one binding relationship entry from the second version from being maliciously read, thereby ensuring data security. Of course, even after cleaning up the session binding relationship set to include binding relationship entries from the first version, the encrypted session binding relationship set includes the encrypted binding relationship entries from the first version. By persistently storing the encrypted session binding relationship set, it is also possible to prevent the binding relationship entries from the first version from being maliciously read, thus ensuring data security.

[0172] After the encrypted session binding set is persistently stored, if the target application is closed, the terminal will unload the inference engine instance of the AI ​​model. After the target application restarts, the terminal can read the encrypted session binding set and restore it.

[0173] like Figure 6 As shown, in response to a target application restart, the terminal loads the encrypted session binding set from persistent storage, decrypts the encrypted session binding set to obtain the decrypted session binding set, and stores the decrypted session binding set in memory. Since the inference engine instance running the AI ​​model is typically unloaded after the target application restarts, after storing the decrypted session binding set in memory, the terminal checks whether the inference engine instance pointed to by each binding entry in the session binding set (i.e., the inference engine instance indicated by the instance identifier in the binding entry) is in memory. If the inference engine instance pointed to by any binding entry is not in memory, it means that the session context of the session indicated by the session identifier in that binding entry is lost. The terminal then deletes that binding entry from the session binding set; this is a reasonable cost of the target application restart. If the inference engine instance pointed to by any binding entry is still in memory, the binding entry is retained in the session binding set, thereby restoring the binding entry so that the session indicated by the session identifier in that binding entry can continue seamlessly.

[0174] For example, suppose that during the transition of the active version of the AI ​​model from version M_A to version M_B, the terminal's operating system terminates the target application process due to insufficient memory (i.e., closes the target application). When the user reopens the target application, a multi-turn session that was previously in progress (e.g., session S123, bound to version M_A) needs to continue. Upon startup of the target application, the SDK in the terminal reads the encrypted session binding relationship set from persistent storage, decrypts it, and the decrypted session binding relationship set contains binding relationship entries {P_A; S123; M_A; xxxx; xxxx}, thus restoring the binding relationship entries corresponding to session S123. The terminal checks whether the inference engine instance P_A is in memory. Since the target application process was previously terminated, the inference engine instance P_A has been unloaded and is no longer in memory. Therefore, the SDK deletes the binding relationship entries {P_A; S123; M_A; xxxx; xxxx} from the session binding relationship set and logs this information. After the active version of the AI ​​model is switched from version M_A to version M_B, if the user initiates a new inference request in session S123, and the session binding relationship set is checked and no binding relationship entry containing the session identifier S123 is found, then session S123 is considered a new session. The terminal binds session S123 to version M_B in the session binding relationship set and processes the inference request through the inference engine instance P_B.

[0175] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0176] Figure 7 This is a logical structure block diagram of a data processing apparatus according to an exemplary embodiment, with reference to... Figure 7 The data processing device 700 includes: The running unit 701 is configured to execute a first version of the AI ​​model running through a first inference engine instance and a second version of the AI ​​model running through a second inference engine instance, wherein the AI ​​model has a first version in an active state and a second version in an inactive state. The receiving unit 702 is configured to receive a first inference request, the first inference request carrying a session identifier of a first session, and the first inference request indicating that an inference task be performed based on the AI ​​model; The query unit 703 is configured to query the session identifier of the first session in at least one binding relationship entry of the second version, the binding relationship entry including the version identifier of the second version and the session identifier of any session; The first processing unit 704 is configured to process the first inference request by running a second inference engine instance of the second version if the session identifier of the first session is found in any binding relationship entry of the second version.

[0177] Optionally, the data processing device 700 further includes: The second processing unit is configured to process the first inference request through the first inference engine instance if the session identifier of the first session is not found in the at least one binding relationship entry.

[0178] Optionally, the second processing unit is configured to execute: If the session identifier of the first session is not found in the at least one binding relationship entry and the first session is not bound to the first version, then bind the first session to the first version; The first inference request is processed using the first inference engine instance.

[0179] Optionally, the data processing device 700 further includes a second processing unit; The receiving unit 702 is further configured to execute a second inference request that does not carry a session identifier, the second inference request indicating that an inference task be performed based on the AI ​​model; The second processing unit is configured to process the second inference request by running the first inference engine instance.

[0180] Optionally, at least one binding relationship entry of the second version is stored in the memory of the terminal; the data processing device 700 further includes: The deletion unit is configured to delete the binding relationship entry in memory if any binding relationship entry of the second version meets the deletion condition.

[0181] Optionally, the deletion conditions include any of the following: The session indicated by the session identifier in the binding relationship entry is explicitly closed; The waiting time for querying the binding relationship entry has reached the preset time.

[0182] Optionally, the data processing device 700 further includes: The adjustment unit is configured to adjust the preset duration based on the remaining storage capacity of the memory.

[0183] Optionally, the data processing device 700 further includes: The encryption unit is configured to perform encryption on at least one binding relationship entry of the second version to obtain at least one encrypted binding relationship entry; The third processing unit is configured to perform persistent storage of at least one of the encrypted binding relationship entries.

[0184] Optionally, the data processing device 700 further includes: The unloading unit is configured to unload the second inference engine instance and release the memory space occupied by the second inference engine instance and the second version if the second version is not bound to any session and the second inference engine instance has no stateless requests being processed. The stateless requests refer to inference requests that do not carry session identifiers.

[0185] Optionally, the data processing device 700 further includes: The switching unit is configured to execute a version switching command in response to the AI ​​model, switching the state of the second version from the active state to the inactive state, and setting the state of the first version to the active state.

[0186] Optionally, the receiving unit 702 is further configured to receive a third inference request, the third inference request indicating that an inference task be performed based on the AI ​​model; The first processing unit 704 is configured to process the third inference request through the second inference engine instance if the state of the first version is switching.

[0187] Regarding the data processing apparatus 700 in the above embodiments, the specific manner in which each unit performs operations has been described in detail in the embodiments of the relevant data processing method, and will not be elaborated here.

[0188] Figure 8 This is a structural block diagram of a terminal according to an exemplary embodiment. Figure 8 The terminal 800 shown may vary considerably due to differences in configuration or performance. It may include one or more Central Processing Units (CPUs) 801 and one or more memories 802. The memory 802 stores at least one line of program code, which is loaded and executed by the processor 801 to implement the data processing methods provided in the various embodiments described above. Of course, the terminal 800 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The terminal 800 may also include other components for implementing device functions, which will not be elaborated upon here.

[0189] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction, which can be executed by a processor in a terminal to complete the data processing method in the above embodiments. Optionally, the computer-readable storage medium may be a non-transitory computer-readable storage medium, such as ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk, and optical data storage device, etc.

[0190] In an exemplary embodiment, a computer program product is also provided, including one or more instructions that can be executed by a terminal's processor to perform the data processing methods provided in the various embodiments described above.

[0191] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0192] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a terminal, the method includes: A first version of the AI ​​model is run through a first inference engine instance, and a second version of the AI ​​model is run through a second inference engine instance, wherein the AI ​​model has a first version in an active state and a second version in an inactive state; Receive a first inference request, the first inference request carrying a session identifier of a first session, the first inference request indicating to perform an inference task based on the AI ​​model; Query the session identifier of the first session in at least one binding relationship entry of the second version, wherein the binding relationship entry includes the version identifier of the second version and the session identifier of any session; If the session identifier of the first session is found in any binding relationship entry of the second version, the first inference request is processed by running the second inference engine instance of the second version.

2. The data processing method according to claim 1, characterized in that, The method further includes: If the session identifier of the first session is not found in the at least one binding relationship entry, the first inference request is processed through the first inference engine instance.

3. The data processing method according to claim 1, characterized in that, If the session identifier of the first session is not found in the at least one binding relationship entry, the first inference request is processed through the first inference engine instance, including: If the session identifier of the first session is not found in the at least one binding relationship entry and the first session is not bound to the first version, then bind the first session to the first version; The first inference request is processed using the first inference engine instance.

4. The data processing method according to claim 1, characterized in that, After running the first version of the AI ​​model through the first inference engine instance, the method further includes: Receive a second inference request that does not carry a session identifier, the second inference request instructing an inference task to be performed based on the AI ​​model; The second inference request is processed by running the first inference engine instance.

5. The data processing method according to claim 1, characterized in that, At least one binding relationship entry of the second version is stored in the memory of the terminal; The method further includes: If any binding relationship entry in the second version meets the deletion condition, delete the binding relationship entry in memory.

6. The data processing method according to claim 5, characterized in that, The deletion criteria include any of the following: The session indicated by the session identifier in the binding relationship entry is explicitly closed; The waiting time for querying the binding relationship entry has reached the preset time.

7. The data processing method according to claim 6, characterized in that, The method further includes: The preset duration is adjusted based on the remaining storage capacity of the memory.

8. The data processing method according to claim 1, characterized in that, The method further includes: Encrypt at least one binding relationship entry in the second version to obtain at least one encrypted binding relationship entry; At least one encrypted binding relationship entry is persistently stored.

9. The data processing method according to claim 1, characterized in that, After running the first version of the AI ​​model through the first inference engine instance, the method further includes: If the second version is not bound to any session and the second inference engine instance is not processing any stateless requests, unload the second inference engine instance and release the memory space occupied by the second inference engine instance and the second version. The stateless request refers to an inference request that does not carry a session identifier.

10. The data processing method according to claim 1, characterized in that, Before running the first version of the AI ​​model through the first inference engine instance, the method further includes: In response to a version switching command for the AI ​​model, the state of the second version is switched from the active state to the inactive state, and the state of the first version is set to the active state.

11. The data processing method according to claim 10, characterized in that, The method further includes: Receive a third inference request, which instructs the execution of an inference task based on the AI ​​model; If the state of the first version is switching, the third inference request is processed through the second inference engine instance.

12. A data processing apparatus, characterized in that, include: The running unit is configured to execute a first version of the AI ​​model running through a first inference engine instance and a second version of the AI ​​model running through a second inference engine instance, wherein the AI ​​model has a first version in an active state and a second version in an inactive state; The receiving unit is configured to receive a first inference request, the first inference request carrying a session identifier of a first session, the first inference request indicating that an inference task be performed based on the AI ​​model; The query unit is configured to query the session identifier of the first session in at least one binding relationship entry of the second version, the binding relationship entry including the version identifier of the second version and the session identifier of any session; The first processing unit is configured to process the first inference request by running a second inference engine instance of the second version if the session identifier of the first session is found in any binding relationship entry of the second version.

13. A terminal, characterized in that, include: One or more processors; One or more memories for storing the one or more processor-executable instructions; The one or more processors are configured to execute the instructions to implement the data processing method as described in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, When at least one instruction in the computer-readable storage medium is executed by one or more processors of the terminal, the computer device is enabled to perform the data processing method as described in any one of claims 1 to 11.

15. A computer program product, characterized in that, It includes one or more instructions, which are executed by one or more processors of the terminal, enabling the terminal to perform the data processing method as described in any one of claims 1 to 11.

Images