Data processing method and device, equipment and storage medium

Abstract

The invention relates to the technical field of intelligent questions and answers, in particular to the technical field of artificial intelligence, and provides a data processing method, device and equipment and a storage medium, which are used for effectively reducing the storage overhead of KV data while ensuring the reply precision. The method comprises the following steps: splitting a problem sequence into a plurality of problem lexical elements, and extracting corresponding semantic key value vectors; performing multi-round iterative processing on the question sequence until a reply answer is output; in each round of iteration, based on the semantic key value vector of each historical lexical element, obtaining an answer lexical element of the round; taking the answer lexical elements as new historical lexical elements, and extracting semantic key value vectors of the new historical lexical elements for storage; when the stored semantic key value vector reaches a storage capacity threshold value, obtaining a lexical element compression quantity based on a problem lexical element quantity and a historical lexical element quantity; and based on the attention evaluation value of each historical lexical element for the answer lexical element of the current round, eliminating the historical lexical elements conforming to the lexical element compression quantity and the corresponding semantic key value vectors.

Description

Technical Field

[0001] This application relates to the field of intelligent question-answering technology, and more particularly to the field of artificial intelligence technology, providing a data processing method, apparatus, device, and storage medium. Background Technology

[0002] Current intelligent question answering tasks mostly employ Large Language Models (LLMs). LLMs need to convert each question word in the input question sequence into a corresponding semantic key-value (KV) vector for storage. Here, the K vector represents the semantic features of the word, and the V vector represents the semantic content of the word; the semantic key-value vector is also known as KV data. The answer is then generated by combining the KV data of each question word. However, the system's memory usage continuously increases as the question answering process progresses. To ensure the response rate and stability, the KV data needs to be compressed to reduce the storage overhead of system resources.

[0003] The industry's compression strategies for key-value (KV) data mainly include the following two: Method 1: KV data compression strategy based on fixed window.

[0004] When using method one, the key-value data of the most recently generated N words will be retained based on a fixed window, or the key-value data of the earliest generated N words will be discarded based on a fixed window.

[0005] While method one is simple to implement, the window size is entirely dependent on manual setting. Setting the size too small will lead to too many discard decisions, increasing additional computational overhead. Moreover, in real-world question-answering scenarios, requesters often extend their questions based on the answer to a particular question. Discarding a segment of key-value data based on a fixed window can result in the loss of crucial contextual information, thereby affecting the response accuracy of the large language model.

[0006] Method 2: Compress KV data using compression algorithms such as low-rank decomposition or sparsification.

[0007] When using method two, a compression algorithm is used to convert high-dimensional, high-volume KV data into low-dimensional, low-volume approximate vectors to reduce overall storage overhead.

[0008] While method two can alleviate the problem of context loss to some extent, its method of transforming key-value data is fixed before model deployment. This means that regardless of whether the current memory overhead is sufficient, the key-value data will be transformed into low-dimensional, low-data-volume data.

[0009] Since the transformation of low-dimensional and low-data-volume data is essentially achieved by reducing the dimensionality or sparsity of the original data, this method will result in varying degrees of information loss in all the saved key-value data, which in turn will affect the response accuracy of the large language model.

[0010] Therefore, how to effectively reduce the storage overhead of KV data while ensuring the accuracy of the response is an urgent problem to be solved. Summary of the Invention

[0011] This application provides a data processing method, apparatus, device, and storage medium for effectively reducing the storage overhead of key-value data while ensuring response accuracy in intelligent question-answering tasks.

[0012] In a first aspect, embodiments of this application provide a data processing method, including: The question sequence is split into multiple question tokens, and the semantic key-value vector of each question token is extracted; The question sequence is iterated multiple times until a response is output. Each iteration includes the following processes: Based on the saved semantic key-value vectors of each historical word, the answer word for this round is obtained; and the answer word is used as a new historical word, and the semantic key-value vector of the answer word is extracted and saved; in the first iteration, each historical word is the multiple question words; When the number of saved semantic key-value vectors reaches the storage threshold, the number of word compressions is obtained based on the current number of problem words and the number of historical words. Based on the attention evaluation value of each historical word for the answer word in this round, historical words that meet the word compression quantity and their corresponding semantic key-value vectors are removed; wherein, each attention evaluation value is determined based on the semantic key-value vector of the corresponding historical word and the answer word, and represents the contribution of the historical word to the generation of the answer word in this round.

[0013] Secondly, embodiments of this application provide a data processing apparatus, including: The semantic extraction unit is configured to split the question sequence into multiple question tokens and extract the semantic key-value vector of each question token. The question response unit is configured to perform multiple rounds of iterative processing on the question sequence until a response answer is output. Each iteration includes the following processes: Based on the saved semantic key-value vectors of each historical word, the answer word for this round is obtained; and the answer word is used as a new historical word, and the semantic key-value vector of the answer word is extracted and saved; in the first iteration, each historical word is the multiple question words; When the number of saved semantic key-value vectors reaches the storage threshold, the number of word compressions is obtained based on the current number of problem words and the number of historical words. Based on the attention evaluation value of each historical word for the answer word in this round, historical words that meet the word compression quantity and their corresponding semantic key-value vectors are removed; wherein, each attention evaluation value is determined based on the semantic key-value vector of the corresponding historical word and the answer word, and represents the contribution of the historical word to the generation of the answer word in this round.

[0014] In one optional implementation, the step of obtaining the word compression quantity based on the current number of question words and the historical number of words is performed, and the question answering unit is specifically configured as follows: Based on the ratio between the number of problem lexical units and the number of historical lexical units, the context complexity corresponding to each historical lexical unit is obtained. The number of lexical compressions is obtained based on the context complexity; wherein the context complexity is negatively correlated with the number of lexical compressions.

[0015] In one optional implementation, the question-answering unit is specifically configured to obtain the number of lexical compressions based on the context complexity, wherein the process is performed as follows: From the historical lexical units, a specified number of reference lexical units are obtained; the specified number is positively correlated with the number of iterations, and each reference lexical unit is an answer lexical unit generated in the iterations. By combining any two reference lexical units, at least one reference lexical group can be obtained; The number of lexical compressions is obtained based on the similarity between the semantic key vectors of the two reference lexical units in each reference lexical unit, combined with the context complexity.

[0016] In one optional implementation, the attention evaluation value based on each historical word for the current round's answer word is executed, and historical words that meet the word compression quantity and their corresponding semantic key-value vectors are cleared. The question answering unit is specifically configured as follows: For each historical word element, the following steps are performed: Based on the attention evaluation value of the historical word element for the answer word element in the current round and the sum of the attention evaluation values ​​of the historical word element for the answer word elements in each of the previous iterations, the global evaluation value of the historical word element is determined. Each answer term in the aforementioned historical terminology is treated as a terminology to be processed. Based on the global evaluation value of each word to be processed, historical words and their corresponding semantic key-value vectors that meet the specified word compression quantity are removed.

[0017] In one optional implementation, before performing the step of treating each answer term in the historical terminology as a terminology to be processed, the question answering unit is further configured to: Based on the global evaluation value of each historical word, and combined with the number of answer words in each historical word, the attention shift of the question sequence is obtained; the attention shift represents the degree of influence of the question sequence on the answer words generated in subsequent iterations. It was determined that the attention offset of the problem sequence did not reach the offset threshold.

[0018] In one optional implementation, the attention shift of the question sequence is obtained by performing the global evaluation based on the individual global evaluation values ​​of each historical lexical unit and combining the number of answer lexical units in each historical lexical unit. The question answering unit is specifically configured as follows: For each answer word in each of the historical word elements, the following steps are performed: obtain the semantic key value vector of the answer word element and the average similarity between it and the semantic key value vector of each question word element; Based on the obtained average similarity values, combined with the global evaluation value of each question word element and its proportion in the global evaluation value of each historical word element, the comprehensive semantic carrying capacity of each answer word element is obtained; the comprehensive semantic carrying capacity represents the degree to which each answer word element reflects the semantic content of the question sequence. Based on the comprehensive semantic carrying capacity and the number of answer lexical units, combined with the number of iterations, the attention shift is obtained.

[0019] In an alternative implementation, the question-answering unit is further configured as follows: When the attention offset reaches the offset threshold, each historical word is treated as a word to be processed.

[0020] In one optional implementation, the attention evaluation value of each historical word element for the answer word element in the current round is determined by the following method: The answer words are feature-encoded to obtain the query vector corresponding to the answer words. Based on the dot product between the query vector and the semantic key vector of each historical word, the semantic relevance between the answer word and each historical word is obtained respectively. For each historical word element, the following steps are performed: based on the proportion of the semantic relevance of the historical word element in the semantic relevance of each historical word element, the attention evaluation value of the historical word element for the answer word element in this round is determined.

[0021] Thirdly, embodiments of this application also provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0022] Fourthly, embodiments of this application also provide an electronic device, including: Memory, used to store computer programs; A processor is configured to invoke a computer program stored in the memory and execute the method described in the first aspect according to the obtained computer program.

[0023] Fifthly, embodiments of this application also provide a computer-readable storage medium storing a computer program for causing a computer to perform the method described in the first aspect.

[0024] The beneficial effects of this application are as follows: The input question sequence is pre-split into multiple question tokens, and the semantic key-value vector (i.e., KV data) of each question token is extracted. Then, the question sequence is processed in multiple rounds of iteration until the answer is output. In each iteration, the answer token for the current round is generated based on the KV data of the saved historical tokens (including question tokens and generated answer tokens), and the answer token is saved as a new historical token in KV, thus forming a complete contextual dependency chain.

[0025] When the amount of saved key-value (KV) data reaches a preset storage threshold, the required number of word segments to be compressed can be dynamically calculated based on the current number of question word segments and the number of historical word segments, ensuring that the storage scale remains within a controllable range. Furthermore, based on the attention evaluation value of each historical word segment in the current round of answer word segment generation, historical word segments with low contribution to current and future generation are identified, and low-contribution word segments and their KV data that meet the compression requirements are preferentially removed. Thus, while ensuring response accuracy, the storage overhead of KV data is effectively reduced.

[0026] Other features and advantages of this application will be set forth in the following description and will become apparent from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 A flowchart illustrating the process of generating answer lexical units for large language models in related technologies; Figure 2 This is a schematic diagram of the KV data compression process in related technologies; Figure 3 This is another schematic diagram of the KV data compression process in related technologies; Figure 4 This is a schematic diagram of the application scenario in the embodiments of this application; Figure 5 This is an overall flowchart of a data processing method according to an embodiment of this application; Figure 6 This is a schematic diagram illustrating the determination of the number of lexical compressions based on CCM in an embodiment of this application; Figure 7 This is another schematic diagram illustrating the determination of the number of lexical compressions based on CCM in the embodiments of this application; Figure 8 This is a schematic diagram of the global semantic similarity calculation process in the embodiments of this application; Figure 9 This is a schematic diagram illustrating the calculation process of the global evaluation value of historical lexical units in this application embodiment; Figure 10 This is an overall flowchart of KV data compression in the embodiments of this application; Figure 11 This is a schematic diagram illustrating the process of KV data compression during intelligent question answering in an embodiment of this application; Figure 12 This is a structural diagram of the data processing apparatus in an embodiment of this application; Figure 13 This is a hardware structure diagram of the electronic device in the embodiments of this application; Figure 14 This is another hardware structure diagram of the electronic device in the embodiments of this application. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments of this application. Obviously, the described embodiments are only some embodiments of the technical solutions of this application, and not all embodiments. Based on the embodiments recorded in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the technical solutions of this application.

[0029] The following describes some of the concepts involved in the embodiments of this application.

[0030] (1) Question tokens are basic semantic units obtained by splitting the question sequence input by the requester through the word segmenter built into the large language model. Each question token corresponds to an initial vector representation, and corresponding KV data (i.e., K vector and V vector) are generated through the pre-trained key projection matrix and value projection matrix, which are used by the model to perform semantic retrieval and context association when generating responses.

[0031] (2) Historical lexical units refer to all lexical units saved during the multiple iterations of generating response answers in the large language model, including the original question lexical units and the answer lexical units generated in each iteration. These lexical units and their corresponding key-value data together constitute the contextual dependency chain required for model inference, which drives the generation of subsequent answer lexical units.

[0032] (3) Attention evaluation value is a numerical value calculated based on the multi-head attention mechanism to measure the semantic contribution of a certain historical word to the currently generated answer word. This value is obtained by performing a dot product operation between the query vector Q and the key vector K of each historical word.

[0033] (4) The number of word segments compressed refers to the number of historical word segments and their KV data that the system needs to clear when the total amount of KV data that has been saved reaches the preset storage threshold. This number is not fixed, but is calculated based on dynamic factors such as the ratio of the number of current problem word segments to the number of historical word segments and the similarity of recently generated content, so as to achieve adaptive memory management.

[0034] (5) Attention shift is a quantitative indicator used to measure whether the model's reasoning focus shifts from the original question sequence to the generated answer content during the generation process. This value comprehensively considers the degree of inheritance of the generated answer from the question semantics, the global contribution ratio of the answer words, and the actual integrity of the retained information. When the attention shift reaches a set threshold, it indicates that the model has the ability to autonomously continue the output based on the existing context. At this time, the question words can be included in the deletable range of KV data to further free up memory resources.

[0035] The design concept of the embodiments of this application is briefly introduced below: Current large language models need to convert each question word in the input question sequence into corresponding key-value data and save it. Then, by combining the key-value data of each question word, an attention mechanism is used to generate the corresponding response answer for the question sequence.

[0036] For ease of understanding, Figure 1 This example illustrates the process of generating response answers corresponding to question sequences using an attention mechanism in a large language model. For instance... Figure 1As shown, the trained large language model mainly includes an embedding layer, an attention layer (Transformer), and an output layer.

[0037] Large language models use a built-in tokenizer in the embedding layer to split the input question sequence into multiple question tokens. Then, each question token is feature-encoded to obtain an initial vector representation (Token) for each question token. i Next, the tokens of each question term will be... i Input to the Transformer layer using a pre-trained key projection matrix W k Sum projection matrix W v For each Token i Perform a linear transformation to generate KV data (i.e., KV) corresponding to each question term. i and V i ).

[0038] The large language model will save each question word and its corresponding key-value data, and use a multi-round iterative process to process the question sequence. In each round of iteration, an answer word will be generated until the corresponding response answer for the question sequence is generated.

[0039] For ease of explanation, this application refers to the lexical units already saved in the large language model as historical lexical units. In the first iteration, the saved historical lexical units are all question lexical units. In subsequent iterations, the saved historical lexical units include all question lexical units and answer lexical units generated in previous iterations.

[0040] In each iteration, the large language model uses the pre-trained query projection matrix W from the Transformer layer. Q Perform a dot product on the latest cached historical term t to obtain the query vector Q corresponding to historical term t. t The query vector Q t This is equivalent to the query signal for this iteration, used to tell the model what information it needs to find to generate the answer words for this round.

[0041] Next, the large language model will use the following formula (1) to calculate the attention evaluation value of each saved historical word for historical word t: (1) in, K represents the attention evaluation value of historical word i for historical word t. i K vector represents the historical word element i; i represents any saved historical word element; j represents the number of saved historical word elements.

[0042] Using the above formula (1), the attention evaluation value of each historical word i for the current historical word t can be calculated. Then, the V vectors of each saved historical word are weighted and summed, and the pre-trained decoding matrix W is used. out The weighted summation result and the attention evaluation value of each historical word for historical word t are decoded by nonlinear transformation to obtain the answer word generated in this round.

[0043] Each round of generated answer words will be treated as new historical words, and their key-value data will be saved as the driver for the next iteration. This continues until a preset iteration stop condition is met (e.g., the iteration reaches a preset number of iterations, or the answer words generated in this round are the end markers indicating the end of the iteration). Then, the answer words generated in each iteration (i.e., ...) will be... Figure 1 The answer tokens (such as answer token 1, answer token 2, etc.) shown are concatenated together to form the final response answer, which is then output and displayed through the output layer.

[0044] As the above process shows, the large language model needs to save the generated answer tokens and corresponding key-value data for each round during the question-and-answer process. Therefore, the system's memory usage will continue to increase as the question-and-answer process progresses.

[0045] To ensure rapid response and stability, the stored key-value (KV) data needs to be compressed to reduce system resource consumption. Industry-standard KV data compression strategies primarily include the following two: Method 1: KV data compression strategy based on fixed window.

[0046] When using method one, the key-value data of the most recently generated N words will be retained based on a fixed window, or the key-value data of the earliest generated N words will be discarded based on a fixed window.

[0047] like Figure 2 As shown in (1), a fixed-size compression window is pre-set at the beginning of the memory space (e.g., a cache queue). When a memory usage alarm is triggered, the cached terms and corresponding key-value data within the compression window will be cleared. For example... Figure 2 As shown in (2), a reservation window is set at the end of the cache queue. When the memory usage alarm is triggered, the words and corresponding KV data outside the reservation window will be cleared.

[0048] While this design approach is simple to implement, the window size is entirely dependent on manual settings. Setting the size too small will lead to an excessive number of discard decisions, increasing additional computational overhead. Moreover, in real-world question-answering scenarios, requesters often extend their questions based on the answer to a particular question. Discarding a segment of key-value data based on a fixed window can result in the loss of crucial contextual information, thereby affecting the accuracy of the large language model's response.

[0049] Method 2: Compress KV data using compression algorithms such as low-rank decomposition or sparsification.

[0050] When using method two, a compression algorithm is used to convert high-dimensional, high-volume KV data into low-dimensional, low-volume approximate vectors to reduce overall storage overhead.

[0051] For example Figure 3 As shown, a compression algorithm (such as a low-rank decomposition algorithm) and an initial dimensionality reduction value (such as a rank of 64, which means reducing the high-dimensional KV data to 64 dimensions) are pre-input into the encoder of the large language model. When the KV data of the word is obtained through the Transformer layer, it is combined with the pre-input compression algorithm to convert the KV data into a low-dimensional, low-data-volume approximate vector. Save it.

[0052] When a memory usage alert is triggered, instead of clearing the saved key-value data, this method performs further dimensionality reduction processing on the saved key-value data, for example, by... Transform into To free up memory space.

[0053] While this approach can alleviate the problem of context loss to some extent, its method of transforming key-value data is fixed before model deployment. This means that regardless of current memory constraints, the key-value data will be transformed into low-dimensional, low-data-volume data. Since this low-dimensional, low-data-volume transformation is essentially data compression achieved through dimensionality reduction or sparsity reduction of the original data, this method will result in varying degrees of information loss in all saved key-value data, thus affecting the response accuracy of large language models.

[0054] In view of this, this application proposes a data processing method, apparatus, device, and storage medium. In the embodiments of this application, the input question sequence is pre-divided into multiple question tokens, and the semantic key-value vector (i.e., KV data) of each question token is extracted. Then, the question sequence is processed iteratively in multiple rounds until the answer is output. In each iteration, the answer token for the current round is generated based on the KV data of the saved historical tokens (including question tokens and generated answer tokens), and this answer token is saved as a new historical token in KV, thus forming a complete contextual dependency chain. When the amount of saved KV data reaches a preset storage threshold, the required token compression amount can be dynamically calculated based on the current number of question tokens and the number of historical tokens, ensuring that the storage scale is always within a controllable range.

[0055] Furthermore, based on the attention evaluation value of each historical word in the current round of answer word generation process, historical words with low contribution to the current and future generation are identified, and low-contribution words and their KV data that meet the compression requirements are preferentially removed. This effectively reduces the storage overhead of KV data while ensuring the accuracy of the response.

[0056] Specifically, compared to using a fixed window to retain or discard key-value data, this application does not simply mechanically clear words based on the generation sequence of words. Instead, it dynamically identifies low-contribution words based on attention evaluation values, thereby avoiding the decrease in response accuracy caused by prematurely discarding key question words or important context. Especially in scenarios where the requester asks extended questions about the answer, the core semantics of the question sequence can still be preserved, ensuring the coherence and accuracy of the model output.

[0057] Compared to global compression algorithms such as low-rank decomposition or sparsification, this application does not force dimensionality reduction or sparsification of KV data, thus avoiding the general information loss caused by fixed compression. It only selectively removes low-contribution words, which significantly reduces memory usage while completely preserving the original KV data of high-contribution words, ensuring that the generation quality is not affected.

[0058] As can be seen, this application achieves efficient management of KV storage through a dynamic and adaptive compression mechanism without the need for manual window presetting or fixed compression strategies. This effectively reduces memory overhead while fully ensuring the model's response accuracy. It is particularly suitable for large language model inference deployment in environments with long sequence generation, multi-turn dialogue, and resource constraints, and has good practicality and promotional value.

[0059] The preferred embodiments of this application are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0060] like Figure 4 The diagram shown illustrates an application scenario in an embodiment of this application. The application scenario diagram includes a client device 410 and a server 420.

[0061] In this application embodiment, the client device 410 includes, but is not limited to, terminal devices with natural language interaction capabilities such as smartphones, tablets, laptops, desktop computers, intelligent voice interaction devices, vehicle terminals, and edge computing devices. Server-side 420 is a backend inference service system for deploying large language models, which can run in a local data center, a distributed computing cluster, or a cloud-based artificial intelligence platform. Server-side 420 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud service platform that provides cloud inference services, model hosting, high-performance computing, GPU / TPU acceleration, big data processing, and artificial intelligence inference engines. This application does not make specific limitations in this regard.

[0062] The client device 410 and the server 420 are connected via a communication network, which can be a wired network or a wireless network, such as 5G, Wi-Fi, or fiber optic network, to transmit the question sequence input by the requester and the response answer returned by the large language model.

[0063] It should be noted that the data processing methods in the various embodiments of this application can be executed by an electronic device, which can be a server 420 or a client device 410. That is, the method can be executed independently by the server 420 or completed collaboratively by the client device 410 and the server 420.

[0064] For example, in a scenario where the server 420 executes the process independently, the client device 410 sends the question sequence input by the requester to the server 420. The server 420 executes the data processing method described in this application based on the deployed large language model: first, it splits the question sequence into multiple question tokens and extracts the key-value (KV) data of each question token. Then, it enters a multi-round iterative generation process, generating the answer token for each round based on the KV data of the saved historical tokens, and saving the newly generated answer token and its KV data as new historical tokens.

[0065] When the total amount of saved key-value (KV) data reaches the preset storage threshold, the system dynamically calculates the number of KV elements to be compressed based on the number of current question KV elements and the number of historical KV elements. Based on the attention evaluation value of each historical KV element to the current answer KV generation process, the system prioritizes the removal of historical KV elements and their KV data with lower contribution, thereby achieving dynamic compression and efficient management of KV storage. Finally, the fully generated response answer is returned to the client device 410 for display.

[0066] In one alternative implementation, the server 420 can continuously monitor the memory usage of the KV storage during the model inference process and dynamically adjust the storage threshold based on the available memory resources of the device to adapt to the deployment requirements of different hardware environments.

[0067] In another optional implementation, the client device 410 has local inference capabilities for a lightweight large language model. The server 420 can distribute the trained attention evaluation mechanism and compression strategy to the client device 410 in the form of a model plugin or inference optimization module, so that the client device 410 can perform key-value storage compression locally, thereby reducing the dependence on network transmission and remote computing resources.

[0068] It should be noted that, Figure 4 The illustrations shown are merely examples; the number and deployment methods of client devices and servers in an actual system are unrestricted and are not specifically limited in this application embodiment. In this application embodiment, when there are multiple servers, the multiple servers can form a distributed inference cluster or constitute consensus nodes in a blockchain network; as disclosed in the data processing method of this application embodiment, key intermediate data such as attention evaluation records, compressed log storage, and context state generation can be selectively stored on the blockchain to enhance the auditability and traceability of the inference process.

[0069] Furthermore, the embodiments of this application can be widely applied to various technical scenarios that rely on large language models for efficient reasoning, including but not limited to intelligent customer service, multi-turn dialogue systems, long text generation, code generation, in-vehicle voice assistants, mobile AI applications, etc. It is especially suitable for high-concurrency, low-latency, and memory-constrained reasoning deployment environments, and has good versatility and engineering implementation value.

[0070] Here are a few typical application scenarios: (a) Efficient Reasoning in Long Contexts of Intelligent Customer Service Systems In intelligent customer service scenarios, requesters often engage in multi-turn dialogues, with questions involving complex contexts such as historical orders, service records, and product parameters. When traditional large language models process such long-sequence interactions, the continuous growth of key-value storage leads to a significant increase in response latency, and may even cause service interruptions due to memory overflow.

[0071] In this embodiment, an inference engine deployed in the customer service backend monitors the KV (Key-Value) storage scale in real time during response generation. When the storage reaches a threshold, based on the attention evaluation value of each historical word element in the current answer generation, low-contribution question or answer words elements are dynamically removed, while key contextual information is retained. For example, when the requester asks, "Why was the logistics of the last order delayed?", the system can retain high-attention words such as "order" and "logistics," compressing only early irrelevant small talk content. This significantly reduces memory usage while ensuring semantic coherence, improving response speed and service stability.

[0072] (II) Lightweight Deployment of Mobile AI Assistants

[0073] When running large language models on resource-constrained devices such as mobile phones and tablets, the limited memory capacity makes it difficult to support long text generation tasks. If a fixed window compression strategy is used, it is easy to lose the problem sequence information, causing the AI ​​assistant to forget the initial intent of the requester.

[0074] In this embodiment, the solution can be integrated into a local inference framework to accurately identify compressible historical lexical units through an attention mechanism. For example, when a requester asks, "Help me write a resignation letter explaining family reasons and remaining polite," and then follows up with, "Add a thank you to the leader," the system, when generating new content, prioritizes retaining high-attention keywords such as "resignation letter," "family reasons," and "thank you to the leader," only cleaning up intermediate transitional expressions to ensure that core instructions are not discarded. This achieves high-quality, long-range dialogue capabilities within limited memory, improving the practicality of mobile AI applications.

[0075] (III) Low-latency interaction optimization of in-vehicle voice assistant

[0076] In intelligent vehicle environments, in-vehicle voice assistants need to respond quickly to driving-related queries, such as navigation routes, vehicle status, and multimedia control, under conditions of high noise and low computing power. Excessive storage of key-value pairs increases inference latency and impacts driving safety.

[0077] This application's embodiments can be applied to in-vehicle voice systems. By dynamically adjusting the storage threshold based on the vehicle's chip performance and available memory, and selectively compressing non-critical context based on attention weights, the system can quickly clear intermediate state words from already executed actions when the requester continuously issues commands such as "change the song" or "turn down the volume." This retains only the current playback theme and the latest control intent, achieving low-latency, high-accuracy continuous interaction and ensuring smooth operation and safety during driving.

[0078] (iv) Context management of programming aids

[0079] In code compilation scenarios, developers often provide detailed context, including function signatures, comments, and calling logic, expecting the model to generate complete code blocks that conform to specifications. Such tasks typically involve long inputs and long outputs, and the key-value storage can easily become bloated.

[0080] In this embodiment, an attention evaluation mechanism can be used to distinguish between low-contribution information such as "core syntax structures" and "temporary variable names," and redundant information can be dynamically released during the generation process. For example, when generating class methods, the system prioritizes retaining high-attention terms such as class names and interface definitions, while compressing secondary descriptions in debug logs or comments. This improves generation efficiency without sacrificing code correctness and is applicable to various development tool scenarios such as IDE plugins and cloud programming assistants.

[0081] In summary, the embodiments of this application can be widely applied in technical fields such as intelligent customer service, mobile AI assistants, in-vehicle voice systems, and code generation tools, which rely on large language models for multi-turn, long-sequence generation. By introducing a dynamic KV storage and compression mechanism based on attention evaluation values, a highly efficient balance between memory overhead and generation quality is achieved, significantly improving inference efficiency while ensuring response accuracy. Furthermore, it can be combined with edge computing to deploy a lightweight attention evaluation module on the terminal side to achieve localized compression decisions; or it can integrate runtime metrics such as memory prediction and device load awareness to further optimize the compression threshold and granularity, expanding to more AI application scenarios with high requirements for inference speed, memory efficiency, and semantic fidelity. It should be noted that the application scenarios listed above are merely simple examples; other application scenarios are also applicable to the embodiments of this application, and will not be elaborated upon here.

[0082] The specific implementation of this application involves data collection and use, such as key-value data of historical terms. When the above embodiments of this application are applied to specific products or technologies, permission or consent from the relevant parties is required. Furthermore, the collection, storage, processing, and analysis of all data shall strictly comply with the privacy protection laws and regulations and data security standards of the relevant countries and regions.

[0083] See Figure 5 The flowchart shown is a data processing method provided in an embodiment of this application, which specifically includes the following steps: S501, split the question sequence into multiple question tokens and extract the semantic key-value vector of each question token; As mentioned above Figure 1 As shown, the large language model splits the input question sequence into multiple question tokens using a built-in tokenizer in the embedding layer. Then, it encodes the features of each question token to obtain an initial vector representation (Token) for each token. i Then, the tokens of each question term i Input to the Transformer layer using a pre-trained key projection matrix W k Sum projection matrix W v For each Token i Perform a linear transformation to generate KV data (i.e., KV) corresponding to each question term. i and V i ).

[0084] Specifically, the bond projection matrix W K Sum projection matrix W V This is a learnable parameter matrix obtained through large-scale corpora during the pre-training phase of the large language model. It belongs to the weight parameters within the model and is related to the query projection matrix W mentioned earlier. QTogether, they form the linear transformation component in the multi-head attention mechanism. These matrices are continuously optimized through backpropagation during model training, enabling the generated K and V vectors to effectively capture the semantic roles and information expression patterns of words in the context.

[0085] For each question word i, its corresponding key vector K i It is the token's own initial vector representation. i With W K The result of matrix multiplication, and its corresponding value vector V. i It is the token's own initial vector representation. i With W v The result of matrix multiplication. Where K i V represents the semantic index feature of this term. i This indicates the complete semantic content carried by the lexical element.

[0086] S502, perform multiple rounds of iterative processing on the question sequence until the answer is output; S503, each iteration includes the following steps S5031~S5033: S5031, based on the semantic key-value vectors of each saved historical word element, obtain the answer word element for this round; and, use the answer word element as a new historical word element, and extract and save the semantic key-value vector of the answer word element; in the first iteration, each historical word element is the multiple question word elements; As mentioned above Figure 5 As shown, in each iteration, the large language model uses the most recently saved historical lexical unit as the historical lexical unit t for this round. Then, it compares the historical lexical unit t with the pre-trained query projection matrix W in the Transformer layer. Q Perform a dot product to obtain the query vector Q for this iteration. t Then the query vector Q t Input the K vectors of each historical word that have been saved into the aforementioned attention calculation formula (1) to obtain the attention evaluation value of each historical word for the current historical word t.

[0087] Subsequently, the V vectors of each saved historical word are weighted and summed, and then the pre-trained decoding matrix W is used. out Yes, the weighted summation result and the attention evaluation value of each historical word are non-linearly transformed to obtain the answer word generated in this round. Finally, the answer word generated in this round is used as a new historical word, and this answer word and the corresponding KV data are added and saved as the driver for the next iteration.

[0088] As can be seen, in this embodiment of the application, each question word and its corresponding key-value data in the question sequence are added and saved in advance. At the same time, during the multi-round iterative processing, the answer word and its corresponding key-value data generated in each round of iteration are added and saved.

[0089] S5032, when the number of saved semantic key-value vectors reaches the storage threshold, the number of word compressions is obtained based on the current number of problem words and the number of historical words. To ensure system stability and response speed, this embodiment pre-sets a storage threshold B based on actual business scenarios. Simultaneously, key-value (KV) data is periodically monitored. When the memory usage of saved KV data (e.g., the number of KV data points) reaches the storage threshold B, or the ratio of KV data memory usage to the storage threshold B exceeds a set range, it indicates that the current storage capacity is approaching the system's preset upper limit. Continuing to accumulate new historical KV data and its corresponding tokens may lead to memory overflow or a significant increase in inference latency, affecting system stability. In this case, the number of tokens to be compressed is calculated by combining the current number of question tokens and the number of historical tokens.

[0090] It should be noted that the number of lexical compressions indicates how many historical lexicals and their corresponding key-value pairs need to be removed from the storage space.

[0091] In reasoning tasks, context complexity (CCM) directly affects the dependence of subsequently generated answer terms on the saved key-value (KV) data. The ratio of the number of question terms to the number of historical terms effectively reflects this dependence and is suitable as a metric for measuring context complexity. Therefore, the ratio of the number of saved question terms to the number of historical terms can be pre-calculated. This ratio can be used as the context complexity (CCM) for each saved historical term, and the term compression quantity can be determined based on this CCM.

[0092] Specifically, a higher ratio between the number of question terms and the number of historical terms indicates fewer currently generated answer terms, meaning subsequent new answer terms are more dependent on context. In this case, the compression of key-value (KV) data should be reduced to improve answer accuracy. Conversely, a lower ratio indicates more currently generated answer terms, meaning subsequent generation tends to be more adaptive and less dependent on KV data. In this case, the compression of KV data can be increased to reduce memory usage.

[0093] Therefore, after determining the CCM corresponding to each historical lexical unit saved in this round, a non-linear mapping function that is negatively correlated with the number of lexical units compressed can be set according to actual business needs. This function can then be used to determine the number of lexical units compressed. In this way, while ensuring response accuracy, the memory overhead can be dynamically controlled by combining CCM, avoiding problems such as context information loss and resource waste caused by fixed compression strategies.

[0094] In one alternative implementation, a nonlinear mapping function f(·) can be designed to calculate the compression intensity C, i.e., C = f(CCM). This function is used to dynamically adjust the aggressiveness of the compression, allowing the compression strategy to adapt to changes in context. For example, when CCM is high, it indicates that the problem context still has a strong influence on the generation process, and the compression intensity should be reduced to retain key information; when CCM is low, it indicates that the model has broken free from problem sequence dependencies, and the compression intensity can be appropriately increased to free up more memory.

[0095] In practice, various compression strength levels (e.g., high / medium / low) can be set based on actual business scenarios. Then, a corresponding CCM mapping range can be assigned to each compression strength. For example... Figure 6 As shown, for high compressive strength, the corresponding CCM mapping range is (0.7, 1]; for medium compressive strength, the corresponding CCM mapping range is (0.4, 0.7]; and for low compressive strength, the corresponding CCM mapping range is [0, 0.6]. This allows us to determine the interval to which the CCM belongs after calculation, and then obtain the current compressive strength based on the above mapping relationship. For example, CCM = 0.5 corresponds to medium compressive strength.

[0096] In addition, a corresponding number of words to be compressed is set for each compression intensity. For example, the number of words to be compressed is 300 for high compression intensity, 200 for medium compression intensity, and 100 for low compression intensity.

[0097] In this way, after determining the CCM corresponding to each saved historical word, the current compression intensity can be determined based on the mapping relationship between CCM and compression intensity, and then the final number of words to be compressed can be determined based on the current compression intensity. This effectively avoids excessive KV data removal, which could affect the quality of subsequent answer word generation, and also alleviates the problem of frequently discarding KV data due to insufficient KV data removal in a single operation.

[0098] In another optional implementation, the mapping relationship between compression intensity and word compression range can be set based on the memory usage of the current KV data. (Combined with the above...) Figure 6 For example, specifically as follows Figure 7As shown, a corresponding word compression range is set for each compression intensity. For example, the word compression range for high compression intensity is 200~350, the word compression range for moderate compression intensity is 100~200, and the word compression range for low compression intensity is 50~100.

[0099] When the memory usage of KV data reaches the storage threshold B, the final number of words to be compressed is determined based on the current memory usage rate of KV data and the word compression range corresponding to the current compression intensity, as shown in the following formula (2): (2) in, C represents the number of lexical units compressed. min C represents the minimum value of the word compression range corresponding to the current compression intensity. max R represents the maximum value of the word compression range corresponding to the current compression intensity; R is the memory usage rate of KV data, which is the ratio of the memory usage of KV data to the storage threshold B.

[0100] For example, when CCM=0.4 and R=0.9, the current CCM corresponds to a low compression intensity. Using the above formula (2), the number of word segments compressed can be calculated as M=50 + (100 - 50) × 0.9 = 95. In this way, not only can the compression intensity be dynamically obtained according to the CCM corresponding to each saved historical word segment, but the compression amount of KV data can also be adaptively adjusted according to the current actual memory usage, thereby improving system stability.

[0101] Considering that the compression strength in the above implementations is determined directly through CCM, in complex question-answering scenarios such as multi-turn question answering or long text question answering, the model's output answers usually contain multiple segments of content with different semantic topics. Therefore, in such scenarios, if the similarity of recently generated answer terms is high, it indicates that subsequently generated answer terms still rely on a large amount of key-value data. In this case, even if the overall proportion of question terms has significantly decreased (i.e., CCM is low), a large amount of key-value data should not be cleared, otherwise it may cause the loss of key semantic content, leading to subsequent answers deviating from the topic.

[0102] Based on this, in order to more accurately capture the semantic change trend in the generation process and improve the fine-grained control capability of the compression strategy, the semantic content of recently generated answer words can be evaluated for similarity, and the compression intensity can be dynamically adjusted in combination with the CCM corresponding to each historical word to obtain a more accurate number of word compressions.

[0103] In one optional implementation, a specified number of reference words can be obtained from each historical word; the specified number is positively correlated with the number of iterations, and each reference word is an answer word generated in an iteration. By arbitrarily combining each pair of reference words, at least one group of reference words is obtained. Then, based on the similarity between the semantic key-value vectors of the two reference words in each group of reference words, and combined with the CCM corresponding to each historical word, the word compression quantity is obtained.

[0104] Specifically, to more accurately determine whether it is safe to clean up historical word units, it is necessary to focus on the model's recent generation behavior, as the model's future output is more likely to be influenced by recently generated content than by earlier generated answer word units. Therefore, a specified number of recently saved answer word units can be selected as reference word units from the saved answer word units. The number of reference word units selected gradually increases with the number of generation rounds. For example, 3 are selected for up to 10 iterations, 5 for 10-20 iterations, and so on. This application does not limit the method of obtaining the number of reference word units.

[0105] Next, the reference lexical units are paired to obtain multiple reference lexical groups. For each reference lexical group, the Euclidean distance between the value vectors (V vectors) of the two reference lexical units within the group is calculated to measure the similarity of the two lexical units in semantic content.

[0106] After obtaining the similarity scores for each reference word group, the average of these scores is calculated to obtain the global semantic similarity score, St. St represents the overall semantic closeness between recently generated content: a larger St indicates that the semantics of the model's continuous outputs are highly consistent, with strong contextual coherence, and the current inference is still in a stable and continuous state, so historical information should be preserved as much as possible; a smaller St indicates that the semantics of the generated content tend to diverge or jump, with weak contextual consistency, and the model may have entered a free expansion phase, at which point the preservation and cleanup efforts can be appropriately increased.

[0107] Figure 8 An exemplary diagram illustrates the calculation process of global semantic similarity St, such as... Figure 8 As shown, assume the specified quantity corresponding to the current iteration round is 3. At this time, the three most recently stored reference lexical units D1, D2, and D3 can be selected from the saved historical lexical units.

[0108] Then, these three reference lexical units are paired to obtain reference lexical group 1 (D1, D2), reference lexical group 2 (D1, D3), and reference lexical group 3 (D2, D3). For each reference lexical group, the similarity is determined based on the Euclidean distance between the value vectors (V vectors) of the two reference lexical units within the group. After obtaining the similarity P1 to P3 for each of the reference lexical groups 1 to 3, the mean of the similarity results for each reference lexical group is taken as the global semantic similarity St, i.e., St = (P1 + P2 + P3) / 3.

[0109] In one alternative implementation, it can be combined with The compression strength C is calculated using the following formula (3) based on the global semantic similarity St: (3) As can be seen, in the above formula (3), exp(-St) decreases as St increases: the smaller St is, the closer exp(-St) is to 1, the greater the compression strength C is, indicating that more data can be released; the larger St is, the closer exp(-St) is to 0, the smaller the compression strength C is, indicating that the amount of KV data compression should be reduced to improve the recovery accuracy. This design can dynamically adjust the cleaning intensity according to the changing trend of the actual output content of the model, avoiding the accidental deletion of key context when semantic coherence is maintained.

[0110] After calculating the compression intensity C using the formula (3) above, the final number of compressed words can be determined according to the mapping relationship between compression intensity and word compression quantity described above. The above process evaluates the semantic similarity of recently generated answer words and incorporates both factors into the calculation, so that the compression intensity can more realistically reflect whether it is appropriate to clear more KV data, thereby ensuring the semantic coherence and accuracy of subsequently generated answer words.

[0111] The following section describes the process of determining the number of lexical units compressed based on global semantic similarity (St) and context complexity (CCM) in this application, using several practical application scenarios as examples: In e-commerce product recommendation scenarios, requests often involve multiple attribute combinations, such as "Find me a Bluetooth headset suitable for outdoor sports, with long battery life, and priced under 2000 yuan." When generating responses, the model typically addresses each condition item by item, with different semantic focuses at each stage. Because the model initially relies heavily on question information in such tasks, the Choice of Customer Competence (CCM) is high. However, as the model explains aspects such as "outdoor performance" and "battery life" item by item, the generated answer expands, and the CCM steadily decreases.

[0112] To prevent prematurely increasing the compression intensity due to a decrease in CCM during the answering process, adjustments can be made when calculating the compression intensity C. Specifically, when the currently generated answer terminology has a high semantic similarity to the key terminology in the question sequence (such as "outdoor" and "battery life"), a higher weight can be added to the CCM in the above formula (3) to reduce the number of terminology compressions. In this way, the problem of prematurely clearing the contextual information of key sections can be effectively alleviated in answering complex questions with multiple conditions as described above, thereby ensuring the coherence of the recommendation logic and improving the generation quality.

[0113] In in-vehicle voice assistant navigation scenarios, the requester typically inputs commands step by step, such as first asking "Go to the nearest gas station," and then asking "Are there any convenience stores along the way?" The system needs to maintain multi-turn task context within limited memory. Since the model output in navigation tasks is mostly short sentences with significant semantic jumps (e.g., jumping from "Turn right in 500 meters" to "There are 3 convenience stores nearby"), the semantic similarity (St) between reference word groups is usually low, leading to a high compression intensity (C). If the number of compressed words is calculated directly based on this compression intensity, it will result in a large number of compressed words, potentially prematurely removing historical words that are still needed to support subsequent responses, causing context breaks and affecting the coherence of the responses and the completeness of the task.

[0114] Therefore, in this scenario, an upper limit (e.g., 15%) can be set for the number of lexical compressions, as well as a threshold for determining whether the CCM is too large. When the CCM exceeds this threshold, the number of lexical compressions is limited to no more than 15% of the total number of currently cached answer lexical units. This avoids clearing a large amount of key-value data due to a low global semantic similarity (St), thereby mitigating the premature clearing of context corresponding to key navigation commands, making the clearing rhythm of key-value caches more stable, and thus improving the inference stability in low-resource environments in vehicles.

[0115] It should be understood that the above examples are for illustrative purposes only and are not intended to limit the specific application scenarios of the present application.

[0116] S5033, based on the attention evaluation value of each historical word element for the answer word element in this round, clear the historical word elements and their corresponding semantic key-value vectors that meet the word element compression quantity; wherein, each attention evaluation value is determined based on the semantic key-value vector of the corresponding historical word element and the answer word element, and represents the contribution of the historical word element to the generation of the answer word element in this round.

[0117] In one alternative implementation, the attention evaluation value of each historical word for the answer word in the current round can be determined in the following manner: The answer tokens generated in this iteration are pre-encoded to obtain the query vector corresponding to the answer token. Then, based on the dot product between the query vector and the semantic key vectors of each historical token, the semantic relevance between the answer token and each historical token is obtained.

[0118] Next, for each historical word, the attention evaluation value of that historical word for the answer word in this round is determined based on the proportion of its semantic relevance in the semantic relevance of all historical words.

[0119] Specifically, for the calculation process of the attention evaluation value of each historical word element for the answer word element in this round, the calculation formula (1) for the attention evaluation value of each historical word element for the latest saved historical word element t can be referred to. When calculating the attention evaluation value of each historical word element for the answer word element in this round, Q in the aforementioned formula (1) is used. t Replace with the query vector Q corresponding to the answer term x in this round. x That's it. For any historical word element, the higher its attention evaluation value for the answer word element in this round, the greater its contribution to the generation of the answer word element in this round.

[0120] This approach allows for the accurate identification and retention of key-value (KV) data for important historical words that contribute significantly to the generation of the current output answer word, based on semantic dependencies during the model's actual inference process. Conversely, it prioritizes removing KV data from historical words that contribute less. This effectively avoids off-topic answers caused by the accidental deletion of crucial contextual information, achieving a better balance between memory efficiency and generation quality. Furthermore, the attention evaluation values ​​of each historical word for the currently generated answer word, calculated during the KV data compression stage, can be cached. In the next iteration, these data can be directly retrieved from the cache as the driving force for generating the next round of answer words, thus reducing computational overhead.

[0121] To improve the stability and accuracy of the cache cleanup strategy, after obtaining the attention evaluation value of each historical word to the answer word in the current round, the attention distribution of each historical word to the answer word generated in each round during multiple iterations can be combined to conduct a more comprehensive evaluation of each historical word, thereby determining the final word to be cleared and the corresponding key-value data.

[0122] In one optional implementation, for each historical word element, the following is performed: based on the sum of the attention evaluation value of the historical word element for the answer word element in the current round and the attention evaluation value of the historical word element for the answer word elements in each of the previous iterations, the global evaluation value of the historical word element is determined.

[0123] Specifically, it can be as follows Figure 9As shown, assume the question sequence is split into question words D1~D6. Let's take this third iteration as an example: In the first iteration, the generated answer word is D7, and the historical words saved in the first round are D1~D7. Then, the attention evaluation values ​​of D1~D7 for D7 are calculated using the above formula (1), and are represented by L1.1~L1.7 respectively.

[0124] In the second iteration, the generated answer word is D8, and the historical words saved in the second round are D1~D8. Then, the attention evaluation values ​​of D1~D8 for D8 are calculated using the above formula (1), and are represented by L2.1~L2.8 respectively.

[0125] In the third iteration, the generated answer word is D9, and the historical words saved in the third round are D1~D9. Then, the attention evaluation values ​​of D1~D9 for D9 are calculated using the above formula (1), and are represented by L3.1~L3.8 respectively.

[0126] Thus, in the current iteration round (i.e., the third iteration), the sum of the attention evaluation values ​​of each historical word for the answer word generated in each iteration round is calculated.

[0127] Continue as Figure 9 As shown, the sum of attention evaluation values ​​for D1 is L1.1+L2.1+L3.1, the sum of attention evaluation values ​​for D2 is L1.2+L2.2+L3.2, ..., the sum of attention evaluation values ​​for D8 is L2.8+L3.8, and the sum of attention evaluation values ​​for D9 is L3.9.

[0128] Next, for each historical word, the ratio between the sum of the attention evaluation values ​​corresponding to that historical word and the number of rounds in which that historical word has been saved is used as the global evaluation value for that historical word. For example, if D1 has been saved for 3 rounds, its global evaluation value is (L1.1 + L2.1 + L3.1) / 3. Similarly, if d2 has been saved for 2 rounds, its global evaluation value is (L2.8 + L3.8) / 2.

[0129] The calculated global evaluation value represents the overall contribution of the historical word element to all generated answers throughout the entire generation process. Continuing with D1 as an example, its global evaluation value is (L1.1 + L2.1 + L3.1) / 3, which represents D1's average contribution to the generation of D7, D8, and D9 in the first three rounds. A lower global evaluation value indicates a smaller impact of the historical word element on the generation of subsequent answer words during the reasoning process, making it redundant information that can be safely cleaned up. Therefore, after obtaining the global evaluation value of each historical word element, each answer word element within each historical word element can be treated as a word element to be processed. Then, based on the global evaluation value of each word element to be processed, historical word elements that meet the word element compression requirements and their corresponding semantic key-value vectors are removed. For example, after determining the word element compression requirements (e.g., removing 3 historical word elements), the 3 answer word elements with the lowest global evaluation values ​​and their corresponding KV data can be removed first.

[0130] Since question terms are the starting point and basis for the entire reasoning task, the model always needs to generate subsequent answers based on the question sequence. Even if, at the current generation stage, some question terms have a low attention correlation with the latest answer terms, it only means that the current output does not directly reference that part of the content, but it does not mean that it will not be used in the future.

[0131] For example, when answering a complex question like "Please analyze the financial situation of a company and provide project recommendations," the initial analysis might focus on financial data, at which point the question-related semantics related to "project recommendations" might score lower. However, in subsequent stages, when generating answer-related semantics for "project recommendations," it's still necessary to trace back to that part of the question semantics. If these short-term associations are prematurely removed, the model will forget the task objective and fail to provide a complete answer.

[0132] Therefore, in the above KV compression process, the saved answer terms are prioritized for removal to ensure the complete preservation of the question sequence information. This design fundamentally ensures that the model's understanding of the task intent is not compromised, effectively preventing off-topic answers, task omissions, or logical breaks caused by accidental deletion of input information. This allows the compression strategy to release memory while maintaining generation quality.

[0133] It should be noted that the above implementation method (i.e. Figure 9In the process illustrated, the global evaluation value of each historical word element includes attention calculation for each newly added historical word element and itself (i.e., the answer word element generated in that round). For example, in the first round, the attention evaluation value of the newly added historical word element d1 with itself is calculated. The advantage of this design is that, for the answer word element, its attention calculation with itself can reflect the semantic strength of that word element. Especially in the early stages of generation, the question word element may not have been fully referenced, but the high attention of the newly generated answer word element with itself can serve as a supplementary signal for its semantic stability, helping to improve the semantic fidelity during the compression process.

[0134] Considering that in some scenarios, more attention should be paid to the actual impact of historical word units on future generated content rather than their own semantic strength, this application also provides an alternative implementation method: in each iteration, the attention evaluation value of newly generated answer word units is not calculated together with their own. (Continuing with the aforementioned...) Figure 9 For example, in the first iteration of this embodiment, only the attention evaluation values ​​of D1~D6 on d1 are calculated; in the second iteration, only the attention evaluation values ​​of D1~d1 on d2 are calculated, and so on.

[0135] At this point, the attention evaluation values ​​of each historical word unit only reflect its contribution to subsequent non-self generation steps. Compared to Figure 9 The advantage of this embodiment, as shown in the process, is that it can more accurately measure the long-term influence of a lexical unit and avoid the retention of redundant newly generated content due to excessive self-attention. It is especially suitable for tasks such as multi-turn dialogue and continuous reasoning, in which the model needs to continuously advance semantic development rather than repeatedly reinforcing existing content, thereby achieving more forward-looking cache cleanup.

[0136] It should be understood that when obtaining the global evaluation value of each historical word, whether to introduce attention calculation between each newly stored historical word and itself can be selected based on the different needs of semantic stability and contextual progression efficiency in the actual business scenario. For tasks that emphasize key information locking and semantic coherence (such as long text summarization and code generation), a scheme that introduces self-attention can be adopted. However, for tasks that emphasize reasoning evolution and efficient memory release (such as high-frequency interactive dialogue and in-vehicle voice assistants), a scheme that does not introduce self-attention can be adopted to achieve more flexible and adaptable KV data compression, and this application does not limit this.

[0137] The following section describes the KV cache compression process based on global evaluation values ​​in this application, using several practical application scenarios as examples: In e-commerce customer service Q&A scenarios, key-value caching compression needs to address the characteristics of complex requests and repetitive responses. Requesters often ask multiple related questions consecutively within a single session, such as "The wrong item I ordered was sent, how do I return or exchange it? How is the shipping cost calculated?". During the model's step-by-step answering process, intermediate content generated (such as "You can apply for an exchange service") may not be referenced in subsequent rounds, resulting in redundant caching.

[0138] In this scenario, the system still uses the global evaluation value as the core indicator, prioritizing the removal of saved answer words with the lowest global evaluation value. Simultaneously, considering that requesters in e-commerce conversations focus on efficiency, and responses are often short and information-dense, the system can be further configured to prioritize removing answer words generated earlier when global evaluation values ​​are similar. For example, when two answer words have global evaluation values ​​in the lowest range, the system prioritizes removing opening phrases like "Hello, I'm sorry" generated in the first round. This strategy, while ensuring that question words are always retained, accelerates the release of early low-value information, adapting to the dual demands of e-commerce scenarios for response speed and resource utilization.

[0139] In the context of in-vehicle voice assistants, the compression of key-value (KV) caches needs to adapt to the short-duration, high-frequency interactions and safety-first principles in the driving environment. The requester typically issues commands in steps while driving, such as first asking "Open the window," then "Set the air conditioning temperature to 22 degrees," and finally "How do I get to the nearest gas station?"

[0140] Such tasks are highly time-dependent and context-dependent; semantic loss at any stage can lead to erroneous operations. Therefore, when performing key-value compression, the system adheres to the basic principle of not clearing problem terms, and only cleans up answer terms after sorting them according to their global evaluation values. Simultaneously, considering the limited memory resources of in-vehicle devices and the unacceptability of information corruption during driving, a compression upper limit threshold can be set, for example, the number of terms compressed in a single operation should not exceed 20% of the total number of generated answer terms, to avoid contextual breaks caused by large-scale cleanup.

[0141] Furthermore, in multi-round navigation or continuous control tasks, when the global evaluation values ​​of multiple answer terms are similar, the answer term furthest from the current round can be prioritized for removal to ensure that the most recently generated information is retained. This strategy maintains task continuity while achieving controllable and predictable memory usage.

[0142] It should be understood that the above examples are for illustrative purposes only and are not intended to limit the specific application scenarios of the present application.

[0143] Considering that in real-world business scenarios, the dependence of subsequently generated answer terms on context changes as the answer responses unfold. Early-generated answer terms primarily rely on the key-value (KV) data of the question sequence. As answer terms are generated, the core semantics of the question sequence are gradually reflected in the answer terms, becoming the actual contextual basis for subsequent reasoning. That is, the longer the iteration, the more it relies on the KV data of already saved answer terms to generate new answer terms. Based on this pattern, to more accurately determine whether question terms still need to be retained, this application introduces attention shift as a dynamic metric to measure the degree to which the model's reasoning focus shifts from the input question to the generated content.

[0144] In one alternative implementation, before treating each answer term in each historical term as a term to be processed, an attention shift of the question sequence can be obtained based on the global evaluation value of each historical term and the number of answer terms in each historical term. This attention shift represents the degree of influence of the question sequence on the answer terms generated in subsequent iterations.

[0145] Specifically, as the number of iterations increases, the number of generated answer terms continues to grow. When generating new content, the model increasingly references previously generated answer terms rather than primarily relying on the question sequence. This means that the dominant role of the question sequence in the current context may gradually be replaced by the generated semantic links.

[0146] Based on this, the attention shift of the question sequence can be obtained by combining the global evaluation value of each historical word and the number of generated answer words. This attention shift can measure whether the overall influence of question words on the subsequently generated answer words is still critical.

[0147] In one alternative implementation, the attention shift of the problem sequence can be determined in the following way: For each answer term, the following steps are performed: Obtain the semantic key-value vector of the answer term and the average similarity between it and the semantic key-value vectors of all question terms. Then, based on the obtained average similarity values, combined with the global evaluation value of each question term and its proportion in the global evaluation value of all historical terms, the comprehensive semantic carrying capacity of each answer term is obtained. Finally, based on this comprehensive semantic carrying capacity, the number of answer terms, and the number of iterations, the attention shift of the question sequence is obtained.

[0148] In practice, the attention shift AOS of the problem sequence can be calculated using the following formula (4): (4) Among them, A qut This represents the sum of the global evaluation values ​​of all saved question terms, A. aserThe sum of the global evaluation values ​​of all saved answer tokens is represented by lb, which represents the logarithm to the base 2 (log2), and L0 represents the number of iterations. qut Sq represents the number of saved answer tokens, and Sq represents the global average similarity of each answer token. The calculation process of Sq is as follows: pre-calculate the V vector of each answer token and the average similarity between it and the V vector of each question token, and then calculate the average similarity of each answer token to obtain Sq.

[0149] To facilitate understanding, the meaning of formula (4) will be explained in detail below: As mentioned earlier, the early-generated answer tokens mainly rely on the key-value (KV) data of the question sequence. As answer tokens are gradually generated, the core semantics of the question sequence are gradually reflected in the answer tokens, becoming the actual contextual basis for subsequent reasoning. That is, the longer the iteration, the more it relies on the KV data of the saved answer tokens to generate new answer tokens.

[0150] Therefore, it can be inferred that when there are enough saved answer words and the saved answer words carry enough core semantics of the question sequence, it means that the model's attention shifts from the question sequence to the saved answer words, and the newly generated answer words depend more on the key-value data of the saved answer words.

[0151] The expression on the left side of the above formula (4) is " "This represents the total semantic content carried by each answer word, indicating the degree to which each answer word reflects the semantic content of the question sequence. The larger this value, the more core semantics of the question sequence are carried by the saved answer words."

[0152] The expression on the right side of the above formula (4) is " This is used to impose dual constraints on the generated length and the preservation of integrity. Since one answer term is generated per iteration, but some answer terms may be deleted due to KV compression, the actual number of answer terms saved, L, is limited. qut ≤ L0.

[0153] Under these conditions, "L" qut The range of values ​​for " / L0" is [0, 1], therefore "1 + L" is... qut The value range of " / L0" is [1, 2]. This design means that when the model has not only experienced a sufficient number of generation rounds (i.e., L0 is large), but also the actual number of retained answer tokens is close to the total number of generated tokens (i.e., L... qut near When ), lb(1 + Lq ut / L0) will then approach 1. If a large number of generated answer words are removed due to frequent compression, it will lead to Lqut Much smaller than L0, at this time lb(1 + L) qut The value of / L0) will be much less than 1, thus suppressing the overall growth of AOS. This design is limited to ensuring that the model has output a sufficiently long set of answer terms before compressing the question terms and the corresponding key-value data, and that these answer terms have not been excessively cleaned up.

[0154] Therefore, the computational mechanism of this Attention Shift (AOS) can accurately identify the evolution trend of the model's dependence on context during dynamic inference. By comprehensively evaluating the degree of inheritance of answer terms from question semantics, their dominance in attention contribution, and the completeness of actual retention, it ensures that question terms are only considered for removal when the answer content has been fully expanded and the semantic link is stable. This design effectively avoids the accidental deletion of input information in the early stages of generation or in the context-sparse stage, preventing off-topic answers, task interruptions, or logical breaks due to the loss of key semantics. Simultaneously, in high-memory-consumption scenarios such as long sequence generation, it provides a safe and quantifiable basis for the efficient cleaning of KV cache, significantly improving the model's memory utilization efficiency and operational stability in multi-turn interactions and continuous inference.

[0155] When the attention offset (AOS) of the question sequence does not reach the offset threshold, it means that the model still highly depends on the question sequence for the generation of subsequent answer words, and the question words still play a key role in the reasoning process. At this time, in order to ensure the consistency between the generated content and the question sequence and avoid task deviation or information omission, each answer word in each historical word is treated as a word to be processed, and the word to be processed that meets the word compression requirement and the corresponding KV data are removed.

[0156] This ensures that the semantic key-value vectors corresponding to the question sequence are completely preserved throughout the early and middle stages of generation, effectively maintaining the model's accurate understanding of the task intent, preventing off-topic answers or logical breaks caused by prematurely clearing input information, and guaranteeing the stability and semantic coherence of the generation process.

[0157] Correspondingly, when AOS reaches the offset threshold, it means that the model has generated enough answer content and its reasoning process mainly relies on the generated answer word sequence. The influence of question words on the current and subsequent generation has been significantly reduced. The model has the ability to autonomously continue the output based on the existing context. At this time, each historical word can be treated as a word to be processed, and the word to be processed and the corresponding KV data that meet the word compression requirements can be removed.

[0158] Therefore, while ensuring that the generation quality is not affected, the scope of KV data that can be cleaned is further expanded. Question words that are weak in generating subsequent answer words can be included in the compression scope. This method is particularly suitable for scenarios that require continuous release of memory resources, such as long text generation, and can further alleviate the memory pressure caused by excessively long context.

[0159] It's important to note that large language model architectures typically contain multiple Transformer layers. Each layer performs different semantic extraction functions during inference and maintains a memory space storing all historical lexical units and their corresponding key-value (KV) data. Simply put, without KV data compression, each Transformer layer stores all question lexical units and their corresponding KV data from the question sequence, as well as the answer lexical units and their corresponding KV data generated in each iteration.

[0160] The process mentioned above in this application, which involves removing unprocessed words and corresponding key-value data from each historical word based on the attention evaluation value of each historical word to the current answer word, can be executed within the memory space of each Transformer layer. That is, step S5033 is executed separately for the memory space of each Transformer layer to achieve key-value data compression.

[0161] Furthermore, to reflect the differences in importance between different layers, in one optional implementation, corresponding layer weights can be set for each layer, and these layer weights directly affect the compression intensity C. Since the built-in parameters of higher-level models directly participate in the generation and decision-making of answer tokens, while lower layers closer to the input layer focus more on semantic understanding of the basic syntax and local structure of the saved key-value (KV) data, this means that more KV data should be preserved in higher layers, thereby improving the quality of subsequent answer token generation.

[0162] Therefore, corresponding layer weights can be set for each layer based on actual business needs, and the layer weights of each layer are negatively correlated with the number of layers. In specific implementation, the aforementioned compression strength calculation formula C=f(CCM)×K can be used; where K is the layer weight of the Transformer layer.

[0163] Based on the above process, it can be seen that because the layer weights of lower Transformer layers are larger, the calculated compression intensity is greater, and more low-contribution historical lexical units and their corresponding key-value pairs are actively cleaned up, significantly reducing the memory usage of this layer. Higher Transformer layers have smaller layer weights, and therefore the calculated compression intensity is smaller, only deleting a very small number of historical lexical units and their corresponding key-value pairs with extremely low global evaluation values.

[0164] In this way, based on the functional characteristics of different Transformer layers, a layered compression approach can be adopted to apply different compression intensities to different layers, thereby realizing a multi-level key-value data compression strategy. This strategy can significantly reduce the memory overhead of lower Transformer layers without affecting inference accuracy, while ensuring the information integrity of key-value data in higher Transformer layers, thereby improving the memory utilization and inference stability of large language models.

[0165] Figure 10 An exemplary embodiment of this application illustrates the process of compressing each historical word and its corresponding key-value pair (KV) data based on the global evaluation value of each historical word, including the following steps: S1001, for each historical word element, perform the following: determine the global evaluation value of the historical word element based on the sum of the attention evaluation value of the historical word element for the answer word element in the current round and the attention evaluation value of the historical word element for the answer word element in each of the previous iterations; S1002, based on the global evaluation value of each historical word and the number of answer words in each historical word, the attention shift of the question sequence is obtained; S1003, determine whether the attention offset reaches the offset threshold; S1004, when the offset threshold is not reached, each answer word in each historical word is treated as a word to be processed; S1005, when the offset threshold is reached, each historical word character is treated as a word character to be processed; S1006. According to the global evaluation value of each word to be processed from low to high, each word to be processed and its corresponding KV data are cleared in sequence until the word to be processed that meets the word compression requirement is cleared.

[0166] The above process achieves dynamic adaptive compression of historical lexical units and corresponding key-value (KV) data. This mechanism, while preserving the semantic integrity of the question sequence, rationally determines whether question lexical units should still be retained based on the changing trend of attention distribution during model generation. When the attention shift is below a threshold, question lexical units are strictly protected from being removed; only those with low contribution are removed from the answer lexical units. Once the model's reasoning focus shifts, question lexical units are included in the scope of removal. The entire process achieves a dynamic balance between generation quality and memory efficiency, preventing answer deviation or task interruption due to premature removal of question lexical units, while also promptly removing question lexical units with low impact on subsequent generation after the reasoning focus shifts, thus preserving high-value KV data and further improving the continuity and stability of model reasoning.

[0167] For ease of understanding, Figure 11An exemplary embodiment of the present application illustrates the overall process of KV data compression during intelligent question answering. For example... Figure 11 As shown in the embodiment of this application, the question sequence input by the requester is obtained in advance, and the large language model will split the input question sequence into multiple question tokens D1~DN through the word segmenter built into the embedding layer.

[0168] Then, the question terms are input into the Transformer layer for feature encoding, and a pre-trained key projection matrix W is used. k Sum projection matrix W v A linear transformation is performed on the initial vector representation of each question term to generate key-value (KV) data corresponding to each question term. Each question term is then treated as a historical term and saved along with its corresponding KV data.

[0169] Next, the question sequence is processed through multiple iterations. In each iteration, the large language model uses the pre-trained query projection matrix W from the Transformer layer. Q Perform a dot product operation on the latest cached historical terms to obtain the query vector Q. t Then, the decoding matrix W is used. out For the query vector Q t The KV data of each cached historical word are nonlinearly transformed and decoded to obtain the answer word d for each round.

[0170] During this period, if a memory alarm is triggered, i.e., the memory usage (i.e. the number of KV data) reaches the storage threshold, the context complexity (CCM) corresponding to each historical word is determined based on the number of problem words and the number of historical words, and the word compression quantity (M) is obtained based on the CCM.

[0171] Furthermore, based on the attention evaluation value L of each historical word element to the answer word elements generated in each iteration, the global evaluation value of each historical word element is determined. Then, from each historical word element, the M words with the lowest global evaluation values ​​and their corresponding key-value pairs are deleted, thereby completing the compression of the key-value pairs.

[0172] The above process accumulates the attention evaluation values ​​of historical lexical units to answer lexical units in each iteration and calculates the global evaluation value. This allows the system to accurately identify historical lexical units that contribute little to the overall generation process. Removing these low-contribution lexical units and their key-value (KV) data effectively releases memory resources while ensuring generation quality. The entire process balances inference efficiency with generation coherence, avoiding the loss of key information due to uniform truncation or random deletion, while maintaining low memory usage in long-sequence generation scenarios. This improves the stability and scalability of large language models in practical applications.

[0173] Based on the same inventive concept, embodiments of this application also provide a data processing apparatus, specifically as follows: Figure 12 As shown, the device may include: The semantic extraction unit 1201 is configured to: split the question sequence into multiple question tokens and extract the semantic key-value vector of each question token; Question response unit 1202 is configured to perform multiple rounds of iterative processing on the question sequence until a response answer is output; Each iteration includes the following processes: Based on the saved semantic key-value vectors of each historical word, the answer word for this round is obtained; and the answer word is used as a new historical word, and the semantic key-value vector of the answer word is extracted and saved; in the first iteration, each historical word is the multiple question words; When the number of saved semantic key-value vectors reaches the storage threshold, the number of word compressions is obtained based on the current number of problem words and the number of historical words. Based on the attention evaluation value of each historical word for the answer word in this round, historical words that meet the word compression quantity and their corresponding semantic key-value vectors are removed; wherein, each attention evaluation value is determined based on the semantic key-value vector of the corresponding historical word and the answer word, and represents the contribution of the historical word to the generation of the answer word in this round.

[0174] In one optional implementation, the step of obtaining the word compression quantity based on the current number of question words and the number of historical words is performed. Specifically, the question answering unit 1202 is configured as follows: Based on the ratio between the number of problem lexical units and the number of historical lexical units, the context complexity corresponding to each historical lexical unit is obtained. The number of lexical compressions is obtained based on the context complexity; wherein the context complexity is negatively correlated with the number of lexical compressions.

[0175] In one optional implementation, the question answering unit 1202 is specifically configured to obtain the number of lexical compressions based on the context complexity, wherein the process is performed as follows: From the historical lexical units, a specified number of reference lexical units are obtained; the specified number is positively correlated with the number of iterations, and each reference lexical unit is an answer lexical unit generated in the iterations. By combining any two reference lexical units, at least one reference lexical group can be obtained; The number of lexical compressions is obtained based on the similarity between the semantic key vectors of the two reference lexical units in each reference lexical unit, combined with the context complexity.

[0176] In one optional implementation, the attention evaluation value based on each historical word for the current round's answer word is executed, and historical words that meet the word compression quantity and their corresponding semantic key-value vectors are cleared. Specifically, the question answering unit 1202 is configured as follows: For each historical word element, the following steps are performed: Based on the attention evaluation value of the historical word element for the answer word element in the current round and the sum of the attention evaluation values ​​of the historical word element for the answer word elements in each of the previous iterations, the global evaluation value of the historical word element is determined. Each answer term in the aforementioned historical terminology is treated as a terminology to be processed. Based on the global evaluation value of each word to be processed, historical words and their corresponding semantic key-value vectors that meet the specified word compression quantity are removed.

[0177] In an optional implementation, before performing the step of treating each answer term in the historical terminology as a terminology to be processed, the question answering unit 1202 is further configured to: Based on the global evaluation value of each historical word, and combined with the number of answer words in each historical word, the attention shift of the question sequence is obtained; the attention shift represents the degree of influence of the question sequence on the answer words generated in subsequent iterations. It was determined that the attention offset of the problem sequence did not reach the offset threshold.

[0178] In one optional implementation, the attention shift of the question sequence is obtained by performing the global evaluation based on the individual historical lexical units and combining the number of answer lexical units in each historical lexical unit. Specifically, the question answering unit 1202 is configured as follows: For each answer word in each of the historical word elements, the following steps are performed: obtain the semantic key value vector of the answer word element and the average similarity between it and the semantic key value vector of each question word element; Based on the obtained average similarity values, combined with the global evaluation value of each question word element and its proportion in the global evaluation value of each historical word element, the comprehensive semantic carrying capacity of each answer word element is obtained; the comprehensive semantic carrying capacity represents the degree to which each answer word element reflects the semantic content of the question sequence. Based on the comprehensive semantic carrying capacity and the number of answer lexical units, combined with the number of iterations, the attention shift is obtained.

[0179] In an alternative implementation, the question answering unit 1202 is further configured to: When the attention offset reaches the offset threshold, each historical word is treated as a word to be processed.

[0180] In one optional implementation, the attention evaluation value of each historical word element for the answer word element in the current round is determined by the following method: The answer words are feature-encoded to obtain the query vector corresponding to the answer words. Based on the dot product between the query vector and the semantic key vector of each historical word, the semantic relevance between the answer word and each historical word is obtained respectively. For each historical word element, the following steps are performed: based on the proportion of the semantic relevance of the historical word element in the semantic relevance of each historical word element, the attention evaluation value of the historical word element for the answer word element in this round is determined.

[0181] After introducing the data processing method and apparatus according to exemplary embodiments of this application, an electronic device according to another exemplary embodiment of this application will be described next.

[0182] Those skilled in the art will understand that various aspects of this application can be implemented as a system, method, or program product. Therefore, various aspects of this application can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, collectively referred to herein as a "circuit," "module," or "system."

[0183] Based on the same inventive concept as the above-described method embodiments, this application also provides an electronic device, which can also be a server, such as... Figure 4 The server-side 420 is shown. In an alternative embodiment, the electronic device can be structured as follows: Figure 13 As shown, it includes a memory 1301, a communication module 1303, and one or more processors 1302.

[0184] The memory 1301 is used to store computer programs executed by the processor 1302. The memory 1301 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and programs required to run instant messaging functions, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0185] Memory 1301 may be volatile memory, such as random-access memory (RAM); memory 1301 may also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1301 may be any other medium capable of carrying or storing a desired computer program having the form of instructions or data structures and accessible by a computer, but is not limited thereto. Memory 1301 may be a combination of the above-described memories.

[0186] The processor 1302 may include one or more central processing units (CPUs) or digital processing units, etc. The processor 1302 is used to implement the above-described data processing method when it calls a computer program stored in the memory 1301.

[0187] The communication module 1303 is used to communicate with playback devices and other servers.

[0188] This application embodiment does not limit the specific connection medium between the memory 1301, communication module 1303, and processor 1302. This application embodiment... Figure 13 The memory 1301 and the processor 1302 are connected via a bus 1304, and the bus 1304 is in Figure 13 The diagram uses thick lines to describe the connections between other components; these are for illustrative purposes only and should not be considered limiting. The 1304 bus can be divided into address bus, data bus, control bus, etc. For ease of description, Figure 13 It is described using only a thick line, but does not indicate that there is only one bus or one type of bus.

[0189] The memory 1301 stores a computer storage medium, which stores computer-executable instructions for implementing the data processing method of this application embodiment. The processor 1302 is used to execute the above-described data processing method.

[0190] In another alternative implementation, the electronic device can also be other electronic devices, such as... Figure 4 The client device 410 is shown. In this embodiment, the electronic device can be structured as follows: Figure 14 As shown, it includes components such as: communication component 1410, memory 1420, display unit 1430, camera 1440, sensor 1450, audio circuit 1460, Bluetooth module 1470, processor 1480, etc.

[0191] The communication component 1410 is used to communicate with the server. In some embodiments, it may include a Circuit-Wireless Fidelity (WiFi) module, which is a short-range wireless transmission technology, and the electronic device can help the requester send and receive information through the WiFi module.

[0192] The memory 1420 can be used to store software programs and data. The processor 1480 executes various functions of the client device 410 and performs data processing by running the software programs or data stored in the memory 1420. The memory 1420 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. The memory 1420 stores an operating system that enables the client device 410 to run. In this application, the memory 1420 may store the operating system and various application programs, and may also store computer programs that execute the data processing methods disclosed in the embodiments of this application.

[0193] The display unit 1430 can also be used to display information input by the requester or information provided to the requester, as well as various menus of the client device 410, in a graphical user interface (GUI). Specifically, the display unit 1430 may include a display screen 1432 disposed on the front of the client device 410. The display screen 1432 may be configured as a liquid crystal display, a light-emitting diode, or the like. The display unit 1430 can be used to display answers to questions in the embodiments of this application.

[0194] The display unit 1430 can also be used to receive input numerical or character information and generate signal inputs related to the requester settings and function control of the client device 410. Specifically, the display unit 1430 may include a touch screen 1431 disposed on the front of the client device 410, which can collect touch operations on or near the requester, such as clicking a button, dragging a scroll bar, etc.

[0195] The touchscreen 1431 can be placed on top of the display screen 1432, or the touchscreen 1431 and the display screen 1432 can be integrated to realize the input and output functions of the client device 410. After integration, it can be referred to as a touch display screen. In this application, the display unit 1430 can display the application and the corresponding operation steps.

[0196] Camera 1440 can be used to capture still images, and the requesting party can publish the images captured by camera 1440 through an application. There can be one or multiple cameras 1440. An object is projected onto a photosensitive element through a lens, generating an optical image. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to processor 1480 to be converted into a digital image signal.

[0197] The playback device may also include at least one sensor 1450, such as an accelerometer 1451, a proximity sensor 1452, a fingerprint sensor 1453, and a temperature sensor 1454. The playback device may also be equipped with a gyroscope, barometer, hygrometer, thermometer, infrared sensor, light sensor, motion sensor, etc.

[0198] Audio circuitry 1460, speaker 1461, and microphone 1462 provide an audio interface between the requester and client device 410. Audio circuitry 1460 converts received audio data into electrical signals, transmits them to speaker 1461, and then converts them into sound signals for output. Client device 410 may also be configured with volume buttons for adjusting the volume of the sound signal. On the other hand, microphone 1462 converts collected sound signals into electrical signals, which are then received by audio circuitry 1460, converted into audio data, and output to communication component 1410 for transmission to, for example, another client device 410, or to memory 1420 for further processing.

[0199] The Bluetooth module 1470 is used to interact with other Bluetooth devices that also have a Bluetooth module via the Bluetooth protocol. For example, a playback device can establish a Bluetooth connection with a wearable electronic device (such as a smartwatch) that also has a Bluetooth module through the Bluetooth module 1470, thereby exchanging data.

[0200] The processor 1480 is the control center of the playback device, connecting various parts of the terminal via various interfaces and lines. It executes various functions and processes data by running or executing software programs stored in the memory 1420 and calling data stored in the memory 1420. In some embodiments, the processor 1480 may include one or more processing units; the processor 1480 may also integrate an application processor and a baseband processor, wherein the application processor mainly handles the operating system, requester interface, and applications, and the baseband processor mainly handles wireless communication. It is understood that the baseband processor may not be integrated into the processor 1480. In this application, the processor 1480 can run the operating system, applications, requester interface display and touch response, and the data processing method of the embodiments of this application. Furthermore, the processor 1480 is coupled to the display unit 1430.

[0201] In an alternative implementation, various aspects of the data processing method provided in this application may also be implemented as a program product comprising a computer program that, when run on an electronic device, causes the electronic device to perform the steps of the data processing method according to the various exemplary embodiments of this application described above.

[0202] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0203] The program product of the embodiments of this application may employ a portable compact disc read-only memory (CD-ROM) and include a computer program, and may run on an electronic device. It should be understood that the program product of this application is not limited thereto. In this application, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with a command execution system, apparatus, or device.

[0204] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying a readable computer program. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with a command execution system, apparatus, or device.

[0205] The computer program contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. The computer program for performing the operations of this application may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc.—and conventional procedural programming languages—such as the "C" language or similar programming languages. The computer program may be executed entirely on the requesting electronic device, partially on the requesting electronic device, as a standalone software package, partially on the requesting electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server.

[0206] In cases involving remote electronic devices, the remote electronic devices can be connected to the requesting electronic device via any type of network, including a local area network (LAN) or a wide area network (WAN), or they can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0207] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0208] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0209] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing a computer-usable computer program.

[0210] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, produce a mechanism for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0211] These computer program commands can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing the commands executed on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0212] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0213] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: The question sequence is split into multiple question tokens, and the semantic key-value vector of each question token is extracted; The question sequence is iterated multiple times until a response is output. Each iteration includes the following processes: Based on the saved semantic key-value vectors of each historical word, the answer word for this round is obtained; and the answer word is used as a new historical word, and the semantic key-value vector of the answer word is extracted and saved; in the first iteration, each historical word is the multiple question words; When the number of saved semantic key-value vectors reaches the storage threshold, the number of word compressions is obtained based on the current number of problem words and the number of historical words. Based on the attention evaluation value of each historical word for the answer word in this round, historical words that meet the word compression quantity and their corresponding semantic key-value vectors are removed; wherein, each attention evaluation value is determined based on the semantic key-value vector of the corresponding historical word and the answer word, and represents the contribution of the historical word to the generation of the answer word in this round.

2. The method according to claim 1, characterized in that, The process of obtaining the number of compressed lexical units based on the current number of problematic lexical units and the historical number of lexical units includes: Based on the ratio between the number of problem lexical units and the number of historical lexical units, the context complexity corresponding to each historical lexical unit is obtained. The number of lexical compressions is obtained based on the context complexity; wherein the context complexity is negatively correlated with the number of lexical compressions.

3. The method according to claim 2, characterized in that, The process of obtaining the number of lexical compressions based on the context complexity includes: From the historical lexical units, a specified number of reference lexical units are obtained; the specified number is positively correlated with the number of iterations, and each reference lexical unit is an answer lexical unit generated in the iterations. By combining any two reference lexical units, at least one reference lexical group can be obtained; The number of lexical compressions is obtained based on the similarity between the semantic key vectors of the two reference lexical units in each reference lexical unit, combined with the context complexity.

4. The method according to claim 1, characterized in that, The step of clearing historical lexical units and their corresponding semantic key-value vectors that meet the specified lexical compression quantity, based on the attention evaluation value of each historical lexical unit for the answer lexical unit in the current round, includes: For each historical word element, the following steps are performed: Based on the attention evaluation value of the historical word element for the answer word element in the current round and the sum of the attention evaluation values ​​of the historical word element for the answer word elements in each of the previous iterations, the global evaluation value of the historical word element is determined. Each answer term in the aforementioned historical terminology is treated as a terminology to be processed. Based on the global evaluation value of each word to be processed, historical words and their corresponding semantic key-value vectors that meet the specified word compression quantity are removed.

5. The method according to claim 4, characterized in that, Before treating each answer term in the historical terminology as a terminology to be processed, the method further includes: Based on the global evaluation value of each historical word, and combined with the number of answer words in each historical word, the attention shift of the question sequence is obtained; the attention shift represents the degree of influence of the question sequence on the answer words generated in subsequent iterations. It was determined that the attention offset of the problem sequence did not reach the offset threshold.

6. The method according to claim 5, characterized in that, The process of obtaining the attention shift of the question sequence based on the global evaluation value of each historical word and the number of answer words in each historical word includes: For each answer word in each of the historical word elements, the following steps are performed: obtain the semantic key value vector of the answer word element and the average similarity between it and the semantic key value vector of each question word element; Based on the obtained average similarity values, combined with the global evaluation value of each question word element and its proportion in the global evaluation value of each historical word element, the comprehensive semantic carrying capacity of each answer word element is obtained; the comprehensive semantic carrying capacity represents the degree to which each answer word element reflects the semantic content of the question sequence. Based on the comprehensive semantic carrying capacity and the number of answer lexical units, combined with the number of iterations, the attention shift is obtained.

7. The method according to claim 5 or 6, characterized in that, The method further includes: When the attention offset reaches the offset threshold, each historical word is treated as a word to be processed.

8. The method according to any one of claims 1-6, characterized in that, The attention evaluation value of each historical term for the answer term in this round is determined in the following way: The answer words are feature-encoded to obtain the query vector corresponding to the answer words. Based on the dot product between the query vector and the semantic key vector of each historical word, the semantic relevance between the answer word and each historical word is obtained respectively. For each historical word element, the following steps are performed: based on the proportion of the semantic relevance of the historical word element in the semantic relevance of each historical word element, the attention evaluation value of the historical word element for the answer word element in this round is determined.

9. A data processing apparatus, characterized in that, The device includes: The semantic extraction unit is configured to split the question sequence into multiple question tokens and extract the semantic key-value vector of each question token. The question response unit is configured to perform multiple rounds of iterative processing on the question sequence until a response answer is output. Each iteration includes the following processes: Based on the saved semantic key-value vectors of each historical word, the answer word for this round is obtained; and the answer word is used as a new historical word, and the semantic key-value vector of the answer word is extracted and saved; in the first iteration, each historical word is the multiple question words; When the number of saved semantic key-value vectors reaches the storage threshold, the number of word compressions is obtained based on the current number of problem words and the number of historical words. Based on the attention evaluation value of each historical word for the answer word in this round, historical words that meet the word compression quantity and their corresponding semantic key-value vectors are removed; wherein, each attention evaluation value is determined based on the semantic key-value vector of the corresponding historical word and the answer word, and represents the contribution of the historical word to the generation of the answer word in this round.

10. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of any of the methods described in claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It includes a computer program that, when run on an electronic device, causes the electronic device to perform the steps of any of the methods described in claims 1 to 8.

12. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of any one of claims 1 to 8.

Images