A data processing method and server

Abstract

The application relates to a data processing method and a server, and relates to the technical field of large models. The method is applied to a computer device, the computer device is integrated with a target large language model including at least two cascaded feature extraction modules and a KV compression module, and the method comprises the following steps: inputting a target prompt text into each feature extraction module to obtain local KV of different target tokens output by each feature extraction module and attention weights of different local KV; for each feature extraction module, performing compression on different local KV of the feature extraction module according to the compression ratio of the feature extraction module and the attention weights of the different local KV through the KV compression module to obtain a target KV compression result corresponding to the feature extraction module; and inputting each target KV compression result into the corresponding feature extraction module to obtain a target response text of the target prompt text. According to the technical scheme, redundant calculation in the reasoning process can be reduced, and the reasoning speed and reasoning accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of large model technology, and in particular to a data processing method and server. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have made significant progress. They can process user-input prompts and output corresponding response texts to meet various user needs, such as knowledge retrieval, text summarization, question answering, and text creation. Currently, reducing memory usage and improving inference efficiency during the use of large language models has become a hot topic in the field of large model technology.

[0003] In traditional techniques, key-value caches (KV caches) are typically compressed dynamically in real time during the inference process of large language models. This is done by updating and compressing the KV cache at each step of the inference process to reduce the number of KV tokens transmitted during inference, thereby reducing memory usage and computational load.

[0004] However, in the aforementioned traditional techniques, the KV cache needs to be repeatedly updated and adjusted during the inference process, which requires frequent memory operations and calculations, leading to increased computational redundancy and latency, and reduced inference accuracy. Summary of the Invention

[0005] This application provides a data processing method and server to reduce redundant calculations in the reasoning process and improve reasoning speed and accuracy.

[0006] In a first aspect, some embodiments provide a data processing method applied to a computer device, the computer device integrating a target large language model, the target large language model including at least two cascaded feature extraction modules and a key-value (KV) compression module, each feature extraction module including a network layer and multiple attention heads connected to the network layer, the data processing method including:

[0007] The target prompt text is input into each feature extraction module to obtain the local key-value pairs (KV) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text. The different target words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by each feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module.

[0008] For each feature extraction module, the KV compression module compresses different local KV values ​​of the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KV values, so as to obtain the target KV compression result corresponding to the feature extraction module.

[0009] The KV compression result of each target is input into the corresponding feature extraction module to obtain the target response text of the target prompt text.

[0010] In the above embodiments, a target large language model is designed, comprising at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. The target prompt text is then input into each feature extraction module, which processes the text to obtain local key-value pairs (KVs) of different target words and their attention weights. The different words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by the feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. Then, for each feature extraction module, the KV compression module compresses the different local KVs output by the feature extraction module according to its compression ratio and the attention weights of the different local KVs, to obtain the target KV compression result corresponding to that feature extraction module. Finally, the target KV compression result corresponding to each feature extraction module is input into the feature extraction module to obtain the target response text of the target prompt text. In this way, on the one hand, before the target large language model enters the inference process, the different local key-value pairs output by each feature extraction module are compressed all at once. This compression of the key-value cache is achieved after the prefilling stage of the target large language model for the target prompt text and before the inference stage, avoiding frequent key-value cache compression and decompression operations during inference, reducing redundant calculations in inference, thereby reducing computational overhead, avoiding increased latency, and improving inference speed. On the other hand, since new key-value caches are not compressed during inference, the increase in inference error caused by compressing new key-value caches is avoided, thus improving inference accuracy.

[0011] Secondly, some embodiments also provide a server integrating a target large language model, the target large language model including at least two cascaded feature extraction modules and a key-value compression module, each feature extraction module including a network layer and multiple attention heads connected to the network layer; the server includes:

[0012] A communication device configured to connect to an external device;

[0013] and at least one processor, connected to the communication device, and configured to:

[0014] The target prompt text is input into each feature extraction module to obtain the local key-value pairs (KV) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text. The different target words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by each feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module.

[0015] For each feature extraction module, the KV compression module compresses different local KV values ​​of the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KV values, so as to obtain the target KV compression result corresponding to the feature extraction module.

[0016] The KV compression result of each target is input into the corresponding feature extraction module to obtain the target response text of the target prompt text.

[0017] In the above embodiments, a target large language model is designed, comprising at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. The target prompt text is then input into each feature extraction module, which processes the text to obtain local key-value pairs (KVs) of different target words and their attention weights. The different words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by the feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. Then, for each feature extraction module, the KV compression module compresses the different local KVs output by the feature extraction module according to its compression ratio and the attention weights of the different local KVs, to obtain the target KV compression result corresponding to that feature extraction module. Finally, the target KV compression result corresponding to each feature extraction module is input into the feature extraction module to obtain the target response text of the target prompt text. In this way, on the one hand, before the target large language model enters the inference process, the different local key-value pairs output by each feature extraction module are compressed all at once. This compression of the key-value cache is achieved after the prefilling stage of the target large language model for the target prompt text and before the inference stage, avoiding frequent key-value cache compression and decompression operations during inference, reducing redundant calculations in inference, thereby reducing computational overhead, avoiding increased latency, and improving inference speed. On the other hand, since new key-value caches are not compressed during inference, the increase in inference error caused by compressing new key-value caches is avoided, thus improving inference accuracy.

[0018] Thirdly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the methods provided in some embodiments of the first aspect.

[0019] Fourthly, a computer program product is provided, comprising: a computer program that, when executed by a processor, implements the steps of the methods provided in some embodiments of the first aspect. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application;

[0022] Figure 2 This is a schematic diagram of the hardware configuration of a display device provided in some embodiments of this application;

[0023] Figure 3 This is a schematic diagram of the hardware configuration of the control device provided in some embodiments of this application;

[0024] Figure 4 This is a schematic diagram of the software configuration of a display device provided in some embodiments of this application;

[0025] Figure 5 A flowchart illustrating a data processing method provided in some embodiments of this application;

[0026] Figure 6 This is a flowchart illustrating the process of determining the local key-value pairs (KVs) of different target words output by each feature extraction module and the attention weights of different local KVs, as provided in some embodiments of this application.

[0027] Figure 7 A flowchart illustrating the process of determining the compression ratio of each feature extraction module, provided for some embodiments of this application;

[0028] Figure 8 A schematic diagram illustrating the process of obtaining the target response text provided in some embodiments of this application;

[0029] Figure 9 A flowchart illustrating the output target response text provided for some embodiments of this application;

[0030] Figure 10 A flowchart illustrating the training objectives of a large language model provided in some embodiments of this application;

[0031] Figure 11 A flowchart illustrating a data processing method provided for other embodiments of this application;

[0032] Figure 12 This is an internal structural diagram of a computer device provided in some embodiments of this application. Detailed Implementation

[0033] The embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described below do not represent all embodiments consistent with this application. They are merely examples of systems and methods consistent with some aspects of this application as detailed in the claims.

[0034] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0035] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0036] The terms “include” and “have”, and any variations thereof, are intended to cover but not exclusively include, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0037] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0038] Traditional techniques typically involve dynamically compressing the key-value cache (KV cache) in real-time during the inference process of a large language model. This is done by updating and compressing the KV cache at each step of the inference process to reduce the number of KV tokens transmitted, thereby reducing memory usage and computational load. However, these traditional techniques require repeated updates and adjustments to the KV cache during inference, leading to frequent memory operations and computations. This results in increased computational redundancy and latency, ultimately reducing inference accuracy.

[0039] Based on this, in some embodiments, a data processing method is provided. This data processing method can be applied to a computer device, which can be a server or a terminal such as a display device. The server can be a standalone server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud services.

[0040] In this embodiment, the display device 200 generally refers to a device with screen display and data processing capabilities. For example, the display device 200 includes, but is not limited to, smart TVs, mobile terminals, computers, monitors, advertising screens, wearable devices, virtual reality devices, augmented reality devices, etc.

[0041] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device provided in some embodiments of this application. For example... Figure 1 As shown, users can operate the display device 200 via touch operation, mobile terminal 300, and control device 100. For example, control device 100 can be a remote control, stylus, gamepad, etc.

[0042] The mobile terminal 300 can serve as a control device for human-computer interaction between the user and the display device 200. The mobile terminal 300 can also serve as a communication device for establishing a communication connection with the display device 200 and exchanging data. In some embodiments, the mobile terminal 300 can install software applications with the display device 200 to establish a connection and communication via network communication protocols, achieving one-to-one control operations and data communication. It can also transmit audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve synchronous display.

[0043] like Figure 1 The diagram also shows that the display device 200 communicates with the server 400 via various communication methods. This allows the display device 200 to communicate via a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0044] Display device 200 can provide broadcast television reception function, and can also be equipped with intelligent network television function that provides computer support, including but not limited to network television, smart television, Internet Protocol television (IPTV), etc.

[0045] Figure 2 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of display device 200.

[0046] In some embodiments, the display device 200 may include at least one of a tuner 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0047] In some embodiments, detector 230 is used to acquire signals from the external environment or to interact with the outside world. For example, detector 230 includes a light receiver, a sensor for acquiring ambient light intensity; or, detector 230 includes an image acquisition device, such as a camera, which can be used to acquire external environmental scenes, user attributes, or user interaction gestures; or, detector 230 includes a sound acquisition device, such as a microphone, for receiving external sounds.

[0048] In some embodiments, the display 260 includes display function components for presenting images and driving components for driving image display. The display 260 is used to receive and display image signals output from the controller 250. For example, the display 260 can be used to display video content, image content, menu control interface components, and user control UI interfaces, etc.

[0049] In some embodiments, the communication device 220 is a component used to communicate with external devices or the server 400 according to various communication protocol types. The display device 200 may have multiple communication devices 220 depending on the supported communication methods. For example, when the display device 200 supports wireless network communication, it may have a communication device 220 with WiFi functionality. When the display device 200 supports Bluetooth connectivity, it needs to have a communication device 220 with Bluetooth functionality.

[0050] The communication device 220 enables the display device 200 to communicate with external devices or the server 400 via wireless or wired connections. Wired connections utilize data cables, interfaces, or other components to connect the display device 200 to external devices. Wireless connections utilize wireless signals or wireless networks. The display device 200 can directly establish a connection with external devices or indirectly through gateways, routers, or other connection devices.

[0051] In some embodiments, the controller 250 may include at least one of a central processing unit, a video processor, an audio processor, a graphics processor, and a power processor, and a first to an nth interface for input / output. The controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0052] In some embodiments, the controller 250 and the tuner 210 may be located in different separate devices, that is, the tuner 210 may also be located in an external device of the main device where the controller 250 is located, such as an external set-top box.

[0053] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0054] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0055] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0056] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0057] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0058] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0059] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0060] The controller 110 includes a processor 112, RAM (Random Access Memory) 113, ROM (Read-Only Memory) 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0061] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC (Near Field Communication) module 133.

[0062] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, and button 144.

[0063] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via WiFi, Bluetooth, or NFC protocols and send them to the display device 200.

[0064] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0065] The power supply 180 is used to provide operating power support for the various components of the control device 100 under the control of the controller.

[0066] In some embodiments, to enable user interaction, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can (control the display device) provide a user interface, allowing users to interact with the display device 200 and supporting the running of various applications.

[0067] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0068] An operating system can be divided into different modules or levels based on the functions it implements.

[0069] For example, such as Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0070] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0071] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0072] like Figure 4 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0073] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0074] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0075] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 4 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB (Universal Serial Bus) driver, HDMI (High Definition Multimedia Interface) driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0076] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0077] As mentioned above, in order to solve the technical problem that in the traditional technology, real-time dynamic compression of KV cache during the reasoning process of large language model requires repeated updating and adjustment of KV cache during the reasoning process, which requires frequent memory operations and calculations, resulting in increased repetitiveness and latency of calculations and reduced reasoning accuracy, a data processing method is provided in some embodiments, which can be applied to servers or the aforementioned display devices.

[0078] In an exemplary embodiment, taking the data processing method applied to a server as an example, the server includes a communication device and at least one processor. The communication connection between the communication device and an external device can be a wired communication connection or a wireless communication connection, and the at least one processor is connected to the communication device.

[0079] The server integrates a target large language model, which includes at least two cascaded feature extraction modules and a key-value compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer.

[0080] The so-called Large Language Model (LLM) refers to a deep learning model trained on a large amount of text data, enabling it to generate natural language text or understand the meaning of language text. Its core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, mimicking the human language cognition and generation process to some extent. Optionally, the Large Language Model uses a Transformer architecture and relies on a self-attention mechanism.

[0081] An attention head is an independent computational unit that captures multi-dimensional semantic associations. Each attention head is responsible for capturing the semantic dependencies between different elements in the input sequence (such as lexical units) from a specific perspective. Multiple attention heads work in parallel, and the results from all attention heads are concatenated and integrated through linear transformations to form a complete multi-head attention output. This enables multi-dimensional sequence information modeling, allowing large language models to simultaneously consider multi-dimensional dependencies and improve their ability to understand complex sequences. For example, when processing the text "Xiaohong went to school today, and she was very sad," one attention head might focus on the referential association between "she" and "Xiaohong," while another attention head might focus on the action-location association between "go" and "school." Ultimately, the results from all attention heads are combined to enable the large language model to fully understand the semantic logic of the text.

[0082] like Figure 5 As shown, a data processing method applied to a processor in a server includes the following steps:

[0083] S 501, input the target prompt text into each feature extraction module respectively, and obtain the local key-value pairs of different target words and the attention weights of different local key-value pairs output by each feature extraction module after processing the target prompt text.

[0084] Among them, different target words are obtained by splitting the target prompt text, and the different local key-value pairs of the same target word output by each feature extraction module are obtained by splitting the global key-value pairs of the target word output by the network layer in the feature extraction module.

[0085] Optionally, the target prompt text can be input by the user to the server through text input, file upload, voice recording, etc. Specifically, when the user inputs text or uploads a file, the server can directly determine the user's output text or uploaded file as the target prompt text; similarly, when the user records voice, the server can convert the user's recorded voice into text and use it as the target prompt text.

[0086] Attention weights are numerical values ​​used in attention mechanisms to measure the strength of the association between the query vector (Q) and the key vector (K). They determine the "attention" that different input tokens should be assigned during the inference process of a large language model. Their core function is to make the large language model "focus" on the input tokens that are most relevant to the current query vector when calculating the value vector (V / alue, V) of the output tokens during the inference process.

[0087] A token is the smallest semantic or grammatical unit obtained after breaking down the original text; it is also called a word example or tag. For example, the original text "I love eating apples" can be broken down into "I", "love", "eat", "apple", and "fruit", a total of five tokens. Optionally, in Chinese text, tokens can be single-character tokens (such as "I") or multi-character tokens (such as "today" and "apple").

[0088] For the target prompt text, it can be segmented into multiple target words. Optionally, the target large language model includes an embedding layer. The target prompt text is input into the embedding layer, which performs word segmentation on the target prompt text to obtain multiple target words. Each target word is then converted into an initial vector (e.g., a 512-dimensional vector). These initial vectors of different target words are then input into each feature extraction module to obtain the local key-value pairs (KV) and attention weights of different local KVs for each target word output by the feature extraction module. Alternatively, the server uses a word segmentation tool to perform word segmentation on the target prompt text to obtain multiple target words. These multiple target words are then input into the embedding layer, which converts each target word into an initial vector (e.g., a 512-dimensional vector). These initial vectors of different target words are then input into each feature extraction module to obtain the local KV and attention weights of different local KVs for each target word output by the feature extraction module.

[0089] For each feature extraction module, the network layer within that module determines the key-value pair (KV) for each target word, which serves as the global KV for that target word. Furthermore, for each target word, the network layer in the feature extraction module can decompose the global KV into multiple distinct local KVs, and assign these local KVs to different attention heads within the feature extraction module. Specifically, for each target word, each attention head in the feature extraction module is assigned a local KV for that target word; different attention heads assign different local KVs, and the local KVs assigned by all attention heads constitute the global KV for that target word. Optionally, the global KV of the target word can be evenly decomposed into multiple local KVs, and these local KVs can be evenly distributed among multiple attention heads. For example, if the global key-value pair of the target word is represented by a 512-dimensional vector and there are 8 attention heads, then each attention head is assigned a local key-value pair that includes 64 dimensions of data. For example, the first attention head is assigned data from dimensions 1 to 64, the second attention head is assigned data from dimensions 65 to 128, the third attention head is assigned data from dimensions 129 to 192, and so on. Each attention head has a local key-value pair that includes 64 dimensions of data.

[0090] Furthermore, for each attention head, it can further determine the attention weight of each local key-value pair assigned to it. Optionally, for each attention head, it can determine the attention weight of that local key-value pair based on the K in each assigned local key-value pair and the Q assigned to that local key-value pair.

[0091] In an optional embodiment, S501 may include, for each feature extraction module, using the network layer of the feature extraction module to extract features from the target prompt text, obtaining global key-value pairs (KVs) for different target words, and assigning different local KVs for each target word to multiple attention heads of the feature extraction module; wherein, each attention head assigns a local KV for a different target word, and the local KVs of the same target word assigned by multiple attention heads constitute the global KV of the target word; using multiple attention heads to calculate attention weights for the assigned local KVs of different target words, obtaining the attention weights of different local KVs for each target word.

[0092] Optionally, the network layer in this feature extraction module can extract features for each target word to obtain the global key-value pair (KV) of that target word. Furthermore, optionally, each attention head can determine the attention weight of each local key-value pair assigned to it based on the contextual relationships of different words in the target prompt text and the order of the target words in the global KV to which its assigned local key-value pairs belong in the target prompt text.

[0093] Thus, for each feature extraction module, the local key-value pairs (KVs) of different target words and the attention weights of different local KVs can be obtained; wherein, the different target words include all target words obtained by splitting the target prompt text, and the different local KVs include all local KVs obtained by splitting the global KV of all target words.

[0094] S 502, for each feature extraction module, the KV compression module compresses different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KVs, to obtain the target KV compression result corresponding to the feature extraction module.

[0095] The so-called KV compression module is a functional module that reduces the amount of local KV of different target words output by the feature extraction modules of the target large language model. This is to reduce the memory usage of the KV cache, reduce the amount of KV data transmitted during the inference process, and improve the inference speed while keeping the inference accuracy of the target large language model unchanged or only slightly reduced.

[0096] Optionally, the compression ratio of different feature extraction modules can be the same preset fixed ratio, or they can be different preset fixed ratios. The preset fixed ratio can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed on this.

[0097] Optionally, the compression ratio of each feature extraction module refers to the proportion of uncompressed local KVs in all local KVs of the feature extraction module after different local KVs of the feature extraction module are compressed.

[0098] Compression of different local key-value pairs (KVs) for each feature extraction module refers to the following: For each feature extraction module, the product of the total number of different local KVs output by that module and the compression ratio of that module is determined. This product represents the number of local KVs remaining after compression. Furthermore, when different local KVs are arranged in descending order of attention weight, this product represents the order of the last local KV that will not be compressed. All local KVs ranked after this last local KV will be compressed. For example, if the total number is 100 and the compression ratio is 20%, then when the different local KVs are arranged in descending order of attention weight, the KV compression module will compress all local KVs ranked from 21st to 100th. Optionally, compression refers to deletion or adding a mask.

[0099] In an optional embodiment, S502 may include, for each feature extraction module, using a KV compression module, compressing different local KVs of the feature extraction module sequentially according to their attention weights from smallest to largest, until the proportion of uncompressed local KVs in all local KVs of the feature extraction module reaches the compression ratio of the feature extraction module. For example, if the total number of all local KVs in the feature extraction module is 100 and the compression ratio is 20%, then the KV compression module starts from the local KV with the smallest attention weight and compresses different local KVs of the feature extraction module sequentially according to their attention weights from smallest to largest, until the number of uncompressed local KVs is 20.

[0100] In another optional embodiment, S502 may include, for each feature extraction module, using a KV compression module to sequentially traverse different local KVs of the feature extraction module in descending order of their attention weights, until the proportion of the traversed local KVs in the total number of local KVs of the feature extraction module reaches the compression ratio of the feature extraction module, and then compressing all the remaining untraversed local KVs. For example, if the total number of local KVs in the feature extraction module is 100 and the compression ratio is 20%, then the KV compression module starts from the local KV with the largest attention weight and sequentially traverses different local KVs of the feature extraction module in descending order of their attention weights, until the number of traversed different local KVs reaches 20, and then compresses all the remaining untraversed local KVs.

[0101] Optionally, a key-value (KV) compression module can be used to compress all local key-value pairs output by all feature extraction modules according to a preset compression ratio and the attention weights of different local key-value pairs. This yields the target key-value compression result for each feature extraction module. The preset compression ratio can be set based on empirical values, experimental results from multiple trials, and the inference accuracy requirements of the target large language model in the actual application scenario. This makes the target key-value compression result for each feature extraction module more controllable in offline scenarios.

[0102] S503, input the KV compression result of each target into the corresponding feature extraction module to obtain the target response text of the target prompt text.

[0103] After obtaining the target key-value compression result corresponding to each feature extraction module, the compression result of each target key-value can be input into its corresponding feature extraction module. This feature extraction module can then determine the compressed local key-values ​​in the global key-value of each target word obtained by the network layer in the module based on the received target key-value compression result. Thus, the uncompressed local key-values ​​in the global key-value of each target word are compressed, such as by adding masks or pruning, to obtain compressed key-values ​​for different target words. Optionally, for target words whose all local key-values ​​are compressed, the global key-value of that target word in the feature extraction module can be directly deleted. Then, the at least two cascaded feature extraction modules included in the target large language model can infer the target response text of the target prompt text based on the compressed key-values ​​of the corresponding different target words and output the inferred target response text. Optionally, the process by which at least two cascaded feature extraction modules included in the target large language model can infer the target response text based on the compressed KV of the corresponding different target words can be called the self-decoding regression process of the target large language model.

[0104] In an optional embodiment, S503 may include: for each feature extraction module, inputting the target KV compression result corresponding to the feature extraction module into the feature extraction module, and having the feature extraction module compress the global KV of different target words according to the target KV compression result to obtain the compressed KV of different target words; obtaining the target response text of the target prompt text; wherein the target response text is obtained by at least two cascaded feature extraction modules processing the compressed KV of the corresponding different target words.

[0105] Optionally, during the process of the target large language model processing the target prompt text to obtain the target response text, metrics such as first byte time, GPU utilization, compression ratio, and key-value hit rate can be monitored and collected throughout the entire process. The orchestration system can then be driven by rules or learning-based strategies. When concurrency surges or GPU load becomes excessively high, the system automatically scales horizontally; when load decreases or cache hit rate continues to improve, it smoothly scales down to release resources. This ensures that the target large language model maintains its predetermined service level under different business peaks, while avoiding long-term over-provisioning and resulting cost waste.

[0106] In the above embodiments, a target large language model is designed, comprising at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. The target prompt text is then input into each feature extraction module, which processes the text to obtain local key-value pairs (KVs) of different target words and their attention weights. The different words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by the feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. Then, for each feature extraction module, the KV compression module compresses the different local KVs output by the feature extraction module according to its compression ratio and the attention weights of the different local KVs, to obtain the target KV compression result corresponding to that feature extraction module. Finally, the target KV compression result corresponding to each feature extraction module is input into the feature extraction module to obtain the target response text of the target prompt text. In this way, on the one hand, before the target large language model enters the inference process, the different local key-value pairs output by each feature extraction module are compressed all at once. This compression of the key-value cache is achieved after the prefilling stage of the target large language model for the target prompt text and before the inference stage, avoiding frequent key-value cache compression and decompression operations during inference, reducing redundant calculations in inference, thereby reducing computational overhead, avoiding increased latency, and improving inference speed. On the other hand, since new key-value caches are not compressed during inference, the increase in inference error caused by compressing new key-value caches is avoided, thus improving inference accuracy.

[0107] Based on the above embodiments, in an exemplary embodiment, the determination of the local key-value pairs (KVs) of different target words output by each feature extraction module and the attention weights of different local KVs are further refined. Optionally, such as Figure 6 As shown, the following steps may be included:

[0108] S601, perform word segmentation on the target prompt text to obtain different target words.

[0109] After obtaining the target prompt text, various methods, such as word segmentation tools, can be used to split the target prompt text into word units to obtain different target word units. The specific implementation method of word unit splitting of the target prompt text is not limited; any specific implementation method capable of splitting the target prompt text into word units falls within the protection scope of this application.

[0110] S602, for each target word, if there are historical words matching the target word in the preset storage space, read from the preset storage space the different local key-value pairs and the attention weights of the different local key-value pairs obtained in each feature extraction module for the historical words matching the target word.

[0111] Understandably, in scenarios involving continuous input and output to a target large language model, such as multi-turn dialogues, the target response text output by the target large language model each time will be influenced by the prompt text and response text from the previous or previous rounds of dialogue. Therefore, in order to further improve the inference speed of the target large language model, after each round of dialogue, the local key-value pairs (KVs) and attention weights of the target words newly appearing in this round of dialogue obtained in different feature extraction modules can be stored. This allows the target large language model to directly use the local KVs and attention weights of the target words obtained in different feature extraction modules during subsequent inference processes, without having to repeatedly process the same target words in each feature extraction module.

[0112] Accordingly, after obtaining different target words in the target prompt text, for each target word, a search can be conducted in a preset storage space to determine if a historical word matching that target word exists. Optionally, for each target word, the search can either determine if the target word exists in the preset storage space, or search for historical words in the preset storage space that have a similarity greater than a preset similarity. The preset similarity can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed on this.

[0113] In this way, for each target word, if there are historical words that match the target word in the preset storage space, the different local key-value pairs and attention weights of the historical words that match the target word can be directly read from the preset storage space in each feature extraction module, without having to repeatedly process the target word in each feature extraction module.

[0114] Optionally, the preset storage space can be storage space in other storage devices on the server besides video memory, or it can be storage space in external storage devices of the server, such as disks or cloud storage. This can reduce video memory usage, thereby reducing memory usage, effectively solving the memory bottleneck problem, and improving the inference efficiency of the target large language model.

[0115] S603, for each target word, if there is no historical word matching the target word in the preset storage space, the target word is input into each feature extraction module respectively, and the different local key values ​​and attention weights of the target word output by each feature extraction module after processing the target word are obtained.

[0116] Accordingly, for each target word, if there is no historical word matching the target word in the preset storage space, the target word needs to be processed by each feature extraction module to obtain the different local key values ​​and attention weights of the target word.

[0117] The specific implementation of S603 is similar to that of S501, and will not be repeated here.

[0118] Optionally, the data processing method further includes, for each target word, if there are no historical words matching the target word in the preset storage space, after obtaining the different local key-value pairs and attention weights of the target word from the output of each feature extraction module, storing the different local key-value pairs and attention weights of the target word in different feature extraction modules into the preset storage space. This allows the target word to be directly read from the preset storage space when it is split again to obtain the target word or other words matching the target word in subsequent use of the target large language model, without having to perform repeated processing by each feature extraction module.

[0119] Optionally, the data processing method further includes storing the different local key-value pairs and attention weights of the newly appearing response words in the target response text in different feature extraction modules into the aforementioned preset storage space. This allows the response words or other words matching the response words to be split again in subsequent use of the target large language model. In this case, the different local key-value pairs and attention weights of the response words obtained in each feature extraction module can be directly read from the aforementioned preset storage space without having to perform repeated processing in each feature extraction module.

[0120] In this embodiment, by storing the different local key-value pairs and attention weights of different local key-value pairs obtained in each feature extraction module for historical word units in a preset storage space, during the processing of the target prompt text by the target large language model, for target word units with matching historical word units in the preset storage space, the different local key-value pairs and attention weights of different local key-value pairs obtained in each feature extraction module for the historical word units matching the target word units can be directly read from the preset storage space, without the need for repeated processing of the target word units by each feature extraction module. Thus, by adopting the KV offload technique, the KV cache generated by the target large language model can be transferred from the video memory to other storage spaces, which can reduce the video memory usage and support larger batch sizes or longer context lengths, thereby further improving the inference efficiency of the target large language model.

[0121] Based on the above embodiments, in one exemplary embodiment, the determination of the compression ratio of each feature extraction module is further refined. Optionally, such as Figure 7 As shown, the following steps may be included:

[0122] S701, determine the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module.

[0123] The distribution parameters include variance and / or entropy.

[0124] Variance is a metric used to measure the dispersion of attention weights in different local key-value pairs output by a feature extraction module. The greater the dispersion, the greater the variance, and vice versa.

[0125] Correspondingly, entropy is an indicator used to measure the disorder of the probability distribution of attention weights of different local key values ​​output by a feature extraction module. The more concentrated the probability distribution (a few values ​​have extremely high probabilities and most values ​​have extremely low probabilities), the smaller the entropy value. Conversely, the more uniform the probability distribution (the closer the probabilities of all values ​​are), the larger the entropy value.

[0126] For each feature extraction module, the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module can be determined based on the attention weights of the different local key values ​​output by the feature extraction module.

[0127] S702, based on the distribution parameters, determine the compression ratio of the feature extraction module.

[0128] For each feature extraction module, the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module can indicate the importance distribution of different local key values ​​output by the feature extraction module in the inference process, and this importance distribution can affect the compression ratio of the feature extraction module.

[0129] For example, if the variance of the attention weights of different local key values ​​(KVs) output by the feature extraction module is small, it indicates that the dispersion of the attention weights of these KVs is small, meaning the differences between their attention weights are small. Consequently, the importance of these different local KVs during inference is relatively consistent. Therefore, to ensure inference accuracy, it is necessary to retain as many different local KVs as possible, resulting in a higher compression ratio for the feature extraction module. Conversely, if the variance of the attention weights of different local KVs output by the feature extraction module is large, it indicates that the dispersion of their attention weights is large, meaning the differences between their attention weights are large. Consequently, the importance of these different local KVs during inference varies greatly; some local KVs are more important than others. Therefore, to improve inference efficiency, it is advisable to compress as many less important local KVs as possible, resulting in a lower compression ratio for the feature extraction module.

[0130] For example, if the entropy value of the attention weights of different local key-value pairs output by the feature extraction module is large, it indicates that the probability distribution of the attention weights of these pairs is relatively uniform, meaning the differences between their attention weights are small. Therefore, the importance of these local key-value pairs in the inference process is relatively consistent. To ensure inference accuracy, it is necessary to retain as many of the different local key-value pairs as possible, resulting in a higher compression ratio for the feature extraction module. Conversely, if the entropy value of the attention weights of different local key-value pairs output by the feature extraction module is small, it indicates that the probability distribution of their attention weights is relatively concentrated, meaning the differences between their attention weights are large. This indicates that the importance of these local key-value pairs varies significantly during inference; some local key-value pairs are more important than others. Therefore, to improve inference efficiency, it is possible to compress as many less important local key-value pairs as possible, resulting in a lower compression ratio for the feature extraction module.

[0131] Based on this, for each feature extraction module, the compression ratio of the feature extraction module can be determined according to the variance and / or entropy of the attention weights of different local key values ​​output by the feature extraction module.

[0132] Optionally, a pre-defined correspondence between different variance ranges and compression ratios can be established. Thus, for each feature extraction module, after obtaining the variance of the attention weights of different local key-value pairs output by that module, the range to which the variance belongs is determined, and the compression ratio corresponding to that range is defined as the compression ratio of that feature extraction module. This correspondence can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed.

[0133] Optionally, a pre-defined correspondence between different entropy value ranges and compression ratios can be established. Thus, for each feature extraction module, after obtaining the entropy of the attention weights of different local key-value pairs output by that module, the value range to which the entropy belongs is determined, and the compression ratio corresponding to that value range is determined as the compression ratio of that feature extraction module. The aforementioned correspondence can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed on this.

[0134] Optionally, a pre-defined correspondence between different combinations of variance and entropy value intervals and compression ratios can be established. Thus, for each feature extraction module, after obtaining the variance and entropy of the attention weights of different local key-value pairs output by that module, the value intervals to which the variance and entropy belong are determined. The compression ratio corresponding to the combination of these value intervals is then determined as the compression ratio for that feature extraction module. This correspondence can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed.

[0135] In this embodiment, for each feature extraction module, the compression ratio of the feature extraction module can be dynamically generated by determining the distribution parameters of the attention weights of different local key-value pairs output by the feature extraction module. This allows the compression ratio of each feature extraction module to be dynamically adjusted according to the actual computational load of the feature extraction module. As a result, not only can the computational resource utilization of the target large language model be improved, but the waste of computational resources caused by a fixed compression ratio can also be avoided. Furthermore, more flexible optimization can be performed for the computational needs of different feature extraction modules, further improving the inference performance and inference efficiency of the target large language model.

[0136] Based on the above embodiments, in one exemplary embodiment, the acquisition of the target response text is further refined. Optionally, such as... Figure 8 As shown, the following steps may be included:

[0137] S801, for each feature extraction module, the feature extraction module processes the existing word KV to obtain the KV of the next predicted word in the predicted response text.

[0138] The predicted response text is the response result of the feature extraction module to the target prompt text. When the feature extraction module is the first feature extraction module, the existing word KV includes the compressed KV of different target words and the KV of the predicted words in the predicted response text. When the feature extraction module is not the first feature extraction module, the existing word KV includes the compressed KV of different target words, the KV of the predicted words in the predicted response text, and the KV of the latest predicted words output by the previous feature extraction module.

[0139] In the reasoning process of the target large language model for the target prompt text, each feature extraction module can obtain a response result for the target prompt text, which serves as the predicted response text for that feature extraction module. Furthermore, the predicted response text of the last feature extraction module is the final output target response text. Similar to the target prompt text, the predicted response text of each feature extraction module is also determined word-by-word. Specifically:

[0140] For the first feature extraction module, the feature extraction module can infer the KV of the next predicted word in the predicted response text based on the compressed KV of the different target words of the feature extraction module and the KV of the predicted words in the predicted response text (i.e. the KV of the words in the predicted response text that have been predicted and belong to the feature extraction module), and transmit the inferred KV of the next predicted word to the next feature extraction module cascaded by the first feature extraction module.

[0141] Accordingly, for each feature extraction module that is not the first one, it can infer the KV of the next predicted word in the predicted response text based on the compressed KV of the target words of that feature extraction module, the KV of the predicted words in the predicted response text, and the KV of the latest predicted word output from the previous feature extraction module cascaded with that feature extraction module. Where this feature extraction module is not the last feature extraction module, it can transmit the inferred KV of the next predicted word to the next feature extraction module cascaded with it.

[0142] Furthermore, for the trailing feature extraction module, the KV of the next predicted token obtained through its prediction is the KV of the token in the target response text.

[0143] S802, acquiring the target response text of the target prompt text output by the trailing feature extraction module based on the KV of different predicted tokens in the corresponding predicted response text.

[0144] Since the predicted response text obtained by prediction of the trailing feature extraction module is exactly the target response text of the target prompt text, after the trailing feature extraction module completes prediction of the full predicted response text, the trailing feature extraction module can convert the KV of different predicted tokens in the predicted response text obtained by its own prediction into response tokens in the text-formatted target response text, and sequentially output the response tokens converted from the KV of different predicted tokens in the order of prediction time of the KV of different predicted tokens from earliest to latest, so that the server acquires the target response text of the target prompt text.

[0145] For example, the target prompt text is "today's weather", different target tokens are "to", "day", "'s", "wea", "ther" respectively, and the target large language model comprises three cascaded feature extraction modules 1 to 3. Specifically:

[0146] Feature extraction module 1 determines the KV of the first predicted token "da" in the predicted response text of feature extraction module 1 according to the compressed KV of different target tokens of feature extraction module 1, and transmits the KV of the predicted "da" to feature extraction module 2; feature extraction module 2 predicts the KV of the first predicted token "y" in the predicted response text of feature extraction module 2 according to the compressed KV of different target tokens of feature extraction module 2 and the KV of the latest predicted token "da" transmitted by feature extraction module 1, and transmits the KV of the predicted "y" to feature extraction module 3; feature extraction module 3 predicts the KV of the first predicted token wea in the predicted response text of feature extraction module 3 according to the compressed KV of different target tokens of feature extraction module 3 and the KV of the latest predicted token "y" transmitted by feature extraction module 2, and thus the KV of "wea" predicted by feature extraction module 3 is also the KV of the first token in the target response text.

[0147] The feature extraction module 1 determines the KV of the second predicted token "气" in the prediction response text of the feature extraction module 1 according to the compressed KV of different target tokens of the feature extraction module 1 and the KV of the predicted token "天", and transmits the KV of the predicted "气" to the feature extraction module 2; the feature extraction module 2 predicts the KV of the second predicted token "天" in the prediction response text of the feature extraction module 2 according to the compressed KV of different target tokens of the feature extraction module 2, the KV of the predicted token "天" and the KV of the latest predicted token "气" transmitted by the feature extraction module 1, and transmits the KV of the predicted "气" to the feature extraction module 3; the feature extraction module 3 predicts the KV of the second predicted token "气" in the prediction response text of the feature extraction module 3 according to the compressed KV of different target tokens of the feature extraction module 3, the KV of the predicted token "天" and the KV of the latest predicted token "气" transmitted by the feature extraction module 2, and thus the KV of "气" predicted by the feature extraction module 3 is also the KV of the second token in the target response text.

[0148] The feature extraction module 1 determines the KV of the third predicted token "晴" in the prediction response text of the feature extraction module 1 according to the compressed KV of different target tokens of the feature extraction module 1 and the KVs of the predicted tokens "天" and "气", and transmits the KV of the predicted "晴" to the feature extraction module 2; the feature extraction module 2 predicts the KV of the third predicted token "晴" in the prediction response text of the feature extraction module 2 according to the compressed KV of different target tokens of the feature extraction module 2, the KVs of the predicted tokens "天" and "气" and the KV of the latest predicted token "晴" transmitted by the feature extraction module 1, and transmits the KV of the predicted "晴" to the feature extraction module 3; the feature extraction module 3 predicts the KV of the third predicted token "晴" in the prediction response text of the feature extraction module 3 according to the compressed KV of different target tokens of the feature extraction module 3, the KVs of the predicted tokens "天" and "气" and the KV of the latest predicted token "晴" transmitted by the feature extraction module 2, and thus the KV of "晴" predicted by the feature extraction module 3 is also the KV of the third token in the target response text.

[0149] Furthermore, the last feature extraction module can sequentially output response tokens in text form obtained by converting the KVs of the predicted tokens "天", "气" and "晴" in the order from earliest to latest of the prediction time of the KVs of the predicted tokens "天", "气" and "晴", that is, the last feature extraction module sequentially outputs the texts "天", "气" and "晴", so that the server can obtain the target response text "天气晴" corresponding to the target prompt text.

[0150] In this embodiment, frequent key-value cache compression and decompression operations are avoided during the inference process of the target large language model, reducing redundant calculations and thus computational overhead, preventing increased latency, and improving inference speed. Furthermore, by not compressing newly added key-value caches during inference, the increased inference error caused by compressing new key-value caches is avoided, thereby improving inference accuracy.

[0151] Based on the above embodiments, in one exemplary embodiment, the output of the target response text is further refined. Optionally, such as... Figure 9 As shown, the following steps may be included:

[0152] S901, Obtain the response tokens corresponding to the KV of the preset number of predicted tokens output by the end feature extraction module when generating KV of the preset number of predicted tokens.

[0153] As mentioned earlier, the predicted response text obtained by the end feature extraction module is the target response text of the target prompt text. Therefore, the end feature extraction module does not need to wait until all the key-value pairs of the predicted words in the predicted response text have been obtained before outputting the response words. Instead, it can output the response words obtained by converting the key-value pairs of each predicted word in the key-value pairs of the predicted words in the order of their generation time from earliest to latest, so that the server can obtain the response words corresponding to the key-value pairs of the predicted words.

[0154] S902, the different response tokens obtained are fused to obtain the target response text.

[0155] The server can obtain multiple sets of response tokens, and each set of response tokens includes a predicted number of response tokens. Therefore, the server can concatenate and merge the multiple sets of response tokens in order of their acquisition time from earliest to latest to obtain the target response text.

[0156] In this embodiment, during the reasoning process of the target large language model, the target words in the obtained target response text can be output step by step according to the reasoning progress and with a preset number as the output granularity. This can reduce the TTFT (Time To First Token) of the target large language model, reduce the latency of the target large language model, improve the reasoning efficiency of the target large language model, and improve the user experience in scenarios such as multi-turn dialogue and long text generation.

[0157] Based on the above embodiments, in an exemplary embodiment, the training of the target large language model is further refined. Optionally, such as... Figure 10 As shown, the following steps may be included:

[0158] S1001, Input the sample prompt text into the initial large language model in the initial model to obtain the sample response text of the sample prompt text.

[0159] Understandably, the initial large language model includes at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. Optionally, the sample prompt text is input into each feature extraction module of the initial large language model to obtain the local key-value pairs (KVs) of different sample words and the attention weights of different local KVs output by each feature extraction module of the initial large language model after processing the sample prompt text. For each feature extraction module of the initial large language model, the KV compression module compresses the different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weights of different local KVs to obtain the sample KV compression result corresponding to the feature extraction module. Each sample KV compression result is input into the corresponding feature extraction module in the initial large language model to obtain the sample response text of the sample prompt text.

[0160] The specific implementation of S1001 is the same as the generation method of the target response text of the target prompt text, and will not be repeated here.

[0161] S1002, Determine the accuracy of the sample response text based on the difference between the sample response text and the sample prompt text and the actual response text.

[0162] After obtaining the sample response text of the sample prompt text, the difference between the sample response text and the actual response text of the sample prompt text can be determined, and then the accuracy of the sample response text can be determined based on this difference.

[0163] Optionally, the similarity between the sample response text and the sample prompt text and the actual response text is determined, and this similarity is used as the accuracy of the sample response text.

[0164] S1003, based on accuracy and the overall compression ratio of the initial large language model, adjusts the additional model parameters of the additional fine-tuning module in the initial model.

[0165] The overall compression ratio refers to the proportion of local key-value pairs in the total key-value compression result of all samples within the total local key-value pairs of the sample word units in the sample prompt text. Optionally, the overall compression ratio is the average of the compression ratios of each feature extraction module in the initial large language model.

[0166] In order to maximize the inference speed of the target large language model while ensuring its inference accuracy, it is often desirable to achieve an accuracy of the sample response text that is not less than the accuracy threshold, even when the overall compression ratio of the sample large language model reaches or falls below the compression ratio threshold. Therefore, the loss value of the loss function of the sample large language model can be determined based on the difference between the accuracy of the sample response text and the accuracy threshold, as well as the difference between the overall compression ratio and the compression ratio threshold. Based on this loss value, the additional model parameters of the fine-tuning module in the initial model can be adjusted. Optionally, the reconstruction error caused by the compressed local key-value pairs can be included as an error component in the loss function of the sample large language model.

[0167] Optionally, if the determined loss value is greater than the loss value threshold, adjust the additional model parameters of the additional fine-tuning module in the initial model and return to step S1001 until the determined loss value is not greater than the loss value threshold. At this point, it can be determined that the initial model has converged, thereby completing the model training process and proceeding to step S1004.

[0168] Optionally, the additional fine-tuning module is the LoRA (Low-Rank Adaptation) module. In essence, it introduces a small number of trainable low-rank matrices as additional model parameters on the basis of the original model structure of the initial large language model to construct the initial model. During the training of the initial model, all or part of the model parameters in the initial large language model are frozen, and only the additional model parameters, or the small number of model parameters that are not frozen and the additional model parameters are adjusted, until the initial model converges.

[0169] The accuracy threshold, compression ratio threshold, and loss value threshold can all be set based on empirical values, experimental results from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios. For example, the accuracy threshold could be 80%, and the compression ratio threshold could be 50%.

[0170] S1004, replace the initial model parameters of the initial large language model with the fusion parameters of the initial model parameters and the additional model parameters to obtain the target large language model.

[0171] After adjusting the additional model parameters of the additional fine-tuning module through multiple rounds of iterations to make the initial model converge, training is stopped. The initial model parameters of the initial large language model are then fused with the additional model parameters of the additional fine-tuning module after the initial model has converged to obtain fused parameters. The initial model parameters of the initial large language model are then replaced with the above fused parameters to obtain the target large language model.

[0172] In this embodiment, by adjusting the additional model parameters of the additional fine-tuning module, the negative impact of local key-value compression on the inference accuracy of the target large language model is resolved. Furthermore, by introducing the additional fine-tuning module, the consistency between training and inference can be improved while maintaining the inference accuracy of the target large language model unaffected by local key-value compression. This allows the target large language model to effectively adapt to the impact of local key-value compression during training and maximizes the preservation of the inference performance of the target large language model.

[0173] Based on the above embodiments, in an exemplary embodiment, such as Figure 11 As shown, the data processing method may include the following steps:

[0174] S1101, The display device sends the target prompt text to the server.

[0175] S1102, after receiving the target prompt text, the server performs word segmentation on the target prompt text to obtain different target words. The server integrates a target large language model, which includes at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer.

[0176] S1103, for each target word, if there are historical words matching the target word in the preset storage space, the server reads the different local key-value pairs and attention weights of the historical words matching the target word obtained in each feature extraction module from the preset storage space.

[0177] S1104, for each target word, if there is no historical word matching the target word in the preset storage space, the server uses the network layer of the feature extraction module to extract features from the target word to obtain the global key-value pair of the target word, and assigns different local key-value pairs of the target word to multiple attention heads of the feature extraction module; the multiple attention heads are used to calculate the attention weights of the assigned local key-value pairs of the target word to obtain the attention weights of the different local key-value pairs of the target word.

[0178] S1105, for each feature extraction module, the server determines the variance and / or entropy of the attention weights of different local key-value pairs output by the feature extraction module, and determines the compression ratio of the feature extraction module based on the variance and / or entropy.

[0179] S1106, For each feature extraction module, the server uses the KV compression module to compress different local KVs in the feature extraction module in ascending order of attention weight, until the proportion of uncompressed local KVs in all local KVs of the feature extraction module reaches the compression ratio of the feature extraction module, thus obtaining the target KV compression result corresponding to the feature extraction module.

[0180] S1107, for each feature extraction module, the server inputs the target KV compression result corresponding to the feature extraction module into the feature extraction module, and the feature extraction module compresses the global KV of different target words according to the target KV compression result to obtain the compressed KV of different target words.

[0181] S1108, for each feature extraction module, the server uses the feature extraction module to process the existing word KV to obtain the KV of the next predicted word in the predicted response text.

[0182] S1109, The server obtains the target response text of the target prompt text based on the KV of different predicted words in the corresponding predicted response text from the end feature extraction module.

[0183] S1110, the server sends the target response text to the display device.

[0184] S1111: After receiving the target response text, the display device controls the display to show the target response text.

[0185] The specific implementation methods of S1101-S1111 are the same as those in the above method embodiments, and will not be repeated here.

[0186] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores PCB product-related data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a PCB product quality inspection method.

[0187] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0193] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a computer device, the computer device integrating a target large language model, the target large language model including at least two cascaded feature extraction modules and a key-value (KV) compression module, each feature extraction module including a network layer and multiple attention heads connected to the network layer, the method comprising: The target prompt text is input into each feature extraction module to obtain the local key-value pairs (KVs) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text. The different target words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by each feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. For each feature extraction module, the key-value (KV) compression module compresses different local key-values ​​based on the compression ratio of the feature extraction module and the attention weights of different local key-values, thereby obtaining the target key-value compression result corresponding to the feature extraction module. The compression ratio of each feature extraction module is determined by the distribution parameters of the attention weights of different local key-values ​​output by the feature extraction module, and these distribution parameters include variance and / or entropy. The KV compression result of each target is input into the corresponding feature extraction module to obtain the target response text of the target prompt text.

2. The method according to claim 1, characterized in that, The step of inputting the target prompt text into each feature extraction module to obtain the local key-value pairs (KV) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text includes: For each feature extraction module, the network layer of the feature extraction module is used to extract features from the target prompt text to obtain global key-value pairs (KVs) for different target words, and different local KVs for each target word are assigned to multiple attention heads of the feature extraction module; wherein, each attention head is assigned a local KV for a different target word, and the local KVs of the same target word assigned by multiple attention heads constitute the global KV of the target word. The attention weights of the local key values ​​of the assigned target words are calculated using the multiple attention heads to obtain the attention weights of the different local key values ​​of each target word.

3. The method according to claim 1, characterized in that, The step of inputting the target prompt text into each feature extraction module to obtain the local key-value pairs (KV) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text includes: The target prompt text is segmented into different target words; For each target word, if there are historical words that match the target word in the preset storage space, read the different local key values ​​and attention weights of the historical words that match the target word in each feature extraction module from the preset storage space. If no historical word matching the target word exists in the preset storage space, the target word is input into each feature extraction module to obtain the different local key values ​​and attention weights of the target word output by each feature extraction module after processing the target word.

4. The method according to any one of claims 1-3, characterized in that, The step involves using the KV compression module to compress different local KV values ​​of the feature extraction module according to the compression ratio of the feature extraction module and the attention weights of different local KV values, thereby obtaining the target KV compression result corresponding to the feature extraction module, including: The KV compression module compresses different local KVs of the feature extraction module in ascending order of their attention weights until the proportion of uncompressed local KVs in all local KVs of the feature extraction module reaches the compression ratio of the feature extraction module.

5. The method according to claim 4, characterized in that, The step of inputting the compression result of each target KV into the corresponding feature extraction module to obtain the target response text of the target prompt text includes: For each feature extraction module, the target key-value compression result corresponding to the feature extraction module is input into the feature extraction module. The feature extraction module then compresses the global key-value pairs of different target words based on the target key-value compression result to obtain the compressed key-value pairs of different target words. Obtain the target response text of the target prompt text; wherein, the target response text is obtained by at least two cascaded feature extraction modules processing the compressed key-value pairs based on the corresponding different target words.

6. The method according to claim 5, characterized in that, The target response text for obtaining the target prompt text includes: For each feature extraction module, the feature extraction module processes the existing word units (KV) to obtain the KV of the next predicted word unit in the predicted response text; wherein, the predicted response text is the response result of the feature extraction module to the target prompt text; when the feature extraction module is the first feature extraction module, the existing word unit KV includes the compressed KV of different target words and the KV of the predicted words in the predicted response text; when the feature extraction module is not the first feature extraction module, the existing word unit KV includes the compressed KV of different target words, the KV of the predicted words in the predicted response text, and the KV of the latest predicted word unit output by the previous feature extraction module; The target response text of the target prompt text is output by the end feature extraction module based on the key-value pairs of different predicted words in the corresponding predicted response text.

7. The method according to claim 6, characterized in that, The target response text of the target prompt text, output by the end feature extraction module based on the key-value pairs (KV) of different predicted words in the corresponding predicted response text, includes: The response word corresponding to the KV of the preset number of predicted word words is output by the end feature extraction module when a preset number of predicted word words are generated; The different response tokens obtained are fused to obtain the target response text.

8. The method according to claim 7, characterized in that, The target large language model is trained in the following way: The sample prompt text is input into the initial large language model in the initial model to obtain the sample response text of the sample prompt text; The accuracy of the sample response text is determined based on the difference between the sample response text and the actual response text of the sample prompt text. Based on the accuracy and the overall compression ratio of the initial large language model, adjust the additional model parameters of the additional fine-tuning module in the initial model; The initial model parameters of the initial large language model are replaced with the fusion parameters of the initial model parameters and the additional model parameters to obtain the target large language model.

9. A server, characterized in that, The server integrates a target large language model, which includes at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. The server includes: A communication device configured to connect to an external device; and at least one processor, connected to the communication device, and configured to: The target prompt text is input into each feature extraction module to obtain the local key-value pairs (KVs) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text. The different target words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by each feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. For each feature extraction module, the key-value (KV) compression module compresses different local key-values ​​based on the compression ratio of the feature extraction module and the attention weights of different local key-values, thereby obtaining the target key-value compression result corresponding to the feature extraction module. The compression ratio of each feature extraction module is determined by the distribution parameters of the attention weights of different local key-values ​​output by the feature extraction module, and these distribution parameters include variance and / or entropy. The KV compression result of each target is input into the corresponding feature extraction module to obtain the target response text of the target prompt text.

Images