Data processing method and server

Abstract

The invention relates to a data processing method and a server, and relates to the technical field of large models. The method is applied to the computer equipment, a target large language model comprising at least two cascaded feature extraction modules and a KV compression module is integrated in the computer equipment, and the method comprises the following steps: respectively inputting a target prompt text into each feature extraction module, obtaining local KV of different target lexical elements output by each feature extraction module and attention weights of different local KV; for each feature extraction module, compressing different local KVs of the feature extraction module through a KV compression module according to a compression ratio of the feature extraction module and attention weights of different local KVs to obtain a target KV compression result corresponding to the feature extraction module; and inputting each target KV compression result into a corresponding feature extraction module to obtain a target response text of the target prompt text. By adopting the technical scheme, redundant calculation in the reasoning process can be reduced, and the reasoning speed and the reasoning precision can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large model, and particularly relates to a data processing method and a server. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, large language models have made significant progress. They can process prompt text input by users and output response text to the prompt text to meet various needs of users, such as knowledge query, text summary extraction, question answering, and text creation. Currently, how to reduce the memory occupation in the use of large language models and improve the inference efficiency has become one of the hot issues in the technical field of large models.

[0003] In the traditional technology, the KV cache (Key-Value Cache) is dynamically compressed in the inference process of the large language model, so that the KV cache is updated and compressed at each step of the inference process to reduce the number of word KV transmitted in the inference process, thereby reducing the memory occupation and computational load in the inference process.

[0004] However, in the above-mentioned traditional technology, the KV cache needs to be repeatedly updated and adjusted in the inference process, so that frequent memory operations and calculations are required, which leads to an increase in the repetition and delay of calculations and reduces the inference accuracy. SUMMARY

[0005] The present application provides a data processing method and a server to reduce redundant calculations in the inference process and improve the inference speed and accuracy.

[0006] In a first aspect, some embodiments provide a data processing method applied to a computer device, the computer device integrating a target large language model, the target large language model including at least two cascaded feature extraction modules and a key-value (KV) compression module, each feature extraction module including a network layer and a plurality of attention heads connected to the network layer, and the data processing method includes:

[0007] inputting a target prompt text into each feature extraction module to obtain local KV of different target word pieces and attention weights of different local KV output by each feature extraction module in processing the target prompt text, wherein the different target word pieces are obtained by splitting the target prompt text, and the different local KV of the same target word piece output by each feature extraction module are obtained by splitting the global KV of the target word piece output by the network layer in the feature extraction module;

[0008] According to the compression ratio of the feature extraction module and the attention weight of different local KVs, the KV compression module is used to compress different local KVs of the feature extraction module to obtain a target KV compression result corresponding to the feature extraction module.

[0009] Each target KV compression result is input into the corresponding feature extraction module to obtain a target response text of the target prompt text.

[0010] In the above embodiment, the target large language model is designed to include at least two cascaded feature extraction modules and a KV compression module, wherein each feature extraction module includes a network layer and a plurality of attention heads connected to the network layer. Then, the target prompt text is input into each feature extraction module to process the target prompt text by the feature extraction module, so as to obtain the local KV of different target tokens output by the feature extraction module and the attention weight of different local KVs. Different tokens are obtained by splitting the target prompt text, and different local KVs of the same target token output by the feature extraction module are obtained by splitting the global KV of the target token output by the network layer in the feature extraction module. Then, for each feature extraction module, the KV compression module is used to compress different local KVs output by the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KVs output by the feature extraction module, to obtain a target KV compression result corresponding to the feature extraction module. Then, the target KV compression result corresponding to each feature extraction module is input into the feature extraction module to obtain a target response text of the target prompt text. In this way, on the one hand, the different local KVs output by each feature extraction module are compressed once before the target large language model enters the inference process, so as to realize the compression of KV cache after the prefilling stage of the target large language model for the target prompt text and before the inference stage, avoid frequent KV cache compression and decompression operations in the inference process, reduce redundant calculations in the inference process, and thus reduce the computational overhead in the inference process, avoid the increase of the inference delay, and improve the inference speed. On the other hand, since the newly added KV cache is not compressed in the inference process, the increase of the inference error caused by the compression of the newly added KV cache is avoided, and the inference accuracy is improved.

[0011] In a second aspect, some embodiments also provide a server integrated with a target large language model, the target large language model including at least two cascaded feature extraction modules and a key-value (KV) compression module, each feature extraction module including a network layer and a plurality of attention heads connected to the network layer; the server includes:

[0012] A communication device configured to be connected with an external device;

[0013] and at least one processor connected with the communication device and configured to:

[0014] input the target prompt text into each feature extraction module respectively, to obtain different local KVs of different target word units and attention weights of different local KVs output by each feature extraction module after processing the target prompt text; wherein the different target word units are obtained by splitting the target prompt text, and the different local KVs of the same target word unit output by each feature extraction module are obtained by splitting the global KV of the target word unit output by the network layer in the feature extraction module;

[0015] for each feature extraction module, compress the different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weights of the different local KVs through the KV compression module, to obtain the target KV compression result corresponding to the feature extraction module;

[0016] input each target KV compression result into the corresponding feature extraction module, to obtain the target response text of the target prompt text.

[0017] In the above embodiments, the target large language model is designed to include at least two cascaded feature extraction modules and a KV compression module, wherein each feature extraction module includes a network layer and a plurality of attention heads connected to the network layer. Then, the target prompt text is input into each feature extraction module respectively, so that the feature extraction module processes the target prompt text to obtain the local KV of different target tokens output by the feature extraction module and the attention weight of different local KVs. Different tokens are obtained by splitting the target prompt text, and different local KVs of the same target token output by the feature extraction module are obtained by splitting the global KV of the target token output by the network layer in the feature extraction module. Then, for each feature extraction module, the KV compression module is used to compress the different local KVs output by the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of the different local KVs, to obtain the target KV compression result corresponding to the feature extraction module. Then, the target KV compression result corresponding to each feature extraction module is input into the feature extraction module to obtain the target response text of the target prompt text. In this way, on the one hand, the different local KVs output by each feature extraction module are compressed once before the target large language model enters the inference process, so that the KV cache is compressed after the prefilling stage of the target large language model for the target prompt text and before the inference stage, avoiding frequent KV cache compression and decompression operations in the inference process, reducing redundant calculations in the inference process, thereby reducing the computational overhead in the inference process, avoiding the increase of the inference delay, and improving the inference speed. On the other hand, since the newly added KV cache is not compressed in the inference process, the increase of the inference error caused by the compression of the newly added KV cache is avoided, and the inference accuracy is improved.

[0018] In a third aspect, a computer-readable storage medium is provided, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the steps of the method provided in some embodiments of the first aspect are implemented.

[0019] In a fourth aspect, a computer program product is provided, and the computer program product includes a computer program. When the computer program is executed by a processor, the steps of the method provided in some embodiments of the first aspect are implemented. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the following will briefly introduce the drawings needed to be used in the description of the embodiments of the present application or the related art. Obviously, the drawings in the following description only some embodiments of the present application, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0021] Figure 1 The schematic diagram of the operation scenario between the display device and the control device provided by some embodiments of the present application is shown.

[0022] Figure 2 The schematic diagram of the hardware configuration of the display device provided by some embodiments of the present application is shown.

[0023] Figure 3 The schematic diagram of the hardware configuration of the control device provided by some embodiments of the present application is shown.

[0024] Figure 4 The schematic diagram of the software configuration of the display device provided by some embodiments of the present application is shown.

[0025] Figure 5 The schematic diagram of the flow of the data processing method provided by some embodiments of the present application is shown.

[0026] Figure 6 The schematic diagram of the flow of determining the local KV of different target tokens output by each feature extraction module and the attention weight of different local KV provided by some embodiments of the present application is shown.

[0027] Figure 7 The schematic diagram of the flow of determining the compression ratio of each feature extraction module provided by some embodiments of the present application is shown.

[0028] Figure 8 The schematic diagram of the flow of obtaining the target response text provided by some embodiments of the present application is shown.

[0029] Figure 9 The schematic diagram of the flow of outputting the target response text provided by some embodiments of the present application is shown.

[0030] Figure 10 The schematic diagram of the flow of training the target large language model provided by some embodiments of the present application is shown.

[0031] Figure 11 The schematic diagram of the flow of the data processing method provided by some embodiments of the present application is shown.

[0032] Figure 12 The internal structure diagram of the computer device provided by some embodiments of the present application is shown. DETAILED DESCRIPTION

[0033] Embodiments will be described in detail below with reference to examples illustrated in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following embodiments are not meant to represent all implementations consistent with the present disclosure. Rather, they are merely examples of implementations consistent with some aspects of the present disclosure as detailed in the claims.

[0034] It should be noted that the brief description of terms in the present application is only for the convenience of understanding the following description of the embodiments, and is not intended to limit the embodiments of the present application. Unless otherwise specified, these terms should be understood according to their ordinary and general meanings.

[0035] The terms "first", "second", "third", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar or like objects or entities, and do not necessarily mean a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchanged under appropriate circumstances.

[0036] The terms "include" and "have" and any variations thereof are intended to cover but not exclusive inclusion, for example, a product or device including a series of components does not have to be limited to all components clearly listed, but can include other components not clearly listed or inherent to these products or devices.

[0037] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or a combination of hardware or / and software code capable of performing a function associated with the element.

[0038] In the conventional technology, the KV cache (Key-Value Cache) is usually dynamically compressed in real time during the inference process of a large language model, so that the KV cache is updated and compressed at each step of the inference process, to reduce the number of word KV transmitted in the inference process, and thus reduce the memory occupation and the calculation load in the inference process. However, in the above-mentioned conventional technology, the KV cache needs to be repeatedly updated and adjusted during the inference process, so that frequent memory operations and calculations are required, which leads to an increase in calculation repetition and delay, and reduces the inference accuracy.

[0039] Based on this, in some embodiments, a data processing method is provided. Wherein the data processing method can be applied to a computer device, which can be a server, a display device or other terminal. Wherein the server can be a stand-alone server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services.

[0040] In the embodiments of the present application, the display device 200 generally refers to a device with picture display and data processing capabilities. For example, the display device 200 includes but is not limited to a smart television, a mobile terminal, a computer, a monitor, an advertising screen, a wearable device, a virtual reality device, an augmented reality device, and the like.

[0041] Figure 1 The present application provides a schematic diagram of the operation scenario between the display device and the control device. As shown in Figure 1 The user can operate the display device 200 through a touch operation, a mobile terminal 300, and a control device 100. For example, the control device 100 can be a remote controller, a stylus, a handle, and the like.

[0042] The mobile terminal 300 can be used as a control device to perform human-computer interaction between the user and the display device 200. The mobile terminal 300 can also be used as a communication device to establish a communication connection with the display device 200 and perform data interaction. In some embodiments, the mobile terminal 300 can install a software application on the display device 200, implement connection communication through a network communication protocol, and achieve the purpose of one-to-one control operation and data communication. The mobile terminal 300 can also transmit audio and video content displayed on the mobile terminal 300 to the display device 200 to achieve a synchronous display function.

[0043] As shown in Figure 1 The display device 200 can also communicate with the server 400 through various communication methods. The display device 200 can be allowed to communicate through a local area network (LAN), a wireless local area network (WLAN), and other networks.

[0044] The display device 200 can provide a broadcast receiving television function, and can additionally provide a smart network television function with computer support, including but not limited to a network television, a smart television, an Internet protocol television (IPTV), and the like.

[0045] Figure 2 The present application provides a schematic diagram of the operation scenario between the display device and the control device. As shown in Figure 1 A hardware configuration block diagram of the display device 200 is shown in

[0046] In some embodiments, the display device 200 can include at least one of a tuning demodulator 210, a communication device 220, a detector 230, a device interface 240, a controller 250, a display 260, an audio output device 270, a memory, a power supply, and a user input interface.

[0047] In some embodiments, the detector 230 is configured to collect signals of the external environment or the external interaction. For example, the detector 230 includes a light receiver configured to collect the intensity of ambient light, or the detector 230 includes an image collector such as a camera configured to collect the scene of the external environment, the attribute of the user, or the interaction gesture of the user, or the detector 230 includes a sound collector such as a microphone configured to receive external sound.

[0048] In some embodiments, the display 260 includes a display component configured to present a picture, and a driving component configured to drive the display of the picture. The display 260 is configured to receive the picture signal output from the controller 250 and display the picture. For example, the display 260 can be configured to display video content, image content, and components of a menu control interface, and a user control UI interface.

[0049] In some embodiments, the communication device 220 is configured to communicate with the external device or the server 400 according to various communication protocol types. The display device 200 can be provided with multiple communication devices 220 according to the supported communication modes. For example, when the display device 200 supports wireless network communication, the display device 200 can be provided with a communication device 220 including a WiFi function. When the display device 200 supports Bluetooth connection communication, the display device 200 needs to be provided with a communication device 220 including a Bluetooth function.

[0050] The communication device 220 can be configured to connect the display device 200 to the external device or the server 400 through wireless or wired connection. The wired connection can be achieved by connecting the display device 200 to the external device through a data line, an interface, or the like. The wireless connection can be achieved by connecting the display device 200 to the external device through a wireless signal or a wireless network. The display device 200 can be directly connected to the external device, or can be indirectly connected to the external device through a gateway, a router, a connection device, or the like.

[0051] In some embodiments, the controller 250 can include at least one of a central processor, a video processor, an audio processor, a graphics processor, a power supply processor, a first interface to an n-th interface for input / output, and the controller 250 can control the operation of the display device and respond to the user's operation by controlling various software control programs stored in the memory. The controller 250 controls the overall operation of the display device 200.

[0052] In some embodiments, the controller 250 and the tuner demodulator 210 can be located in different split devices, i.e., the tuner demodulator 210 can also be located in an external device of the main device where the controller 250 is located, such as an external set-top box or the like.

[0053] In some embodiments, a user can input user commands through a graphical user interface (GUI) displayed on a display 260, and the user input interface receives user input commands through the graphical user interface (GUI).

[0054] In some embodiments, the audio output device 270 can be a built-in speaker of the display device 200 or an external audio output device connected to the display device 200. For the external audio output device connected to the display device 200, the display device 200 may also be provided with an external audio output terminal, through which the audio output device can be connected to the display device 200 to output sound from the display device 200.

[0055] In some embodiments, the user input interface 280 can be used to receive instructions from user input.

[0056] Figure 3 Provided for some embodiments of this application Figure 1 Hardware configuration block diagram of the central control device. (Example) Figure 3 As shown, the control device 100 may include: a controller 110, a communication interface 130, a user input / output interface, a memory, and a power supply.

[0057] The control device 100 is configured to control the display device 200, and to receive user input operation commands and convert the operation commands into commands that the display device 200 can recognize and respond to, thus acting as an intermediary for interaction between the user and the display device 200.

[0058] In some embodiments, the control device 100 may be an intelligent device. For example, the control device 100 may be equipped with various applications for controlling the display device 200 according to user needs.

[0059] In some embodiments, such as Figure 1 As shown, the mobile terminal 300 or other smart electronic devices can perform similar functions to the control device 100 after installing the application of the control display device 200.

[0060] The controller 110 includes a processor 112, RAM (Random Access Memory) 113, ROM (Read-Only Memory) 114, a communication interface 130, and a communication bus. The controller 110 is used to control the operation of the control device 100, as well as the communication and cooperation between internal components and the external and internal data processing functions.

[0061] Under the control of the controller 110, the communication interface 130 enables communication of control signals and data signals with the display device 200. The communication interface 130 may include at least one of other near-field communication modules such as WiFi chip 131, Bluetooth module 132, and NFC (Near Field Communication) module 133.

[0062] User input / output interface 140, wherein the input interface includes at least one of other input interfaces such as microphone 141, touchpad 142, sensor 143, and button 144.

[0063] In some embodiments, the control device 100 includes at least one of a communication interface 130 and an input / output interface 140. The control device 100 is configured with the communication interface 130, such as a WiFi, Bluetooth, or NFC module, which can encode user input commands via WiFi, Bluetooth, or NFC protocols and send them to the display device 200.

[0064] The memory 190 is used to store various operating programs, data, and applications for driving and controlling the control device 100 under the control of the controller. The memory 190 can also store various control signal instructions input by the user.

[0065] The power supply 180 is used to provide operating power support for the various components of the control device 100 under the control of the controller.

[0066] In some embodiments, to enable user interaction, the display device 200 may run an operating system. The operating system is a computer program used to manage and control the hardware and software resources of the display device 200. The operating system can (control the display device) provide a user interface, allowing users to interact with the display device 200 and supporting the running of various applications.

[0067] It should be noted that the operating system can be a native operating system based on a specific operating platform, a third-party operating system that is deeply customized based on a specific operating platform, or an independent operating system specifically developed for display devices.

[0068] An operating system can be divided into different modules or levels based on the functions it implements.

[0069] For example, such as Figure 4 As shown, in some embodiments, the system is divided into four layers, from top to bottom: the Applications layer (referred to as the "Application Layer"), the Application Framework layer (referred to as the "Framework Layer"), the System Library layer, and the Kernel layer.

[0070] In some embodiments, the application layer provides services and interfaces for applications, enabling the display device 200 to run applications and interact with the user based on the applications. The application layer may contain at least one application, which may be a built-in Windows program, system settings program, or clock program of the operating system; or it may be an application developed by a third-party developer. In specific implementations, the application packages in the application layer are not limited to the examples above.

[0071] The framework layer provides application programming interfaces (APIs) and a programming framework for applications. The application framework layer includes predefined functions. It acts as a central processing unit, determining the actions taken by applications within the application layer. Through the API, applications can access system resources and obtain system services during execution.

[0072] like Figure 4 As shown, the application framework layer in this embodiment includes a view system, managers, and content providers. The view system designs and implements the application's interface and interactions, and includes lists, grids, text boxes, and buttons. The managers include at least one of the following modules: an activity manager for interacting with all running activities in the system; a location manager for providing system services or applications with access to system location services; a package manager for retrieving various information related to application packages currently installed on the device; a notification manager for controlling the display and clearing of notification messages; and a window manager for managing icons, windows, toolbars, wallpapers, and desktop widgets on the user interface.

[0073] In some embodiments, the Activity Manager manages the lifecycle of individual applications and common navigation and back functions, such as controlling application exit, opening, and back actions. The Window Manager manages all window programs, such as obtaining the screen size, determining if a status bar is present, locking the screen, capturing the screen, and controlling changes to the display window, such as shrinking the display window, shaking the display, or distorting the display.

[0074] In some embodiments, the system runtime library layer can provide support for the framework layer. When the framework layer is used, the operating system runs the instruction library contained in the system runtime library layer, such as the C / C++ instruction library, to implement the functions to be performed by the framework layer.

[0075] In some embodiments, the kernel layer is a functional layer situated between the hardware and software of the display device 200. The kernel layer can implement functions such as hardware abstraction, multitasking, and memory management. For example, ... Figure 4 As shown, hardware drivers can be configured in the kernel layer. The drivers included in the kernel layer can be at least one of the following: audio driver, display driver, Bluetooth driver, camera driver, WIFI driver, USB (Universal Serial Bus) driver, HDMI (High Definition Multimedia Interface) driver, sensor driver (such as fingerprint sensor, temperature sensor, pressure sensor, etc.), and power driver, etc.

[0076] It should be noted that the above examples are merely a simple division of operating system functions and do not limit the specific form of the operating system of the display device 200 in this application embodiment. Depending on the function of the display device, the type of operating system, and other factors, the number of levels and the specific level type of the operating system may be expressed in other forms.

[0077] As mentioned above, in order to solve the technical problem that in the traditional technology, real-time dynamic compression of KV cache during the reasoning process of large language model requires repeated updating and adjustment of KV cache during the reasoning process, which requires frequent memory operations and calculations, resulting in increased repetitiveness and latency of calculations and reduced reasoning accuracy, a data processing method is provided in some embodiments, which can be applied to servers or the aforementioned display devices.

[0078] In an exemplary embodiment, taking the data processing method applied to a server as an example, the server includes a communication device and at least one processor. The communication connection between the communication device and an external device can be a wired communication connection or a wireless communication connection, and the at least one processor is connected to the communication device.

[0079] The server integrates a target large language model, which includes at least two cascaded feature extraction modules and a key-value compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer.

[0080] The so-called Large Language Model (LLM) refers to a deep learning model trained on a large amount of text data, enabling it to generate natural language text or understand the meaning of language text. Its core idea is to learn the patterns and structures of natural language through large-scale unsupervised training, mimicking the human language cognition and generation process to some extent. Optionally, the Large Language Model uses a Transformer architecture and relies on a self-attention mechanism.

[0081] An attention head is an independent computational unit that captures multi-dimensional semantic associations. Each attention head is responsible for capturing the semantic dependencies between different elements in the input sequence (such as lexical units) from a specific perspective. Multiple attention heads work in parallel, and the results from all attention heads are concatenated and integrated through linear transformations to form a complete multi-head attention output. This enables multi-dimensional sequence information modeling, allowing large language models to simultaneously consider multi-dimensional dependencies and improve their ability to understand complex sequences. For example, when processing the text "Xiaohong went to school today, and she was very sad," one attention head might focus on the referential association between "she" and "Xiaohong," while another attention head might focus on the action-location association between "go" and "school." Ultimately, the results from all attention heads are combined to enable the large language model to fully understand the semantic logic of the text.

[0082] like Figure 5 As shown, a data processing method applied to a processor in a server includes the following steps:

[0083] S 501, input the target prompt text into each feature extraction module respectively, and obtain the local key-value pairs of different target words and the attention weights of different local key-value pairs output by each feature extraction module after processing the target prompt text.

[0084] Among them, different target words are obtained by splitting the target prompt text, and the different local key-value pairs of the same target word output by each feature extraction module are obtained by splitting the global key-value pairs of the target word output by the network layer in the feature extraction module.

[0085] Optionally, the target prompt text can be input by the user to the server through text input, file upload, voice recording, etc. Specifically, when the user inputs text or uploads a file, the server can directly determine the user's output text or uploaded file as the target prompt text; similarly, when the user records voice, the server can convert the user's recorded voice into text and use it as the target prompt text.

[0086] Attention weights are numerical values ​​used in attention mechanisms to measure the strength of the association between the query vector (Q) and the key vector (K). They determine the "attention" that different input tokens should be assigned during the inference process of a large language model. Their core function is to make the large language model "focus" on the input tokens that are most relevant to the current query vector when calculating the value vector (V / alue, V) of the output tokens during the inference process.

[0087] A token is the smallest semantic or grammatical unit obtained after breaking down the original text; it is also called a word example or tag. For example, the original text "I love eating apples" can be broken down into "I", "love", "eat", "apple", and "fruit", a total of five tokens. Optionally, in Chinese text, tokens can be single-character tokens (such as "I") or multi-character tokens (such as "today" and "apple").

[0088] For the target prompt text, it can be segmented into multiple target words. Optionally, the target large language model includes an embedding layer. The target prompt text is input into the embedding layer, which performs word segmentation on the target prompt text to obtain multiple target words. Each target word is then converted into an initial vector (e.g., a 512-dimensional vector). These initial vectors of different target words are then input into each feature extraction module to obtain the local key-value pairs (KV) and attention weights of different local KVs for each target word output by the feature extraction module. Alternatively, the server uses a word segmentation tool to perform word segmentation on the target prompt text to obtain multiple target words. These multiple target words are then input into the embedding layer, which converts each target word into an initial vector (e.g., a 512-dimensional vector). These initial vectors of different target words are then input into each feature extraction module to obtain the local KV and attention weights of different local KVs for each target word output by the feature extraction module.

[0089] For each feature extraction module, the network layer within that module determines the key-value pair (KV) for each target word, which serves as the global KV for that target word. Furthermore, for each target word, the network layer in the feature extraction module can decompose the global KV into multiple distinct local KVs, and assign these local KVs to different attention heads within the feature extraction module. Specifically, for each target word, each attention head in the feature extraction module is assigned a local KV for that target word; different attention heads assign different local KVs, and the local KVs assigned by all attention heads constitute the global KV for that target word. Optionally, the global KV of the target word can be evenly decomposed into multiple local KVs, and these local KVs can be evenly distributed among multiple attention heads. For example, if the global key-value pair of the target word is represented by a 512-dimensional vector and there are 8 attention heads, then each attention head is assigned a local key-value pair that includes 64 dimensions of data. For example, the first attention head is assigned data from dimensions 1 to 64, the second attention head is assigned data from dimensions 65 to 128, the third attention head is assigned data from dimensions 129 to 192, and so on. Each attention head has a local key-value pair that includes 64 dimensions of data.

[0090] Furthermore, for each attention head, it can further determine the attention weight of each local key-value pair assigned to it. Optionally, for each attention head, it can determine the attention weight of that local key-value pair based on the K in each assigned local key-value pair and the Q assigned to that local key-value pair.

[0091] In an optional embodiment, S501 may include, for each feature extraction module, using the network layer of the feature extraction module to extract features from the target prompt text, obtaining global key-value pairs (KVs) for different target words, and assigning different local KVs for each target word to multiple attention heads of the feature extraction module; wherein, each attention head assigns a local KV for a different target word, and the local KVs of the same target word assigned by multiple attention heads constitute the global KV of the target word; using multiple attention heads to calculate attention weights for the assigned local KVs of different target words, obtaining the attention weights of different local KVs for each target word.

[0092] Optionally, the network layer in this feature extraction module can extract features for each target word to obtain the global key-value pair (KV) of that target word. Furthermore, optionally, each attention head can determine the attention weight of each local key-value pair assigned to it based on the contextual relationships of different words in the target prompt text and the order of the target words in the global KV to which its assigned local key-value pairs belong in the target prompt text.

[0093] Thus, for each feature extraction module, the local key-value pairs (KVs) of different target words and the attention weights of different local KVs can be obtained; wherein, the different target words include all target words obtained by splitting the target prompt text, and the different local KVs include all local KVs obtained by splitting the global KV of all target words.

[0094] S 502, for each feature extraction module, the KV compression module compresses different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KVs, to obtain the target KV compression result corresponding to the feature extraction module.

[0095] The so-called KV compression module is a functional module that reduces the amount of local KV of different target words output by the feature extraction modules of the target large language model. This is to reduce the memory usage of the KV cache, reduce the amount of KV data transmitted during the inference process, and improve the inference speed while keeping the inference accuracy of the target large language model unchanged or only slightly reduced.

[0096] Optionally, the compression ratio of different feature extraction modules can be the same preset fixed ratio, or they can be different preset fixed ratios. The preset fixed ratio can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed on this.

[0097] Optionally, the compression ratio of each feature extraction module refers to the proportion of uncompressed local KVs in all local KVs of the feature extraction module after different local KVs of the feature extraction module are compressed.

[0098] Compression of different local key-value pairs (KVs) for each feature extraction module refers to the following: For each feature extraction module, the product of the total number of different local KVs output by that module and the compression ratio of that module is determined. This product represents the number of local KVs remaining after compression. Furthermore, when different local KVs are arranged in descending order of attention weight, this product represents the order of the last local KV that will not be compressed. All local KVs ranked after this last local KV will be compressed. For example, if the total number is 100 and the compression ratio is 20%, then when the different local KVs are arranged in descending order of attention weight, the KV compression module will compress all local KVs ranked from 21st to 100th. Optionally, compression refers to deletion or adding a mask.

[0099] In an optional embodiment, S502 may include, for each feature extraction module, using a KV compression module, compressing different local KVs of the feature extraction module sequentially according to their attention weights from smallest to largest, until the proportion of uncompressed local KVs in all local KVs of the feature extraction module reaches the compression ratio of the feature extraction module. For example, if the total number of all local KVs in the feature extraction module is 100 and the compression ratio is 20%, then the KV compression module starts from the local KV with the smallest attention weight and compresses different local KVs of the feature extraction module sequentially according to their attention weights from smallest to largest, until the number of uncompressed local KVs is 20.

[0100] In another optional embodiment, S502 may include, for each feature extraction module, using a KV compression module to sequentially traverse different local KVs of the feature extraction module in descending order of their attention weights, until the proportion of the traversed local KVs in the total number of local KVs of the feature extraction module reaches the compression ratio of the feature extraction module, and then compressing all the remaining untraversed local KVs. For example, if the total number of local KVs in the feature extraction module is 100 and the compression ratio is 20%, then the KV compression module starts from the local KV with the largest attention weight and sequentially traverses different local KVs of the feature extraction module in descending order of their attention weights, until the number of traversed different local KVs reaches 20, and then compresses all the remaining untraversed local KVs.

[0101] Optionally, a key-value (KV) compression module can be used to compress all local key-value pairs output by all feature extraction modules according to a preset compression ratio and the attention weights of different local key-value pairs. This yields the target key-value compression result for each feature extraction module. The preset compression ratio can be set based on empirical values, experimental results from multiple trials, and the inference accuracy requirements of the target large language model in the actual application scenario. This makes the target key-value compression result for each feature extraction module more controllable in offline scenarios.

[0102] S503, input the KV compression result of each target into the corresponding feature extraction module to obtain the target response text of the target prompt text.

[0103] After obtaining the target key-value compression result corresponding to each feature extraction module, the compression result of each target key-value can be input into its corresponding feature extraction module. This feature extraction module can then determine the compressed local key-values ​​in the global key-value of each target word obtained by the network layer in the module based on the received target key-value compression result. Thus, the uncompressed local key-values ​​in the global key-value of each target word are compressed, such as by adding masks or pruning, to obtain compressed key-values ​​for different target words. Optionally, for target words whose all local key-values ​​are compressed, the global key-value of that target word in the feature extraction module can be directly deleted. Then, the at least two cascaded feature extraction modules included in the target large language model can infer the target response text of the target prompt text based on the compressed key-values ​​of the corresponding different target words and output the inferred target response text. Optionally, the process by which at least two cascaded feature extraction modules included in the target large language model can infer the target response text based on the compressed KV of the corresponding different target words can be called the self-decoding regression process of the target large language model.

[0104] In an optional embodiment, S503 may include: for each feature extraction module, inputting the target KV compression result corresponding to the feature extraction module into the feature extraction module, and having the feature extraction module compress the global KV of different target words according to the target KV compression result to obtain the compressed KV of different target words; obtaining the target response text of the target prompt text; wherein the target response text is obtained by at least two cascaded feature extraction modules processing the compressed KV of the corresponding different target words.

[0105] Optionally, during the process of the target large language model processing the target prompt text to obtain the target response text, metrics such as first byte time, GPU utilization, compression ratio, and key-value hit rate can be monitored and collected throughout the entire process. The orchestration system can then be driven by rules or learning-based strategies. When concurrency surges or GPU load becomes excessively high, the system automatically scales horizontally; when load decreases or cache hit rate continues to improve, it smoothly scales down to release resources. This ensures that the target large language model maintains its predetermined service level under different business peaks, while avoiding long-term over-provisioning and resulting cost waste.

[0106] In the above embodiments, a target large language model is designed, comprising at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. The target prompt text is then input into each feature extraction module, which processes the text to obtain local key-value pairs (KVs) of different target words and their attention weights. The different words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by the feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. Then, for each feature extraction module, the KV compression module compresses the different local KVs output by the feature extraction module according to its compression ratio and the attention weights of the different local KVs, to obtain the target KV compression result corresponding to that feature extraction module. Finally, the target KV compression result corresponding to each feature extraction module is input into the feature extraction module to obtain the target response text of the target prompt text. In this way, on the one hand, before the target large language model enters the inference process, the different local key-value pairs output by each feature extraction module are compressed all at once. This compression of the key-value cache is achieved after the prefilling stage of the target large language model for the target prompt text and before the inference stage, avoiding frequent key-value cache compression and decompression operations during inference, reducing redundant calculations in inference, thereby reducing computational overhead, avoiding increased latency, and improving inference speed. On the other hand, since new key-value caches are not compressed during inference, the increase in inference error caused by compressing new key-value caches is avoided, thus improving inference accuracy.

[0107] Based on the above embodiments, in an exemplary embodiment, the determination of the local key-value pairs (KVs) of different target words output by each feature extraction module and the attention weights of different local KVs are further refined. Optionally, such as Figure 6 As shown, the following steps may be included:

[0108] S601, perform word segmentation on the target prompt text to obtain different target words.

[0109] After obtaining the target prompt text, various methods, such as word segmentation tools, can be used to split the target prompt text into word units to obtain different target word units. The specific implementation method of word unit splitting of the target prompt text is not limited; any specific implementation method capable of splitting the target prompt text into word units falls within the protection scope of this application.

[0110] S602, for each target word, if there are historical words matching the target word in the preset storage space, read from the preset storage space the different local key-value pairs and the attention weights of the different local key-value pairs obtained in each feature extraction module for the historical words matching the target word.

[0111] Understandably, in scenarios involving continuous input and output to a target large language model, such as multi-turn dialogues, the target response text output by the target large language model each time will be influenced by the prompt text and response text from the previous or previous rounds of dialogue. Therefore, in order to further improve the inference speed of the target large language model, after each round of dialogue, the local key-value pairs (KVs) and attention weights of the target words newly appearing in this round of dialogue obtained in different feature extraction modules can be stored. This allows the target large language model to directly use the local KVs and attention weights of the target words obtained in different feature extraction modules during subsequent inference processes, without having to repeatedly process the same target words in each feature extraction module.

[0112] Accordingly, after obtaining different target words in the target prompt text, for each target word, a search can be conducted in a preset storage space to determine if a historical word matching that target word exists. Optionally, for each target word, the search can either determine if the target word exists in the preset storage space, or search for historical words in the preset storage space that have a similarity greater than a preset similarity. The preset similarity can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed on this.

[0113] In this way, for each target word, if there are historical words that match the target word in the preset storage space, the different local key-value pairs and attention weights of the historical words that match the target word can be directly read from the preset storage space in each feature extraction module, without having to repeatedly process the target word in each feature extraction module.

[0114] Optionally, the preset storage space can be storage space in other storage devices on the server besides video memory, or it can be storage space in external storage devices of the server, such as disks or cloud storage. This can reduce video memory usage, thereby reducing memory usage, effectively solving the memory bottleneck problem, and improving the inference efficiency of the target large language model.

[0115] S603, for each target word, if there is no historical word matching the target word in the preset storage space, the target word is input into each feature extraction module respectively, and the different local key values ​​and attention weights of the target word output by each feature extraction module after processing the target word are obtained.

[0116] Accordingly, for each target word, if there is no historical word matching the target word in the preset storage space, the target word needs to be processed by each feature extraction module to obtain the different local key values ​​and attention weights of the target word.

[0117] The specific implementation of S603 is similar to that of S501, and will not be repeated here.

[0118] Optionally, the data processing method further includes, for each target word, if there are no historical words matching the target word in the preset storage space, after obtaining the different local key-value pairs and attention weights of the target word from the output of each feature extraction module, storing the different local key-value pairs and attention weights of the target word in different feature extraction modules into the preset storage space. This allows the target word to be directly read from the preset storage space when it is split again to obtain the target word or other words matching the target word in subsequent use of the target large language model, without having to perform repeated processing by each feature extraction module.

[0119] Optionally, the data processing method further includes storing the different local key-value pairs and attention weights of the newly appearing response words in the target response text in different feature extraction modules into the aforementioned preset storage space. This allows the response words or other words matching the response words to be split again in subsequent use of the target large language model. In this case, the different local key-value pairs and attention weights of the response words obtained in each feature extraction module can be directly read from the aforementioned preset storage space without having to perform repeated processing in each feature extraction module.

[0120] In this embodiment, by storing the different local key-value pairs and attention weights of different local key-value pairs obtained in each feature extraction module for historical word units in a preset storage space, during the processing of the target prompt text by the target large language model, for target word units with matching historical word units in the preset storage space, the different local key-value pairs and attention weights of different local key-value pairs obtained in each feature extraction module for the historical word units matching the target word units can be directly read from the preset storage space, without the need for repeated processing of the target word units by each feature extraction module. Thus, by adopting the KV offload technique, the KV cache generated by the target large language model can be transferred from the video memory to other storage spaces, which can reduce the video memory usage and support larger batch sizes or longer context lengths, thereby further improving the inference efficiency of the target large language model.

[0121] Based on the above embodiments, in one exemplary embodiment, the determination of the compression ratio of each feature extraction module is further refined. Optionally, such as Figure 7 As shown, the following steps may be included:

[0122] S701, determine the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module.

[0123] The distribution parameters include variance and / or entropy.

[0124] Variance is a metric used to measure the dispersion of attention weights in different local key-value pairs output by a feature extraction module. The greater the dispersion, the greater the variance, and vice versa.

[0125] Correspondingly, entropy is an indicator used to measure the disorder of the probability distribution of attention weights of different local key values ​​output by a feature extraction module. The more concentrated the probability distribution (a few values ​​have extremely high probabilities and most values ​​have extremely low probabilities), the smaller the entropy value. Conversely, the more uniform the probability distribution (the closer the probabilities of all values ​​are), the larger the entropy value.

[0126] For each feature extraction module, the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module can be determined based on the attention weights of the different local key values ​​output by the feature extraction module.

[0127] S702, determine the compression ratio of the feature extraction module based on the distribution parameters.

[0128] For each feature extraction module, the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module can indicate the importance distribution of different local key values ​​output by the feature extraction module in the inference process, and this importance distribution can affect the compression ratio of the feature extraction module.

[0129] For example, if the variance of the attention weights of different local key values ​​(KVs) output by the feature extraction module is small, it indicates that the dispersion of the attention weights of these KVs is small, meaning the differences between their attention weights are small. Consequently, the importance of these different local KVs during inference is relatively consistent. Therefore, to ensure inference accuracy, it is necessary to retain as many different local KVs as possible, resulting in a higher compression ratio for the feature extraction module. Conversely, if the variance of the attention weights of different local KVs output by the feature extraction module is large, it indicates that the dispersion of their attention weights is large, meaning the differences between their attention weights are large. Consequently, the importance of these different local KVs during inference varies greatly; some local KVs are more important than others. Therefore, to improve inference efficiency, it is advisable to compress as many less important local KVs as possible, resulting in a lower compression ratio for the feature extraction module.

[0130] For example, if the entropy value of the attention weights of different local key-value pairs output by the feature extraction module is large, it indicates that the probability distribution of the attention weights of these pairs is relatively uniform, meaning the differences between their attention weights are small. Therefore, the importance of these local key-value pairs in the inference process is relatively consistent. To ensure inference accuracy, it is necessary to retain as many of the different local key-value pairs as possible, resulting in a higher compression ratio for the feature extraction module. Conversely, if the entropy value of the attention weights of different local key-value pairs output by the feature extraction module is small, it indicates that the probability distribution of their attention weights is relatively concentrated, meaning the differences between their attention weights are large. This indicates that the importance of these local key-value pairs varies significantly during inference; some local key-value pairs are more important than others. Therefore, to improve inference efficiency, it is possible to compress as many less important local key-value pairs as possible, resulting in a lower compression ratio for the feature extraction module.

[0131] Based on this, for each feature extraction module, the compression ratio of the feature extraction module can be determined according to the variance and / or entropy of the attention weights of different local key values ​​output by the feature extraction module.

[0132] Optionally, a pre-defined correspondence between different variance ranges and compression ratios can be established. Thus, for each feature extraction module, after obtaining the variance of the attention weights of different local key-value pairs output by that module, the range to which the variance belongs is determined, and the compression ratio corresponding to that range is defined as the compression ratio of that feature extraction module. This correspondence can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed.

[0133] Optionally, a pre-defined correspondence between different entropy value ranges and compression ratios can be established. Thus, for each feature extraction module, after obtaining the entropy of the attention weights of different local key-value pairs output by that module, the value range to which the entropy belongs is determined, and the compression ratio corresponding to that value range is determined as the compression ratio of that feature extraction module. The aforementioned correspondence can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed on this.

[0134] Optionally, a pre-defined correspondence between different combinations of variance and entropy value intervals and compression ratios can be established. Thus, for each feature extraction module, after obtaining the variance and entropy of the attention weights of different local key-value pairs output by that module, the value intervals to which the variance and entropy belong are determined. The compression ratio corresponding to the combination of these value intervals is then determined as the compression ratio for that feature extraction module. This correspondence can be set based on empirical values, experimental values ​​from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios; no specific limitations are imposed.

[0135] In this embodiment, for each feature extraction module, the compression ratio of the feature extraction module can be dynamically generated by determining the distribution parameters of the attention weights of different local key-value pairs output by the feature extraction module. This allows the compression ratio of each feature extraction module to be dynamically adjusted according to the actual computational load of the feature extraction module. As a result, not only can the computational resource utilization of the target large language model be improved, but the waste of computational resources caused by a fixed compression ratio can also be avoided. Furthermore, more flexible optimization can be performed for the computational needs of different feature extraction modules, further improving the inference performance and inference efficiency of the target large language model.

[0136] Based on the above embodiments, in one exemplary embodiment, the acquisition of the target response text is further refined. Optionally, such as... Figure 8 As shown, the following steps may be included:

[0137] S801, for each feature extraction module, the feature extraction module processes the existing word KV to obtain the KV of the next predicted word in the predicted response text.

[0138] The predicted response text is the response result of the feature extraction module to the target prompt text. When the feature extraction module is the first feature extraction module, the existing word KV includes the compressed KV of different target words and the KV of the predicted words in the predicted response text. When the feature extraction module is not the first feature extraction module, the existing word KV includes the compressed KV of different target words, the KV of the predicted words in the predicted response text, and the KV of the latest predicted words output by the previous feature extraction module.

[0139] In the reasoning process of the target large language model for the target prompt text, each feature extraction module can obtain a response result for the target prompt text, which serves as the predicted response text for that feature extraction module. Furthermore, the predicted response text of the last feature extraction module is the final output target response text. Similar to the target prompt text, the predicted response text of each feature extraction module is also determined word-by-word. Specifically:

[0140] For the first feature extraction module, the feature extraction module can infer the KV of the next predicted word in the predicted response text based on the compressed KV of the different target words of the feature extraction module and the KV of the predicted words in the predicted response text (i.e. the KV of the words in the predicted response text that have been predicted and belong to the feature extraction module), and transmit the inferred KV of the next predicted word to the next feature extraction module cascaded by the first feature extraction module.

[0141] Accordingly, for each feature extraction module that is not the first one, it can infer the KV of the next predicted word in the predicted response text based on the compressed KV of the target words of that feature extraction module, the KV of the predicted words in the predicted response text, and the KV of the latest predicted word output from the previous feature extraction module cascaded with that feature extraction module. Where this feature extraction module is not the last feature extraction module, it can transmit the inferred KV of the next predicted word to the next feature extraction module cascaded with it.

[0142] Furthermore, for the end feature extraction module, the KV of the next predicted token it predicts is the KV of the token in the target response text.

[0143] S802, obtain the target response text of the target prompt text output by the end feature extraction module based on the KV of different predicted tokens in the corresponding predicted response text.

[0144] Since the predicted response text predicted by the end feature extraction module is the target response text of the target prompt text, after the end feature extraction module predicts the complete predicted response text, the end feature extraction module can convert the KV of different predicted tokens in the predicted response text it predicts into response tokens in the target response text in text form, and output the response tokens after conversion of the KV of different predicted tokens in order from the earliest to the latest prediction time of the KV of different predicted tokens, so that the server can obtain the target response text of the target prompt text.

[0145] For example, the target prompt text is "Today's weather", the different target tokens are "今", "天", "的", "天", and "气", and the target large language model includes three cascaded feature extraction modules 1 to 3. Specifically:

[0146] The feature extraction module 1 determines the KV of the first predicted token "天" in the predicted response text of the feature extraction module 1 according to the compressed KV of different target tokens of the feature extraction module 1, and transmits the predicted KV of "天" to the feature extraction module 2; the feature extraction module 2 predicts the KV of the first predicted token "天" in the predicted response text of the feature extraction module 2 according to the compressed KV of different target tokens of the feature extraction module 2 and the latest predicted token "天" KV transmitted by the feature extraction module 1, and transmits the predicted KV of "天" to the feature extraction module 3; the feature extraction module 3 predicts the KV of the first predicted token "天" in the predicted response text of the feature extraction module 3 according to the compressed KV of different target tokens of the feature extraction module 3 and the latest predicted token "天" KV transmitted by the feature extraction module 2, then the KV of "天" predicted by the feature extraction module 3 is also the KV of the first token in the target response text.

[0147] The feature extraction module 1 determines the KV of the second predicted token "qi" in the predicted response text of the feature extraction module 1 based on the compressed KVs of different target tokens of the feature extraction module 1 and the KV of the already predicted token "tian", and transmits the predicted KV of "qi" to the feature extraction module 2; the feature extraction module 2 predicts the KV of the second predicted token "tian" in the predicted response text of the feature extraction module 2 based on the compressed KVs of different target tokens of the feature extraction module 2, the KV of the already predicted tokens "tian" and "qi", and the latest predicted KV of "qi" transmitted by the feature extraction module 1, and transmits the predicted KV of "qi" to the feature extraction module 3; the feature extraction module 3 predicts the KV of the second predicted token "qi" in the predicted response text of the feature extraction module 3 based on the compressed KVs of different target tokens of the feature extraction module 3, the KV of the already predicted tokens "tian" and "qi", and the latest predicted KV of "qi" transmitted by the feature extraction module 2, then the KV of "qi" predicted by the feature extraction module 3 is also the KV of the second token in the target response text.

[0148] The feature extraction module 1 determines the KV of the third predicted token "qing" in the predicted response text of the feature extraction module 1 based on the compressed KVs of different target tokens of the feature extraction module 1 and the KVs of the already predicted tokens "tian" and "qi", and transmits the predicted KV of "qing" to the feature extraction module 2; the feature extraction module 2 predicts the KV of the third predicted token "qing" in the predicted response text of the feature extraction module 2 based on the compressed KVs of different target tokens of the feature extraction module 2, the KVs of the already predicted tokens "tian" and "qi", and the latest predicted KV of "qing" transmitted by the feature extraction module 1, and transmits the predicted KV of "qing" to the feature extraction module 3; the feature extraction module 3 predicts the KV of the third predicted token "qing" in the predicted response text of the feature extraction module 3 based on the compressed KVs of different target tokens of the feature extraction module 3, the KVs of the already predicted tokens "tian" and "qi", and the latest predicted KV of "qing" transmitted by the feature extraction module 2, then the KV of "qing" predicted by the feature extraction module 3 is also the KV of the third token in the target response text.

[0149] Furthermore, the last feature extraction module can sequentially output the response tokens in text form obtained by converting the KVs of the already predicted tokens "tian", "qi", and "qing" in the order of the prediction times of the KVs of the already predicted tokens "tian", "qi", and "qing" from earliest to latest, that is, the last feature extraction module sequentially outputs the texts "tian", "qi", and "qing". Thus, the server can obtain the target response text "tian qi qing" of the target prompt text.

[0150] In this embodiment, frequent key-value cache compression and decompression operations are avoided during the inference process of the target large language model, reducing redundant calculations and thus computational overhead, preventing increased latency, and improving inference speed. Furthermore, by not compressing newly added key-value caches during inference, the increased inference error caused by compressing new key-value caches is avoided, thereby improving inference accuracy.

[0151] Based on the above embodiments, in one exemplary embodiment, the output of the target response text is further refined. Optionally, such as... Figure 9 As shown, the following steps may be included:

[0152] S901, Obtain the response word corresponding to the KV of the preset number of predicted word words output by the end feature extraction module when generating KV of the preset number of predicted word words.

[0153] As mentioned earlier, the predicted response text obtained by the end feature extraction module is the target response text of the target prompt text. Therefore, the end feature extraction module does not need to wait until all the key-value pairs of the predicted words in the predicted response text have been obtained before outputting the response words. Instead, it can output the response words obtained by converting the key-value pairs of each predicted word in the key-value pairs of the predicted words in the order of their generation time from earliest to latest, so that the server can obtain the response words corresponding to the key-value pairs of the predicted words.

[0154] S902, the different response tokens obtained are fused to obtain the target response text.

[0155] The server can obtain multiple sets of response tokens, and each set of response tokens includes a predicted number of response tokens. Therefore, the server can concatenate and merge the multiple sets of response tokens in order of their acquisition time from earliest to latest to obtain the target response text.

[0156] In this embodiment, during the reasoning process of the target large language model, the target words in the obtained target response text can be output step by step according to the reasoning progress and with a preset number as the output granularity. This can reduce the TTFT (Time To First Token) of the target large language model, reduce the latency of the target large language model, improve the reasoning efficiency of the target large language model, and improve the user experience in scenarios such as multi-turn dialogue and long text generation.

[0157] Based on the above embodiments, in an exemplary embodiment, the training of the target large language model is further refined. Optionally, such as... Figure 10 As shown, the following steps may be included:

[0158] S1001, Input the sample prompt text into the initial large language model in the initial model to obtain the sample response text of the sample prompt text.

[0159] Understandably, the initial large language model includes at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. Optionally, the sample prompt text is input into each feature extraction module of the initial large language model to obtain the local key-value pairs (KVs) of different sample words and the attention weights of different local KVs output by each feature extraction module of the initial large language model after processing the sample prompt text. For each feature extraction module of the initial large language model, the KV compression module compresses the different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weights of different local KVs to obtain the sample KV compression result corresponding to the feature extraction module. Each sample KV compression result is input into the corresponding feature extraction module in the initial large language model to obtain the sample response text of the sample prompt text.

[0160] The specific implementation of S1001 is the same as the generation method of the target response text of the target prompt text, and will not be repeated here.

[0161] S1002, Determine the accuracy of the sample response text based on the difference between the sample response text and the sample prompt text and the actual response text.

[0162] After obtaining the sample response text of the sample prompt text, the difference between the sample response text and the actual response text of the sample prompt text can be determined, and then the accuracy of the sample response text can be determined based on this difference.

[0163] Optionally, the similarity between the sample response text and the sample prompt text and the actual response text is determined, and this similarity is used as the accuracy of the sample response text.

[0164] S1003, based on accuracy and the overall compression ratio of the initial large language model, adjusts the additional model parameters of the additional fine-tuning module in the initial model.

[0165] The overall compression ratio refers to the proportion of local key-value pairs in the total key-value compression result of all samples within the total local key-value pairs of the sample word units in the sample prompt text. Optionally, the overall compression ratio is the average of the compression ratios of each feature extraction module in the initial large language model.

[0166] In order to maximize the inference speed of the target large language model while ensuring its inference accuracy, it is often desirable to achieve an accuracy of the sample response text that is not less than the accuracy threshold, even when the overall compression ratio of the sample large language model reaches or falls below the compression ratio threshold. Therefore, the loss value of the loss function of the sample large language model can be determined based on the difference between the accuracy of the sample response text and the accuracy threshold, as well as the difference between the overall compression ratio and the compression ratio threshold. Based on this loss value, the additional model parameters of the fine-tuning module in the initial model can be adjusted. Optionally, the reconstruction error caused by the compressed local key-value pairs can be included as an error component in the loss function of the sample large language model.

[0167] Optionally, if the determined loss value is greater than the loss value threshold, adjust the additional model parameters of the additional fine-tuning module in the initial model and return to step S1001 until the determined loss value is not greater than the loss value threshold. At this point, it can be determined that the initial model has converged, thereby completing the model training process and proceeding to step S1004.

[0168] Optionally, the additional fine-tuning module is the LoRA (Low-Rank Adaptation) module. In essence, it introduces a small number of trainable low-rank matrices as additional model parameters on the basis of the original model structure of the initial large language model to construct the initial model. During the training of the initial model, all or part of the model parameters in the initial large language model are frozen, and only the additional model parameters, or the small number of model parameters that are not frozen and the additional model parameters are adjusted, until the initial model converges.

[0169] The accuracy threshold, compression ratio threshold, and loss value threshold can all be set based on empirical values, experimental results from multiple trials, and the inference accuracy requirements of the target large language model in actual application scenarios. For example, the accuracy threshold could be 80%, and the compression ratio threshold could be 50%.

[0170] S1004, replace the initial model parameters of the initial large language model with the fusion parameters of the initial model parameters and the additional model parameters to obtain the target large language model.

[0171] After adjusting the additional model parameters of the additional fine-tuning module through multiple rounds of iterations to make the initial model converge, training is stopped. The initial model parameters of the initial large language model are then fused with the additional model parameters of the additional fine-tuning module after the initial model has converged to obtain fused parameters. The initial model parameters of the initial large language model are then replaced with the above fused parameters to obtain the target large language model.

[0172] In this embodiment, by adjusting the additional model parameters of the additional fine-tuning module, the negative impact of local key-value compression on the inference accuracy of the target large language model is resolved. Furthermore, by introducing the additional fine-tuning module, the consistency between training and inference can be improved while maintaining the inference accuracy of the target large language model unaffected by local key-value compression. This allows the target large language model to effectively adapt to the impact of local key-value compression during training and maximizes the preservation of the inference performance of the target large language model.

[0173] Based on the above embodiments, in an exemplary embodiment, such as Figure 11 As shown, the data processing method may include the following steps:

[0174] S1101, The display device sends the target prompt text to the server.

[0175] S1102, after receiving the target prompt text, the server performs word segmentation on the target prompt text to obtain different target words. The server integrates a target large language model, which includes at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer.

[0176] S1103, for each target word, if there are historical words matching the target word in the preset storage space, the server reads the different local key-value pairs and attention weights of the historical words matching the target word obtained in each feature extraction module from the preset storage space.

[0177] S1104, for each target word, if there is no historical word matching the target word in the preset storage space, the server uses the network layer of the feature extraction module to extract features from the target word to obtain the global key-value pair of the target word, and assigns different local key-value pairs of the target word to multiple attention heads of the feature extraction module; the multiple attention heads are used to calculate the attention weights of the assigned local key-value pairs of the target word to obtain the attention weights of the different local key-value pairs of the target word.

[0178] S1105, for each feature extraction module, the server determines the variance and / or entropy of the attention weights of different local key-value pairs output by the feature extraction module, and determines the compression ratio of the feature extraction module based on the variance and / or entropy.

[0179] S1106, For each feature extraction module, the server uses the KV compression module to compress different local KVs in the feature extraction module in ascending order of attention weight, until the proportion of uncompressed local KVs in all local KVs of the feature extraction module reaches the compression ratio of the feature extraction module, thus obtaining the target KV compression result corresponding to the feature extraction module.

[0180] S1107, for each feature extraction module, the server inputs the target KV compression result corresponding to the feature extraction module into the feature extraction module, and the feature extraction module compresses the global KV of different target words according to the target KV compression result to obtain the compressed KV of different target words.

[0181] S1108, for each feature extraction module, the server uses the feature extraction module to process the existing word KV to obtain the KV of the next predicted word in the predicted response text.

[0182] S1109, The server obtains the target response text of the target prompt text based on the KV of different predicted words in the corresponding predicted response text from the end feature extraction module.

[0183] S1110, the server sends the target response text to the display device.

[0184] S1111: After receiving the target response text, the display device controls the display to show the target response text.

[0185] The specific implementation methods of S1101-S1111 are the same as those in the above method embodiments, and will not be repeated here.

[0186] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 12As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores PCB product-related data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a PCB product quality inspection method.

[0187] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0188] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0189] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0190] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0191] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0192] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0193] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data processing method, characterized in that, Applied to a computer device, the computer device integrating a target large language model, the target large language model including at least two cascaded feature extraction modules and a key-value (KV) compression module, each feature extraction module including a network layer and multiple attention heads connected to the network layer, the method comprising: The target prompt text is input into each feature extraction module to obtain the local key-value pairs (KVs) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text. The different target words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by each feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. For each feature extraction module, the KV compression module compresses different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KVs, to obtain the target KV compression result corresponding to the feature extraction module. The KV compression result of each target is input into the corresponding feature extraction module to obtain the target response text of the target prompt text.

2. The method according to claim 1, characterized in that, The step of inputting the target prompt text into each feature extraction module to obtain the local key-value pairs (KV) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text includes: For each feature extraction module, the network layer of the feature extraction module is used to extract features from the target prompt text to obtain global key-value pairs (KVs) for different target words, and different local KVs for each target word are assigned to multiple attention heads of the feature extraction module; wherein, each attention head is assigned a local KV for a different target word, and the local KVs of the same target word assigned by multiple attention heads constitute the global KV of the target word. The attention weights of the local key values ​​of the assigned target words are calculated using the multiple attention heads to obtain the attention weights of the different local key values ​​of each target word.

3. The method according to claim 1, characterized in that, The step of inputting the target prompt text into each feature extraction module to obtain the local key-value pairs (KV) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text includes: The target prompt text is segmented into different target words; For each target word, if there are historical words that match the target word in the preset storage space, read the different local key values ​​and attention weights of the historical words that match the target word in each feature extraction module from the preset storage space. If no historical word matching the target word exists in the preset storage space, the target word is input into each feature extraction module to obtain the different local key values ​​and attention weights of the target word output by each feature extraction module after processing the target word.

4. The method according to claim 1, characterized in that, The compression ratio of each feature extraction module is determined as follows: Determine the distribution parameters of the attention weights of different local key values ​​output by the feature extraction module; wherein the distribution parameters include variance and / or entropy; The compression ratio of the feature extraction module is determined based on the distribution parameters.

5. The method according to any one of claims 1-4, characterized in that, The step involves using the KV compression module to compress different local KV values ​​of the feature extraction module according to the compression ratio of the feature extraction module and the attention weights of different local KV values, thereby obtaining the target KV compression result corresponding to the feature extraction module, including: The KV compression module compresses different local KVs of the feature extraction module in ascending order of their attention weights until the proportion of uncompressed local KVs in all local KVs of the feature extraction module reaches the compression ratio of the feature extraction module.

6. The method according to any one of claims 1-5, characterized in that, The step of inputting the compression result of each target KV into the corresponding feature extraction module to obtain the target response text of the target prompt text includes: For each feature extraction module, the target key-value compression result corresponding to the feature extraction module is input into the feature extraction module. The feature extraction module then compresses the global key-value pairs of different target words based on the target key-value compression result to obtain the compressed key-value pairs of different target words. Obtain the target response text of the target prompt text; wherein, the target response text is obtained by at least two cascaded feature extraction modules processing the compressed key-value pairs based on the corresponding different target words.

7. The method according to claim 6, characterized in that, The target response text for obtaining the target prompt text includes: For each feature extraction module, the feature extraction module processes the existing word units (KV) to obtain the KV of the next predicted word unit in the predicted response text; wherein, the predicted response text is the response result of the feature extraction module to the target prompt text; when the feature extraction module is the first feature extraction module, the existing word unit KV includes the compressed KV of different target words and the KV of the predicted words in the predicted response text; when the feature extraction module is not the first feature extraction module, the existing word unit KV includes the compressed KV of different target words, the KV of the predicted words in the predicted response text, and the KV of the latest predicted word unit output by the previous feature extraction module; The target response text of the target prompt text is output by the end feature extraction module based on the key-value pairs of different predicted words in the corresponding predicted response text.

8. The method according to claim 7, characterized in that, The target response text of the target prompt text, output by the end feature extraction module based on the key-value pairs (KV) of different predicted words in the corresponding predicted response text, includes: The response word corresponding to the KV of the preset number of predicted word words is output by the end feature extraction module when a preset number of predicted word words are generated; The different response terms obtained are fused to obtain the target response text.

9. The method according to any one of claims 1-8, characterized in that, The target large language model is trained in the following way: The sample prompt text is input into the initial large language model in the initial model to obtain the sample response text of the sample prompt text; The accuracy of the sample response text is determined based on the difference between the sample response text and the actual response text of the sample prompt text. Based on the accuracy and the overall compression ratio of the initial large language model, adjust the additional model parameters of the additional fine-tuning module in the initial model; The initial model parameters of the initial large language model are replaced with the fusion parameters of the initial model parameters and the additional model parameters to obtain the target large language model.

10. A server, characterized in that, The server integrates a target large language model, which includes at least two cascaded feature extraction modules and a key-value (KV) compression module. Each feature extraction module includes a network layer and multiple attention heads connected to the network layer. The server includes: A communication device configured to connect to an external device; and at least one processor, connected to the communication device, and configured to: The target prompt text is input into each feature extraction module to obtain the local key-value pairs (KVs) of different target words and the attention weights of different local KVs output by each feature extraction module after processing the target prompt text. The different target words are obtained by splitting the target prompt text, and the different local KVs of the same target word output by each feature extraction module are obtained by splitting the global KV of the target word output by the network layer in the feature extraction module. For each feature extraction module, the KV compression module compresses different local KVs of the feature extraction module according to the compression ratio of the feature extraction module and the attention weight of different local KVs, to obtain the target KV compression result corresponding to the feature extraction module. The KV compression result of each target is input into the corresponding feature extraction module to obtain the target response text of the target prompt text.

Images