Data processing method and device based on large model, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and device based on a large model, a storage medium and electronic equipment. The method relates to the technical field of artificial intelligence, and comprises the following steps: processing description information through a first reasoning model in a first graphics processor corresponding to a pre-filling node to obtain key value information and a first reply character output by the first reasoning model in the first graphics processor, and obtaining a second reply character based on a key value logic block identifier corresponding to the key value information; storing the key value information belonging to the same key value logic block identifier to a target video memory area in the first graphics processor, and processing the key value information and the first reply character in the target video memory area through a second reasoning model in a second graphics processor corresponding to the decoding node, and obtaining target reply information corresponding to the description information. The problem that the data processing efficiency is low is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and more specifically, to a data processing method and device based on a large model, a storage medium, and an electronic device. Background Art

[0002] For generative large model inference services, reusing the kv cache (key-value cache) as much as possible to reduce the required computing resources is the core to improve the overall throughput. For the separated inference framework, separate prefilling and decoding so that each can run independently. An important process in the separated architecture is to transfer the reusable kv cache generated in the prefill stage to the selected decode. This process needs to be as fast as possible, otherwise it will affect the response speed of the inference service.

[0003] In the prior art, in order to facilitate the forward propagation calculation of the model layer by layer, the physical cache structure of the kv cache is designed so that each layer of the model uses a continuous video memory area. For example, the data structure of the kv cache is: , each list element represents a single layer of continuous kv cache. This design brings about the problem of inter-block discreteness. Since the send API and recv API of the centralized communication library (centralized communication library, used for communication between multiple GPUs) can only transfer a continuous kv cache in the video memory space at each call, the number of API calls required for kv cache transmission is directly related to the number of discrete memory blocks, which will cause significant transmission delays. For example, the number of discrete memory blocks that a block of kv cache may have is the number of layers (64 layers in the 32B model) multiplied by 2 (for k or v), and 128 API atomic capability calls are required to transfer one block. This discreteness has an adverse impact on overall computing performance.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a data processing method and device, a storage medium and an electronic device based on a large model, so as to at least solve the technical problem that when pre-filling the large model separate reasoning, the kv output by each layer of the model is stored in a continuous video memory area, and when the kv needs to be transmitted to the decoding part for decoding, only one kv in the continuous video memory space can be transmitted each time, resulting in transmission delays, which in turn leads to relatively low data processing efficiency.

[0006] According to one aspect of an embodiment of the present application, a large model-based data processing method is provided, including: processing the descriptive information input by the target object through the first inference model in the first graphics processor corresponding to the pre-filled node, and obtaining the key-value information and the first reply character output by the first inference model in the first graphics processor; based on the key-value logic block identifier corresponding to the key-value information, storing the key-value information belonging to the same key-value logic block identifier into the target video memory area in the first graphics processor, wherein the storage addresses of the key-value information belonging to the same key-value logic block identifier in the target video memory area are continuous; processing the key-value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node, and obtaining the target reply information corresponding to the descriptive information.

[0007] Furthermore, before processing the description information through the first inference model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first inference model in the first graphics processor, the method also includes: determining whether the weight parameters corresponding to the target reasoning model need to be split; if the weight parameters corresponding to the target reasoning model need to be split, determining the first number of required first graphics processors and the second number of required second graphics processors based on the description information; splitting the weight parameters of the target reasoning model to obtain a first number of first inference models, and determining the first graphics processor based on the first number of first inference models; splitting the weight parameters of the target reasoning model to obtain a second number of second inference models, and determining the second graphics processor based on the second number of second inference models.

[0008] Furthermore, before processing the key-value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, the method also includes: determining whether the number of the first graphics processors is the same as the number of the second graphics processors; if the number of the first graphics processors is the same as the number of the second graphics processors, sending the key-value information in the target video memory area to the second target graphics processor through the first graphics processor, wherein the second target graphics processor is the second graphics processor corresponding to the decoding node that requires the key-value information stored in the first graphics processor to be decoded.

[0009] Furthermore, sending the key value information in the target video memory area to the second graphics processor through the first graphics processor includes: determining the second target graphics processor through the first graphics processor based on a mapping relationship between the first graphics processor and the second graphics processor; determining an ID of the first target key value information in the target video memory area required by the second target graphics processor; reading the first target key value information from the target video memory area based on the ID of the first target key value information through the first graphics processor; and sending the first target key value information to the second target graphics processor through the first graphics processor.

[0010] Furthermore, after sending the first target key value information to the second target graphics processor through the first graphics processor, the method further includes: receiving the key value information and determining a video memory address in the second target graphics processor for storing the first target key value information through the second target graphics processor; and writing the first target key value information into the video memory address through the second target graphics processor.

[0011] Furthermore, after determining whether the number of the first graphics processors is the same as the number of the second graphics processors, the method also includes: if the number of the first graphics processors is not the same as the number of the second graphics processors, determining the ID of the second target key-value information to be sent in the first graphics processor; reading the second target key-value information from the target video memory area based on the ID of the second target key-value information through a parallelized replication operator, and writing the second target key-value information in parallel into a preset sending buffer area; and sending the third target key-value information in the sending buffer area to the second image processor in the decoding node through a preset sending function.

[0012] Furthermore, after sending the third target key value information in the sending buffer area to the multiple decoding nodes through a preset sending function, the method also includes: receiving the third target key value information through a preset receiving function, and writing the third target key value information into a preset receiving buffer area; determining the fourth target key value information required by the second graphics processor corresponding to the decoding node; splitting the key value information in the receiving buffer area through a parallelized replication operator to obtain the fourth target key value information, and writing the fourth target key value information into the video memory area in the second graphics processor corresponding to the decoding node.

[0013] Furthermore, before the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filled node, the method also includes: for the first graphics processor, dividing the description information into key-value logical blocks according to a preset key-value block size to obtain key-value logical block identifiers of the characters in the description information; and determining the key-value logical block identifier corresponding to the key-value information to be generated based on the key-value logical block identifiers of the characters in the description information.

[0014] According to another aspect of an embodiment of the present application, a data processing device based on a large model is also provided, including: a first processing unit, used to process the description information through the first inference model in the first graphics processor corresponding to the pre-fill node, and obtain the key value information and the first reply character output by the first inference model in the first graphics processor, wherein the first inference model in the first graphics processor corresponding to the pre-fill node is determined by the target inference model; a storage unit, used to store the key value information belonging to the same key value logic block identifier to the target video memory area in the first graphics processor based on the key value logic block identifier corresponding to the key value information, wherein the storage address of the key value information belonging to the same key value logic block identifier in the target video memory area is continuous; a second processing unit, used to process the key value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node, and obtain the target reply information corresponding to the description information, wherein the second inference model in the second graphics processor corresponding to the decoding node is determined by the target inference model.

[0015] Furthermore, the device also includes: a first judgment unit, used to judge whether the weight parameters corresponding to the target reasoning model need to be split before processing the description information through the first reasoning model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first reasoning model in the first graphics processor; a first determination unit, used to determine the first number of required first graphics processors and the second number of required second graphics processors based on the description information if the weight parameters corresponding to the target reasoning model need to be split; a first splitting unit, used to split the weight parameters of the target reasoning model to obtain a first number of first reasoning models, and determine the first graphics processor based on the first number of first reasoning models; a second splitting unit, used to split the weight parameters of the target reasoning model to obtain a second number of second reasoning models, and determine the second graphics processor based on the second number of second reasoning models.

[0016] Furthermore, the device also includes: a second judgment unit, used to judge whether the number of the first graphics processors is the same as the number of the second graphics processors before processing the key-value information and the first reply character in the target video memory area through a second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information; a first sending unit, used to send the key-value information in the target video memory area to the second target graphics processor through the first graphics processor if the number of the first graphics processors is the same as the number of the second graphics processors, wherein the second target graphics processor is the second graphics processor corresponding to the decoding node that needs to decode the key-value information stored in the first graphics processor.

[0017] Furthermore, the sending unit includes: a first determination module, used to determine the second target graphics processor based on a mapping relationship between the first graphics processor and the second graphics processor through the first graphics processor; a second determination module, used to determine the ID of the first target key value information in the target video memory area required by the second target graphics processor; a reading module, used to read the first target key value information from the target video memory area through the first graphics processor based on the ID of the first target key value information; and a sending module, used to send the first target key value information to the second target graphics processor through the first graphics processor.

[0018] Furthermore, the device also includes: a first receiving unit, used to receive the key value information and determine a video memory address in the second target graphics processor where the first target key value information is stored after the first target key value information is sent to the second target graphics processor through the first graphics processor; and a writing unit, used to write the first target key value information into the video memory address through the second target graphics processor.

[0019] Furthermore, the device also includes: a second determination unit, used to determine the ID of the second target key value information to be sent in the first graphics processor after determining whether the number of the first graphics processors is the same as the number of the second graphics processors, if the number of the first graphics processors is not the same as the number of the second graphics processors; a reading unit, used to read the second target key value information from the target video memory area based on the ID of the second target key value information through a parallelized replication operator, and write the second target key value information in parallel into a preset sending buffer area; a second sending unit, used to send the third target key value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function.

[0020] Furthermore, the device also includes: a second receiving unit, which is used to receive the third target key value information in the sending buffer area through a preset receiving function after sending the third target key value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function, and write the third target key value information into a preset receiving buffer area; a third determining unit, which is used to determine the fourth target key value information required by the second graphics processor corresponding to the decoding node; and a third splitting unit, which is used to split the key value information in the receiving buffer area through a parallelized replication operator to obtain the fourth target key value information, and write the fourth target key value information into the video memory area of ​​the second graphics processor corresponding to the decoding node.

[0021] Furthermore, the device also includes: a division unit, used to divide the description information into key-value logical blocks according to a preset key-value block size for the first graphics processor before processing the description information through the first inference model in the first graphics processor corresponding to the pre-filled node, so as to obtain the key-value logical block identifier of the characters in the description information; a fourth determination unit, used to determine the key-value logical block identifier corresponding to the key-value information to be generated based on the key-value logical block identifier of the characters in the description information.

[0022] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the storage medium is located is controlled to execute any one of the above-mentioned large model-based data processing methods.

[0023] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising a memory storing an executable program; and a processor for running the program, wherein the program executes any one of the above-mentioned large model-based data processing methods when running.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is also provided, wherein the computer program product includes a stored computer program, and when the computer program is executed by a processor, any one of the above-mentioned large model-based data processing methods is implemented.

[0025] In an embodiment of the present application, the description information is processed by a first reasoning model in a first graphics processor corresponding to a pre-filled node to obtain key-value information and a first reply character output by the first reasoning model in the first graphics processor; based on a key-value logic block identifier corresponding to the key-value information, the key-value information belonging to the same key-value logic block identifier is stored in a target video memory area in the first graphics processor, wherein the storage addresses of the key-value information belonging to the same key-value logic block identifier in the target video memory area are continuous; the key-value information and the first reply character in the target video memory area are processed by a second reasoning model in a second graphics processor corresponding to a decoding node. The invention processes characters to obtain target reply information corresponding to the description information, and stores the key-value information belonging to the same key-value logic block identifier in the continuous target video memory area in the first graphics processor, so as to reduce the number of times the kv is transmitted, thereby achieving the technical effect of improving the data processing efficiency of the inference model, and further solves the problem that when pre-filling the large model separate inference, the kv output by each layer of the model is stored in a continuous video memory area, and when the kv needs to be transmitted to the decoding part for decoding, only one kv in the continuous video memory space can be transmitted each time, resulting in a delay in transmission, and then resulting in the technical problem of relatively low data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0027] Figure 1 is a hardware structure block diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0028] Figure 2 This is the process of the data processing method based on the large model provided in Example 1 of the present application Figure 1 ;

[0029] Figure 3 This is the process of the data processing method based on the large model provided in Example 1 of the present application Figure 2 ;

[0030] Figure 4 This is the process of the data processing method based on the large model provided in Example 1 of the present application Figure 3 ;

[0031] Figure 5 It is a schematic diagram of a large model-based data processing device provided according to the second embodiment of the present application.

[0032] Figure 6 It is a structural block diagram of an electronic device provided according to Embodiment 3 of the present application. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this application.

[0034] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] First, some nouns or terms that appear in the process of describing the embodiments of the present application are subject to the following explanations.

[0036] kv block: used to store a fixed number of tokens attention key and value cache.

[0037] num_blocks: the number of kv blocks.

[0038] num_layers: number of model layers.

[0039] block_size: the size of the kv block.

[0040] num_kv_heads: The number of headers in the kv cache.

[0041] head_size: The size of the kv cache header.

[0042] cache_dim: the tensor dimension of the kv cache actually stored in a single kvblock;

[0043] .

[0044] Tensor Parallelism (TP): is a technology for distributed training and reasoning. In TensorParallelism, the parameters of the model (such as the weight matrix) are divided into multiple small blocks and distributed to different GPUs. Different GPUs are responsible for processing different parts.

[0045] GPU rank: usually refers to the unique identifier (ID or serial number) of each GPU in a multi-GPU system.

[0046] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0047] Example 1

[0048] According to an embodiment of the present application, a data processing method based on a large model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0049] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a data processing method based on a large model is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, and the processor set 102 may include a processor set, Figure 1 102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission module 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0050] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0051] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data processing method based on the large model in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, realizing the above-mentioned data processing method based on the large model. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0052] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] The display may be a touch screen type liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0054] Under the above operating environment, this application provides Figure 2 The data processing method based on the large model is shown. Figure 2This is the process of the data processing method based on the large model according to the first embodiment of the present application Figure 1 .

[0055] Step S201: Process the description information input by the target object through the first reasoning model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first reasoning model in the first graphics processor.

[0056] Step S202: based on the key-value logical block identifier corresponding to the key-value information, the key-value information belonging to the same key-value logical block identifier is stored in the target video memory area of ​​the first graphics processor, wherein the key-value information belonging to the same key-value logical block identifier has a continuous storage address in the target video memory area.

[0057] Optionally, the split inference framework splits the pre-filling and decoding of the inference model, which are processed by different GPU (graphics processing unit) nodes. The specific inference process is as follows Figure 3 As shown, the front-end interface receives the description information (e.g., the question information to be answered) input by the target object, and sends the description information to the corresponding GPU node for pre-filling. In the pre-filling stage, the received description information to be answered is converted into kv and stored in its own video memory area, and the first reply character, i.e., first token, is output. kv and first taken are sent to the corresponding GPU node for decoding for decoding until the final reply information is obtained.

[0058] In the prior art, in order to facilitate the forward propagation calculation of the model layer by layer, the physical cache structure of the kv cache is designed so that each layer of the model uses a continuous video memory area. For example, the data structure of the kv cache is: , each list element represents a single layer of continuous kv cache. This design brings about the problem of inter-block discreteness. Since the send API and recv API of the centralized communication library (centralized communication library, used for communication between multiple GPUs) can only transfer a continuous kv cache in the video memory space at each call, the number of API calls required for kv cache transmission is directly related to the number of discrete memory blocks, which will cause significant transmission delays. For example, the number of discrete memory blocks that a block of kv cache may have is the number of layers (64 layers in the 32B model) multiplied by 2 (for k or v), and transferring one block requires 128 API atomic calls.

[0059] In order to solve the above problems, in the large model-based data processing method provided in Example 1 of the present application, the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first inference model in the first graphics processor corresponding to the pre-filled node.

[0060] It should be noted that the first inference model in the first graphics processor corresponding to the above-mentioned pre-filled node is determined by the target inference model. The target inference model is a large model, which refers to a machine learning model with large-scale parameters and complex computing structure, usually constructed by a deep neural network.

[0061] After obtaining the above-mentioned kv cache (i.e., the above-mentioned key-value information), the cache structure of the kv cache is modified, and for the first graphics processor corresponding to the pre-filled node, the key-value information belonging to the same key-value logical block identifier in the kv cache output by the first graphics processor is stored in the target video memory area in the first graphics processor. It should be noted that the storage address of the key-value information belonging to the same key-value logical block identifier in the target video memory area is continuous, that is, the physical address of the key-value information belonging to the same key-value logical block identifier is continuous. The kv cache cache structure is transformed into a continuous space tensor, as shown below.

[0062]

[0063] Compared with the existing technology This improvement can effectively reduce the number of network API calls, because the kv cache of a block is at the same physical address, and there is no need to read the kv cache of a block from each layer. Therefore, the number of network API calls for each kv block can be reduced by "the number of layers multiplied by 2 (for k or v)".

[0064] Step S203, processing the key value information and the first reply character in the target video memory area through the second reasoning model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, wherein the second reasoning model in the second graphics processor corresponding to the decoding node is determined by the target reasoning model.

[0065] Optionally, after obtaining the corresponding reusable kv cache and first token, the key value information and the first reply character in the target video memory area are processed by a second inference model in a second graphics processor corresponding to the decoding node to obtain target reply information corresponding to the description information.

[0066] It should be noted that the number of first graphics processors corresponding to the pre-filling node can be one or more, and the number of second graphics processors corresponding to the decoding node can be one or more. When the number of first graphics processors is multiple, for any first graphics processor, only the kv cache output by itself is stored.

[0067] To sum up, after the key-value information outputted from the first inference model in the first graphics processor corresponding to the pre-filled node is outputted, the key-value information belonging to the same key-value logic block identifier in the key-value information outputted by the first graphics processor is stored in a video memory area with a continuous address. When the kv in a key-value logic block needs to be transferred to the decoding part for decoding, there is no need to repeatedly read the kv from the video memory area, which can effectively reduce the number of kv transmissions and improve the kv transmission efficiency, thereby achieving the effect of improving data processing efficiency.

[0068] In order to improve the efficiency of parallel processing, in the large model-based data processing method provided in Example 1 of the present application, before the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character outputted in the first inference model in the first graphics processor corresponding to the pre-filled node, the method also includes: judging whether the weight parameters corresponding to the target inference model need to be split; if the weight parameters corresponding to the target inference model need to be split, determining the first number of required first graphics processors and the second number of required second graphics processors based on the description information; splitting the weight parameters of the target inference model to obtain a first number of first inference models, and determining the first graphics processor based on the first number of first inference models; splitting the weight parameters of the target inference model to obtain a second number of second inference models, and determining the second graphics processor based on the second number of second inference models.

[0069] Optionally, in order to increase the speed of pre-filling and decoding, multiple first GPUs can be set at the pre-filling node, and multiple second GPUs can be set at the decoding node. It can be determined whether the weight parameters corresponding to the target inference model need to be split according to actual needs, that is, whether it is necessary to pre-fill in parallel through multiple first GPUs and whether it is necessary to decode in parallel through multiple second GPUs.

[0070] If the weight parameters corresponding to the labeled inference model need to be split, the first number of the first graphics processors and the second number of the second graphics processors required are determined according to the type of description information. For example, the long text summary question-answering task is a typical long input and short output, and the content creation generation task is a short input and long output. For the task type with long input and short output, the number of pre-filled nodes can be set to be larger and the number of decoding nodes can be set to be smaller. For the task type with short input and long output, the number of pre-filled nodes can be set to be smaller and the number of decoding nodes can be set to be larger.

[0071] After obtaining the above-mentioned first quantity and second quantity, the weight parameters of the target inference model can be split by Tensor Parallelism (TP) to obtain the first quantity of first inference models, and then the above-mentioned first graphics processor can be obtained based on the first quantity of first inference models. And the weight parameters of the target inference model can be split by Tensor Parallelism (TP) to obtain the second quantity of second inference models, and the second graphics processor can be determined according to the second quantity of second inference models.

[0072] By splitting the weight parameters of the target inference model, parallel pre-filling and parallel decoding can be achieved, thereby improving the efficiency of data processing by the target inference model.

[0073] It should be noted that if the weight parameters of the target inference model do not need to be split, the first inference model in the first GPU corresponding to a pre-filled node is the target inference model, and the second inference model in the second GPU corresponding to a decoding node is also the target inference model. And the number of GPUs corresponding to a pre-filled node is one, and the number of GPUs corresponding to a decoding node is also one.

[0074] In order to improve the efficiency of data processing, in the large model-based data processing method provided in Example 1 of the present application, before the key-value information and the first reply character in the target video memory area are processed by the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, the method also includes: determining whether the number of first graphics processors is the same as the number of second graphics processors; if the number of first graphics processors is the same as the number of second graphics processors, sending the key-value information in the target video memory area to the second target graphics processor through the first graphics processor, wherein the second target graphics processor is the second image processor corresponding to the decoding node that needs to decode the key-value information stored in the first graphics processor.

[0075] Optionally, since the last layer of the original KV cache changes when TP is enabled for the inference service, as shown below.

[0076] Depend on

[0077] Change to

[0078] This is because TP will be split according to the dimension of the head and the corresponding kv cache will be calculated on the corresponding prefill. TP size refers to the number of tensor parallelism, which is also the first number or the second number mentioned above. When transmitting kvcache, if the number of prefill and decode tensor parallelism is inconsistent, when different GPUs write data to the kv cache of the decoding node, they cannot write directly in a single transmission and reception due to dimension mismatch. Therefore, in order to improve transmission efficiency, you can first determine whether the number of the first graphics processor is the same as the number of the second graphics processor, that is, whether the number of tensor parallelism between prefill and decode is the same.

[0079] If the number of tensor parallelism between prefill and decode is the same, that is, the TP size is the same, then the kv cache structure of all nodes in the cluster is guaranteed to be consistent, and the key value information in the target video memory area is directly sent to the second target graphics processor through the first graphics processor. It should be noted that the second target graphics processor is the second target graphics processor corresponding to the decoding node that needs to decode the key value information stored in the first graphics processor. The second target graphics processor may be one or more.

[0080] By determining whether the number of the first graphics processors is the same as the number of the second graphics processors, it can be ensured that the dimensions of prefill and decode are consistent when the kv cache is transmitted, thereby ensuring that the kv cache is correctly transmitted and received.

[0081] It should be noted that if TP is not enabled for the inference service (i.e., the weight parameters of the target inference model do not need to be split), then the number of GPUs corresponding to a prefill node is one, and the number of GPUs corresponding to a decoding node is also one. At this time, the number of tensor parallelism between prefill and decode is the same. Therefore, the key-value information in the target video memory area is also sent directly to the second graphics processor through the first graphics processor.

[0082] If the number of tensor parallelism between prefill and decode is consistent, in the large model-based data processing method provided in Example 1 of the present application, sending the key value information in the target video memory area to the second target graphics processor through the first graphics processor includes: determining the second target graphics processor based on the mapping relationship between the first graphics processor and the second graphics processor through the first graphics processor; determining the ID of the first target key value information in the target video memory area required by the second target graphics processor; reading the first target key value information from the target video memory area based on the ID of the first target key value information through the first graphics processor; and sending the first target key value information to the second target graphics processor through the first graphics processor.

[0083] After sending the first target key value information to the second target graphics processor through the first graphics processor, the key value information is received by the second target graphics processor and a video memory address for storing the first target key value information in the second target graphics processor is determined; and the first target key value information is written into the video memory address by the second target graphics processor.

[0084] Optionally, before data processing is performed through the pre-filling node and the decoding node, if weight splitting is performed, the mapping relationship between the first graphics processor and the second graphics processor is determined according to the split weight parameters, that is, the kv cache meta information list is determined, and the kv cache meta information list is divided into a sending list send_meta and a receiving list recv_meta. The sending list send_meta and the receiving list recv_meta include the mapping relationship of Gpurank between sending and receiving and the index information of the kv block (that is, the ID of the kv block to be sent or the ID of the kvblock to be received).

[0085] Therefore, when the key value information in the target video memory area is sent to the second target graphics processor through the first graphics processor, the following steps are included: the first graphics processor determines the second target graphics processor and the ID of the first target key value information in the target video memory area required by the second target graphics processor based on the mapping relationship between the first graphics processor and the second graphics processor, that is, the sending node (that is, the above-mentioned first graphics processor) indicates the block id index to be sent and the current number of fragments and the total number of fragments (used for processing the transmission and reception when the data is divided into small blocks due to TP). It should be noted that, since TP may cause the kv cache of a block to be divided into small blocks, the number of fragments currently transmitted and the total number of fragments corresponding to the kv cache of the block need to be stated during transmission.

[0086] Then, the first graphics processor executes the corresponding send API to send the kv cache corresponding to the address of the block index stored in itself to the GPU receiving the rank (ie, the second target graphics processor mentioned above).

[0087] For the receiving node (i.e., the second target graphics processor mentioned above), the corresponding recv API is executed to accept the tensor from the sending rank GPU (i.e., the first graphics processor mentioned above), and directly write it to the address of the corresponding block index, that is, the second target graphics processor receives the key value information and determines the video memory address for storing the first target key value information in the second target graphics processor; the first target key value information is written into the video memory address through the second target graphics processor.

[0088] The accurate transmission of the kv cache between the pre-filling node and the decoding node can be achieved through the mapping relationship between the pre-filling node and the decoding node.

[0089] If the number of first graphics processors is different from the number of second graphics processors, in the large model-based data processing method provided in Example 1 of the present application, after determining whether the number of first graphics processors is the same as the number of second graphics processors, the method also includes: determining the ID of the second target key value information to be sent in the first graphics processor; reading the second target key value information from the target video memory area based on the ID of the second target key value information through a parallelized replication operator, and writing the second target key value information in parallel into a preset sending buffer area; sending the third target key value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function.

[0090] After sending the third target key value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function, the third target key value information is received through a preset receiving function, and the third target key value information is written into the preset receiving buffer area; the fourth target key value information required by the second graphics processor corresponding to the decoding node is determined; the key value information in the receiving buffer area is split through a parallelized replication operator to obtain the fourth target key value information, and the fourth target key value information is written into the video memory area of ​​the second graphics processor corresponding to the decoding node.

[0091] Optionally, if the number of parallel prefill and decode tensors is inconsistent, when different first graphics processors write kv cache to the second graphics processor in the decoding node, they cannot write directly in a single transmission and reception due to dimension mismatch, which will cause the kvcache from prefill to decoding to be restricted by discrete memory during transmission. Therefore, in order to avoid the above problems, when the number of parallel prefill and decode tensors is inconsistent, the ID of the second target key value information to be sent in the first graphics processor is determined by sending the list send_meta, and then the kvcache of the scattered kv block is written to the sending buffer area according to the order in send_meta through the parallel copy operator. It should be noted that the size of the reserved kvcache sending buffer is determined according to the model size and the gpu video memory resources.

[0092] After completing the parallel writing, execute the nccl.send atomic operation (that is, the preset sending function mentioned above) to send the contents in the send buffer to the GPU of the receiving rank.

[0093] The receiving node (i.e., the second graphics processor mentioned above) executes the nccl.recv atomic operation (i.e., the preset receiving function mentioned above) to receive the kv cache of the sending rank GPU device, and writes the received kv cache (i.e., the third target key value information mentioned above) into the preset receiving buffer area. After writing, the fourth target key value information required by the second graphics processor corresponding to the decoding node is determined according to the kv cache meta information list (recv_meta), and then the continuous kv cache in the receiving buffer area is split into the corresponding second graphics processor's video memory area through the Triton operator (i.e., the parallelized replication operator mentioned above). It should be noted that if the size of the kv cache tensor to be sent exceeds the reserved send and receive buffer size, multiple write + send + receive operations are performed.

[0094] The above steps can effectively solve the low-latency kv cache transmission with consistent or inconsistent parallel numbers of prefill / decode tensors, improve transmission efficiency, and thus achieve the effect of improving data processing efficiency.

[0095] In the large model-based data processing method provided in the first embodiment of the present application, before the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filled node, the method also includes: for the first graphics processor, dividing the description information into key-value logical blocks according to a preset key-value block size to obtain key-value logical block identifiers of the characters in the description information; and determining the key-value logical block identifier corresponding to the key-value information to be generated based on the key-value logical block identifiers of the characters in the description information.

[0096] Optionally, before the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filled node, the description information is divided into key-value logical blocks according to a preset key-value block size (i.e., the size of the kv block), that is, the key-value logical block identifier corresponding to the token in the description information is determined, and then the key-value logical block identifier corresponding to the key-value information to be generated corresponding to the token is determined, so that the kvcache belonging to the same kv block can be subsequently stored in continuous physical addresses according to the key-value logical block identifier, that is, the storage addresses of the key-value information belonging to the same key-value logical block identifier in the target video memory area are continuous.

[0097] It should be noted that the size of the kv block refers to the number of kvs stored in a kv block, and the key-value logical block refers to the logical address corresponding to the kv belonging to the same kv block. In large model reasoning, the kv cache will correspond to the logical address and the physical address. When stored in the target video memory area in the first graphics processor, the corresponding physical address is the address used for data operations. Before generating kv, the logical address corresponding to the kv will be divided first, and then the corresponding physical address will be determined.

[0098] It should be noted that due to the change in the kv cache cache structure, the corresponding operators used to read and calculate kvcache also need to be changed accordingly. The operators use thread blocks and threads to process data in parallel. Each thread calculates the address of the data to be accessed in the global video memory through the thread index combined with the offset and step size. The thread index is calculated based on the thread ID and data layout. When the dimension of the data structure changes, the way to calculate this global index theoretically changes accordingly. In addition, in the operator, the offset and step size are used to calculate the position of the element. When the dimension or layout of the data structure changes, the formula for calculating the offset and step size of each element can be adjusted accordingly.

[0099] In an optional embodiment, the Figure 4 The flowchart shown in the figure realizes the transmission of kv cache. Figure 4As shown in the figure, the kv cache structure is adjusted, and the kv cache of a kv block is aggregated into a continuous block. The operation operator is adjusted according to the adjustment of the kvcache structure. Get the sent and received kv cache meta information list, namely send_meta and recv_meta. Determine whether the parallel number of prefill and decode tensors is consistent. If the parallel number of prefill and decode tensors is consistent, the kv block of the prefill node can be directly transmitted in the inplace mode (that is, directly operate on the original storage location without establishing a kv cache send and receive buffer). If the parallel number of prefill and decode tensors is inconsistent, establish a kv cache send and receive buffer, and then optimize the data reading and writing process through parallel processing. The Triton parallel replication operator writes the scattered kv caches of each block into the sending buffer in the order of send_meta. The receiving node receives the kv cache from the sending rank GPU device and writes it directly into the receiving buffer. The Triton parallel replication operator disassembles the continuous kv cache of the send and receive buffer and writes it back to the address of the corresponding block index.

[0100] In the large model-based data processing method provided in Example 1 of the present application, the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filling node to obtain the key value information and the first reply character output by the first inference model in the first graphics processor; based on the key value logic block identifier corresponding to the key value information, the key value information belonging to the same key value logic block identifier is stored in the target video memory area in the first graphics processor, wherein the storage addresses of the key value information belonging to the same key value logic block identifier in the target video memory area are continuous; the key value information and the first reply character in the target video memory area are processed by the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, which solves the problem that in the related technology, when pre-filling the large model separate reasoning, the kv output by each layer of the model is stored in a continuous video memory area, and when the kv needs to be transmitted to the decoding part for decoding, only one kv in the continuous video memory space can be transmitted each time, resulting in transmission delays, and then resulting in the problem of relatively low data processing efficiency. In the present scheme, after the key-value information outputted from the first inference model in the first graphics processor corresponding to the pre-filled node is obtained, for the first graphics processor, the key-value information belonging to the same key-value logic block identifier in the key-value information outputted by the first graphics processor is stored in a video memory area with a continuous address. When the kv in a key-value logic block needs to be transferred to the decoding part for decoding, there is no need to repeatedly read the kv from the video memory area, which can effectively reduce the number of kv transmissions and improve the kv transmission efficiency, thereby achieving the effect of improving data processing efficiency.

[0101] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0102] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, disk, CD), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0103] Example 2

[0104] According to an embodiment of the present application, a large model-based data processing device for implementing the above-mentioned large model-based data processing method is also provided. Figure 5 As shown, the device includes: a first processing unit 501, a storage unit 502 and a second processing unit 503.

[0105] The first processing unit 501 is used to process the description information input by the target object through the first reasoning model in the first graphics processor corresponding to the pre-filled node, and obtain the key value information and the first reply character output by the first reasoning model in the first graphics processor;

[0106] The storage unit 502 is used to store the key-value information belonging to the same key-value logical block identifier into the target video memory area in the first graphics processor based on the key-value logical block identifier corresponding to the key-value information, wherein the storage addresses of the key-value information belonging to the same key-value logical block identifier in the target video memory area are continuous;

[0107] The second processing unit 503 is used to process the key value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information.

[0108] In the data processing device based on the large model provided in the second embodiment of the present application, the first processing unit 501 processes the description information through the first inference model in the first graphics processor corresponding to the pre-filling node, and obtains the key value information and the first reply character output by the first inference model in the first graphics processor; the storage unit 502 stores the key value information belonging to the same key value logic block identifier to the target video memory area in the first graphics processor based on the key value logic block identifier corresponding to the key value information, wherein the storage address of the key value information belonging to the same key value logic block identifier in the target video memory area is continuous; the second processing unit 503 processes the key value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node, and obtains the target reply information corresponding to the description information, which solves the problem that in the related technology, when pre-filling the large model separate reasoning, the kv output by each layer of the model is stored in a continuous video memory area, and when the kv needs to be transmitted to the decoding part for decoding, only one kv in the continuous video memory space can be transmitted each time, resulting in transmission delays, and then resulting in low data processing efficiency. In the present scheme, after the key-value information outputted from the first inference model in the first graphics processor corresponding to the pre-filled node is obtained, for the first graphics processor, the key-value information belonging to the same key-value logic block identifier in the key-value information outputted by the first graphics processor is stored in a video memory area with a continuous address. When the kv in a key-value logic block needs to be transferred to the decoding part for decoding, there is no need to repeatedly read the kv from the video memory area, which can effectively reduce the number of kv transmissions and improve the kv transmission efficiency, thereby achieving the effect of improving data processing efficiency.

[0109] Optionally, in the large model-based data processing device provided in Example 2 of the present application, the device also includes: a first judgment unit, used to judge whether the weight parameters corresponding to the target reasoning model need to be split before processing the description information through the first inference model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first inference model in the first graphics processor; a first determination unit, used to determine the first number of required first graphics processors and the second number of required second graphics processors based on the description information if the weight parameters corresponding to the target reasoning model need to be split; a first splitting unit, used to split the weight parameters of the target reasoning model to obtain a first number of first inference models, and determine the first graphics processor based on the first number of first inference models; a second splitting unit, used to split the weight parameters of the target reasoning model to obtain a second number of second inference models, and determine the second graphics processor based on the second number of second inference models.

[0110] Optionally, in the large model-based data processing device provided in Example 2 of the present application, the device also includes: a second judgment unit, used to process the key-value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, and determine whether the number of first graphics processors is the same as the number of second graphics processors; a first sending unit, used to send the key-value information in the target video memory area to the second target graphics processor through the first graphics processor if the number of first graphics processors is the same as the number of second graphics processors, wherein the second target graphics processor is the second graphics processor corresponding to the decoding node that needs to decode the key-value information stored in the first graphics processor.

[0111] Optionally, in the large model-based data processing device provided in Example 2 of the present application, the sending unit includes: a first determination module, used to determine the second target graphics processor through the first graphics processor based on the mapping relationship between the first graphics processor and the second graphics processor; a second determination module, used to determine the ID of the first target key value information in the target video memory area required by the second target graphics processor; a reading module, used to read the first target key value information from the target video memory area through the first graphics processor based on the ID of the first target key value information; and a sending module, used to send the first target key value information to the second target graphics processor through the first graphics processor.

[0112] Optionally, in the large model-based data processing device provided in Embodiment 2 of the present application, the device further includes: a first receiving unit, used to, after sending the first target key value information to the second target graphics processor through the first graphics processor, receive the key value information through the second target graphics processor and determine a video memory address in the second target graphics processor where the first target key value information exists; and a writing unit, used to write the first target key value information into the video memory address through the second target graphics processor.

[0113] Optionally, in the large model-based data processing device provided in Example 2 of the present application, the device also includes: a second determination unit, used to determine the ID of the second target key-value information to be sent in the first graphics processor after determining whether the number of the first graphics processors is the same as the number of the second graphics processors, if the number of the first graphics processors is not the same as the number of the second graphics processors; a reading unit, used to read the second target key-value information from the target video memory area based on the ID of the second target key-value information through a parallelized replication operator, and write the second target key-value information in parallel into a preset sending buffer area; a second sending unit, used to send the third target key-value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function.

[0114] Optionally, in the large model-based data processing device provided in Example 2 of the present application, the device also includes: a second receiving unit, which is used to receive the third target key value information in the sending buffer area through a preset receiving function after sending the third target key value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function, and write the third target key value information into a preset receiving buffer area; a third determination unit, which is used to determine the fourth target key value information required by the second graphics processor corresponding to the decoding node; and a third splitting unit, which is used to split the key value information in the receiving buffer area through a parallelized replication operator to obtain the fourth target key value information, and write the fourth target key value information into the video memory area of ​​the second graphics processor corresponding to the decoding node.

[0115] Optionally, in the large model-based data processing device provided in Example 2 of the present application, the device also includes: a division unit, used to divide the description information into key-value logical blocks according to a preset key-value block size for the first graphics processor before processing the description information through the first inference model in the first graphics processor corresponding to the pre-filled node, so as to obtain key-value logical block identifiers of the characters in the description information; and a fourth determination unit, used to determine the key-value logical block identifier corresponding to the key-value information to be generated based on the key-value logical block identifiers of the characters in the description information.

[0116] It should be noted that the first processing unit 501, the storage unit 502 and the second processing unit 503 correspond to steps S201 to S203 in the first embodiment, and the three units and the corresponding steps implement the same examples and application scenarios, but are not limited to the contents disclosed in the first embodiment. It should be noted that the above units, as part of the device, can be run in the computer terminal 10 provided in the first embodiment.

[0117] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0118] Example 3

[0119] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0120] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0121] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the large model: the description information is processed by the first inference model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first inference model in the first graphics processor; based on the key value logic block identifier corresponding to the key value information, the key value information belonging to the same key value logic block identifier is stored in the target video memory area in the first graphics processor, wherein the storage address of the key value information belonging to the same key value logic block identifier in the target video memory area is continuous; the key value information and the first reply character in the target video memory area are processed by the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information.

[0122] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on a large model: before processing the description information through the first reasoning model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output in the first reasoning model in the first graphics processor corresponding to the pre-filled node, the method also includes: judging whether the weight parameters corresponding to the target reasoning model need to be split; if the weight parameters corresponding to the target reasoning model need to be split, based on the description information, determining the first number of required first graphics processors and the second number of required second graphics processors; splitting the weight parameters of the target reasoning model to obtain a first number of first reasoning models, and determining the first graphics processor based on the first number of first reasoning models; splitting the weight parameters of the target reasoning model to obtain a second number of second reasoning models, and determining the second graphics processor based on the second number of second reasoning models.

[0123] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on a large model: before processing the key value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, the method also includes: determining whether the number of first graphics processors is the same as the number of second graphics processors; if the number of first graphics processors is the same as the number of second graphics processors, sending the key value information in the target video memory area to the second target graphics processor through the first graphics processor, wherein the second target graphics processor is the second graphics processor corresponding to the decoding node that needs to decode the key value information stored in the first graphics processor.

[0124] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the large model: sending the key value information in the target video memory area to the second target graphics processor through the first graphics processor includes: determining the second target graphics processor based on the mapping relationship between the first graphics processor and the second graphics processor through the first graphics processor; determining the ID of the first target key value information in the target video memory area required by the second target graphics processor; reading the first target key value information from the target video memory area based on the ID of the first target key value information through the first graphics processor; sending the first target key value information to the second target graphics processor through the first graphics processor.

[0125] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the large model: after sending the first target key value information to the second target graphics processor through the first graphics processor, the method also includes: receiving the key value information through the second target graphics processor and determining the video memory address for storing the first target key value information in the second target graphics processor; writing the first target key value information into the video memory address through the second target graphics processor.

[0126] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the large model: after determining whether the number of the first graphics processor is the same as the number of the second graphics processor, the method also includes: if the number of the first graphics processor is not the same as the number of the second graphics processor, determining the ID of the second target key value information to be sent in the first graphics processor; reading the second target key value information from the target video memory area based on the ID of the second target key value information through a parallelized replication operator, and writing the second target key value information in parallel into a preset sending buffer area; sending the third target key value information in the sending buffer area to the second graphics processor corresponding to the decoding node through a preset sending function.

[0127] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on the large model: after sending the third target key value information in the sending buffer area to multiple decoding nodes through a preset sending function, the method also includes: receiving the third target key value information through a preset receiving function, and writing the third target key value information into a preset receiving buffer area; determining the fourth target key value information required by the second graphics processor corresponding to the decoding node; splitting the key value information in the receiving buffer area through a parallelized replication operator to obtain the fourth target key value information, and writing the fourth target key value information into the video memory area of ​​the second graphics processor corresponding to the decoding node.

[0128] The above-mentioned electronic device can execute the program code of the following steps in the data processing method based on a large model: before processing the description information through the first inference model in the first graphics processor corresponding to the pre-filled node, the method also includes: for the first graphics processor, dividing the description information into key value logical blocks according to a preset key value block size to obtain the key value logical block identifiers of the characters in the description information; based on the key value logical block identifiers of the characters in the description information, determining the key value logical block identifier corresponding to the key value information to be generated.

[0129] Optionally, Figure 6 is a structural block diagram of a computer terminal according to an embodiment of the present application. Figure 6 As shown, the electronic device 60 may include: one or more ( Figure 6 Only one is shown in the figure) processor 602, memory 604. The electronic device 60 may also include a storage controller, through which the memory 604 is controlled and managed; the electronic device 60 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0130] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the data processing method and device based on the large model in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, realizing the above-mentioned data processing method based on the large model. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories can be connected to the electronic device 60 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0131] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: process the description information through the first inference model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first inference model in the first graphics processor; based on the key value logic block identifier corresponding to the key value information, store the key value information belonging to the same key value logic block identifier to the target video memory area in the first graphics processor, wherein the storage address of the key value information belonging to the same key value logic block identifier in the target video memory area is continuous; process the key value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information.

[0132] Optionally, the processor may also execute program code for the following steps: before processing the description information through the first reasoning model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character outputted in the first reasoning model in the first graphics processor corresponding to the pre-filled node, the method further includes: determining whether the weight parameters corresponding to the target reasoning model need to be split; if the weight parameters corresponding to the target reasoning model need to be split, determining the first number of required first graphics processors and the second number of required second graphics processors based on the description information; splitting the weight parameters of the target reasoning model to obtain a first number of first reasoning models, and determining the first graphics processor based on the first number of first reasoning models; splitting the weight parameters of the target reasoning model to obtain a second number of second reasoning models, and determining the second graphics processor based on the second number of second reasoning models.

[0133] Optionally, the processor may also execute program code for the following steps: before processing the key-value information and the first reply character in the target video memory area through the second inference model in the second graphics processor corresponding to the decoding node to obtain the target reply information corresponding to the description information, the method further includes: determining whether the number of first graphics processors is the same as the number of second graphics processors; if the number of first graphics processors is the same as the number of second graphics processors, sending the key-value information in the target video memory area to the second target graphics processor through the first graphics processor, wherein the second target graphics processor is the second graphics processor corresponding to the decoding node that requires decoding of the key-value information stored in the first graphics processor.

[0134] Optionally, the processor may also execute program code of the following steps: sending the key value information in the target video memory area to the second target graphics processor through the first graphics processor includes: determining the second target graphics processor through the first graphics processor based on the mapping relationship between the first graphics processor and the second graphics processor; determining the ID of the first target key value information in the target video memory area required by the second target graphics processor; reading the first target key value information from the target video memory area based on the ID of the first target key value information through the first graphics processor; and sending the first target key value information to the second target graphics processor through the first graphics processor.

[0135] Optionally, the processor may also execute program code of the following steps: after sending the first target key value information to the second target graphics processor through the first graphics processor, the method further includes: receiving the key value information through the second target graphics processor and determining a video memory address in the second target graphics processor where the first target key value information exists; and writing the first target key value information into the video memory address through the second target graphics processor.

[0136] Optionally, the processor may also execute the program code of the following steps: after determining whether the number of the first graphics processors is the same as the number of the second graphics processors, the method further includes: if the number of the first graphics processors is not the same as the number of the second graphics processors, determining the ID of the second target key value information to be sent in the first graphics processor; reading the second target key value information from the target video memory area based on the ID of the second target key value information through a parallelized replication operator, and writing the second target key value information in parallel into a preset sending buffer area; sending the third target key value information in the sending buffer area to the second image processor corresponding to the decoding node through a preset sending function.

[0137] Optionally, the processor may also execute the program code of the following steps: after sending the third target key value information in the sending buffer to multiple decoding nodes through a preset sending function, the method also includes: receiving the third target key value information through a preset receiving function, and writing the third target key value information into a preset receiving buffer; determining the fourth target key value information required by the second graphics processor corresponding to the decoding node; splitting the key value information in the receiving buffer through a parallelized replication operator to obtain the fourth target key value information, and writing the fourth target key value information into the video memory area of ​​the second graphics processor corresponding to the decoding node.

[0138] Optionally, the processor may also execute the program code of the following steps: before processing the description information through the first inference model in the first graphics processor corresponding to the pre-filled node, the method also includes: for the first graphics processor, dividing the description information into key value logical blocks according to a preset key value block size to obtain key value logical block identifiers of the characters in the description information; determining the key value logical block identifier corresponding to the key value information to be generated based on the key value logical block identifiers of the characters in the description information.

[0139] It can be understood by those skilled in the art that Figure 6 The structure shown is for illustration only, and the electronic device 60 may also be a terminal device such as a smart phone (such as an Android phone, an IOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), or a PAD. Figure 6 The structure of the electronic device is not limited. For example, the electronic device 60 may also include Figure 6 More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 6 Different configurations are shown.

[0140] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0141] Example 4

[0142] An embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the above-mentioned computer program product can be used to store the program code executed by the large model-based data processing method provided in the above-mentioned embodiment 1.

[0143] Example 5

[0144] The embodiment of the present application also provides a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the large model-based data processing method provided in the first embodiment.

[0145] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0146] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0147] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0152] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method based on a large model, characterized in that: include: Processing the description information input by the target object through the first reasoning model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first reasoning model in the first graphics processor; Based on the key-value logical block identifier corresponding to the key-value information, the key-value information belonging to the same key-value logical block identifier is stored in a target video memory area in the first graphics processor, wherein the storage addresses of the key-value information belonging to the same key-value logical block identifier in the target video memory area are continuous; The key value information and the first reply character in the target video memory area are processed by a second inference model in a second graphics processor corresponding to the decoding node to obtain target reply information corresponding to the description information.

2. The method according to claim 1, characterized in that Before the description information is processed by the first reasoning model in the first graphics processor corresponding to the pre-filled node to obtain the key value information and the first reply character output by the first reasoning model in the first graphics processor, the method further includes: Determine whether the weight parameters corresponding to the target inference model need to be split; If the weight parameters corresponding to the target reasoning model need to be split, determining a first number of first graphics processors required and a second number of second graphics processors required based on the description information; Splitting the weight parameters of the target inference model to obtain a first number of first inference models, and determining the first graphics processor based on the first number of first inference models; The weight parameters of the target inference model are split to obtain a second number of second inference models, and the second graphics processor is determined based on the second number of second inference models.

3. The method according to claim 1, characterized in that Before processing the key value information and the first reply character in the target video memory area by a second inference model in a second graphics processor corresponding to the decoding node to obtain target reply information corresponding to the description information, the method further includes: Determining whether the number of the first graphics processors is the same as the number of the second graphics processors; If the number of the first graphics processors is the same as the number of the second graphics processors, the key value information in the target video memory area is sent to the second target graphics processor via the first graphics processor, wherein the second target graphics processor is the second graphics processor corresponding to the decoding node that needs to perform decoding processing on the key value information stored in the first graphics processor.

4. The method according to claim 3, characterized in that Sending the key value information in the target video memory area to the second target graphics processor through the first graphics processor includes: Determining the second target graphics processor by the first graphics processor based on a mapping relationship between the first graphics processor and the second graphics processor; Determine the ID of the first target key value information in the target video memory area required by the second target graphics processor; Reading the first target key value information from the target video memory area based on the ID of the first target key value information by the first graphics processor; The first target key value information is sent to the second target graphics processor through the first graphics processor.

5. The method according to claim 4, characterized in that After sending the first target key value information to the second target graphics processor through the first graphics processor, the method further includes: receiving the key value information through the second target graphics processor and determining a video memory address in the second target graphics processor for storing the first target key value information; The first target key value information is written into the video memory address through the second target graphics processor.

6. The method according to claim 3, characterized in that: After determining whether the number of the first graphics processors is the same as the number of the second graphics processors, the method further includes: If the number of the first graphics processors is different from the number of the second graphics processors, determining the ID of the second target key-value information to be sent in the first graphics processor; Reading the second target key value information from the target video memory area based on the ID of the second target key value information through a parallelized replication operator, and writing the second target key value information into a preset sending buffer area in parallel; The third target key value information in the sending buffer is sent to the second graphics processor corresponding to the decoding node through a preset sending function.

7. The method according to claim 6, characterized in that After sending the third target key value information in the sending buffer to the second graphics processor corresponding to the decoding node through a preset sending function, the method further includes: Receiving the third target key value information through a preset receiving function, and writing the third target key value information into a preset receiving buffer area; Determine fourth target key value information required by a second graphics processor corresponding to the decoding node; The key value information in the receiving buffer area is split by a parallelized copy operator to obtain the fourth target key value information, and the fourth target key value information is written into a graphics memory area in the second graphics processor corresponding to the decoding node.

8. The method according to claim 1, characterized in that: Before processing the description information by using a first inference model in a first graphics processor corresponding to the pre-filled node, the method further includes: For the first graphics processor, dividing the description information into key value logic blocks according to a preset key value block size to obtain key value logic block identifiers of characters in the description information; According to the key value logical block identifier of the character in the description information, the key value logical block identifier corresponding to the key value information to be generated is determined.

9. A data processing device based on a large model, characterized in that: include: A first processing unit, configured to process the description information inputted by the target object through a first reasoning model in a first graphics processor corresponding to the pre-filled node, and obtain key value information and a first reply character outputted by the first reasoning model in the first graphics processor; a storage unit, configured to store the key-value information belonging to the same key-value logical block identifier into a target video memory area in the first graphics processor based on the key-value logical block identifier corresponding to the key-value information, wherein the storage addresses of the key-value information belonging to the same key-value logical block identifier in the target video memory area are continuous; The second processing unit is used to process the key value information and the first reply character in the target video memory area through a second inference model in a second graphics processor corresponding to the decoding node to obtain target reply information corresponding to the description information.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute the large model-based data processing method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: include: A memory storing an executable program; A processor is used to run the program, wherein the program, when running, executes the large model-based data processing method described in any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises a stored computer program, and when the computer program is executed by a processor, the large model-based data processing method according to any one of claims 1 to 8 is implemented.

Images