Approximate inference method, apparatus, electronic device, and readable storage medium

Abstract

The application discloses a kind of approximate inference method, device, electronic equipment and readable storage medium.The method comprises: receiving target input data by large language model;According to the dimension of the basic unit of target input data, the target input data is processed by block, and a plurality of data blocks are obtained, wherein each data block in the plurality of data blocks includes a plurality of basic units;Determine the key-value pair corresponding to each of the plurality of basic units included in each data block, and select the representative key-value pair from the key-value pair corresponding to each of the plurality of basic units included in each data block as the target key-value pair of the corresponding data block;Determine the attention score corresponding to each of the plurality of data blocks based on the target key-value pair corresponding to each of the plurality of data blocks, and infer according to the attention score, to obtain inference result.The application can effectively reduce the data amount of key-value pair that needs to be loaded, reduce the storage and memory overhead, reduce the data transmission cost, so that the inference efficiency is more efficient.

Description

Technical Field

[0001] This application relates to the field of neural network technology, and more specifically, to a method, apparatus, electronic device, and readable storage medium for approximate reasoning. Background Technology

[0002] With the development of artificial intelligence technology, reasoning using large language models has become a trend. However, the process of reasoning using large language models involves a series of complex matrix operations and nonlinear activation functions to generate output, which places high demands on computing resources, especially GPU memory. Currently, reasoning using large language models suffers from high resource consumption and low inference efficiency. Therefore, how to improve the inference efficiency of large language models while reducing their resource consumption has become an urgent problem to be solved. Summary of the Invention

[0003] In view of the above problems, this application proposes a method, apparatus, electronic device and readable storage medium for approximate reasoning to solve the above problems.

[0004] In a first aspect, embodiments of this application provide an approximate reasoning method, the method comprising: receiving target input data through a large language model; dividing the target input data into blocks according to the dimension of the basic units of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units; determining the key-value pairs corresponding to each of the multiple basic units included in each data block, and selecting representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as target key-value pairs for the corresponding data block; determining the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and performing reasoning based on the attention scores to obtain a reasoning result.

[0005] Secondly, this application provides an approximate reasoning apparatus, comprising: a target input data receiving module for receiving target input data through a large language model; a block segmentation module for segmenting the target input data into multiple data blocks based on the dimensions of the basic units of the target input data, wherein each of the multiple data blocks includes multiple basic units; a target key-value pair determination module for determining the key-value pairs corresponding to each of the multiple basic units included in each data block, and selecting representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as target key-value pairs for the corresponding data block; and a reasoning module for determining the attention scores corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and performing reasoning based on the attention scores to obtain a reasoning result.

[0006] Thirdly, embodiments of this application provide an electronic device, including a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor performs the above-described method.

[0007] Fourthly, embodiments of this application provide a computer-readable storage medium storing program code, which can be invoked by a processor to execute the above-described method.

[0008] The approximate inference method, apparatus, electronic device, and readable storage medium provided in this application divide the target input data into blocks according to the dimension of the basic units of the target input data received by the large language model, resulting in multiple data blocks. Then, it determines the key-value pairs corresponding to each of the multiple basic units included in each data block, and selects representative key-value pairs from these as the target key-value pairs for the corresponding data block. Furthermore, it determines the attention scores for each of the multiple data blocks based on the target key-value pairs, and finally performs approximate inference based on these attention scores to obtain the inference result. In this application, by dividing the target input data into blocks, the excessive computational overhead of standard attention mechanisms is avoided, making the large language model more efficient in processing target input data. Moreover, by selecting representative key-value pairs as the target key-value pairs for the corresponding data blocks and performing attention inference based on these target key-value pairs, the amount of key-value pairs that need to be loaded is effectively reduced. It reduces storage and memory overhead, further lowers data transfer costs, makes inference more efficient, shortens inference time, improves response speed, reduces user waiting time, and enhances user experience. Attached Figure Description

[0009] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0010] Figure 1 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0011] Figure 2 This illustration shows a schematic diagram of attention pass calculation based on target key-value pairs according to an embodiment of this application;

[0012] Figure 3A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0013] Figure 4 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0014] Figure 5 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0015] Figure 6 A schematic diagram of the prefilling acceleration of segmented approximate attention provided in an embodiment of this application is shown;

[0016] Figure 7 This illustration shows a schematic diagram of reasoning combining an auxiliary model and a main model, provided in an embodiment of this application.

[0017] Figure 8 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0018] Figure 9 This illustration shows a schematic diagram of a tree-based greedy prediction method combined with single-point matching failure backtracking provided in an embodiment of this application;

[0019] Figure 10 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0020] Figure 11 A schematic diagram illustrating computation sharing provided in an embodiment of this application is shown;

[0021] Figure 12 This illustration shows a schematic diagram of storing data in different storage locations according to an embodiment of this application;

[0022] Figure 13 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown.

[0023] Figure 14 A block diagram of an approximate reasoning apparatus provided in one embodiment of this application is shown;

[0024] Figure 15 A block diagram of an electronic device for performing a data acquisition method according to an embodiment of this application is shown;

[0025] Figure 16 A storage unit for storing or carrying program code implementing the boot exception handling method according to an embodiment of the present application is shown. Detailed Implementation

[0026] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0027] To better understand the solutions of the embodiments of this application, the technical terms used in the embodiments of this application will be explained below.

[0028] LLM: Large Language Model.

[0029] Transformer: The core architecture of LLMs.

[0030] KV Cache: Key-Value Cache.

[0031] Currently, existing solutions for accurate inference on large models include DeepSpeed ​​and VLLM. DeepSpeed, an open-source deep learning optimization library developed by Microsoft, focuses on optimizing the inference process of large models. By combining a high-performance custom inference kernel, communication optimization, and heterogeneous memory technology, it achieves inference at unprecedented scales while achieving unparalleled reductions in latency, throughput, and cost. VLLM is a large model serving framework designed for batch processing scenarios. Its proposed PagedAttention algorithm efficiently manages intermediate variables during the inference process of large models, saving GPU memory and improving throughput. PagedAttention utilizes manually managed paging technology, allowing the storage of consecutive keys and values ​​in non-contiguous memory space. This means that when processing multiple requests, the sequence length of the requests does not need to be the same, thereby improving GPU memory utilization. However, they mainly focus on exact inference scenarios, and therefore only have specific optimizations for exact LLM inference scenarios, without specific designs for efficient approximate LLM inference. However, in many existing long sequence high-resource inference scenarios, due to the inference overhead of the LLM architecture increasing quadratically with the sequence length, users often focus more on efficiency and resource requirements, while sacrificing some of the generation quality requirements.

[0032] Large model inference tasks typically involve extremely complex computations, especially when handling large-scale batch inference, where computational resources and time consumption are enormous, easily leading to performance bottlenecks. This is mainly because large models rely on a large number of parameters and complex matrix operations. The main stages of large model inference computation are generally divided into Prefilling and Decoding.

[0033] Prefilling refers to the stage where the model processes the input sequence at the beginning of generation inference. This is usually the first time the model sees the input, it performs batch processing on the entire input sequence, including calculating and storing all queries, keys, and values. This step is usually used to initialize the context of inference, such as processing prompts or dialogue history, to ensure that the model has enough contextual information for subsequent generation. However, there are bottleneck problems in long sequence inference: (1) Computational complexity: The complexity of the self-attention mechanism is the square of the sequence length. Therefore, when the input sequence is long, the computational cost increases exponentially, especially for the self-attention computation of long sequences, which consumes a lot of system resources. (2) Storage burden: Long sequences mean that more key-value pairs need to be stored in the cache, which consumes a lot of memory and storage space, putting a burden on hardware resources, especially in the KV caching mechanism. (3) High initial delay: Because all input sequences need to be processed first, this will result in a long initial delay in the model response. Users need to wait a long time when performing generation tasks, especially in the case of long sequences.

[0034] Decoding refers to the process of progressively outputting the generated sequence; that is, the model decodes new content step by step based on the previous generated results. Decoding is usually autoregressive, meaning that each time a new token is generated, it is added to the context as input for the next generation, and this process continues until the generation task is completed. However, long sequence inference suffers from the problem of KV cache growth: during the decoding phase, each new token generated adds a new value to the KV cache, causing the cache to grow continuously. For long generated sequences, the KV cache will occupy more and more storage space, increasing memory pressure and management costs.

[0035] Existing large-model split scheduling solutions include Mooncake, which is the service platform of Kimi, an LLM service provided by Moonshot AI. It employs a split architecture centered on KVCache, separating the prefill and decoding processes into separate clusters and utilizing the CPU, DRAM, and SSD resources of the GPU cluster for split caching of the KVCache. Mooncake reduces computational resource waste by predicting load and rejecting overloaded requests early. In long-context scenarios, Mooncake achieves a 525% throughput improvement in simulated scenarios compared to baseline methods and handles 75% more requests under real-world loads. However, this split scheduling architecture itself faces significant scheduling overhead; a unified local scheduling solution would be a better choice.

[0036] In traditional pre-trained model inference architectures, prefilling and decoding are two independent stages. The prefilling stage typically handles parallel processing of input data, resulting in high computational complexity, while the decoding stage generates output tokens sequentially through autoregression. This difference in computational characteristics complicates resource scheduling, hindering efficient hardware resource utilization and leading to uneven inference performance. Furthermore, the system needs to frequently switch between these two stages, increasing management and scheduling complexity and incurring additional computational and resource overhead. This stage-switching overhead is more pronounced under high load or batch request scenarios, limiting system response speed and throughput.

[0037] To address the aforementioned problems, the inventors, through long-term research, discovered and proposed the approximate inference method, apparatus, electronic device, and readable storage medium provided in the embodiments of this application. By segmenting the target input data into blocks, the excessive computational overhead of the standard attention mechanism is avoided, making large language models more efficient in processing target input data. Furthermore, by selecting representative key-value pairs as the target key-value pairs for the corresponding data blocks and performing attention inference based on these target key-value pairs, the amount of key-value pairs that need to be loaded is effectively reduced. This reduces storage and memory overhead, further lowers data transmission costs, makes inference more efficient, shortens inference time, improves response speed, reduces user waiting time, and enhances user experience. The specific approximate inference method is described in detail in subsequent embodiments.

[0038] Please see Figure 1 , Figure 1 A flowchart illustrating an embodiment of the approximate inference method provided in this application is shown. This method avoids the excessive computational overhead of standard attention mechanisms by dividing the target input data into blocks, making large language models more efficient in processing target input data. Furthermore, by selecting representative key-value pairs as the target key-value pairs for corresponding data blocks and performing attention inference based on these target key-value pairs, the amount of key-value pairs that need to be loaded is effectively reduced. This reduces storage and memory overhead, further lowers data transmission costs, makes inference more efficient, shortens inference time, improves response speed, reduces user waiting time, and enhances user experience. In specific embodiments, this approximate inference method is applied to, for example... Figure 14 The approximate reasoning device 200 and the electronic device 100 equipped with the approximate reasoning device 200 are shown. Figure 15The following will use an electronic device as an example to illustrate the specific process of this embodiment. Of course, it is understood that the electronic device used in this embodiment may include smartphones, tablets, wearable electronic devices, etc., and is not limited thereto. The following will focus on... Figure 1 The process shown will be explained in detail. The approximate reasoning method may specifically include the following steps:

[0039] Step S110: Receive target input data through the large language model.

[0040] One approach is to use a large language model to infer meaning from user input based on different requests, thereby obtaining the desired result. For example, a user inputs an English paper, and the large language model performs translation inference on the input paper to obtain a translation result. Here, the target input data is the data entered by the user based on their actual needs.

[0041] In some embodiments, the target input data includes at least one of text data, image data, audio data, and video data.

[0042] Optionally, when the target input data received by the large language model is multimodal data, that is, data including at least two types of data (text data, image data, audio data and video data), the large language model can perform reasoning on each type of data separately, and then perform reasoning based on the interactive attention mechanism of different modalities and the context of the data of different modalities.

[0043] Step S120: Divide the target input data into blocks according to the dimensions of the basic units of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units.

[0044] One approach is to first determine the tokens within the target input data after obtaining it, and then determine the dimensions of those tokens. This allows for the segmentation of the target input data based on the dimensions of the tokens. In the large language model, the token is the smallest unit of data processing, i.e., the basic unit of the target input data.

[0045] Optionally, if the target input data is text data, the word dimension in the target input data is determined by determining the position of each character in the text data, and the target input data is then segmented based on the word dimension. If the target input data is image data, the pixel dimension of the image data is determined, and the dimension of the basic unit of the target input data is determined by determining the position of each pixel in the image data, or by determining the color channel corresponding to each pixel (such as the red, green, and blue dimensions in RGB mode), and the target input data is then segmented based on the dimension of the basic unit. If the target input data is audio data, the dimension of the basic unit of the target input data can be determined by determining the time dimension or the amplitude of each audio sampling point. The target input data can then be segmented based on the dimension of the basic unit. If the target input data is video data, the dimension of the basic unit of the target input data can be determined by determining the pixel distribution in the horizontal (width) and vertical (height) directions of each image frame, or the color channel dimension of each image frame, or the time dimension of consecutive frames. The target input data can then be segmented based on the dimension of the basic unit.

[0046] Optionally, the block size of the target input data can be determined in real time by assessing the complexity of the target input data, the importance of the context, or the complexity of the inference request corresponding to the target input request. For example, if the target input data is image data, the complexity of inference based on image data is high due to the large number of pixels. Therefore, the size of each data block can be reduced to ensure that each data block can be inferred quickly, avoiding the reduction in inference efficiency caused by excessive information in the data block.

[0047] Step S130: Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block.

[0048] As a method, key-value pairs are a crucial component of attention mechanisms. Initially applied primarily in Natural Language Processing (NLP) tasks such as machine translation and text generation, attention mechanisms have since been widely adopted in other fields like computer vision. Their core purpose is to enable the model to focus on relevant parts of the input data, much like how humans selectively focus on important information. Taking the Transformer architecture in NLP as an example, it is built upon a self-attention mechanism. In this architecture, the input text sequence is transformed into a series of vector representations, including a query vector, a key vector, and a value vector. The key and value vectors together form key-value pairs.

[0049] Optionally, the basic units included in each data block can be represented by vectors, and then the query vector, key vector and value vector corresponding to each of the multiple basic units can be determined by linear transformation. Then, the key-value pairs corresponding to each of the multiple basic units can be determined based on the key vector and value vector corresponding to each of the multiple basic units.

[0050] Optionally, the target key-value pair can be determined by the maximum and / or minimum value among the key-value pairs corresponding to each of the multiple basic units in each data block. Alternatively, cluster analysis can be performed on the values ​​of the key-value pairs corresponding to all basic units within each data block to obtain multiple clusters of key-value pairs. Then, the cluster center key-value pairs corresponding to each type in each data block can be selected, and one key-value pair can be randomly selected as the target key-value pair. Another option is to determine the average value of the key-value pairs in each data block, and then determine the key-value pair with the smallest difference between the key-value pairs corresponding to each of the multiple basic units and the average value of the key-value pairs in the data block.

[0051] Step S140: Determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and perform inference based on the attention scores to obtain the inference result.

[0052] As one approach, large language models based on attention mechanisms calculate attention scores for each basic unit of the input data during inference. However, when the target input data is large and complex, the number of basic units increases, leading to very slow inference efficiency. Furthermore, the storage of key-value pairs for each basic unit during inference increases the computational complexity, further reducing efficiency. Therefore, after determining the target key-value pairs for each data block, the data block is treated as a whole, and the target key-value pairs are used as the key-value pairs for that data block. Then, attention scores for multiple data blocks are determined using the target key-value pairs for each data block. This approximate attention scores are used to generate an importance score between the target input data and the data blocks. Finally, inference is performed based on these attention scores to obtain the inference result.

[0053] Optionally, by determining the attention scores corresponding to each of the multiple data blocks through the target key-value pairs corresponding to each of the multiple data blocks, that is, by reducing the attention scores corresponding to the basic units that need to be calculated, the computational load of the model is reduced, thus realizing structured pruning in the attention mechanism and improving the efficiency of large language models in reasoning.

[0054] Optionally, after determining the target key-value pairs corresponding to each of the multiple data blocks, the attention score can be obtained by multiplying the query values ​​corresponding to each of the multiple basic units in each data block with the key-value pairs corresponding to each of the multiple basic units. Since in this embodiment, representative key-value pairs are selected from the key-value pairs corresponding to the multiple basic units in each data block as the target key-value pairs, the attention score for that data block can be obtained simply by multiplying the query values ​​corresponding to the target key-value pairs. Figure 2As shown, for each query, multiple key-value pairs are determined from the initial key set based on the corresponding sample. Then, representative key-value pairs are selected from these multiple key-value pairs. Finally, K important keys (the retained tokens and selected tokens in the diagram) are selected from the initial key set to form the Top-K key set. During this process, some tokens (white blocks) are retained in storage resources, while some tokens (blocks with dashed borders, i.e., deleted tokens) are reduced during computation. Ultimately, based on the above operations, the estimated output (the estimated output result corresponding to each query) is obtained, which is the attention score corresponding to each data block. Each query corresponds to one sample, some empty, some marked with numbers, possibly representing sample information or specific identifiers related to the query. Compared to current standard attention calculation methods, determining the attention score for each data block based on the target key-value pairs eliminates the need for attention calculation on every basic unit in the target input data. It removes the storage of unimportant key-value pairs for each data block, thereby reducing redundant computational operations and improving the inference speed of large language models. The attention calculation only needs to be performed on these representative elements, rather than processing each token.

[0055] Optionally, after determining the attention scores corresponding to each of the multiple data blocks, the attention scores of the multiple data blocks are aggregated by methods such as average aggregation and weighted aggregation to obtain the attention score of the target input data. Then, inference is performed based on the attention scores according to the inference task corresponding to the target input data.

[0056] As another approach, quantization inference technology is also incorporated to further reduce computational costs and memory requirements. Quantization technology compresses the weights and activation function values ​​in the attention mechanism into a low-precision format, such as compressing 16-bit floating-point numbers into 4 bits. This reduces the computational resource consumption during attention calculations on multiple data blocks, making it particularly suitable for large-model inference scenarios on low-power devices.

[0057] As another approach, at the hardware implementation level, inference for large language models can be further optimized through targeted hardware accelerators (such as specific acceleration units on FPGAs or GPUs). By accelerating segmentation operations and target key-value pair selection processes at the hardware level, inference efficiency can be significantly improved. Especially for large-scale distributed clusters, computational tasks can be parallelized and distributed across different hardware nodes to reduce network communication overhead.

[0058] In the embodiments of this application, the target input data is segmented according to the dimension of the basic units of the target input data received by the large language model, resulting in multiple data blocks. Then, the key-value pairs corresponding to the multiple basic units included in each data block are determined. Representative key-value pairs are selected from these pairs as the target key-value pairs for the corresponding data block. Furthermore, attention scores for each data block are determined based on these target key-value pairs. Finally, approximate inference is performed based on the attention scores for each data block to obtain the inference result. In this application, segmenting the target input data avoids the excessive computational overhead of standard attention mechanisms, making the large language model more efficient in processing target input data. By selecting representative key-value pairs as the target key-value pairs for the corresponding data blocks and performing attention inference based on these target key-value pairs, the amount of key-value pairs that need to be loaded is effectively reduced. This reduces storage and memory overhead, further lowers data transmission costs, makes inference more efficient, shortens inference time, improves response speed, reduces user waiting time, and enhances user experience.

[0059] Please see Figure 3 , Figure 3 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown. The following will focus on... Figure 3 The process shown will be explained in detail. The approximate reasoning method may specifically include the following steps:

[0060] Step S210: Receive target input data through the large language model.

[0061] Step S220: Divide the target input data into blocks according to the dimensions of the basic units of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units.

[0062] Step S230: Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and determine the maximum key-value pair and the minimum key-value pair among the key-value pairs corresponding to each of the multiple basic units included in each data block.

[0063] In one approach, after determining the key-value pairs corresponding to the multiple basic units included in each data block, the key-value pairs corresponding to the multiple basic units can be compared to find the maximum and minimum values ​​among the key-value pairs corresponding to the multiple basic units in each data block, thereby determining the maximum and minimum key-value pairs.

[0064] Step S240: Determine the maximum key-value pair and the minimum key-value pair as the target key-value pair.

[0065] As a method, since the maximum and minimum key-value pairs highlight the key and edge information in the data block, they can well reflect the characteristics of the entire data segment or data block. In order to select representative key-value pairs that can represent the characteristic information in the corresponding data block from the key-value pairs corresponding to each of the multiple basic units, the maximum and minimum key-value pairs can be determined as the target key-value pairs.

[0066] In some embodiments, after step S240, the method further includes: storing the target key-value pair for use in the next inference operation.

[0067] As one approach, in attention-based reasoning methods, key-value pairs can be stored to facilitate quick location and retrieval of corresponding values ​​during subsequent reasoning. Only target key-value pairs can be stored, reducing the waste of storage space and storage pressure caused by storing key-value pairs that have little impact on the reasoning process.

[0068] Step S250: Determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and perform inference based on the attention scores to obtain the inference result.

[0069] The specific steps of steps S210-S220 and S250 can be found in steps S110-S120 and S140, and will not be repeated here.

[0070] In this embodiment, by selecting the largest and smallest key-value pairs as target key-value pairs, attention scores are calculated based on these target key-value pairs. This avoids the low computational efficiency caused by calculating attention scores for all key-value pairs. Furthermore, storing the target key-value pairs reduces storage pressure and improves inference efficiency.

[0071] Please see Figure 4 , Figure 4 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown. The following will focus on... Figure 4 The process shown will be described in detail. If the target input data is the text data, the approximate reasoning method may specifically include the following steps:

[0072] Step S310: Receive target input data through the large language model.

[0073] Step S320: Determine the text length of the text data.

[0074] As one approach, large language models need to process a large amount of information and perform long document analysis in scenarios involving long texts. However, in order to ensure the efficiency of large language models in processing long texts, some unimportant information needs to be ignored. In scenarios involving short texts, large language models should calculate each token as much as possible. Therefore, when receiving text data as target input data, the length of the text data is first determined, and based on the text length, it is determined whether the scenario in which the large language model performs inference is a long text scenario or a short text scenario.

[0075] Step S330: If it is determined that the length of the text is greater than the length threshold, the target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain the plurality of data blocks, wherein each data block in the plurality of data blocks includes a plurality of basic units.

[0076] As one approach, when the text length is determined to be greater than a length threshold, the large language model is in the scenario of long text reasoning. In this case, the target input data can be divided into blocks, and the target key-value pairs can be determined by the representative key-value pairs of the key-value pairs corresponding to the multiple basic units in each data block. Approximate reasoning can then be performed based on the target key-value pairs corresponding to multiple data blocks, thereby improving the efficiency of the large language model in reasoning about long texts.

[0077] Step S340: Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block.

[0078] Step S350: Determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and perform inference based on the attention scores to obtain the inference result.

[0079] For a detailed description of steps S310 and S340-S350, please refer to steps S110-S150, which will not be repeated here.

[0080] In some embodiments, after step S310, the method further includes:

[0081] Determine the dialogue turn corresponding to the text data, and determine the dialogue complexity corresponding to the text data.

[0082] As one approach, large language models need to process a large amount of information when performing inference in scenarios involving long, multi-turn dialogues with users. To ensure inference efficiency, approximate inference methods can be used. However, for dialogues with fewer turns, as much detailed information as possible is needed to ensure accuracy, so approximate inference methods are unnecessary. Therefore, after receiving the target input data, the dialogue turn and dialogue complexity corresponding to the text data as the target input data can be determined first. This allows for determining whether approximate inference is needed by performing operations such as segmenting the text data based on the turn and dialogue complexity.

[0083] Optionally, the dialogue length of each conversation in the text data can be determined first, and the dialogue complexity corresponding to the text data can be determined based on this dialogue length. For example, the dialogue complexity can be determined based on the number of long dialogues and their lengths by statistically analyzing the dialogue lengths in the text data.

[0084] If the number of dialogue rounds is greater than the number of rounds threshold and the dialogue complexity is greater than the complexity threshold, then the target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain the multiple data blocks.

[0085] As one approach, when the number of dialogue rounds exceeds a round threshold and the dialogue complexity exceeds a complexity threshold, the text data can be identified as text data generated in a long-term, multi-round dialogue scenario. Therefore, it can be determined that the target input data can be divided into blocks, and the target key-value pairs can be determined by identifying representative key-value pairs from the key-value pairs corresponding to multiple basic units in each data block. Based on the target key-value pairs corresponding to multiple data blocks, approximate inference can be performed, thereby improving the efficiency of large language models in reasoning about long texts.

[0086] In this embodiment, the decision to trigger block processing of the target input data and subsequent key-value pair selection is determined based on the text length of the text data or the corresponding dialogue turn and the corresponding dialogue complexity of the text data. This makes the large language model more applicable to different scenarios and automatically enables approximate calculation under certain conditions without manual user intervention, thereby improving the user experience.

[0087] Please see Figure 5 , Figure 5 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown. The following will focus on... Figure 5The process is described in detail below. The large language model includes an auxiliary model and a main model. The model parameters of the main model are greater than those of the auxiliary model. The auxiliary model is used for inference based on the attention score, and the main model is used for verification based on the output of the auxiliary model. The approximate inference method may specifically include the following steps:

[0088] Step S410: Receive target input data through the large language model.

[0089] Step S420: Divide the target input data into blocks according to the dimensions of the basic units of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units.

[0090] Step S430: Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block.

[0091] The specific steps of S410-S430 can be found in steps S110-S130, and will not be repeated here.

[0092] Step S440: Determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and perform inference based on the attention scores through the auxiliary model to obtain multiple prediction results, and send the multiple prediction results to the main model.

[0093] As one approach, the auxiliary model can be a lightweight model of a small-parameter draft model, while the main model can be a model with high accuracy and many parameters. Since the auxiliary model has fewer parameters and a relatively simple structure, it can quickly generate some possible intermediate results or predictions of subsequent words based on the attention scores corresponding to each of the multiple data blocks, thereby obtaining multiple prediction results.

[0094] Step S450: Determine the probability corresponding to each of the multiple prediction results using the master model.

[0095] As one approach, since the main model has a large number of parameters and high accuracy, its inference probability can be determined by validating multiple predictions obtained from the auxiliary model. Optionally, the main model can determine the probability of each prediction by semantically matching the multiple predictions output by the auxiliary model.

[0096] Step S460: Determine the inference result based on the probabilities corresponding to each of the multiple prediction results.

[0097] As one approach, if a prediction with a probability greater than a probability threshold is found among the probabilities corresponding to multiple prediction results, then the initial prediction of the auxiliary model is determined to be correct. Subsequent inference can then be performed based on the prediction results with a probability greater than the probability threshold, thereby skipping some originally required calculation steps, reducing unnecessary calculations, improving inference efficiency, and reducing the computational pressure on large language models.

[0098] As a general approach, the reasoning process of large language models currently mainly consists of two processes: prefilling and decoding. Typically, the large language model first performs prefilling, where each token in the input data is mapped into low-dimensional, dense word vectors through the model's embedding layer. These word vectors are then processed sequentially through multiple layers (such as the Transformer architecture's multi-head attention layer and feedforward neural network layers) according to the model's architecture (e.g., the Transformer architecture) to obtain the overall feature representation of the entire input data. Based on this overall feature representation obtained during the prefilling process, subsequent output tokens are gradually generated to construct the complete reasoning result. In other words, in traditional reasoning architectures, prefilling and decoding are usually considered two independent stages. Prefilling processes the input sequence, while decoding generates the output based on the processed input sequence. This results in the decoding stage only being able to proceed after the prefilling stage is complete, leading to a decrease in the efficiency of large language models in reasoning. Therefore, speculative inference can be used, employing a main model and an auxiliary model with different parameters to perform inference together. The auxiliary model makes initial predictions based on attention scores from multiple data blocks, which are then validated by the main model for parallel inference, achieving a several-fold increase in inference speed. Furthermore, during the auxiliary model's initial prediction, the target input data can be divided into blocks. For each data block, representative key-value pairs are identified as target key-value pairs for prediction. Attention scores for each data block are calculated using these target key-value pairs, and these calculated scores are used as approximate attention scores for each data block in the prediction process. This simplifies traditional intensive self-attention computation while maintaining computational efficiency. Figure 6 As shown, calculations are performed between query segments and target key-value pairs. The criticality between each query segment and each target key-value pair is estimated by calculating the maximum and minimum values ​​(indicated by "MAX; MIN" in the diagram), yielding values ​​such as 0.3, 0.02, 0.01, and 0.5. Then, based on the estimated criticality, non-critical calculations in the self-attention mechanism are pruned. Figure 6(The pruned parts are marked with "×") to remove unnecessary computations and improve computational efficiency. By having the auxiliary model and the main model perform inference together, the autoregressive process is transformed into a prefilling-like process, forming the objective conditions for a unified inference framework. Prefilling and decoding computations can share the same contextual information, and attention computation is performed on target key-value pairs during inference, reducing the need for redundant data loading and storage, eliminating the switching overhead between the prefilling and decoding stages, and making the inference process smoother and more efficient. Specifically, in the prefilling computation stage, by dividing the input data into several data blocks and then selecting key representative key-value pairs in each block, attention computation is performed only on these representative key-value pairs, significantly reducing the computational latency of the large model preprocessing stage. In the decoding stage, a piecewise approximate attention mechanism is used to parallelize the computation of a batch of token sequences generated by the auxiliary and main models, improving the parallelism of the decoding stage computation and increasing the generation rate of the inference stage while maintaining a certain quality.

[0099] Furthermore, by combining the auxiliary model with the main model for inference, the decoding stage is further divided into two phases: prediction (draft) and verification (verify). In the prediction phase, important data is extracted based on the query value corresponding to the currently processed data block, and only this extracted important information is used to predict multiple subsequent tokens. In the verification phase, a segmented approximate attention method is used to verify the predicted multiple tokens in parallel, significantly improving the parallelism of decoding while maintaining a certain level of decoding quality. After obtaining multiple prediction results from the auxiliary model, the main model can divide these prediction results and context into multiple segments, and perform attention calculations on these segments using target key-value pairs to obtain the probabilities corresponding to each prediction result. This method ensures parallel computation of multiple segments. By utilizing parallelized segmented approximate attention, representative query and target key-value pairs are selected and simplified, significantly reducing the attention computation burden while maintaining computational accuracy. Then, based on the multiple prediction results obtained, it is determined whether the decoding result matches the speculative prediction. If a match is found, the matching prediction result is determined as the inference result. If the results do not match, the system will revert to the conventional autoregressive path, which involves inference through the main model and correcting erroneous generated results. This reduces redundant computation and improves inference efficiency by pruning non-critical target key-value pairs. Figure 7As shown, the process first involves using an auxiliary model to perform draft inference, generating multiple prediction results. This allows for the rapid generation of preliminary inference content using a lightweight model. Then, the main model calculates attention scores based on the key-value pairs corresponding to each of the multiple data blocks, and determines the probabilities of each prediction result based on these attention scores, resulting in values ​​such as 0.5, 0.08, 0.3, and 0.02. The process then uses a pruned self-attention mechanism and the probabilities of each prediction result to filter and verify the results (with "√" and "×" indicating the verification results). The filtered results are then concatenated with a portion of the results from the smaller model to obtain the final inference result.

[0100] As another approach, in heterogeneous computing environments, different hardware platforms (such as CPUs, GPUs, and TPUs) are used to handle different parts of the prefilling and decoding stages. For example, simpler attention segmentation computations can be handled by low-power CPUs, while complex, intensive computations are handled by GPUs or TPUs. Through load balancing and task allocation in heterogeneous computing architectures, the utilization of computing resources during the inference process can be optimized.

[0101] In this embodiment, multiple prediction results are obtained by using an auxiliary model for initial prediction. Then, the probability corresponding to each of the multiple prediction results is determined by the main model. Based on the probability, the inference result is determined, which further improves the efficiency of the large language model for inference, while ensuring the consistency of the calculation process and the efficiency of resource scheduling.

[0102] Please see Figure 8 , Figure 8 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown. The following will focus on... Figure 8 The process shown will be explained in detail. The approximate reasoning method may specifically include the following steps:

[0103] Step S510: Receive target input data through the large language model.

[0104] Step S520: Divide the target input data into blocks according to the dimension of the basic unit of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units.

[0105] Step S530: Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block.

[0106] Step S540: Determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and perform inference based on the attention scores to obtain the inference result. The specific steps of steps S510-S540 can be found in steps S110-S140, and will not be repeated here.

[0107] Step S550: Determine multiple combinations of basic units based on the reasoning results.

[0108] As one approach, since reasoning on target input data involves combining and assembling basic reasoning units at each step to obtain a complete reasoning result, a tree-structured greedy reasoning method can be used to ensure the accuracy of the reasoning result. This method involves reasoning on the target input data in the form of a tree structure. In each step of the tree-structured greedy reasoning method, the optimal option is selected to construct a tree structure representing possible results or relationships. This allows the reasoning result to include multiple basic combinations, with different combinations of basic units yielding different reasoning content. Since it is difficult to guarantee the complete accuracy of the prediction of the target input data's basic units, multiple possibilities for subsequent basic units can be recorded for verification, allowing for multiple choices. This enables the determination of multiple combinations of randomly generated basic units in the reasoning result.

[0109] Step S560: Determine the single-point matching results of each of the various basic unit combinations, and correct the historical prediction basic units based on the single-point matching results.

[0110] As one approach, when a large language model performs inference, the input data is segmented into a series of basic units. When predicting the next basic unit, the large language model compares and matches the predicted basic units one by one with the historical predicted basic units in the historical prediction results, thereby obtaining single-point matching results for various combinations of basic units, and then correcting the historical predicted basic units based on the single-point matching results.

[0111] Optionally, if a single-point matching result indicates a single-point mismatch among multiple basic unit combinations (e.g., two strings mismatched by only one character), then the historical prediction basic unit corresponding to the single-point mismatch is determined in the historical prediction record. The frequency of occurrence of each basic unit with the single-point mismatch is then determined within the historical prediction basic units. The basic unit with the higher frequency of occurrence is identified as the correctly predicted basic unit, and all historical prediction basic units are corrected to this corrected basic unit. This facilitates subsequent inference using the corrected historical basic units. Figure 9As shown, the input data is first predicted using a tree-based greedy prediction method, resulting in various combinations of basic units. As illustrated, a tree structure is constructed starting from the initial words "Alan" and "Waker". Taking "Alan" as an example, it has a branch called "Turing", which is further divided into two sub-branches: "is" and "was". The "is" branch then extends to "a", which in turn has possibilities such as "girl", "man", and "good". The branch structure of "Waker" is similar, with subsequent branches such as "is" and "feel". Then, the single-point matching results of various basic unit combinations are determined, and corrections are made based on these results. For example, the sentence "Alan Turing is not a girl" is used for verification. Here, "single-point failure correction" is marked, which determines the single-point matching result of each token in the sentence. If a token is found to be inconsistent with a certain rule or condition (possibly due to grammatical or semantic issues), it is corrected. The diagram highlights the token "not," suggesting that issues with its relationship to surrounding words or its overall semantics were discovered during verification, necessitating adjustments. The revised "not" then serves as a new starting point for further expansion and speculation within the tree structure. Following "not" are branches such as "is" and "a," with "a" further extending to possibilities like "man," "girl," and "good."

[0112] In this embodiment, the single-point matching results of various basic unit combinations are determined by determining the inference results. Based on the single-point matching results, the historical prediction basic units are corrected to ensure the accuracy of subsequent inference by the large language model.

[0113] Please see Figure 10 , Figure 10 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown. The following will focus on... Figure 10 The process shown will be explained in detail. The approximate reasoning method may specifically include the following steps:

[0114] Step S610: Obtain the preprocessing queue and determine the request prefix and request length corresponding to each of the multiple input data included in the preprocessing queue, wherein the multiple input data includes at least the target input data.

[0115] As one approach, when processing large amounts of input data, many similar requests may exist across different input data sets. These requests involve numerous identical computational processes during processing. Therefore, to improve the processing performance of large language models, it's possible to first determine the request prefixes and request lengths for each of the multiple input data sets included in the preprocessing queue. This allows for batch processing of large amounts of input data. Specifically, when a user inputs data into the large language model, a request is typically added specifying the specific operation the model will perform on that input data, such as translating a paper, analyzing an article, or interpreting a dialogue. The terms "translate," "analyze," and "interpret" constitute the request prefixes, and the request length is the text length of the request entered by the user based on the input data.

[0116] Step S620: Group the preprocessing queue according to the request prefix and the request length to obtain at least one preprocessing group.

[0117] One approach is to group input data with the same request prefix and request length into a single group, thereby facilitating centralized processing of input data with the same request prefix and request length.

[0118] Step S630: If it is determined that there is data in the cache data that is the same as the request prefix according to the request prefix corresponding to the at least one preprocessing group, then the target data that is the same as the request prefix is ​​obtained from the cache data, and incremental calculation is performed on each input data in the at least one preprocessing group according to the target data to obtain the inference result.

[0119] As one approach, a prefix tree can be constructed for at least one preprocessing group to cache and manage the input data of that preprocessing group. Since the large language model stores historical inference data during the inference process, the stored data can be obtained by first determining whether a request with the same request prefix has been processed in the stored data based on the request prefix corresponding to at least one preprocessing group. In this way, when it is determined that a request with the same request prefix has been processed or when there is completely identical input data, the target data generated when processing the same request prefix can be directly obtained from the stored data, or the inference result corresponding to the completely identical input data can be directly used as the output. This avoids the waste of computing resources caused by repeated calculation of input data with the same request prefix.

[0120] Optionally, since different input data may have the same request prefix but different parts, in order to ensure the accuracy of inference on the input data, incremental calculation is performed on the different parts of each input data in at least one preprocessing group in accordance with the method of steps S110-S150, combined with the target data with the same request prefix, to obtain the inference result.

[0121] Step S640: If it is determined that there is no data in the cached data that is the same as the request prefix according to the request prefix corresponding to the at least one preprocessing group, then any input data is selected from the at least one preprocessing group for processing to obtain reference data. The reference data is used as the target data of the at least one preprocessing group, and incremental calculation is performed on each input data in the at least one preprocessing group according to the target data to obtain the inference result.

[0122] As one approach, if it is determined that no requests with the same request prefix have been processed, inference can be performed on the input data in at least one preprocessing group according to steps S110-S150. Data corresponding to the same request prefix is ​​identified as reference data, which is then used as the target data for inference on the input data in the at least one preprocessing group. Then, the different parts corresponding to each input data in the at least one preprocessing group are combined with the target data with the same request prefix and incrementally calculated according to steps S110-S150 to obtain the inference result. This effectively reduces the time spent on repetitive calculations and improves processing efficiency. Figure 11 As shown, received user requests are added to a preprocessing queue. When processing other user requests, the requests in the preprocessing queue are sorted in batches, with requests with similar prefixes and lengths grouped together, constructing a prefix tree for caching and data management. After preprocessing, requests are added to a waiting queue. The main loop continuously retrieves requests from the waiting queue for processing; if no requests are available, they are directly retrieved from the preprocessing queue for processing. Based on the constructed prefix tree, a group of requests with the same prefix can be extracted. For these requests, the main loop first checks if there are any identical requests that have already been processed in the cache. If so, the cached result is returned directly; otherwise, the calculations for these identical prefixes are shared. Input data with the same prefix uses the same calculation result. Finally, incremental calculations are performed on each request, including incremental calculations on parts that are not cached. This effectively reduces the time spent on redundant calculations and improves processing efficiency.

[0123] In some embodiments, the method further includes:

[0124] Determine the usage frequency of each of the multiple data items in the cached data.

[0125] As one approach, given the limited cache size of large language models and the devices hosting them, reducing storage resource consumption while retaining essential information can be achieved by prioritizing the storage of important data on disks with faster read / write speeds and storing less important data on disks with slower read / write speeds. Therefore, the usage frequency of each data item in the cache can be determined first. Optionally, this usage frequency can be the number of times each data item in the cache is hit within a preset time period.

[0126] The storage location of each of the multiple data is determined according to the usage frequency, and the multiple data are stored based on the storage location, wherein the multiple data in the cache data are stored in a prefix tree manner.

[0127] One approach is to pre-set the usage frequency range for different storage locations, and then determine the storage location of each piece of data based on the usage frequency of the cached data and the usage frequency range of each storage location. This allows data with different usage frequencies to be stored in storage locations with different read / write speeds, ensuring cache hit rate while improving processing efficiency.

[0128] Optionally, a prefix tree-based LRU-like cache eviction policy can be used to store cached data. This involves evicting cached data that hasn't been used for a long time, ensuring a high cache hit rate while freeing up space for other cached data. Alternatively, to accelerate the caching process, a hierarchical cache management strategy can be employed. Evicted cached data from the GPU is moved to the CPU, evicted cached data from the CPU is moved to the SSD, and evicted data from the SSD is moved to a slower disk, ensuring a high cache hit rate while improving system processing efficiency. For example... Figure 12 As shown, since the data corresponding to cache block 1, cache block 14, cache block 15 and cache block 10 are the data corresponding to the prefix tree, they are used more frequently. Therefore, they are stored in the GPU with the fastest read and write speed. The data in the other cache blocks are the data after incremental calculation, which are used less frequently. Therefore, they are stored in the SSD with the slowest read and write speed.

[0129] As an alternative approach, dynamic paging technology can be used to manage the memory and computational resources of large language models based on vLLM (Virtual Large Language Model) and paged attention. This enables efficient scheduling of key-value pair caches during inference, reducing unnecessary computational overhead. vLLM allows for flexible switching between memory and computational resources of the large language model, without relying on the separation of hardware storage and computation. Paged attention further optimizes the resource usage of attention computation by computing attention in blocks and using paging technology to page less important key-value pair caches to lower memory levels. This approach reduces the consumption of hardware computational resources while retaining necessary attention information, improving the efficiency of approximate segmented attention inference.

[0130] In this embodiment, the preprocessing queue is grouped according to the request prefix and request length corresponding to each of the multiple input data included in the determined preprocessing queue to obtain at least one preprocessing group. Then, based on the request prefix corresponding to each of the at least one preprocessing group, it is determined whether there is data in the cached data that is the same as the request prefix. If data with the same request prefix is ​​found, target data with the same request prefix is ​​obtained from the cached data, and incremental calculation is performed on each input data in the at least one preprocessing group based on the target data to obtain an inference result. Alternatively, if data with the same request prefix is ​​found, any input data is selected from the at least one preprocessing group for processing to obtain reference data. The reference data is used as the target data for the at least one preprocessing group, and incremental calculation is performed on each input data in the at least one preprocessing group based on the target data to obtain an inference result. This effectively reduces the time spent on repeated calculations and improves processing efficiency.

[0131] Please see Figure 13 , Figure 13 A flowchart illustrating an embodiment of the approximate reasoning method provided in this application is shown. The following will focus on... Figure 13 The process shown will be explained in detail. The approximate reasoning method may specifically include the following steps:

[0132] Step S710: Receive target input data through the large language model.

[0133] Step S720: Determine the real-time load rate of the large language model.

[0134] As one approach, to ensure that the large language model can perform approximate attention calculation on the target input data based on the actual situation, that is, to further improve the adaptability and performance of the large language model by dynamically adjusting the block segmentation strategy, the real-time load rate of the large language model can be determined first, and then the size of the blocks for the target input data can be determined based on the real-time load rate of the large language model.

[0135] Step S730: Determine the block size based on the dimension of the basic unit of the target input data and the real-time load rate, and perform block processing on the target input data according to the block size to obtain the plurality of data blocks, wherein each of the plurality of data blocks includes a plurality of basic units.

[0136] As one approach, when the real-time load rate of a large language model is high, the corresponding data blocks to be processed contain more data. Therefore, reducing the number of data blocks increases the sparsity of the target input data, ensuring the large language model can quickly process multiple data blocks. Conversely, when the real-time load rate of the large language model is low, its computing power is sufficient, and the size of the data blocks can be reduced. This increases the number of data blocks, reduces the sparsity of the target input data, and ensures the quality and performance of the large language model in processing multiple data blocks. Therefore, the block size can be determined by combining the real-time load rate of the large language model and the dimensionality of the basic units of the target input data, and the target input data can be processed by dividing it into blocks based on this block size.

[0137] Step S740: Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block.

[0138] Step S750: Determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and perform inference based on the attention scores to obtain the inference result.

[0139] The specific steps of S710 and S740-S750 can be found in steps S110 and S130-S140, and will not be repeated here.

[0140] In this embodiment, the block size is determined based on the real-time load rate of the large language model and the dimension of the basic unit of the target input data. This allows the target input data to be segmented based on the block size, thereby improving the adaptability and performance of the large language model in approximate reasoning.

[0141] Please see Figure 14 , Figure 14 A block diagram of an approximate reasoning apparatus according to an embodiment of this application is shown. The following will address... Figure 14 The block diagram shown illustrates that the approximate reasoning device 200 includes: a target input data receiving module 210, a block segmentation module 220, a target key-value pair determination module 230, and a reasoning module 240, wherein:

[0142] The target input data receiving module 210 is used to receive target input data through a large language model;

[0143] The segmentation module 220 is used to segment the target input data according to the dimension of the basic unit of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units.

[0144] The target key-value pair determination module 230 is used to determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block.

[0145] The inference module 240 is used to determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and to perform inference based on the attention score to obtain the inference result.

[0146] Further, the target key-value pair determination module 230 includes a first determination unit and a second determination unit, wherein:

[0147] The first determining unit is used to determine the maximum key-value pair and the minimum key-value pair among the key-value pairs corresponding to the plurality of basic units included in each data block.

[0148] The second determining unit is used to determine the maximum key-value pair and the minimum key-value pair as the target key-value pair.

[0149] Furthermore, the approximate reasoning apparatus 200 also includes a storage module for storing the target key-value pairs for use in the next reasoning operation.

[0150] Furthermore, the target input data includes at least one of text data, image data, audio data, and video data.

[0151] Furthermore, if the target input data is the text data, the segmentation module includes a text length determination unit and a first segmentation unit, wherein:

[0152] The text length determination unit is used to determine the text length of the text data;

[0153] The first segmentation unit is used to segment the target input data according to the dimension of the basic unit of the target input data if it is determined that the length of the text is greater than the length threshold, so as to obtain the multiple data blocks.

[0154] Furthermore, if the target input data is the text data, the chunking module includes a dialogue complexity determination unit and a second chunking unit, wherein:

[0155] A dialogue complexity determination unit is used to determine the dialogue turn corresponding to the text data and to determine the dialogue complexity corresponding to the text data.

[0156] The second segmentation unit is used to segment the target input data according to the dimension of the basic unit of the target input data to obtain the multiple data blocks if the number of dialogue rounds is greater than the number of rounds threshold and the dialogue complexity is greater than the complexity threshold.

[0157] Furthermore, the large language model includes an auxiliary model and a main model, wherein the model parameters of the main model are greater than those of the auxiliary model, the auxiliary model is used for inference based on the attention score, and the main model is used for verification based on the output of the auxiliary model.

[0158] Furthermore, the reasoning module includes a prediction unit, a probability determination unit, and a reasoning result determination unit, wherein:

[0159] The prediction unit is used to perform inference based on the attention score through the auxiliary model, obtain multiple prediction results, and send the multiple prediction results to the main model;

[0160] A probability determination unit is used to determine the probability corresponding to each of the multiple prediction results through the main model;

[0161] The reasoning result determination unit is used to determine the reasoning result based on the probability corresponding to each of the multiple prediction results.

[0162] Furthermore, the reasoning result is obtained based on a tree-based greedy prediction method, and the approximate reasoning device 200 further includes a basic unit combination determination module and a correction module, wherein:

[0163] A basic unit combination determination module is used to determine multiple basic unit combinations based on the reasoning results;

[0164] The correction module is used to determine the single-point matching results of each of the various basic unit combinations, and to correct the historical prediction basic units based on the single-point matching results.

[0165] Furthermore, the approximate reasoning device 200 further includes a first determining module, a grouping module, a second determining module, and a third determining module, wherein:

[0166] The first determining module is used to obtain a preprocessing queue and determine the request prefix and request length corresponding to each of the multiple input data included in the preprocessing queue, wherein the multiple input data includes at least the target input data;

[0167] A grouping module is used to group the preprocessing queue according to the request prefix and the request length to obtain at least one preprocessing group;

[0168] The second determining module is configured to, if it is determined from the request prefix corresponding to the at least one preprocessing group that there is data in the cached data that is the same as the request prefix, then obtain target data that is the same as the request prefix from the cached data, and perform incremental calculation on each input data in the at least one preprocessing group based on the target data to obtain an inference result; or

[0169] The third determining module is used to select any input data from the at least one preprocessing group for processing if it is determined that there is no data in the cached data that is the same as the request prefix based on the request prefix corresponding to the at least one preprocessing group, to obtain reference data, to use the reference data as the target data of the at least one preprocessing group, and to perform incremental calculation on each input data in the at least one preprocessing group based on the target data to obtain the inference result.

[0170] Furthermore, the approximate reasoning apparatus 200 also includes a frequency determination module and a storage location determination module, wherein:

[0171] A frequency determination module is used to determine the usage frequency of each of the multiple data items in the cached data.

[0172] The storage location determination module is used to determine the storage location of each of the plurality of data according to the usage frequency, and to store the plurality of data based on the storage location, wherein the plurality of data in the cache data are stored in a prefix tree manner.

[0173] Further, the segmentation module 220 is characterized in that it includes a real-time load rate determination unit and a third segmentation unit, wherein:

[0174] A real-time load rate determination unit is used to determine the real-time load rate of the large language model;

[0175] The third segmentation unit is used to determine the segmentation size based on the dimension of the basic unit of the target input data and the real-time load rate, and to segment the target input data according to the segmentation size to obtain the multiple data blocks.

[0176] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0177] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0178] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0179] Please see Figure 15 This document illustrates a structural block diagram of an electronic device 100 provided in an embodiment of this application. The electronic device 100 can be a smartphone, tablet computer, e-reader, or other electronic device capable of running applications. The electronic device 100 in this application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications can be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more applications are configured to perform the methods described in the foregoing method embodiments.

[0180] The processor 110 may include one or more processing cores. The processor 110 connects to various parts within the electronic device 100 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 120, and by calling data stored in the memory 120. Optionally, the processor 110 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 110 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content to be displayed; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 110 and may be implemented separately using a communication chip.

[0181] The memory 120 may include random access memory (RAM) or read-only memory (ROM). The memory 120 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 120 may include a program storage area and a data storage area. The program storage area may store instructions for implementing an operating system, instructions for implementing functions (such as touch functionality, sound playback functionality, image playback functionality, etc.), and instructions for implementing the various method embodiments described below. The data storage area may also store data created by the electronic device 100 during use (such as phonebook data, audio and video data, chat log data, etc.).

[0182] Please see Figure 16 This diagram illustrates a structural block diagram of a computer-readable storage medium provided in an embodiment of this application. The computer-readable medium 400 stores program code that can be called by a processor to execute the methods described in the above method embodiments.

[0183] The computer-readable storage medium 400 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read-Only Memory), EPROM, hard disk, or ROM. Optionally, the computer-readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 400 has storage space for program code 410 that performs any of the method steps described above. This program code can be read from or written to one or more computer program products. The program code 410 may be compressed, for example, in a suitable form.

[0184] In summary, the multi-person group photo generation method, diffusion model training method, and apparatus provided in this application determine a reference image including multiple reference figures, extract the head coordinate information corresponding to each of the multiple reference figures from the reference image, obtain multiple headshot images input based on the multiple reference figures, and extract the headshot feature vectors corresponding to each of the multiple headshot images. There is a one-to-one correspondence between the headshot coordinate information corresponding to each of the multiple reference figures and the headshot feature vectors corresponding to each of the multiple headshot images. The headshot coordinate information corresponding to each of the multiple reference figures and the headshot feature vectors corresponding to each of the multiple headshot images are input into the control network and enhanced diffusion network of the trained diffusion model for attention calculation, thereby obtaining a target group photo image output by the trained diffusion model that matches the multiple headshot images. Therefore, by extracting the headshot coordinate positions corresponding to each of the multiple reference figures in the reference image and the headshot feature vectors corresponding to each of the multiple headshot images, and jointly injecting them into the control network and enhanced diffusion network of the diffusion model for attention calculation, it is possible to ensure that the headshot features can be accurately compiled, thus improving the generation effect of multi-person group photos.

[0185] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for approximate reasoning, characterized in that, The method includes: Receive target input data through a large language model; The target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units. Determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block. Based on the target key-value pairs corresponding to each of the multiple data blocks, the attention score corresponding to each of the multiple data blocks is determined, and inference is performed based on the attention score to obtain the inference result.

2. The method according to claim 1, characterized in that, The step of selecting representative key-value pairs from the key-value pairs corresponding to the multiple basic units included in each data block as the target key-value pairs for the corresponding data block includes: Among the key-value pairs corresponding to the multiple basic units included in each data block, the largest key-value pair and the smallest key-value pair are determined; The maximum key-value pair and the minimum key-value pair are determined as the target key-value pair.

3. The method according to claim 1, characterized in that, After selecting representative key-value pairs from the key-value pairs corresponding to the multiple basic units included in each data block as the target key-value pairs for the corresponding data block, the method further includes: The target key-value pairs are stored so that they can be used for reasoning in the next inference process.

4. The method according to any one of claims 1-3, characterized in that, The target input data includes at least one of text data, image data, audio data, and video data.

5. The method according to claim 4, characterized in that, If the target input data is text data, the target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain multiple data blocks, including: Determine the text length of the text data; If the text length is determined to be greater than the length threshold, the target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain the multiple data blocks.

6. The method according to claim 4, characterized in that, If the target input data is text data, the target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain multiple data blocks, including: Determine the dialogue turn corresponding to the text data, and determine the dialogue complexity corresponding to the text data; If the number of dialogue rounds is greater than the number of rounds threshold and the dialogue complexity is greater than the complexity threshold, then the target input data is divided into blocks according to the dimension of the basic unit of the target input data to obtain the multiple data blocks.

7. The method according to any one of claims 1-3, characterized in that, The large language model includes an auxiliary model and a main model, wherein the model parameters of the main model are greater than those of the auxiliary model. The auxiliary model is used for inference based on the attention score, and the main model is used for verification based on the output of the auxiliary model.

8. The method according to claim 7, characterized in that, The reasoning based on the attention score to obtain the reasoning result includes: The auxiliary model performs inference based on the attention score to obtain multiple prediction results, and sends the multiple prediction results to the main model; The probability corresponding to each of the multiple prediction results is determined by the main model. The inference result is determined based on the probability corresponding to each of the multiple prediction results.

9. The method according to any one of claims 1-3, characterized in that, The inference result is obtained based on a tree-based greedy prediction method. After inferring the result based on the attention score, the method further includes: Based on the reasoning results, various combinations of basic units are determined; The single-point matching results of each of the various basic unit combinations are determined, and the historical prediction basic units are corrected based on the single-point matching results.

10. The method according to any one of claims 1-3, characterized in that, The method further includes: Obtain a preprocessing queue and determine the request prefix and request length corresponding to each of the multiple input data included in the preprocessing queue, wherein the multiple input data includes at least the target input data; The preprocessing queue is grouped according to the request prefix and the request length to obtain at least one preprocessing group; If it is determined that data with the same request prefix exists in the cached data based on the request prefix corresponding to the at least one preprocessing group, then target data with the same request prefix is ​​obtained from the cached data, and incremental calculation is performed on each input data in the at least one preprocessing group based on the target data to obtain the inference result; or If it is determined that there is no data in the cached data that is the same as the request prefix based on the request prefix corresponding to the at least one preprocessing group, then any input data is selected from the at least one preprocessing group for processing to obtain reference data. The reference data is used as the target data of the at least one preprocessing group, and incremental calculation is performed on each input data in the at least one preprocessing group based on the target data to obtain the inference result.

11. The method according to claim 10, characterized in that, The method further includes: Determine the usage frequency of each of the multiple data items in the cached data; The storage location of each of the multiple data is determined according to the usage frequency, and the multiple data are stored based on the storage location, wherein the multiple data in the cache data are stored in a prefix tree manner.

12. The method according to any one of claims 1-3, characterized in that, The target input data is divided into blocks according to the dimensions of the basic units of the target input data to obtain multiple data blocks, including: Determine the real-time load rate of the large language model; The block size is determined based on the dimension of the basic unit of the target input data and the real-time load rate, and the target input data is divided into blocks according to the block size to obtain the multiple data blocks.

13. A device for approximate reasoning, characterized in that, The device includes: The target input data receiving module is used to receive target input data through a large language model; The block segmentation module is used to segment the target input data into blocks according to the dimension of the basic unit of the target input data to obtain multiple data blocks, wherein each of the multiple data blocks includes multiple basic units. The target key-value pair determination module is used to determine the key-value pairs corresponding to each of the multiple basic units included in each data block, and select representative key-value pairs from the key-value pairs corresponding to each of the multiple basic units included in each data block as the target key-value pairs of the corresponding data block. The inference module is used to determine the attention score corresponding to each of the multiple data blocks based on the target key-value pairs corresponding to each of the multiple data blocks, and to perform inference based on the attention scores to obtain the inference result.

14. An electronic device, characterized in that, The method includes a memory and a processor, the memory being coupled to the processor, the memory storing instructions, and the processor performing the method as described in any one of claims 1-12 when the instructions are executed by the processor.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium contains program code that can be invoked by a processor to execute the method as described in any one of claims 1-12.

Images