Multi-candidate response generation method and device, electronic equipment and storage medium

Abstract

This invention provides a method, apparatus, electronic device, and storage medium for generating multiple candidate responses, belonging to the field of artificial intelligence technology. The method includes: inputting a target prompt into a large language model; generating multiple candidate responses for the target prompt through multiple parallel generation threads; during the generation of multiple candidate responses word-by-word, obtaining hidden state features corresponding to each parallel generation thread using the network layer of the large language model; performing batch-dimensional attention calculations on each hidden state feature using a cross-thread shared layer to obtain shared state features; and using the output layer of the large language model to predict the output words of each parallel generation thread based on the shared state features. This invention breaks the independence between parallel generation threads by introducing a cross-thread shared layer between the network layer and the output layer, enabling interaction and fusion of feature dimensions among the threads during the generation process. This effectively utilizes the global information of the parallel group, improving the diversity and quality of generated candidate responses.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device, and storage medium for generating multiple candidate responses. Background Technology

[0002] In natural language generation tasks such as intelligent dialogue, text creation, and logical reasoning, in order to improve the quality and robustness of responses or obtain diverse results, it is often not necessary to rely on a single output. Instead, multiple different candidate responses need to be generated for the same input prompt, so that the optimal result can be selected through mechanisms such as reward model scoring, re-ranking, or majority voting to meet the user's demand for high-quality generated content.

[0003] To meet the above requirements, existing technologies typically utilize the parallel computing capabilities of graphics processing units (GPUs) and employ batching techniques for inference to improve generation efficiency. Specifically, this involves constructing multiple generation tasks for the same input prompt into a batch of data input into a large language model. During the generation of candidate responses word-by-word, the large language model uses a self-attention mechanism to handle long-distance dependencies within each sequence. That is, each parallel generation thread independently calculates its hidden state based solely on its own historical context information (Prompt and already generated words), and predicts the probability distribution of the next word based on this independent hidden state, thereby outputting multiple candidate responses in parallel within the same time step.

[0004] However, this existing parallel generation method has limitations in feature utilization. Although multiple generation threads are physically synchronous, strict data isolation is maintained between them in the model's feature extraction and computation logic. This mutual closure at the feature level prevents the model from integrating the overall information of the parallel population during the generation process. Consequently, the generation process of each candidate response lacks a global perspective based on the population state, limiting the richness and accuracy of the generated candidate responses in terms of feature representation. Summary of the Invention

[0005] This invention provides a method, apparatus, electronic device, and storage medium for generating multiple candidate responses, which addresses the shortcomings of existing technologies where parallel generation threads are isolated at the feature level, preventing the model from integrating global information from the parallel group and thus limiting the diversity and quality of generated candidate responses. This invention breaks down the information barriers between threads during the generation process, integrates global information from the parallel group through feature interaction, and effectively improves the diversity and overall quality of generated candidate responses.

[0006] This invention provides a method for generating multiple candidate responses, comprising: The target prompt is input into a large language model, and multiple candidate responses generated by the large language model through multiple parallel generation threads are obtained. In the process of generating the multiple candidate responses word-by-word meta-analysis from the large language model: The hidden state features corresponding to each of the parallel generation threads are obtained by utilizing the network layer of the large language model. By utilizing the cross-thread shared layer of the large language model, batch-dimensional attention calculations are performed on each of the hidden state features to obtain the shared state features; Using the output layer of the large language model, the output lexical units of each of the parallel generation threads are predicted based on the shared state features.

[0007] According to a multi-candidate response generation method provided by the present invention, the network layer of the large language model includes multiple stacked Transformer decoder blocks, and each candidate response is composed of multiple lexical units arranged in sequence; The step of using the network layer of the large language model to obtain the hidden state features corresponding to each of the parallel generation threads includes: Construct multiple input sequences, each of which corresponds to a parallel generation thread and includes the target prompt and the word sequence already generated by the parallel generation thread; All the input sequences are input as a batch into the stacked Transformer decoder blocks; From the output of the last Transformer decoder block of the network layer, extract the feature vector corresponding to the current generation position of each input sequence, and use it as the hidden state feature corresponding to each parallel generation thread.

[0008] According to a multi-candidate response generation method provided by the present invention, the cross-thread shared layer is disposed between the network layer and the output layer; The method utilizes the cross-thread shared layer of the large language model to perform batch-dimensional attention calculations on each hidden state feature to obtain shared state features, including: Stack the hidden state features corresponding to all the parallel generation threads in the batch dimension to construct a combined feature matrix; Using the cross-thread shared layer, self-attention calculation is performed on the combined feature matrix to obtain a fusion feature matrix containing cross-thread interaction information; The fused feature matrix is ​​split along the batch dimension to obtain the shared state features corresponding to each of the parallel generation threads.

[0009] According to a multi-candidate response generation method provided by the present invention, the method utilizes the cross-thread shared layer to perform self-attention calculation on the combined feature matrix to obtain a fusion feature matrix containing cross-thread interaction information, including: Using the query weight matrix, key weight matrix, and value weight matrix, linear mapping is performed on the combined feature matrix to obtain the query matrix, key matrix, and value matrix, respectively. The product of the query matrix and the transpose of the key matrix is ​​calculated to obtain the attention score matrix, which represents the correlation between the hidden state features of each of the parallel generation threads; The value matrix is ​​weighted and summed using the attention score matrix to generate the fusion feature matrix.

[0010] According to a multi-candidate response generation method provided by the present invention, the cross-thread shared layer is configured with an interaction mask matrix, the interaction mask matrix is ​​a fully visible mask matrix, the dimension of the fully visible mask matrix is ​​consistent with the dimension of the initial score matrix, and the element values ​​in the fully visible mask matrix corresponding to the positions between different parallel generation threads are reserved identifier values, the reserved identifier values ​​make the corresponding attention scores retained in subsequent calculations; The step of calculating the product between the query matrix and the transpose of the key matrix to obtain the attention score matrix includes: Calculate the product of the query matrix and the transpose of the key matrix to obtain the initial score matrix; The attention score matrix is ​​obtained by adding the fully visible mask matrix to the initial score matrix element by element.

[0011] According to a multi-candidate response generation method provided by the present invention, the cross-thread shared layer further includes a residual connection structure; The step of using the attention score matrix to perform a weighted summation of the value matrix to generate the fused feature matrix includes: The value matrix is ​​weighted and summed using the attention score matrix to obtain the intermediate interaction feature matrix; The intermediate interactive feature matrix and the combined feature matrix are added element by element to obtain the fused feature matrix.

[0012] According to a multi-candidate response generation method provided by the present invention, the step of stacking the hidden state features corresponding to all the parallel generation threads in the batch dimension to construct a combined feature matrix includes: Obtain the position encoding vectors corresponding to each of the parallel generation threads, and the position encoding vectors are used to identify the index positions of each of the parallel generation threads in the batch dimension; Each of the aforementioned location encoding vectors is fused with its corresponding hidden state feature to obtain location-aware features; The location-aware features corresponding to all the parallel generation threads are stacked in the batch dimension to construct the combined feature matrix.

[0013] According to a multi-candidate response generation method provided by the present invention, each of the parallel generation threads is configured with different sampling parameters, and the candidate responses generated by different parallel generation threads are different; The step of using the output layer of the large language model to predict the output lexical units of each of the parallel generation threads based on the shared state features includes: The output layer is used to map the shared state features corresponding to each of the parallel generation threads into a probability distribution on the vocabulary dimension; For any parallel generation thread, the probability distribution is adjusted using the configured sampling parameters to determine the output lexical based on the adjusted probability distribution.

[0014] According to the multi-candidate response generation method provided by the present invention, the sampling parameters include temperature parameters; The step of adjusting the probability distribution using the configured sampling parameters for any parallel generation thread, and determining the output lexical unit based on the adjusted probability distribution, includes: The probability distribution is smoothed or sharpened using the temperature parameter configured by any of the parallel generation threads to obtain the adjusted probability distribution; The output lexical units are sampled based on the adjusted probability distribution.

[0015] According to a multi-candidate response generation method provided by the present invention, in the process of processing the input sequence using the stacked Transformer decoder blocks, the method further includes: In each step of word-by-word generation, the key vector and value vector calculated by the Transformer decoder block for the generated word sequence of each parallel generation thread are stored in the key-value cache. In the step of generating the current new lexical unit, the key vector and value vector stored in the key-value cache are called, and the corresponding query vector, key vector and value vector are calculated only for the current new lexical unit; Based on the query vector, key vector, and value vector of the current new lexical, as well as the retrieved key vector and value vector, the self-attention calculation within the network layer is performed.

[0016] According to a multi-candidate response generation method provided by the present invention, the step of performing self-attention calculation within the network layer includes: Construct a causal mask matrix, wherein the element value in the region located in the upper right corner of the main diagonal is the masking identifier value, and the element value in the region located in the lower left corner of the main diagonal is the retention identifier value. After calculating the attention score inside the network layer, the causal mask matrix is ​​added element by element to the attention score inside the network layer to obtain the masked attention score. The masked attention scores are normalized to obtain the final attention weights used for weighted summation.

[0017] According to a multi-candidate response generation method provided by the present invention, the key-value cache is configured in a shared mode; The step of calling the key vector and value vector stored in the key-value cache includes: Identify the common portions of each input sequence that belong to the target prompt; For the common part, each of the parallel generation threads shares the same key-value cache data.

[0018] According to a multi-candidate response generation method provided by the present invention, after generating multiple candidate responses for the target cue, the method further includes: A reward model is used to evaluate the quality of each candidate response, and a quality score is obtained for each candidate response. Based on the quality score, a target response is selected from all the candidate responses, or a weighted combination of all the candidate responses is performed to generate a target response.

[0019] According to a multi-candidate response generation method provided by the present invention, the cross-thread shared layer is inserted as an adapter module into the initial large language model; The adapter module includes trainable parameters. During the training process of the initial large language model to obtain the large language model, the parameters of the network layers of the initial large language model are frozen, and the trainable parameters are updated based on the cross-thread interactive loss function.

[0020] The present invention also provides a multi-candidate response generation device, which mainly includes a response generation unit. The response generation unit is used to input the target prompt into a large language model and obtain multiple candidate responses generated by the large language model through multiple parallel generation threads. In the process of generating the multiple candidate responses word-by-word from the large language model: The hidden state features corresponding to each of the parallel generation threads are obtained by utilizing the network layer of the large language model. By utilizing the cross-thread shared layer of the large language model, batch-dimensional attention calculations are performed on each of the hidden state features to obtain the shared state features; Using the output layer of the large language model, the output lexical units of each of the parallel generation threads are predicted based on the shared state features.

[0021] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the multi-candidate response generation method as described above.

[0022] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-candidate response generation method as described above.

[0023] The multi-candidate response generation method, apparatus, electronic device, and storage medium provided by this invention break the independence between parallel generation threads by introducing a cross-thread shared layer between the network layer and the output layer. This enables each thread to interact and fuse feature dimensions during the generation process, thereby effectively utilizing the global information of the parallel group and improving the diversity and quality of generated candidate responses. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the multi-candidate response generation method provided by the present invention.

[0026] Figure 2 This is a schematic diagram of the structure of the large language model provided by the present invention.

[0027] Figure 3 This is a schematic diagram of the process for obtaining hidden state features in a network layer, as provided by the present invention.

[0028] Figure 4 This is a schematic diagram of the process provided by the present invention, in which a cross-thread shared layer performs attention calculation on hidden state features to obtain shared state features.

[0029] Figure 5 This is a schematic diagram of the process for obtaining the fusion feature matrix using self-attention calculation provided by the present invention.

[0030] Figure 6This is a schematic diagram of the process provided by the present invention for stacking the hidden state features of parallel generation threads in the batch dimension to generate a combined feature matrix.

[0031] Figure 7 This is a schematic diagram illustrating the operating principle of the key-value cache provided by the present invention.

[0032] Figure 8 This is a schematic diagram of the self-attention calculation process within the network layer provided by the present invention.

[0033] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0035] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Those skilled in the art will understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0036] It should be noted that all actions involving the acquisition of medical and patient information or data in this invention are carried out in compliance with the relevant data protection laws and policies of the country where the invention is located, and with the authorization granted by the owner of the relevant device.

[0037] In the medical field, doctor-patient question-and-answer systems are crucial intelligent platforms connecting patients with medical resources. Widely used in scenarios such as disease consultation, medication guidance, preliminary symptom assessment, and post-operative rehabilitation advice, they break down the time and space limitations of medical services, providing patients with immediate and accessible health consultations. Simultaneously, they help medical institutions divert non-emergency consultation needs, alleviate pressure on outpatient services, and optimize the allocation efficiency of medical service resources. With the deep penetration of big data modeling technology in the medical field, traditional doctor-patient question-and-answer systems are gradually upgrading from rule-matching responses to generative intelligent interaction. These systems can generate more logical answers based on patients' described symptoms, medical history, and other information, playing a significant role in improving the patient's medical experience and reducing the cost of obtaining health information.

[0038] Because doctor-patient question-and-answer systems need to adapt to individual patient differences, the safety of medical advice, and the diversity of communication scenarios, single responses are prone to problems such as insufficient adaptation and one-sided information. In contrast, multi-candidate response generation can output answers in a multi-dimensional and differentiated manner, taking into account both personalized needs and the comprehensiveness and reliability of medical advice, making it a more suitable generation method for medical scenarios.

[0039] However, current doctor-patient question-and-answer methods and related systems still face many challenges in generating multiple candidate responses in real-world medical scenarios.

[0040] First, the responses output by the doctor-patient question-and-answer system are generated independently, making it impossible to share thinking and reasoning information, which can easily lead to contradictory content (such as inconsistent dosage recommendations for the same drug), increasing the risk of misunderstanding for patients.

[0041] Second, existing doctor-patient question-and-answer systems have difficulty reusing correct reasoning fragments from different responses when generating multiple candidate responses. If a single response has an oversight, other responses cannot fill in the gaps, resulting in insufficient comprehensiveness of the answer.

[0042] Third, existing doctor-patient question-and-answer systems are unable to fully integrate and dynamically adjust to patients' personalized information (such as allergy history and underlying diseases), and the multi-response system is still too general and cannot match differentiated medical needs.

[0043] In terms of analysis, existing multi-candidate response generation relies on the generation capabilities of Large Language Models (LLMs). Based on patient input such as symptoms and medical history, multiple independent responses are generated through autoregression. During multi-candidate response generation, each response calls the LLM for parallel computation, with no information exchange during the generation process; it only relies on the LLM's single inference output to produce different results. Specifically, in existing technical solutions, each generation thread's token sampling at time step s depends solely on its own history. That is, for B independent parallel generation threads targeting the same prompt, when the model generates the token for the next time step s+1, the sampling probability is entirely determined by the current thread's own information. ; Where b represents the b-th parallel generation thread, ranging from 1 to B; s represents the current time step, and s+1 is the time step of the next token to be generated; q represents the input prompt, which is the common input of all threads based on the same prompt word; This represents all the tokens generated by the b-th parallel generation thread between time step 1 and s, i.e., the historical generation trajectory of this parallel generation thread.

[0044] Existing methods, when deriving to step s+1, only refer to the input prompts and the derivation process of the previous s steps in the current thread, without considering the derivation content and thought process of the other B-1 threads. Then, after all parallel responses are generated, existing methods perform post-processing on the multiple responses (such as keyword filtering and simple voting) to filter or integrate the responses to obtain the final output response. Although existing methods utilize the generation results of multiple responses through subsequent integration, because the response generation processes are still independent, they currently fail to effectively reuse effective information from the inference process, thus leaving room for improvement in accuracy.

[0045] Therefore, traditional LLM-based multi-candidate response generation methods have limitations in processing the hidden state features (i.e., feature vectors of the input passed through the neural network) of different batches of responses. For a three-dimensional vector of shape B×S×D (B is the number of parallel generation threads, S is the sequence length, and D is the dimension of the hidden state features), it is only regarded as B independent stacks of two-dimensional vectors of shape S×D. Therefore, the way to generate the output from the hidden state inference is to perform independent inference on B individual two-dimensional vectors. This processing method directly leads to the problem of isolated parallel generation threads, wasting the potentially useful information between threads and ignoring the global correlation features that exist between different threads.

[0046] To address this, the present invention provides a novel multi-candidate response generation method that focuses on fully utilizing the additional information between multiple responses, breaking the inherent isolation problem between multiple threads in traditional methods, and further improving the performance of LLM in medical doctor-patient dialogue systems.

[0047] Compared to traditional methods, the multi-candidate response method proposed in this invention activates the ability of LLM to view the hidden state features of different batches as a holistic three-dimensional structure: the model no longer isolates the features of each generation thread, but treats the hidden state of the entire batch as a complete three-dimensional vector, allowing the model to transfer information along the batch axis (i.e., the B-axis). Specifically, at the same time step, the features corresponding to tokens from different threads can reference each other. In this way, the decoding process of LLM is equivalent to autoregressively generating B×D dimensional features for each time step s, thus enabling cross-thread feature fusion by utilizing information from different generation threads when generating each token. Therefore, compared to traditional solutions, the proposed solution only requires lightweight structural modifications, effectively improving the accuracy and rationality of doctors' responses with only a small number of additional parameters.

[0048] The multi-candidate response generation method of the present invention can be implemented by electronic devices, such as servers, computer clusters, or terminal devices configured with graphics processing units (GPUs) or dedicated artificial intelligence (AI) chips. In the following embodiments, a medical AI server deployed in the cloud will be used as the execution subject for description.

[0049] This embodiment provides a method for generating multiple candidate responses, which is particularly suitable for scenarios that require multiple interpretation angles or expression styles for professional issues (such as medical consultations) to help professionals (such as doctors) better communicate with service recipients (such as patients).

[0050] Specifically, the provided method for generating multiple candidate responses mainly includes inputting target prompts into a large language model and obtaining multiple candidate responses generated by the large language model through multiple parallel generation threads.

[0051] The target prompt is the initial input data that initiates the task of generating multiple candidate responses. It can take the form of a natural language command entered by a doctor or patient through the human-computer interaction interface of a medical information system, or a text string automatically invoked by other medical service modules. For example, in a doctor-patient communication scenario, the target prompt could be: "Please explain the causes and daily management suggestions for 'type 2 diabetes' in a simple and easy-to-understand way."

[0052] Large language models can employ a deep learning model with natural language understanding and generation capabilities, pre-trained on massive text datasets, such as corpora containing general and medical domains. This could be a generative pre-trained model based on the Transformer architecture. The large language model is configured to support batch inference, enabling it to process multiple input sequences simultaneously.

[0053] Multiple parallel generation threads refer to multiple logical computation paths created within the same computation batch during model inference to generate different candidate responses. For example, if a doctor wants to obtain three different styles of interpretation, three parallel generation threads can be set up, each responsible for generating one candidate response. These parallel generation threads physically utilize the parallel computing cores of GPUs or AI chips to execute simultaneously.

[0054] Multiple candidate responses refer to multiple different text sequences for the same target prompt. For example, for the prompt about "type 2 diabetes" mentioned above, three different candidate responses might be obtained: Candidate response A focuses on reassuring metaphors, such as: "You can think of your body as a city, and insulin as the trucks that transport energy (blood sugar). Right now, the factories (cells) in the city are a bit 'ignoring' the trucks, causing energy to get stuck on the road. Don't worry, this is common, and we can get the factories 'back up' by adjusting our diet and exercising more."

[0055] Candidate response B focuses on action points, such as: "Simply put, your body is using blood sugar less efficiently. The key to managing it is three things: First, pay attention to your diet, eat more vegetables and less sweets; second, stick to a daily walk or do some exercise; third, monitor your blood sugar regularly. We will work together to develop an easy and feasible plan."

[0056] Candidate response C focuses on simplifying the mechanism, such as: "This is mainly due to 'insulin resistance,' which means that although your body secretes enough insulin, its effect is reduced. This is related to genetics and lifestyle habits, and in most cases, it can be well controlled through lifestyle interventions."

[0057] After receiving the target prompt from the input terminal, the medical AI server invokes the deployed large language model to execute the service. During the invocation, the number of candidate responses to be generated (i.e., the number of parallel generation threads N, where N≥2) can be specified. Upon receiving the target prompt, the large language model creates initial states for each of the N parallel generation threads, each initial state containing the target prompt, and then begins the word-by-word meta-generation process.

[0058] Figure 1 This is a flowchart illustrating the multi-candidate response generation method provided by the present invention, as shown below. Figure 1 As shown, in the process of obtaining the word-by-word meta-response generated by the large language model, this embodiment specifically performs the following operations: Step 11: Using the network layer of the large language model, obtain the hidden state features corresponding to each of the parallel generation threads.

[0059] Step 12: Utilize the cross-thread shared layer of the large language model to perform batch-dimensional attention calculations on each of the hidden state features to obtain shared state features.

[0060] Step 13: Using the output layer of the large language model, predict the output lexical units of each of the parallel generation threads based on the shared state features.

[0061] Token generation refers to the autoregressive generation mechanism used by large language models. At each generation time step, the next token is predicted based on the existing text sequence of all parallel generation threads (initially the target prompt, and subsequently the target prompt plus the generated tokens). Here, a token refers to the basic unit of text processed by the model, which can be a word, a Chinese character, or a subword.

[0062] Network layers are the main structure of large language models. They are usually composed of multiple stacked deep neural network modules (such as Transformer decoder blocks) and are responsible for deep encoding and feature extraction of the input sequence.

[0063] Hidden state features refer to the high-dimensional feature vectors output at specific positions after the input sequence has been processed by network layers. These vectors are typically the last position of the input sequence, representing the next word to be predicted. This high-dimensional feature vector can be understood as the culmination of the model's deep semantic understanding and consideration of the current input context.

[0064] In this embodiment, at any generation time step, for N parallel generation threads, each thread contains its own current sequence. For example, the input sequence of thread A might be "...city, insulin is delivery". The network layer of the large language model processes these N input sequences in batches in parallel, thereby outputting a corresponding hidden state feature for each parallel generation thread. These N hidden state features are independent of each other at this moment, each representing the semantic state of the N parallel generation threads.

[0065] The cross-thread shared layer of the large language model is a key component introduced in this invention. It can be regarded as a special neural network layer. Its design purpose is to break down the information barriers between parallel generation threads and promote feature interaction.

[0066] Performing batch-dimensional attention computation on each hidden state feature is the core operation performed across the thread-shared layer. Unlike the self-attention computation performed only in the sequence dimension within the network layer, it extends the computation scope to the batch dimension.

[0067] The specific principle is as follows: the N hidden state features obtained by the N parallel generation threads in sub-step one are aggregated and input into the attention mechanism as a new sequence. This allows the hidden state features of each parallel generation thread to perform attention score calculation and information weighting with the hidden state features of all other parallel generation threads. In this way, the features of thread A, which is generating a "metaphorical reassurance" style response, can be borrowed from the hidden state features of thread B, which is generating a "action point" style response.

[0068] Shared state features refer to the new feature vectors obtained after the attention interaction computation described above. For each parallel generation thread, the output shared state features not only contain its own original semantic information but also incorporate contextual information from other parallel generation threads. Therefore, compared to the original hidden state features, shared state features have a broader global perspective.

[0069] The output layer of a large language model can be a linear mapping layer, which is responsible for mapping high-dimensional feature vectors to the model's vocabulary space, and usually combines functions such as Softmax to calculate the probability of each word in the vocabulary being generated at the current time step.

[0070] The output lexical units of each parallel generation thread predicted based on the shared state features are the lexical units that the output layer of the large language model ultimately determines for each parallel generation thread at this time step.

[0071] In this embodiment, N shared state features corresponding to N parallel generation threads are respectively fed into the output layer. Based on these shared state features that incorporate global information, the output layer calculates a probability distribution for each parallel generation thread and determines its respective output tokens according to the probability distribution (e.g., by sampling or selecting the maximum probability). These output tokens are then appended to the end of the sequence of the corresponding thread as part of the input for the next time step generation process. This process is repeated until all parallel generation threads have generated complete candidate responses, such as encountering a terminator or reaching a preset length.

[0072] In summary, the multi-candidate response generation method provided by this invention introduces a cross-thread shared layer between the network layer and the output layer, breaking the independence between parallel generation threads. This allows each thread to interact and fuse feature dimensions during the generation process, thereby effectively utilizing the global information of the parallel group and improving the diversity and quality of generated candidate responses.

[0073] Figure 2 This is a schematic diagram of the structure of the large language model provided by the present invention, such as... Figure 2 As shown, in a preferred embodiment of the present invention, the specific structure of the network layer of the large language model, the relationship between candidate responses and lexical units, and the specific process of obtaining hidden state features are described in detail.

[0074] The large language model used in this embodiment is an innovative extension based on the standard Transformer architecture. Its overall data flow can be summarized as follows: after the input data is processed by multiple core Transformer blocks, its output features will enter a new part, namely the cross-thread shared layer, to complete feature interaction before subsequent processing.

[0075] Specifically, the architecture of the large language model includes: (1) Input module, responsible for receiving a three-dimensional tensor as input features. The three dimensions of this tensor are: Batch dimension, Sequence / Step dimension, and Dimension dimension. Among them, the Batch dimension corresponds to the number of parallel generation threads in the parallel generation task. For example, if N candidate responses are generated simultaneously, the size of the Batch dimension is N; the Sequence / Step dimension corresponds to the length of the input sequence of each parallel generation thread; the Dimension dimension corresponds to the dimension of the feature vector represented by each word.

[0076] (2) The network layer, composed of multiple stacked Transformer decoder blocks, is the core feature extraction part of the large language model. For example... Figure 2 As shown, each Transformer decoder block mainly contains two sub-layers: an attention layer and a feedforward layer.

[0077] The attention layer is used to first perform normalization (such as Layer Normalization) on the input features, and then perform attention calculation (such as multi-head self-attention). Its output is then residually connected to the input of the layer. Figure 2 (As shown by the ⊕ symbol).

[0078] The feedforward layer is mainly used to receive the output from the attention layer. It also performs normalization once, and then passes it through a feedforward neural network. Its output is then residually connected to the input of the same layer.

[0079] Optionally, the network layer can be composed of multiple such Transformer decoder blocks stacked sequentially.

[0080] (3) Figure 2 The newly added cross-thread shared layer shown is located at the end of the network layer and is used to realize the interaction of core features. It also contains structures such as normalization, multi-threaded attention calculation and residual connection.

[0081] Meanwhile, in order to clarify the basic structure of the generated content, this embodiment further clarifies that each finally generated candidate response is actually a text sequence composed of multiple words arranged in order.

[0082] Based on the architecture of the large language model described above, this invention provides a specific implementation method for obtaining the hidden state features corresponding to each of the parallel generation threads by utilizing the network layer of the large language model.

[0083] Figure 3This is a schematic diagram of the process for obtaining hidden state features in a network layer according to the present invention, as shown below. Figure 3 As shown, the main implementation steps include, but are not limited to, the following: Step 31: Construct multiple input sequences, where each input sequence corresponds to a parallel generation thread and includes the target prompt and the word sequence already generated by the parallel generation thread.

[0084] Step 32: Input all the input sequences as a batch into the stacked Transformer decoder blocks.

[0085] Step 33: Extract the feature vector corresponding to the current generation position of each input sequence from the output of the last Transformer decoder block of the network layer, and use it as the hidden state feature corresponding to each parallel generation thread.

[0086] In this embodiment, at any generation time step, a dedicated input sequence is prepared for each parallel generation thread. This input sequence is constructed by concatenating the target prompt shared by all parallel generation threads with the lexical sequence already generated by that parallel generation thread. Taking the doctor-patient communication scenario regarding "type 2 diabetes" in the previous embodiment as an example, at a certain intermediate generation step, the input sequence constructed for the three parallel generation threads, after lexicalization and embedding, will form an input tensor with dimensions [3, Sequence Length, Dimension], corresponding to... Figure 2 The input content in the input module.

[0087] Subsequently, the constructed input sequence is fed into the network layer of the large language model. The input sequence flows sequentially through multiple stacked Transformer decoder blocks in the network layer. The attention layer and feedforward layer in each Transformer decoder block perform deep feature extraction and encoding on all input sequences in the batch in parallel but independently.

[0088] The extraction operation of extracting the feature vector corresponding to each input sequence at the current generation position occurs after the data leaves the last Transformer decoder block and before entering the newly added cross-thread shared layer.

[0089] After all Transformer decoder blocks have been computed for the batch data, an output tensor with the same dimension as the input tensor is obtained. For this output tensor, this embodiment extracts the feature vector at the last position of the sequence (i.e., the current generation position) for each parallel generation thread along the batch dimension. The N extracted feature vectors are precisely defined as the hidden state features of each parallel generation thread at the current time step. At this point, these hidden state features each encapsulate the complete contextual information of their respective parallel generation thread.

[0090] This invention provides an implementation method for constructing input data for each parallel generation thread, performing efficient batch processing, and accurately extracting hidden state features from the output of the last Transformer decoder block as the basis for subsequent interactions during the parallel generation process. This ensures that the acquired hidden state features can accurately reflect the independent generation state of each parallel generation thread, providing an accurate data foundation for subsequent cross-thread features.

[0091] Based on the above embodiments, as an optional embodiment, the specific configuration of the cross-thread shared layer and the core process of performing batch-level attention calculations to obtain shared state features are described in detail.

[0092] The physical location of the cross-thread shared layer within the architecture of the large language model is explicitly set between the network layer and the output layer. This setting ensures that cross-thread feature interactions occur after each parallel generation thread has completed deep semantic encoding (through all Transformer decoder blocks) and before the final prediction of the next lexical unit (entering the output layer). This results in the highest quality and most concise information of the hidden state features used in the interaction, while also ensuring that the impact of the interaction can directly and effectively affect the final lexical unit prediction, avoiding information attenuation or distortion during transmission through subsequent network layers.

[0093] Figure 4 This is a schematic diagram of the process provided by the present invention, in which a cross-thread shared layer performs attention calculations on hidden state features to obtain shared state features, as shown below. Figure 4 As shown, the step of utilizing the cross-thread shared layer of the large language model to perform batch-dimensional attention calculations on each of the hidden state features to obtain shared state features may include, but is not limited to: Step 41: Stack the hidden state features corresponding to all the parallel generation threads in the batch dimension to construct a combined feature matrix.

[0094] Step 42: Using the cross-thread shared layer, perform self-attention calculation on the combined feature matrix to obtain a fused feature matrix containing cross-thread interaction information.

[0095] Step 43: Split the fused feature matrix in the batch dimension to obtain the shared state features corresponding to each of the parallel generation threads.

[0096] Building upon the detailed implementation of N parallel generation threads acquiring N independent hidden state features (each hidden state feature being a one-dimensional vector) in the preceding embodiments, in this embodiment, these N hidden state features are stacked or concatenated along the batch dimension to form a two-dimensional combined feature matrix of size [N, Dimension]. This operation is equivalent to aggregating the feature vectors, originally scattered along the batch dimension and representing individual thought processes, into a new feature set that can be processed collectively.

[0097] As a key innovation of this invention, unlike traditional self-attention calculations performed on the sequence dimension within network layers, the self-attention calculation in this embodiment operates on a combined feature matrix composed of N hidden state features. During the calculation, each hidden state feature simultaneously acts as a query, key, and value (or its mapping), and performs attention scores with all other hidden state features. This means that for the hidden state features of parallel generation thread A generating a "metaphorical reassurance" style response, its correlation with the hidden state features of parallel generation thread B generating a "action point" style response and parallel generation thread C generating a "mechanism simplification" style response is calculated, and feature information from parallel generation threads B and C is selectively absorbed based on this correlation. Finally, by weighted summing of all value vectors, the original hidden state feature at each position is transformed into a new feature vector containing cross-thread interaction information, and all these new feature vectors together constitute the fused feature matrix.

[0098] This invention establishes a dynamic roundtable mechanism within an abstract semantic feature space. In a medical question-and-answer scenario, if the "metaphor" explored by one parallel generation thread (e.g., parallel generation thread A) is too obscure, while the "action points" of another parallel generation thread (e.g., parallel generation thread B) are very clear and explicit, then during self-attention computation, the features of parallel generation thread A may receive more attention and be fused with the hidden state features of parallel generation thread B because of their higher semantic relevance. This fusion is not a simple averaging or splicing, but rather a selective, weighted information borrowing through an attention mechanism. This allows potentially mediocre or erroneous generation paths to be dynamically inspired by other, better paths, thereby improving the overall lower bound and average quality of the candidate response set.

[0099] Furthermore, in this embodiment, the fused feature matrix of size [N, Dimension] obtained in the previous step is split along the batch dimension to obtain N independent one-dimensional feature vectors. At this time, each of these N one-dimensional feature vectors is no longer the initial hidden state feature, but a shared state feature that has absorbed the advantages of other parallel generation threads. For example, each shared state feature contains a comprehensive understanding of the core issue of "how to better explain type 2 diabetes" from multiple perspectives.

[0100] This invention, by clearly defining the architectural position of the cross-thread shared layer and elaborating on its core process of "stacking-self-attention computation-splitting," achieves a highly efficient and profound cross-thread feature interaction mechanism. This goes beyond simple feature mixing; by dynamically adjusting the mutual influence between hidden state features of different parallel generation threads through attention weights, it not only effectively generates semantically differentiated but also consistently high-quality candidate responses. Furthermore, when facing complex and open-ended problems (such as popular explanations in professional fields), it significantly enhances the robustness and creativity of the model generation solution through the emergence of collective intelligence.

[0101] Based on the above embodiments, in an optional embodiment of the present invention, the self-attention calculation process inside the cross-thread shared layer is explained in more detail, clarifying the mathematical operation details of generating the fusion feature matrix, and is compared with conventional attention layer calculation to highlight its innovation.

[0102] Figure 5 This is a schematic diagram of the process for obtaining the fused feature matrix using self-attention calculation provided by the present invention, as shown below. Figure 5 As shown, it mainly includes, but is not limited to: Step 51: Using the query weight matrix, key weight matrix, and value weight matrix, perform linear mapping on the combined feature matrix to obtain the query matrix, key matrix, and value matrix, respectively.

[0103] Step 52: Calculate the product between the query matrix and the transpose of the key matrix to obtain the attention score matrix, which represents the correlation between the hidden state features of each of the parallel generation threads.

[0104] Step 53: Use the attention score matrix to perform a weighted summation on the value matrix to generate the fusion feature matrix.

[0105] To accurately describe the operations on data in different dimensions, this embodiment first defines a three-dimensional data tensor. XThe dimensions are batch dimension B, sequence dimension S, and feature dimension D. Here, B represents Batch, corresponding to the number of parallel generation threads; S represents Sequence / Step, corresponding to the sequence length; and D represents Dimension, corresponding to the hidden state feature dimension.

[0106] by Tensor X A slice on batch dimension B, namely This represents the S×D two-dimensional data matrix corresponding to the b-th parallel generation thread in the three-dimensional tensor; Tensor X A slice along the sequence dimension S, i.e. This represents the two-dimensional data matrix of dimension B×D corresponding to the s-th time step in the three-dimensional tensor. This matrix is ​​the combined feature matrix.

[0107] The computation of a conventional attention layer (such as a self-attention module within a network layer) operates on the sequence dimension S. Its operation can be represented as performing attention calculations on data of dimension S×D in each parallel generation thread b, with the mathematical formula as follows: ; This operation ensures that information flows only within the sequence of a single parallel generation thread.

[0108] In contrast, the core innovation of the cross-thread shared layer in this embodiment lies in applying attention computation to the batch dimension B. Specifically, it performs attention computation on the combined feature matrix at each time step s. The mathematical formula for calculating attention can be precisely expressed as: ; The specific steps for implementing this formula are broken down as follows: First, the combined feature matrix is ​​linearly mapped using the query weight matrix, key weight matrix, and value weight matrix to obtain the query matrix, key matrix, and value matrix.

[0109] Combined feature matrix (denoted as) X Its dimensions are [N, D] model ], where N is the number of parallel generation threads, D model The dimension of the hidden state features. Query the weight matrix (denoted as...). W Q ), key weight matrix (denoted as W K ) and value weight matrix (denoted as W V ) is a trainable parameter matrix contained in the cross-thread shared layer.

[0110] The process of performing a linear mapping can be achieved using matrix multiplication, with the specific mathematical relationships as follows: Query Matrix Q = X W Q ; Key matrix K = X W K ; Value matrix V = X W V .

[0111] Through independent linear transformations, three new matrices with distinct functions are derived from the original combined feature matrix. (Query matrix) Q This can be understood as a query signal emitted by each parallel generation thread to find information about other related threads; the key matrix. K Represents the queried identifier information for each parallel generation thread; value matrix V This represents the actual content information to be weighted and merged carried by each parallel generation thread.

[0112] Further, the product of the query matrix and the transpose of the key matrix is ​​calculated to obtain the attention score matrix. This step corresponds to the formula above. Partially. The calculation result is an attention score matrix with dimensions [N, N], where N is the number of parallel generation threads, and each element represents the feature correlation between different parallel generation threads.

[0113] Finally, the value matrix is ​​weighted and summed using the attention score matrix to generate the fusion feature matrix.

[0114] This step corresponds to applying the softmax function and scaling factor. ,in The dimension of the key vector, the processed attention score matrix and value matrix Multiply. The final result. This is the fusion feature matrix.

[0115] Existing multi-candidate response generation for doctor answers relies on the generation capabilities of large models. Based on patient input such as symptoms and medical history, multiple independent responses are generated through autoregression. During multi-candidate response generation, each response calls the model for parallel computation, with no information exchange during the generation process; the different results are only output by the model in a single inference. Specifically, in existing technical solutions, each thread's token sampling at time step s depends solely on its own history. That is, for B parallel and independent response generation threads targeting the same prompt, the sampling probability when the model generates the token for the next time step s+1 is entirely determined by the current thread's own information. ; in, This represents all tokens generated by the b-th thread between time step 1 and s, i.e., the historical generation trajectory of any generating thread.

[0116] Unlike existing technologies, the embodiments of this invention, through the attention calculation performed at the batch dimension, fundamentally change the sampling probability p of the large language model when generating the lexical unit at the next time step s+1. For any parallel generation thread b, its next lexical unit... The generation probability no longer depends solely on its own history, but rather on the complete historical information of all parallel generation threads (b' from 1 to B). Its conditional probability can be expressed as: ; in, `b` is the input prompt; `b'` is an iteration variable used to iterate through all parallel generation threads. This formula clearly shows that the next decision of any parallel generation thread is determined by the collective information of all parallel generation threads.

[0117] The embodiments of this invention rigorously define the essential difference between the cross-thread shared layer and the conventional attention layer as the transformation of the computational dimension (from the sequence dimension S to the batch dimension B). This transformation mathematically ensures that information can be effectively interacted and fused between different parallel generation threads. Its effect is directly reflected in the final conditional probability formula, proving that this invention effectively realizes collaborative generation based on parallel group information.

[0118] Based on the above embodiments, as an optional embodiment of the present invention, further explanation of the masking mechanism is provided for the self-attention calculation process of the cross-thread shared layer. This aims to structurally illustrate that the information channel between parallel generation threads is completely open, which is fundamentally different from the masking mechanism used to restrict information flow in conventional attention layers.

[0119] In this embodiment, the step of calculating the product between the query matrix and the transpose of the key matrix to obtain the attention score matrix may include, but is not limited to: calculating the product between the query matrix and the transpose of the key matrix to obtain an initial score matrix; and adding the fully visible mask matrix to the initial score matrix element by element to obtain the attention score matrix.

[0120] This implementation uses matrix multiplication. An initial score matrix of dimension [N, N] is obtained, where N is the number of parallel generation threads. Each element of this initial score matrix is ​​unprocessed and directly reflects the raw correlation score between the hidden state features of any two parallel generation threads.

[0121] Subsequently, a fully visible mask matrix is ​​introduced to explicitly define the scope of interaction for attention computation in a cross-thread shared layer.

[0122] Structurally, the dimension of the fully visible mask matrix is ​​exactly the same as the dimension of the initial fraction matrix, which is also [N, N]; in terms of content, the element values ​​of the fully visible mask matrix are set to reserved identifier values.

[0123] In typical neural network implementations, the reserved flag value is usually 0. A mask with a value of 0, when added element-wise to the initial score, will not change the value of the initial score. Therefore, the purpose of the reserved flag value is to ensure that the corresponding attention score is completely and undiminished in subsequent normalization calculations (such as softmax).

[0124] Operationally, this fully visible mask matrix is ​​added element-wise to the initial score matrix obtained in the previous step, and the result is the final attention score matrix. Since all elements of the fully visible mask matrix retain the label values, the addition operation is numerically equivalent to not modifying the initial score matrix at all.

[0125] In the Transformer decoder block of this embodiment, a causal mask matrix is ​​typically used. The elements of the upper triangular region (representing future positions) of this causal mask matrix are extremely large negative numbers (such as negative infinity). By adding this to the initial score matrix, the attention weights of future positions can be effectively reduced to zero after softmax, thereby preventing information from flowing from the future to the past. In stark contrast to the conventional masking mechanisms used to block information flow, the fully visible mask matrix employed in this embodiment preserves all identifier values. This design explicitly guarantees computationally that the attention scores of any parallel generation thread (any row of the attention score matrix) on all other parallel generation threads (all columns of that row) will not be suppressed. It fundamentally eliminates the computational barriers between parallel generation threads, providing a structural prerequisite for feature weighted fusion based on attention scores. In other words, by applying such a mask that does not perform any masking, this embodiment enforces fully-connected interaction between parallel generation threads at the architectural level, enabling large language models to learn and utilize the collective information of all parallel threads.

[0126] In summary, the embodiments of the present invention, by defining and applying a fully visible mask matrix, provide an unrestricted range of interaction for attention computation in a cross-thread shared layer. This is not only fundamentally different from conventional masking mechanisms used to isolate information in terms of technical implementation, but more importantly, it provides the necessary and sufficient structural guarantee for the effective fusion of features between parallel generation threads.

[0127] In large-scale deployments, to maximize the utilization of computing resources such as GPUs, a single computation batch may contain parallel generation threads serving multiple different target prompts (e.g., processing medical advice for user A and legal advice for user B simultaneously). In this case, a fully visible mask may no longer be suitable because it can lead to erroneous feature interactions between threads belonging to different target prompts.

[0128] To address this issue, this implementation introduces a target-cue-based grouped interaction mask to ensure that feature interactions occur only within a subset of parallel generation threads that share the same target cues.

[0129] Specifically, when calculating cross-thread attention at each time step s, the initial score matrix is ​​calculated, i.e. An additional mask matrix M will then be applied. The elements of this mask matrix M... It is determined by the following rules: ; Where b and b' are the indices of the parallel generation thread in the batch; and These are the original target hints corresponding to the b-th and b'-th parallel generation threads; A value of 0 indicates that attention scores are retained, allowing the attention scores between the two parallel generation threads to take effect; when... When the value is -∞, it represents occlusion. By adding it to the initial score, the attention score between the two parallel generation threads will approach zero after the subsequent softmax calculation, thus effectively preventing information interaction between them.

[0130] After adding this mask matrix M to the initial score matrix, the final attention calculation formula for the cross-thread shared layer is precisely corrected as follows: .

[0131] This invention, by introducing and applying this target-based grouping interaction mask, achieves a balance between logical correctness and computational efficiency in complex mixed batch processing scenarios.

[0132] On the one hand, by strictly isolating thread groups with different target prompts, it is ensured that the feature fusion of the model always occurs within a semantically relevant context, avoiding information pollution and model performance degradation caused by processing irrelevant tasks.

[0133] On the other hand, instead of creating a separate small batch for each target cue, a large number of generation tasks from different sources can be aggregated into a large batch for processing, making full use of the hardware's parallel computing capabilities.

[0134] Based on the above embodiments, as an optional embodiment of the present invention, the internal architecture of the cross-thread shared layer is further optimized by introducing a residual connection structure.

[0135] You can refer to this. Figure 2 As shown, after the attention layer and feedforward layer of the Transformer block, there is an addition operation represented by the symbol "⊕", which is the embodiment of residual connection. This embodiment also applies this structure to the newly added cross-thread shared layer. After introducing this structure, the value matrix is ​​weighted and summed using the attention score matrix to generate the fused feature matrix, which may include, but is not limited to, the following implementation steps: The value matrix is ​​weighted and summed using the attention score matrix to obtain an intermediate interaction feature matrix. The normalized attention score matrix is ​​then compared with the value matrix V. sMatrix multiplication is performed to obtain the intermediate interaction feature matrix. This intermediate interaction feature matrix represents the incremental or change information generated by the hidden state features of each parallel generation thread after absorbing information from all other parallel generation threads, encapsulating all the new knowledge learned through cross-thread self-attention computation.

[0136] The intermediate interaction feature matrix and the combined feature matrix are added element-wise to obtain the fused feature matrix. In this embodiment, the intermediate interaction feature matrix obtained in the previous step is added element-wise to the original input before entering the cross-thread shared layer, i.e., the combined feature matrix (composed of stacked independent hidden state features from all parallel generation threads). Since both the intermediate interaction feature matrix and the combined feature matrix have dimensions [N, D], the fusion feature matrix is ​​obtained through this addition. model (N is the number of parallel generation threads, D) model (where is the feature dimension), this element-wise addition operation is mathematically feasible. The result of the addition is defined as the final output fused feature matrix. This final fused feature matrix can be physically understood as: the set of the original features and the incremental features generated by the interaction.

[0137] The embodiments of the present invention bring the following technical advantages by introducing a residual connection structure: (1) It ensures the integrity of the information flow. By directly adding the original combined feature matrix to the result after interaction, it ensures that the unmodified hidden state information of each parallel generation thread can be passed down as the backbone information without loss. The intermediate interaction feature matrix is ​​used as bypass information to supplement and correct it. This structure effectively avoids the problem that the original important information is diluted or lost due to multiple nonlinear transformations in deep networks.

[0138] (2) Improved training stability and convergence speed of large language models. Residual connections provide a highway for gradients during backpropagation, which can effectively alleviate the common gradient vanishing or gradient explosion problems in deep neural networks, making it easier to optimize and train new modules such as cross-thread shared layers.

[0139] (3) Adaptive feature fusion is achieved. This residual connection structure gives the large language model an inherent flexibility. During training, if cross-thread interaction is beneficial to the current task, the model can learn the corresponding weights to generate an information-rich intermediate interaction feature matrix; conversely, if cross-thread interaction is harmful or useless, the large language model can turn off or bypass the interaction module by making the output value of the intermediate interaction feature matrix approach zero, so that the output fused feature matrix is ​​approximately equal to the input combined feature matrix. This adaptive capability makes the feature interaction mechanism of this invention more robust and efficient.

[0140] Based on the above embodiments, as an optional embodiment of the present invention, the construction process of the combined feature matrix is ​​further enhanced by introducing a position encoding vector before stacking the hidden state features.

[0141] In the aforementioned embodiments, the obtained combined feature matrix is ​​constructed by directly stacking the hidden state features corresponding to all parallel generation threads along the batch dimension. While this approach aggregates all features, it presents a potential information loss problem: the self-attention mechanism itself is permutation invariant, meaning that if the order of the input sequence is shuffled, its output remains unchanged. For attention computation in cross-thread shared layers, this means that the model cannot distinguish by default which hidden state feature comes from the first parallel generation thread, which comes from the second parallel generation thread, and so on.

[0142] To address this issue, this embodiment provides an implementation method that stacks the hidden state features corresponding to all the parallel generation threads in the batch dimension to construct a combined feature matrix.

[0143] Figure 6 This is a schematic diagram of the process provided by the present invention for stacking the hidden state features of parallel generation threads in the batch dimension to generate a combined feature matrix, as shown below. Figure 6 As shown, the specific steps include, but are not limited to, the following: Step 61: Obtain the position encoding vector corresponding to each of the parallel generation threads. The position encoding vector is used to identify the index position of each of the parallel generation threads in the batch dimension.

[0144] A positional encoding vector is a special type of vector with the same dimension as the model features. Its purpose is to provide a unique identifier for each position in the input sequence.

[0145] In the context of this implementation, this position does not refer to the time step position of the token in the input sequence, but rather to the index position of the parallel generation thread in the batch. For example, for a batch containing N parallel generation threads, N different position encoding vectors can be predefined, corresponding to indices 0, 1, 2, ..., N-1.

[0146] These positional encoding vectors can be fixed and generated based on sine and cosine functions, or they can be learnable parameters, such as a thread ID embedding.

[0147] Step 62: Fuse each of the location encoding vectors with the corresponding hidden state features to obtain location-aware features.

[0148] The fusion operation performed can be an element-wise vector addition, which adds the hidden state feature vector of the i-th parallel generation thread to its corresponding i-th position encoding vector.

[0149] The new vector obtained after addition is defined as the position-aware feature, which not only contains the original semantic information (from the hidden state features), but also additionally encodes the identity information of its parallel generation thread in the batch (from the position encoding vector).

[0150] Step 63: Stack the position-aware features corresponding to all the parallel generation threads in the batch dimension to construct the combined feature matrix.

[0151] The N location-aware features obtained in the previous step are stacked along the batch dimension to form the final feature with dimensions [N, D]. model The combined feature matrix of ].

[0152] The embodiments of the present invention introduce positional encoding of parallel generation threads, which brings crucial structural information to the self-attention computation of cross-thread shared layers.

[0153] On the one hand, it breaks the permutation invariance, giving it thread identity awareness. By attaching a unique positional identifier to the features of each parallel-generated thread, the large language model can distinguish the source of information when performing cross-thread attention computation. For example, it can learn to discover that the features of the thread with index 0 (usually the main candidate or reference thread) are more worthy of attention, or that there may be a stronger correlation between threads with similar indices.

[0154] On the other hand, it enables more complex and structured interaction patterns. With position-aware capabilities, the cross-thread shared layer is no longer simply a weighted average of an unordered set of features, but can learn biased interaction patterns based on thread indices. For example, a large language model can learn a broadcast pattern (all threads pay attention to a specific thread) or a polling pattern (threads alternately pay attention to each other). This structured information exchange capability makes the collaborative methods of parallel groups richer and more flexible, enabling them to adapt to more complex generation tasks.

[0155] Based on the above embodiments, as an optional embodiment of the present invention, further limitations are provided on the process of finally predicting and determining the output lexical units. The core of this limitation is to introduce differentiated generation strategies for different parallel generation threads, so as to ensure from a mechanism perspective that the multiple candidate responses generated in the end are different.

[0156] A key feature of this embodiment is that different sampling parameters are configured for each of the parallel generation threads. Sampling parameters are a set of hyperparameters used to control the balance between randomness and determinism in the lexical selection stage when the large language model generates text. In this embodiment, they include, but are not limited to, temperature parameters, core sampling Top-p, and Top-k.

[0157] The temperature parameter adjusts the smoothness of the probability distribution. A higher temperature parameter increases the chance of selecting low-probability words, thus enhancing the randomness and creativity of the generation; a lower temperature parameter amplifies the advantage of high-probability words, making the generation results more deterministic and conservative. Top-p controls sampling from the smallest set of words whose total probability reaches a threshold p, introducing a certain degree of diversity while ensuring generation quality. Top-k controls sampling only from the k words with the highest probabilities.

[0158] Different sampling parameters mean that a unique set of parameters can be specified for each parallel generation thread. For example, in a medical communication scenario, the following configuration can be used: Configure a lower temperature parameter (e.g., 0.2) for parallel generation thread A to generate a baseline response that is rigorously worded and highly consistent with medical facts; or configure a higher temperature parameter (e.g., 0.8) for parallel generation thread B to generate a more popular explanation that uses more metaphors and is more creative; or configure a moderate Top-p value (e.g., 0.9) for parallel generation thread C to generate a response that strikes a balance between factual accuracy and expressive flexibility.

[0159] Based on the above configuration, the prediction of output lexical units of each of the parallel generation threads using the output layer of the large language model based on the shared state features may include, but is not limited to: The output layer is used to map the shared state features corresponding to each of the parallel generation threads into a probability distribution on the vocabulary dimension; For any parallel generation thread, the probability distribution is adjusted using the configured sampling parameters to determine the output lexical based on the adjusted probability distribution.

[0160] Specifically, in this embodiment, after obtaining N shared state features that have undergone cross-thread interaction, these N shared state features are respectively input into the output layer of the large language model. The output layer processes each shared state feature to generate a corresponding probability distribution covering the entire vocabulary. This probability distribution represents the likelihood that each word in the vocabulary will be the next output word in the current context. At this point, N independent probability distributions can be obtained, each corresponding to one of the N parallel generation threads.

[0161] As a post-processing procedure to personalize the N probability distributions obtained in the previous step, taking any parallel generation thread (e.g., parallel generation thread A) as an example, this embodiment retrieves the sampling parameters (e.g., temperature parameter 0.2) pre-configured for that parallel generation thread. Then, the probability distribution corresponding to parallel generation thread A is mathematically transformed using this specific parameter. For example, applying the temperature parameter involves performing an exponential operation on each probability value in the original probability distribution and then re-normalizing it.

[0162] After obtaining the adjusted probability distribution, the final output word is determined from the new probability distribution. The determination method can be random sampling (i.e., randomly selecting a word based on the new probability) or deterministic selection, such as selecting the word with the highest adjusted probability. This process is executed separately and independently for all N parallel generation threads. Because the sampling parameters configured for each parallel generation thread are different, the adjustment process of their respective probability distributions also differs, greatly increasing the possibility of differences in the final selected output word.

[0163] This invention provides a controllable and explicit differentiation guide for the parallel generation process. By ensuring that all candidate responses possess high collective intelligence across a cross-thread shared layer, and by configuring different sampling parameters for each parallel generation thread, it achieves structured exploration of different regions of the solution space. Compared to existing methods that rely solely on randomness to generate differences, its generation strategy is more purposeful and controllable, significantly enhancing the overall value of the generated multi-candidate response set. This is particularly relevant in professional question-and-answer scenarios requiring a balance between rigor and accessibility (such as medical and legal questions), providing users with richer, more practical, and higher-quality alternatives.

[0164] Based on the above embodiments, as an optional embodiment of the present invention, the specific types of the sampling parameters and how they adjust the probability distribution are further clarified. The sampling parameters specifically include key temperature parameters that control the level of randomness in the generated text. Therefore, adjusting the probability distribution using the configured sampling parameters for any parallel generation thread to determine the output lexical based on the adjusted probability distribution may include, but is not limited to: The probability distribution is smoothed or sharpened using the temperature parameter configured by any of the parallel generation threads to obtain the adjusted probability distribution; The output lexical units are sampled based on the adjusted probability distribution.

[0165] The probability distribution refers to the original probability distribution directly generated by the output layer without any modification, which can be denoted as P=(p1, p2, ..., p...). V), where V is the size of the vocabulary, and p i is the original probability of the i-th token.

[0166] Denote the temperature parameter configured for any parallel generation thread as T. Then, for different parallel generation threads, the value of T is different.

[0167] Smoothing or sharpening is a mathematical transformation process, and its specific formula is: ; where, is the probability value of the i-th token in the original probability distribution; T is the temperature parameter configured for the parallel generation thread; is the result of temperature scaling the original probability of the i-th token; is the sum of performing the same operation of (1 / T) power on all V tokens in the vocabulary, serving as the normalization factor to ensure that the sum of all adjusted probabilities is still 1.

[0168] When T > 1, this transformation will smooth the original probability distribution, which corresponds to generating more random, creative, and diverse text. In the original probability distribution, the gap between high-probability values and low-probability values will be narrowed, making tokens with relatively low original probabilities have non-negligible chances of being selected.

[0169] When 0 < T < 1, this transformation will sharpen the original probability distribution, which corresponds to generating text that is more inclined to be deterministic, conservative, and factual. The gap between high-probability values and low-probability values will be widened, making the advantages of the few tokens with the highest original probabilities more prominent.

[0170] When T = 1, the adjusted probability distribution is the same as the original probability distribution.

[0171] By setting different values of the temperature parameter for different parallel generation threads, after processing, the same original probability distribution will result in two distinct adjusted probability distributions.

[0172] After obtaining the adjusted probability distribution P'=(p'1, p'2,..., p' V ), the large language model will sample from this distribution to determine the final output token. The sampling method used can be multinomial sampling (MultinomialSampling), that is, according to the adjusted probability p' of each token i as the weight of its being drawn, randomly draw a token as the output at this time step. Since the adjusted probability distributions of different parallel generation threads are different, even if the same random sampling method is used, the finally sampled output tokens are very likely to be different, thus achieving differentiation in the generated content.

[0173] This invention specifies the sampling parameter as a temperature parameter and describes how it acts on the probability distribution, detailing how to achieve fine control over the randomness of generation by adjusting the temperature parameter. Applying this mechanism to different parallel generation threads provides a concrete and effective means of implementing differentiated generation strategies, enabling the invention to purposefully and controllably generate text content with different levels of innovation or rigor while ensuring the overall quality of candidate responses.

[0174] In the standard word-by-word generation process, for each new word generated, the large language model needs to concatenate it with all previous historical words (including target prompts and already generated words) to form a new, longer sequence, and then perform a complete forward propagation computation on this new sequence. This means that as the length of the generated sequence increases, the computational cost at each step increases linearly, because the processing of historical information is repeatedly performed, resulting in a significant waste of computational resources, especially when generating long text responses, which leads to a significant increase in inference latency.

[0175] To address the aforementioned computational efficiency issues in the autoregressive generation process of large language models, this invention provides an optimization method.

[0176] Figure 7 This is a schematic diagram illustrating the operating principle of the key-value cache provided by the present invention. It serves as an optional embodiment of the present invention, such as... Figure 7 As shown, the process of processing the input sequence using the stacked Transformer decoder blocks may include, but is not limited to: Step 71: In each step of word-by-word generation, the key vector and value vector calculated by the Transformer decoder block for each of the parallel generation threads are stored in the key-value cache. Step 72: In the step of generating the current new lexical unit, the key vector and value vector stored in the key-value cache are called, and the corresponding query vector, key vector and value vector are calculated only for the current new lexical unit; Step 73: Based on the query vector, key vector, and value vector of the current new lexical unit, and the retrieved key vector and value vector, perform self-attention calculation within the network layer.

[0177] At a generation time step (e.g., time step t-1), after the large language model has processed the input sequence and computed the query vector, key vector, and value vector corresponding to all lexical units, the key vectors and value vectors associated with the historical lexical units (from the 1st to the (t-1th)th lexical unit) are extracted. These extracted key vectors and value vectors are stored in a dedicated memory area, namely the key-value cache.

[0178] It should be emphasized that, considering that the word sequences generated by each parallel generation thread are different, this embodiment will maintain an independent key-value cache for each parallel generation thread.

[0179] At the end of time step t-1, the key-value buffer of each parallel generation thread stores its complete set of key and value vectors from the 1st to the (t-1th)th lexical. In the next generation time step (time step t), the goal of the large language model is to generate the (t+1th)th lexical. At this point, the input to the large language model is no longer the entire sequence of length t, but only the current new lexical at the t-th position.

[0180] This embodiment further calls the key-value cache of the corresponding parallel generation thread to retrieve the key vectors and value vectors of all stored historical lexical units (1 to t-1). Meanwhile, the large language model only needs to perform a lightweight forward propagation once on the current new lexical unit to calculate the query vector, key vector, and value vector that belong only to that new lexical unit.

[0181] When performing self-attention computation within the network layer (i.e., within the attention layer of each Transformer decoder block), the required inputs include: (1) Use the query vector from the current new word element and the key vector called out to concatenate with the key vector from the current new word element to form a complete key set containing all historical and current position information.

[0182] (2) The retrieved value vector is concatenated with the value vector from the current new word to form a complete value set.

[0183] In this way, large language models can achieve attention computation effects that are completely equivalent to processing complete sequences with minimal computational cost, meaning that the current new word can pay attention to all historical words, including itself.

[0184] By introducing a key-value caching mechanism, the computational complexity of generating new lexical units in each step is reduced from being proportional to the total sequence length t to being proportional to a constant. Since large language models no longer need to repeatedly process increasingly longer historical sequences, this reduction in complexity directly translates into a significant improvement in inference speed and a substantial reduction in latency. This makes the multi-candidate response generation method of the present invention suitable for applications that handle long contexts or require fast responses.

[0185] Figure 8 This is a schematic diagram of the self-attention calculation process within the network layer provided by the present invention, as shown below. Figure 8As shown, this embodiment provides further constraints on the masking mechanism for the self-attention calculation process within the network layer to ensure the logical correctness of the calculation and strict adherence to the autoregressive generation pattern, specifically including but not limited to: Step 81: Construct a causal mask matrix. The element values ​​in the region located in the upper right corner of the main diagonal of the causal mask matrix are masking flag values, and the element values ​​in the region located in the lower left corner of the main diagonal are retention flag values. Step 82: After calculating the attention score inside the network layer, the causal mask matrix is ​​added element by element to the attention score inside the network layer to obtain the masked attention score. Step 83: Normalize the masked attention score to obtain the final attention weights for weighted summation.

[0186] The causal mask matrix is ​​a matrix with the same dimensions as the attention score matrix, specifically designed for autoregressive generation tasks in this embodiment. Its structure includes a region located in the upper right of the main diagonal and a region located in the lower left of the main diagonal.

[0187] In this system, the element values ​​in the region located to the upper right of the main diagonal are set as occlusion flags. In actual network computation, this occlusion flag can be an extremely large negative number (e.g., negative infinity). Physically, this region corresponds to the case in attention computation where the query position index is greater than the key position index, indicating an attempt to focus on future positions. Conversely, the element values ​​in the region located to the lower left of the main diagonal are set as retention flags, typically 0. This region corresponds to the case where the query position index is greater than or equal to the key position index, indicating a focus on current or historical positions.

[0188] The attention score within the network layer refers to the raw score obtained by performing a dot product between the query vector of the current word and the key vectors of all historical and current words. In this embodiment, the causal mask matrix constructed in the previous step is added element-wise to this raw attention score matrix. Since the masking flag value is an extremely large negative number, it is kept at 0. The effect of this addition operation is that for the attention score at a future position, adding an extremely large negative number makes its value also extremely negative; for the attention scores at the current and historical positions, adding 0 leaves their values ​​unchanged.

[0189] The occluded attention score obtained in the previous step is input into the Softmax function for normalization. The Softmax function compresses the output probability of inputs with extremely negative values ​​(i.e., the occluded future position score) to almost 0, while inputs with normal values ​​(i.e., the retained current and historical position scores) are normally normalized to a probability distribution that sums to 1.

[0190] The resulting probability distribution with a future position weight of 0 is the final attention weight used for weighted summation.

[0191] By introducing and applying a well-defined causal mask matrix, this invention provides strict directional constraints for the self-attention computation within the network layer. This ensures that when a large language model generates any word, its information source is strictly limited to the sequence content preceding (including itself) that word, completely eliminating the possibility of information leaking from the future to the past.

[0192] Based on the above embodiments, as another optional embodiment of the present invention, the configuration and usage of the key-value cache have been further optimized, aiming to significantly improve the computational efficiency and resource utilization in the parallel generation process, especially in scenarios involving long target hints.

[0193] The aforementioned embodiments introduced a key-value caching mechanism. However, if a completely independent key-value cache is maintained for each parallel generation thread, a problem arises: since all parallel generation threads share the same initial target hint, the operation of calculating and storing the key vector and value vector for the target hint will be repeated N times (N is the number of parallel generation threads). This leads to unnecessary computational redundancy and waste of GPU memory resources.

[0194] To address this issue, this embodiment configures the key-value cache in shared mode. Based on this shared mode, accessing the key vectors and value vectors stored in the key-value cache includes: Identify the common portions of each input sequence that belong to the target prompt; For the common part, each of the parallel generation threads shares the same key-value cache data.

[0195] At the start of the generation task, this embodiment explicitly knows that the input sequences of all parallel generation threads share the same target prompt as a prefix, and this common prefix is ​​identified as the common part. Taking a medical consultation scenario as an example, regardless of the style of response subsequently generated, the input sequence of all parallel generation threads begins with the text "Please explain the causes and daily management suggestions of 'type 2 diabetes' to an anxious patient in a simple and easy-to-understand way," and this text is the aforementioned common part.

[0196] Before generating the first lexical unit, the common part (i.e., the target cue) is computed only once. During this computation, the key and value vectors generated by the network layers (i.e., multiple stacked Transformer decoder blocks) for all lexical units of the common part are stored in a single, shared key-value cache data block.

[0197] Subsequently, when the N parallel generation threads begin generating their first new lexical units, during the self-attention computation within the network layers, they all reference or point to the same key-value cache data to obtain contextual information about the target cues. From the generation of the first new lexical unit onwards, each parallel generation thread creates and maintains its own private, non-shared key-value cache data for its newly generated, unique lexical sequence.

[0198] Therefore, from an overall perspective, the key-value cache used by each parallel generation thread actually consists of two parts: a read-only part shared by all threads that corresponds to the target hint, and an independent read-write part that corresponds to the subsequently generated content.

[0199] By configuring the key-value cache in shared mode and sharing the same cached data for common target hints, this embodiment of the invention achieves significant technical optimization.

[0200] First, for a length of L p The target hint, with N parallel generation threads, reduces the video memory required to store its key-value cache from being proportional to N to a fixed value independent of N. This optimization effect increases with the number of parallel generation threads N and the target hint length L. p The increased complexity of this process makes it increasingly significant, enabling larger-scale parallel generation or processing of longer contexts with limited hardware resources. Furthermore, the pre-filling stage for processing input prompts is one of the most computationally intensive parts of the entire generation process. By reducing the number of calculations in this stage from N to 1, the initial time required from input prompts to generating the first word is significantly shortened, improving system responsiveness and user experience.

[0201] Through the foregoing embodiments, the multi-candidate response generation method provided by this invention can efficiently generate multiple high-quality and diverse candidate responses for the same target prompt. However, for end users or downstream applications, a single, optimal output result is usually required. Manually selecting from multiple candidate responses is not only inefficient but also relies on the subjectivity of human judgment.

[0202] To address this issue, in another optional embodiment of the invention, after generating multiple candidate responses for the target prompt, the method further includes: A reward model is used to evaluate the quality of each candidate response, and a quality score is obtained for each candidate response. Based on the quality score, a target response is selected from all the candidate responses, or a weighted combination of all the candidate responses is performed to generate a target response.

[0203] The reward model can be a standalone, specially trained rating model whose input is a cue-response pair and whose output is a scalar value (i.e., a quality score) that quantifies the extent to which the response satisfies the cue's requirements.

[0204] Reward models are typically trained based on large amounts of human preference data. For example, for the same cue, human annotators may rank multiple different responses, and the model learns these ranking relationships to master the criteria for evaluating text quality. These criteria may include, but are not limited to, factual accuracy, logical coherence, language fluency, harmlessness, and the degree of adherence to instructions.

[0205] In this implementation, the generated N candidate responses are paired one by one with their corresponding original target cue, and then these N cue-response pairs are input into the pre-loaded reward model.

[0206] The reward model scores each candidate response, resulting in N corresponding quality scores. For example, in a medical scenario, the reward model might award the highest score to candidate response B, which emphasizes key action points and is clear and easy to understand.

[0207] Selecting the target response is the most straightforward strategy. It simply compares all N quality scores, determines the candidate response with the highest score as the final target response, and outputs it to the user or downstream application. This approach ensures that the output result is the one with the highest quality after model evaluation among all candidates.

[0208] A more complex ensemble strategy is to weight and combine all the candidate responses to generate the target response. For example, instead of directly selecting a single best response, one could use quality scores as weights to perform some form of fusion across all candidate responses. In some advanced applications, such as code generation or mathematical derivation, the target response could be selected by comparing the final answers of multiple candidate responses and voting based on their respective quality scores.

[0209] This invention introduces an automated evaluation and selection mechanism based on a reward model after the generation process, constructing a complete "generation-evaluation-selection" closed loop. This transforms the selection step, which previously required manual intervention, into a process that can be automatically executed by a machine, significantly improving the overall system's usability and end-to-end efficiency. Furthermore, by generating multiple candidates and performing secondary verification using independent reward models, accidental errors during the generation process can be effectively filtered out, significantly improving the reliability of the final output. This is particularly important in professional fields with extremely high accuracy requirements (such as medicine and finance).

[0210] Directly modifying the internal structure of a large, pre-trained language model and performing full fine-tuning is not only computationally expensive but also prone to causing catastrophic performance degradation on the original task. To address this issue, this invention proposes a parameter-efficient fine-tuning (PEFT) strategy, which involves inserting cross-thread shared layers as adapter modules into the initial large language model.

[0211] Here, the initial large language model refers to a base model that has been pre-trained on a large-scale corpus and possesses general language capabilities. The adapter module is a small, pluggable neural network module. In this embodiment, the cross-thread shared layer is designed as an adapter module. The insertion operation refers to adding this cross-thread shared layer, acting as an adapter module, to the data flow path of the initial large language model without changing any weights or structure of the original network layers. Specifically, it can be inserted between the last Transformer decoder block and the output layer of the initial large language model's network layers.

[0212] During the scenario-based training phase of the initial large language model, the aim is to enable the initial large language model to acquire cross-thread collaborative generation capabilities, thereby obtaining the final large language model that can be used for reasoning. To this end, this embodiment implements a specific parameter update strategy.

[0213] First, the adapter module contains trainable parameters and shares the weight matrix within the layer across threads, for example, querying the weight matrix ( W Q ), key weight matrix ( W K ) and value weight matrix ( W V These parameters are set to a trainable state. The number of these parameters is very small relative to the total number of parameters in the entire initial large language model (usually less than 1%).

[0214] Secondly, the parameters of the network layers of the initial large language model are frozen during the training process. Specifically, freezing means that the original, large number of network layer parameters of the initial large language model (such as the weights of all Transformer decoder blocks) remain unchanged throughout the entire training process and do not participate in gradient updates.

[0215] Finally, the trainable parameters are updated based on the loss function for cross-thread interaction. The goal of training is to optimize the performance of the adapter module (cross-thread shared layer). This embodiment designs one or more specific cross-thread interaction loss functions to guide the initial large language model to learn how to effectively utilize cross-thread information, such as diversity loss, consistency loss, or cooperative loss.

[0216] Diversity loss is used to penalize situations where multiple parallel generation threads produce overly similar outputs, encouraging them to explore different semantic directions. For example, this can be achieved by calculating the cosine similarity between feature vectors of different candidate responses and minimizing that similarity.

[0217] Consistency or cooperation loss refers to the desire for all candidate responses to remain consistent in following certain high-level instructions in some tasks. It can be calculated based on some aggregated characteristics of all candidate responses to encourage them to cooperate in completing the task.

[0218] The dataset used for training contains multiple "hint-multiple high-quality response" pairs. Hints are input into the model to obtain multiple responses generated in parallel. The loss function described above is then calculated based on the comparison of these responses with the standard answer and the relationships between them. The trainable parameters in the adapter module are updated using only the gradients generated by the calculated loss.

[0219] Because only a very small number of adapter parameters need to be trained and stored, rather than the entire model with billions or even hundreds of billions of parameters, this approach makes it possible to extend the functionality of large language models on ordinary hardware, significantly lowering the technical application threshold. Since the main parameters of the initial large language model are frozen, the general language knowledge and capabilities learned during the pre-training phase are fully preserved. Furthermore, the trained cross-thread shared layer adapter can be stored and distributed as a separate file. During inference, this adapter can be dynamically loaded into the base model as needed to enable the functionality of this invention, or unloaded to restore the original behavior of the base model, providing great flexibility for model management and deployment.

[0220] Based on the above embodiments, as an optional embodiment, a multi-candidate response generation apparatus is further provided, mainly comprising: A response generation unit is used to input the target prompt into a large language model and obtain multiple candidate responses generated by the large language model through multiple parallel generation threads. In the process of generating the multiple candidate responses word-by-word from the large language model: The hidden state features corresponding to each of the parallel generation threads are obtained by utilizing the network layer of the large language model. By utilizing the cross-thread shared layer of the large language model, batch-dimensional attention calculations are performed on each of the hidden state features to obtain the shared state features; Using the output layer of the large language model, the output lexical units of each of the parallel generation threads are predicted based on the shared state features.

[0221] It should be noted that the multi-candidate response generation device provided by the present invention can execute the multi-candidate response generation method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0222] The multi-candidate response generation device provided by this invention breaks the independence between parallel generation threads by introducing a cross-thread shared layer between the network layer and the output layer. This enables each thread to interact and fuse feature dimensions during the generation process, thereby effectively utilizing the global information of the parallel group and improving the diversity and quality of generated candidate responses.

[0223] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a multi-candidate response generation method, which includes: inputting a target prompt into a large language model, obtaining multiple candidate responses generated by the large language model through multiple parallel generation threads; in the process of obtaining the multiple candidate responses generated by the large language model word-by-word: using the network layer of the large language model to obtain the hidden state features corresponding to each of the parallel generation threads; using the cross-thread shared layer of the large language model to perform batch-dimensional attention calculation on each of the hidden state features to obtain shared state features; and using the output layer of the large language model to predict the output words of each of the parallel generation threads based on the shared state features.

[0224] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0225] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the multi-candidate response generation method provided in the above embodiments, the method comprising: inputting a target cue into a large language model, obtaining multiple candidate responses generated by the large language model through multiple parallel generation threads; in the process of obtaining the multiple candidate responses generated by the large language model word-by-word: using the network layer of the large language model to obtain hidden state features corresponding to each of the parallel generation threads; using the cross-thread shared layer of the large language model to perform batch-dimensional attention calculation on each of the hidden state features to obtain shared state features; and using the output layer of the large language model to predict the output words of each of the parallel generation threads based on the shared state features.

[0226] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-candidate response generation method provided in the above embodiments. The method includes: inputting a target cue into a large language model; obtaining multiple candidate responses generated by the large language model through multiple parallel generation threads; during the process of obtaining the multiple candidate responses generated per word by the large language model: using the network layer of the large language model to obtain hidden state features corresponding to each of the parallel generation threads; using the cross-thread shared layer of the large language model to perform batch-dimensional attention calculation on each of the hidden state features to obtain shared state features; and using the output layer of the large language model to predict the output words of each of the parallel generation threads based on the shared state features.

[0227] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0228] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0229] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-candidate response generation method, characterized by, include: The target prompt is input into a large language model, and multiple candidate responses generated by the large language model through multiple parallel generation threads are obtained. In the process of generating the multiple candidate responses word-by-word meta-analysis from the large language model: The hidden state features corresponding to each of the parallel generation threads are obtained by utilizing the network layer of the large language model. By utilizing the cross-thread shared layer of the large language model, batch-dimensional attention calculations are performed on each of the hidden state features to obtain the shared state features; Using the output layer of the large language model, the output lexical units of each of the parallel generation threads are predicted based on the shared state features.

2. The multi-candidate response generation method of claim 1, wherein, The network layer of the large language model includes multiple stacked Transformer decoder blocks, and each candidate response is composed of multiple lexical units arranged in sequence; The step of using the network layer of the large language model to obtain the hidden state features corresponding to each of the parallel generation threads includes: Construct multiple input sequences, each of which corresponds to a parallel generation thread and includes the target prompt and the word sequence already generated by the parallel generation thread; All the input sequences are input as a batch into the stacked Transformer decoder blocks; From the output of the last Transformer decoder block of the network layer, extract the feature vector corresponding to the current generation position of each input sequence, and use it as the hidden state feature corresponding to each parallel generation thread.

3. The multi-candidate response generation method of claim 1, wherein, The cross-thread shared layer is configured between the network layer and the output layer; The method utilizes the cross-thread shared layer of the large language model to perform batch-dimensional attention calculations on each hidden state feature to obtain shared state features, including: Stack the hidden state features corresponding to all the parallel generation threads in the batch dimension to construct a combined feature matrix; Using the cross-thread shared layer, self-attention calculation is performed on the combined feature matrix to obtain a fusion feature matrix containing cross-thread interaction information; The fused feature matrix is ​​split along the batch dimension to obtain the shared state features corresponding to each of the parallel generation threads.

4. The multi-candidate response generation method of claim 3, wherein, The step of utilizing the cross-thread shared layer to perform self-attention computation on the combined feature matrix to obtain a fused feature matrix containing cross-thread interaction information includes: Using the query weight matrix, key weight matrix, and value weight matrix, linear mapping is performed on the combined feature matrix to obtain the query matrix, key matrix, and value matrix, respectively. The product of the query matrix and the transpose of the key matrix is ​​calculated to obtain the attention score matrix, which represents the correlation between the hidden state features of each of the parallel generation threads; The value matrix is ​​weighted and summed using the attention score matrix to generate the fusion feature matrix.

5. The multi-candidate response generation method of claim 4, wherein, The cross-thread shared layer is configured with an interaction mask matrix, which is a fully visible mask matrix. The dimension of the fully visible mask matrix is ​​the same as the dimension of the initial score matrix. The element values ​​in the fully visible mask matrix corresponding to the positions between different parallel generation threads are reserved identifier values. The reserved identifier values ​​ensure that the corresponding attention scores are retained in subsequent calculations. The step of calculating the product between the query matrix and the transpose of the key matrix to obtain the attention score matrix includes: Calculate the product of the query matrix and the transpose of the key matrix to obtain the initial score matrix; The attention score matrix is ​​obtained by adding the fully visible mask matrix to the initial score matrix element by element.

6. The multi-candidate response generation method of claim 4, wherein, The cross-thread shared layer also includes a residual connection structure; The step of using the attention score matrix to perform a weighted summation of the value matrix to generate the fused feature matrix includes: The value matrix is ​​weighted and summed using the attention score matrix to obtain the intermediate interaction feature matrix; The intermediate interactive feature matrix and the combined feature matrix are added element by element to obtain the fused feature matrix.

7. The multi-candidate response generation method according to claim 3, characterized in that, The step of stacking the hidden state features corresponding to all the parallel generation threads in the batch dimension to construct a combined feature matrix includes: Obtain the position encoding vectors corresponding to each of the parallel generation threads, and the position encoding vectors are used to identify the index positions of each of the parallel generation threads in the batch dimension; Each of the aforementioned location encoding vectors is fused with its corresponding hidden state feature to obtain location-aware features; The location-aware features corresponding to all the parallel generation threads are stacked in the batch dimension to construct the combined feature matrix.

8. The multi-candidate response generation method according to claim 1, characterized in that, Each of the parallel generation threads is configured with different sampling parameters, and the candidate responses generated by different parallel generation threads are different; The step of using the output layer of the large language model to predict the output lexical units of each of the parallel generation threads based on the shared state features includes: The output layer is used to map the shared state features corresponding to each of the parallel generation threads into a probability distribution on the vocabulary dimension; For any parallel generation thread, the probability distribution is adjusted using the configured sampling parameters to determine the output lexical based on the adjusted probability distribution.

9. The multi-candidate response generation method according to claim 8, characterized in that, The sampling parameters include temperature parameters; The step of adjusting the probability distribution using the configured sampling parameters for any parallel generation thread, and determining the output lexical unit based on the adjusted probability distribution, includes: The probability distribution is smoothed or sharpened using the temperature parameter configured by any of the parallel generation threads to obtain the adjusted probability distribution; The output lexical units are sampled based on the adjusted probability distribution.

10. The multi-candidate response generation method according to claim 2, characterized in that, The process of processing the input sequence using the stacked Transformer decoder blocks also includes: In each step of word-by-word generation, the key vector and value vector calculated by the Transformer decoder block for the generated word sequence of each parallel generation thread are stored in the key-value cache. In the step of generating the current new lexical unit, the key vector and value vector stored in the key-value cache are called, and the corresponding query vector, key vector and value vector are calculated only for the current new lexical unit; Based on the query vector, key vector, and value vector of the current new lexical, as well as the retrieved key vector and value vector, the self-attention calculation within the network layer is performed.

11. The multi-candidate response generation method according to claim 10, characterized in that, The execution of self-attention calculation within the network layer includes: Construct a causal mask matrix, wherein the element value in the region located in the upper right corner of the main diagonal is the masking identifier value, and the element value in the region located in the lower left corner of the main diagonal is the retention identifier value. After calculating the attention score inside the network layer, the causal mask matrix is ​​added element by element to the attention score inside the network layer to obtain the masked attention score. The masked attention scores are normalized to obtain the final attention weights used for weighted summation.

12. The multi-candidate response generation method according to claim 10, characterized in that, The key-value cache is configured in shared mode; The step of calling the key vector and value vector stored in the key-value cache includes: Identify the common portions of each input sequence that belong to the target prompt; For the common part, each of the parallel generation threads shares the same key-value cache data.

13. The multi-candidate response generation method according to any one of claims 1 to 12, characterized in that, After generating multiple candidate responses for the target prompt, the process further includes: A reward model is used to evaluate the quality of each candidate response, and a quality score is obtained for each candidate response. Based on the quality score, a target response is selected from all the candidate responses, or a weighted combination of all the candidate responses is performed to generate a target response.

14. The multi-candidate response generation method according to any one of claims 1 to 12, characterized in that, The cross-thread shared layer is inserted as an adapter module into the initial large language model; The adapter module includes trainable parameters. During the training process of the initial large language model to obtain the large language model, the parameters of the network layers of the initial large language model are frozen, and the trainable parameters are updated based on the cross-thread interactive loss function.

15. A multi-candidate response generation device, characterized in that, include: A response generation unit is used to input the target prompt into a large language model and obtain multiple candidate responses generated by the large language model through multiple parallel generation threads. In the process of generating the multiple candidate responses word-by-word from the large language model: The hidden state features corresponding to each of the parallel generation threads are obtained by utilizing the network layer of the large language model. By utilizing the cross-thread shared layer of the large language model, batch-dimensional attention calculations are performed on each of the hidden state features to obtain the shared state features; Using the output layer of the large language model, the output lexical units of each of the parallel generation threads are predicted based on the shared state features.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the multi-candidate response generation method as described in any one of claims 1 to 14.

17. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the multi-candidate response generation method as described in any one of claims 1 to 14.

Images