Information processing method and device and electronic equipment

Abstract

The invention discloses an information processing method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the steps of inputting first hidden state information of each lexical element in first information into a first model, generating a first key value pair through a network layer of the first model, and obtaining the first lexical element based on the first key value pair; generating a second key value pair based on the first lexical element and the first key value pair generated by each network layer through the network layer of the first model, and obtaining second information based on the second key value pair; the first lexical element is a starting lexical element of the second information; the types of network layers of the first model comprise a reference layer, a prediction layer and a sharing layer; the key value pair generated by the prediction layer is obtained based on the hidden state information generated by the first reference layer; the first reference layer is a reference layer which is ranked in the front and is closest to the prediction layer; the key value pair generated by the sharing layer is a key value pair generated by a previous network layer adjacent to the sharing layer.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence technology, specifically relating to an information processing method, apparatus, and electronic device. Background Technology

[0002] With the powerful capabilities of large language models in tasks such as text generation and dialogue interaction, deploying them to mobile terminals to achieve intelligent applications with low latency, high privacy protection, and offline availability has become an important trend in the industry.

[0003] Most current large language models are based on the decoder-only Transformer architecture. This architecture typically divides inference into two stages: a pre-filling stage and a decoding stage. In the pre-filling stage, the network layers process the entire query information in parallel, generating key and value vectors for each word in the query information. Then, based on the key-value pairs generated by the network, the first word of the response information is generated. In the decoding stage, the network layers use the generated first word, along with the key and value vectors generated in the pre-filling stage, to generate subsequent words.

[0004] However, in the pre-filling and decoding stages of current large language models, when calculating the key-value pairs for each word, the subsequent network layer needs to obtain the key-value pairs generated by the previous network layer and calculate the corresponding key-value pairs based on the key-value pairs generated by the previous network layer. As a result, when the input and output information is long, the computational load of the network layers of the large language model will increase dramatically, leading to low efficiency in obtaining the response information. Summary of the Invention

[0005] The purpose of this application is to provide an information processing method, apparatus, and electronic device that can improve the calculation speed of key-value pairs, thereby improving the efficiency of obtaining reply information.

[0006] In a first aspect, embodiments of this application provide an information processing method, the method comprising: The first hidden state information of each word element in the first information is input into the first model. The first key-value pair is generated through the network layer of the first model, and the first word element is obtained based on the first key-value pair. Through the network layers of the first model, a second key-value pair is generated based on the first word and the first key-value pair generated by each network layer, and the second information is obtained based on the second key-value pair; Wherein, the first word is the starting word of the second information; the network layer of the first model includes a base layer, a prediction layer, and a shared layer; the key-value pairs generated by the prediction layer are obtained based on the hidden state information generated by the first base layer; the first base layer is the base layer that is ranked first and is closest to the prediction layer; the key-value pairs generated by the shared layer are the key-value pairs generated by the network layer adjacent to the shared layer; the first information is query information, and the second information is the response information to the first information.

[0007] Thus, by inputting the first hidden state information of each word in the first information into the first model, the network layers of the first model generate the first key-value pair, and the first word is obtained based on the first key-value pair. Then, the network layers of the first model generate the second key-value pair based on the first word and the first key-value pair generated by each network layer, and the second information is obtained based on the second key-value pair. The network layers of the first model include a base layer, a prediction layer, and a shared layer. The key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer, which is the first layer in the order and closest to the prediction layer. The key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer. In this way, during the processing of the first information by the network layers of the first model, the key-value pair generated by the prediction layer can be predicted based on the hidden state information generated by the first base layer, which is the first layer in the order and closest to the prediction layer. The prediction layer does not need to perform the hidden state information generation process, but only needs to calculate the key-value pair, thus reducing the amount of computation. Furthermore, the key-value pairs generated by the shared layer are the same as those generated by the preceding network layer adjacent to the shared layer. In this way, the shared layer does not need to perform the key-value pair generation process and can directly reuse the key-value pairs generated by the preceding network layer, thereby improving the calculation speed of key-value pairs and thus improving the efficiency of obtaining response information.

[0008] In some embodiments, before inputting the first hidden state information of each lexical in the first information into the first model, generating a first key-value pair through the network layer of the first model, and obtaining the first lexical based on the first key-value pair, the method further includes: The second hidden state information of each word element in the third information is input into the first model. The third hidden state information is generated through the network layer of the first model, and the second word element is obtained based on the third hidden state information. The third information is query information and is different from the first information. Through the network layers of the first model, based on the second lexical unit and the third hidden state information generated by each network layer, fourth hidden state information is generated, and based on the fourth hidden state information, fourth information is obtained; wherein, the second lexical unit is the starting lexical unit of the fourth information; The network layer type of each network layer is determined based on the hidden state information generated by each network layer. In the first model, any network layer can be a normal layer, a baseline layer, a prediction layer, or a shared layer.

[0009] Thus, before using the first model, the network layer type of each network layer in the first model is objectively determined in advance using third information that differs from the first information, as well as the corresponding response information, rather than being determined based on user-defined criteria. This improves the objectivity of determining the network layer type of each network layer in the first model. Furthermore, pre-determining the network layer type of each network layer in the first model avoids the problems of low efficiency in obtaining the first term and second information that would result from determining the network layer type of each network layer during actual use of the first model.

[0010] In some embodiments, determining the network layer type of each network layer based on the hidden state information generated by each network layer includes: Based on the hidden state information generated by each network layer, determine the average similarity value of the hidden state information of each network layer; Network layers whose average similarity value of the hidden state information is less than a first threshold are defined as ordinary layers. N network layers out of M network layers are determined as the baseline layers; wherein, the average similarity value of the hidden state information of the M network layers is greater than a first threshold; the N network layers are the network layers ranked first in the first model; M and N are both positive integers, and N < M; Q of the M network layers are determined as prediction layers or shared layers; the Q network layers are the network layers other than the reference layer among the M network layers; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

[0011] Thus, since the hidden state information of the shallower network layers at the beginning of the sequence tends to capture basic, general grammatical and semantic features, their semantic representations are relatively stable and reliable. Choosing these network layers as the baseline layer means that the "source data" used to predict the deep key-value pair cache is of higher quality and more fundamental, reducing prediction errors introduced by the abstract or unstable representations of the baseline layer itself. On the other hand, the hidden state information of the deeper network layers at the end of the sequence tends to be more advanced and abstract semantic reasoning. The semantic representations in these deep network layers are redundant. Therefore, predicting or sharing this part is the most cost-effective. Choosing these network layers as prediction layers or sharing layers is equivalent to precisely optimizing regions with high redundancy and high computational cost, avoiding potentially risky approximate calculations in the key shallow feature extraction stage. The prediction layer undertakes the main task of saving computation by bypassing the complete self-attention calculation through prediction, significantly reducing latency. The sharing layer undertakes the main task of saving memory by directly reusing the key-value pair cache of the prediction layer, without performing the complete self-attention calculation or adding storage, achieving a dual "zero-cost" optimization of computation and memory.

[0012] In some embodiments, determining the average similarity value of the hidden state information of each network layer based on the hidden state information generated by each network layer includes: Calculate the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information to obtain P first similarity values; wherein, the fifth hidden state information is the hidden state information generated by the g-th network layer, the P sixth hidden state information is the hidden state information generated by P network layers, and the P network layers are the P network layers in the first model located after the g-th network layer, where g and P are positive integers; When P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; Wherein, the g-th network layer is any network layer in the first model.

[0013] In this way, the network layer type can be determined by the average similarity value of the hidden state information corresponding to each network layer. This allows for precise quantification of the redundancy of the semantic information represented within the first model. Similarity analysis can accurately identify which network layers in the first model have highly redundant outputs and which have unique and necessary outputs, providing strong support for determining the network layer type in the first model. Furthermore, based on the determined network layer type, the hidden state information of each word in the first information can be calculated and stored, making the computational and storage resource investment of the first model more "cost-effective" and avoiding waste of computational and storage resources.

[0014] In some embodiments, generating the first key-value pair through the network layer of the first model includes: Based on the first and second learning weights corresponding to the i-th network layer in the first model, and the first key-value pair generated by the (i-1)-th network layer, the first key-value pair of the i-th network layer is obtained; wherein, the i-th network layer is a base layer or a normal layer; i is a positive integer and i≥1; when i=1, the first key-value pair generated by the (i-1)-th network layer is the key vector and value vector in the first hidden state information; Based on the first learning weight and the second learning weight corresponding to the j-th network layer in the first model, and the first key-value pair generated by the second reference layer, the first key-value pair of the j-th network layer is obtained; the second reference layer is the reference layer that is closest to the j-th network layer in the sorted order, and the j-th network layer is the prediction layer; j is a positive integer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information.

[0015] Thus, for key-value pairs in the normal layer and the base layer, a complete self-attention calculation is required. However, for the prediction layer, a complete self-attention calculation is not performed. Instead, it can be calculated directly based on the hidden state information of the base layer, which is closest to the prediction layer. There is no need to perform the dot product calculation between the query vector and the key vector. In other words, in this embodiment, each network layer in the first model no longer performs self-attention calculation independently. Instead, it utilizes the redundancy in the deep representations of the first model, allowing some deep layers to directly predict and generate the required key-value pairs from the hidden states already calculated in the shallow layers, and caches the generated key-value pairs. This mechanism bypasses the quadratic complexity calculation of some layers, reducing computational latency at the source, saving computational resources of electronic devices, and thus improving the determination efficiency of the first term. Furthermore, allowing some deep network layers to directly reuse key-value pairs from shallow network layers reduces memory usage at the source.

[0016] In some embodiments, determining Q network layers out of the M network layers as prediction layers or shared layers includes: Based on the first, second, and third learning weights corresponding to the k-th network layer, the hidden state information generated by the (k-1)-th network layer, the sequence length of the third information, and the embedding vector of the third information, the first contribution of the k-th network layer is obtained; wherein, the k-th network layer is any one of the Q network layers; the first contribution is used to indicate the probability value that the k-th network layer is a prediction layer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information; and the third learning weight is the learning weight of the query vector in the hidden state information. If the first contribution is greater than the second threshold, the kth network layer is determined as the prediction layer; If the first contribution is less than or equal to the second threshold, the kth network layer is determined as a shared layer.

[0017] Thus, by calculating the first contribution of the k-th network layer, the k-th network layer is determined to be either a prediction layer or a shared layer, rather than simply assigning the first few layers of the Q network layers to prediction layers and the last few to shared layers. Instead, it dynamically determines whether any one of the Q network layers is a prediction layer or a shared layer based on the third information actually input into the first model. That is, in this embodiment, the first model can dynamically decide for each network layer whether to execute a shared path with low computational and memory overhead or a prediction path with high computational and memory overhead, based on specific circumstances such as the sequence length of the input data. This adaptive capability enables the first model to dynamically adjust the allocation of computing resources according to the task complexity, achieving an optimal allocation of computation and storage. This ensures that the first model maximizes memory utilization efficiency in simple scenarios and retains higher performance in complex scenarios, achieving a balance between efficiency and effectiveness when processing information.

[0018] In some embodiments, generating a second key-value pair based on the first word and the first key-value pair generated by each network layer of the first model, and obtaining second information based on the second key-value pair, includes: Through each network layer of the first model, a second key-value pair of the first word is generated based on the first word and the first key-value pair generated by each network layer; Using the first model, a third word is obtained based on the second key-value pair of the first word generated by the last network layer in the first model; wherein, the third word is the second word in the second information; If the third word is the same as the reference end word, the first word is identified as the second information.

[0019] Thus, after obtaining the first lexical unit, it is not compared with the reference end lexical unit. Instead, the third lexical unit obtained based on the first lexical unit is compared with the reference end lexical unit. This is because after the first model generates the first lexical unit, it has no self-generated context to rely on and must rely purely on the first information for contextual understanding. This may lead to inaccurate contextual understanding. By comparing the third lexical unit obtained based on the first lexical unit with the reference end lexical unit, the third lexical unit and the first lexical unit can have a better contextual understanding. Thus, the second information obtained is more consistent with the first information.

[0020] In some embodiments, after obtaining the third lexical unit, the method further includes: The first key-value pair generated by each network layer and the second key-value pair of the first word generated by the network layer are combined to obtain the combined key-value pair of the network layer. The synthesized key-value pair generated in the h-th iteration of the network layer is determined as the first key-value pair of the network layer in the (h+1)-th iteration. If the third lexical unit is different from the reference end lexical unit, the third lexical unit is input into the first model. Through the network layer of the first model, a second key-value pair is generated based on the third lexical unit and the first key-value pair generated by each network layer at the (h+1)th iteration. Based on the second key-value pair, a fourth lexical unit is obtained. The fourth lexical unit is the third lexical unit in the second information.

[0021] Thus, when the third lexical unit differs from the reference end lexical unit, the third lexical unit is input into the first model. Through the network layers of the first model, a second key-value pair is generated based on the third lexical unit and the first key-value pair generated by each network layer at the (h+1)th iteration. Based on the second key-value pair, the fourth lexical unit is obtained. This process is repeated to continuously update the key-value pairs cached by each network layer. When predicting the next lexical unit, more key-value pairs of historical lexical units can be referenced. The resulting next lexical unit has a higher contextual understanding compared to previous historical lexical units, thereby improving the prediction accuracy of the next lexical unit.

[0022] In some embodiments, before inputting the first hidden state information of each lexical in the first information into the first model, generating a first key-value pair through the network layer of the first model, and obtaining the first lexical based on the first key-value pair, the method further includes: Obtain the hidden state information of each word element in the sample information and the reference information of the sample information; the reference information is the reference response information of the sample information; The hidden state information of each word in the sample information is input into the teacher model. The first teacher key-value pair is generated through the network layer of the teacher model, and the first teacher word is obtained based on the first teacher key-value pair. Through the network layers of the teacher model, a second teacher key-value pair is generated based on the first teacher lexical unit and the first teacher key-value pair generated by each network layer, and teacher information is obtained based on the second teacher key-value pair; The hidden state information of each word in the sample information is input into the student model. The first student key-value pair is generated through the network layer of the student model, and the first student word is obtained based on the first student key-value pair. Through the network layers of the student model, based on the first student word and the first student key-value pairs generated by each network layer, a second student key-value pair is generated, and based on the second student key-value pair, student information is obtained; The student model is trained based on the reference information, the teacher information, the student information, the teacher key-value pairs, and the student key-value pairs to obtain the first model; The teacher model and the student model have the same model structure; the teacher key-value pairs are generated based on the first network layer of the teacher model, and the student key-value pairs are generated based on the second network layer of the student model, with the first network layer and the second network layer corresponding to each other.

[0023] In this way, by using the trained model with frozen parameters as the teacher model and the untrained model as the student model, the student model is trained using sample information. This allows the output of the student model to continuously approach that of the teacher model, resulting in the first model. This avoids the problem of training the student model without a reference, which would lead to inaccurate response information if the student model were used directly. By continuously optimizing the student model and making it closer to the trained teacher model with frozen parameters, the training effect of the student model has a reference. This ensures that the trained student model is consistent with the teacher model, thus improving the reliability of model training.

[0024] In some embodiments, training the student model based on the reference information, the teacher information, the student information, teacher key-value pairs, and student key-value pairs to obtain a first model includes: Based on the teacher information and the student information, the first loss function value is obtained; The second loss function value is obtained based on the teacher key-value pairs and the student key-value pairs; Based on the student information and the reference information, the third loss function value is obtained; The student model is trained based on the first loss function value, the second loss function value, and the third loss function value to obtain the first model.

[0025] Thus, by directly optimizing the student information output by the student model through the first loss function value, aligning it with the teacher information output by the teacher model, it ensures that the student information generated by the student model matches the teacher information output by the teacher model, guaranteeing the correctness of the features represented internally by the student model. The second loss function value requires the student key-value pairs generated by the student model to be as close as possible to the teacher key-value pairs generated by the teacher model. Since key-value pairs are the core of the attention mechanism and directly relate to the model's ability to understand context, forcing alignment of key-value pairs is equivalent to providing guidance on the model's "thinking process" or "intermediate steps," ensuring that even if the student model skips certain attention mechanism calculations, its internally generated intermediate representations remain correct and effective. This fundamentally compensates for the accuracy loss that may result from key-value pair prediction and sharing, ensuring that the semantic space of the key-value pairs predicted by the student model aligns with that generated by the teacher model, allowing the student model to retain the teacher model's capabilities to the maximum extent, and ensuring that the training of the student model does not sacrifice the model's core performance. The third loss function value ensures that the student model's autoregressive language modeling ability is not compromised, thus guaranteeing that the student model's basic capabilities are not unbalanced and preventing the student model from losing its autoregressive computational ability in the process of excessively imitating the teacher model.

[0026] Secondly, embodiments of this application provide an information processing apparatus, which includes: The information input module is used to input the first hidden state information of each word element in the first information into the first model, generate the first key-value pair through the network layer of the first model, and obtain the first word element based on the first key-value pair; The generation module is used to generate a second key-value pair based on the first word and the first key-value pair generated by each network layer of the first model, and to obtain second information based on the second key-value pair; wherein, the first word is the starting word of the second information; the network layers of the first model include a base layer, a prediction layer, and a shared layer; the key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer; the first base layer is the base layer that is ranked first and is closest to the prediction layer; the key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer; the first information is query information, and the second information is the response information to the first information.

[0027] In some embodiments, the information input module is further configured to input the second hidden state information of each word element in the third information into the first model, generate the third hidden state information through the network layer of the first model, and obtain the second word element based on the third hidden state information; wherein, the third information is query information, and the third information is different from the first information; The generation module is further configured to generate fourth hidden state information based on the second lexical unit and the third hidden state information generated by each network layer through the network layer of the first model, and obtain fourth information based on the fourth hidden state information; wherein, the second lexical unit is the starting lexical unit of the fourth information; The device further includes: a first determining module, used to determine the network layer type of each network layer based on the hidden state information generated by each network layer; wherein, the network layer type of any network layer in the first model is a normal layer, a base layer, a prediction layer, or a shared layer.

[0028] In some embodiments, the first determining module is specifically used for: Based on the hidden state information generated by each network layer, determine the average similarity value of the hidden state information of each network layer; Network layers whose average similarity value of the hidden state information is less than a first threshold are defined as ordinary layers. N network layers out of M network layers are determined as the baseline layers; wherein, the average similarity value of the hidden state information of the M network layers is greater than a first threshold; the N network layers are the network layers ranked first in the first model; M and N are both positive integers, and N < M; Q of the M network layers are determined as prediction layers or shared layers; the Q network layers are the network layers other than the reference layer among the M network layers; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

[0029] In some embodiments, the first determining module is specifically used for: Calculate the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information to obtain P first similarity values; wherein, the fifth hidden state information is the hidden state information generated by the g-th network layer, the P sixth hidden state information is the hidden state information generated by P network layers, and the P network layers are the P network layers in the first model located after the g-th network layer, where g and P are positive integers; When P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; Wherein, the g-th network layer is any network layer in the first model.

[0030] In some embodiments, the information input module is specifically used for: Based on the first and second learning weights corresponding to the i-th network layer in the first model, and the first key-value pair generated by the (i-1)-th network layer, the first key-value pair of the i-th network layer is obtained; wherein, the i-th network layer is a base layer or a normal layer; i is a positive integer and i≥1; when i=1, the first key-value pair generated by the (i-1)-th network layer is the key vector and value vector in the first hidden state information; Based on the first learning weight and the second learning weight corresponding to the j-th network layer in the first model, and the first key-value pair generated by the second reference layer, the first key-value pair of the j-th network layer is obtained; the second reference layer is the reference layer that is closest to the j-th network layer in the sorted order, and the j-th network layer is the prediction layer; j is a positive integer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information.

[0031] In some embodiments, the first determining module is specifically used for: Based on the first, second, and third learning weights corresponding to the k-th network layer, the hidden state information generated by the (k-1)-th network layer, the sequence length of the third information, and the embedding vector of the third information, the first contribution of the k-th network layer is obtained; wherein, the k-th network layer is any one of the Q network layers; the first contribution is used to indicate the probability value that the k-th network layer is a prediction layer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information; and the third learning weight is the learning weight of the query vector in the hidden state information. If the first contribution is greater than the second threshold, the kth network layer is determined as the prediction layer; If the first contribution is less than or equal to the second threshold, the kth network layer is determined as a shared layer.

[0032] In some embodiments, the generation module is specifically used for: Through each network layer of the first model, a second key-value pair of the first word is generated based on the first word and the first key-value pair generated by each network layer; Using the first model, a third word is obtained based on the second key-value pair of the first word generated by the last network layer in the first model; wherein, the third word is the second word in the second information; If the third word is the same as the reference end word, the first word is identified as the second information.

[0033] In some embodiments, the apparatus further includes: A synthesis module is used to synthesize the first key-value pair generated by each network layer and the second key-value pair of the first word generated by the network layer to obtain the synthesized key-value pair of the network layer. The second determining module is used to determine the synthesized key-value pair generated by the h-th iteration of the network layer as the first key-value pair of the network layer at the (h+1)-th iteration. The generation module is further configured to, when the third lexical element is different from the reference end lexical element, input the third lexical element into the first model, generate a second key-value pair through the network layer of the first model based on the third lexical element and the first key-value pair generated by each network layer at the (h+1)th iteration, and obtain a fourth lexical element based on the second key-value pair; the fourth lexical element is the third lexical element in the second information.

[0034] In some embodiments, the apparatus further includes: The acquisition module is used to acquire the hidden state information of each word in the sample information and the reference information of the sample information; the reference information is the reference response information of the sample information. The information input module is also used to input the sample latent state information of each word in the sample information into the teacher model, generate the first teacher key-value pair through the network layer of the teacher model, and obtain the first teacher word based on the first teacher key-value pair; The generation module is further configured to generate a second teacher key-value pair based on the first teacher lexical and the first teacher key-value pair generated by each network layer through the network layer of the teacher model, and obtain teacher information based on the second teacher key-value pair; The information input module is also used to input the sample hidden state information of each word in the sample information into the student model, generate the first student key-value pair through the network layer of the student model, and obtain the first student word based on the first student key-value pair; The generation module is further configured to generate a second student key-value pair based on the first student word and the first student key-value pair generated by each network layer through the network layer of the student model, and obtain student information based on the second student key-value pair; The model training module is used to train the student model based on the reference information, the teacher information, the student information, the teacher key-value pairs, and the student key-value pairs to obtain a first model; wherein the teacher model and the student model have the same model structure; the teacher key-value pairs are generated based on the first network layer of the teacher model, and the student key-value pairs are generated based on the second network layer of the student model, and the first network layer and the second network layer correspond to each other.

[0035] In some embodiments, the model training module is specifically used for: Based on the teacher information and the student information, the first loss function value is obtained; The second loss function value is obtained based on the teacher key-value pairs and the student key-value pairs; Based on the student information and the reference information, the third loss function value is obtained; The student model is trained based on the first loss function value, the second loss function value, and the third loss function value to obtain the first model.

[0036] Thirdly, embodiments of this application provide an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can run on the processor, and the programs or instructions, when executed by the processor, implement the method as described in the first aspect.

[0037] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the method described in the first aspect.

[0038] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0039] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0040] In this embodiment, the first hidden state information of each word in the first information is input into the first model. The network layer of the first model generates a first key-value pair, and the first word is obtained based on the first key-value pair. Then, the network layer of the first model generates a second key-value pair based on the first word and the first key-value pair generated by each network layer, and the second information is obtained based on the second key-value pair. The network layer of the first model includes a base layer, a prediction layer, and a shared layer. The key-value pair generated by the prediction layer is based on the key-value pair generated by the first base layer, which is the closest to the prediction layer in the order. The key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer. Thus, during the processing of the first information by the network layer of the first model, the key-value pair generated by the prediction layer can be predicted based on the hidden state information generated by the base layer, which is the closest to the prediction layer in the order. The prediction layer does not need to perform the hidden state information generation process, but only needs to calculate the key-value pair, thereby reducing the amount of computation. Furthermore, the key-value pairs generated by the shared layer are the same as those generated by the preceding network layer adjacent to the shared layer. In this way, the shared layer does not need to perform the key-value pair generation process and can directly reuse the key-value pairs generated by the preceding network layer, thereby improving the calculation speed of key-value pairs and thus improving the efficiency of obtaining response information.

[0041] Understandably, the information processing apparatus provided in the second aspect, the electronic device provided in the third aspect, the readable storage medium provided in the fourth aspect, the chip provided in the fifth aspect, and the computer program product provided in the sixth aspect are all used to execute the method provided in this application. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods, and will not be repeated here. Attached Figure Description

[0042] Figure 1 This is a flowchart illustrating the information processing method provided in some embodiments of this application; Figure 2 These are schematic diagrams of AI assistant interfaces provided in some embodiments of this application; Figure 3 This is a schematic diagram of the division of network layers in the first model provided by some embodiments of this application; Figure 4 This is a schematic diagram of the division of network layers in the first model provided by some embodiments of this application; Figure 5 This is a schematic diagram of the average similarity values ​​of the hidden state information of each network layer provided in some embodiments of this application; Figure 6 These are schematic diagrams illustrating the structure of an information processing apparatus according to some embodiments of this application; Figure 7 These are schematic diagrams illustrating the structure of an electronic device according to some embodiments of this application; Figure 8 It is a schematic diagram of the hardware structure of an electronic device shown in some embodiments of the present application. Specific embodiments

[0043] Next, the technical solutions in the embodiments of the present application will be clearly described in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art belong to the scope of protection of the present application.

[0044] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, rather than to describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are usually of the same type, and the number of objects is not limited. For example, the first object can be one or N. In addition, "and / or" in the specification and claims means at least one of the connected objects, and the character " / " generally represents an "or" relationship between the associated objects before and after.

[0045] The terms used in the embodiments section of the present application are only used to explain the specific embodiments of the present application, rather than to limit the present application. The terms related to the embodiments of the present invention will be explained below.

[0046] Lexeme: In natural language processing, a lexeme is the basic unit after the text is segmented and processed by the model.

[0047] Lexicalization: It refers to the process of splitting a text sequence into a series of lexemes. For example, for the query information "What's the weather like today", it is divided into a series of lexemes, and the obtained lexemes are respectively "今", "天", "天", "气", "怎", "么", and "样".

[0048] Hidden state information: In the Transformer model, hidden state information refers to the intermediate data generated during the calculation of each layer of the model. Hidden state information is a high-dimensional, dense vector or matrix, which represents the "understanding" and "representation" of the information in the input model by the model, and carries rich semantic information of the information in the input model.

[0049] Large Language Models (LLMs) are deep learning models trained on massive amounts of text data, enabling them to generate natural language text or understand the meaning of language text. These models can provide in-depth knowledge and language production on a wide range of topics through training on large datasets. The core idea is to learn patterns and structures of natural language through large-scale unsupervised training, mimicking human language cognition and generation processes to some extent. Large language models employ a similar Transformer architecture and pre-training objectives as small models; the main difference lies in increasing model size, training data, and computational resources. Compared to traditional Natural Language Processing (NLP) models, large language models are better able to understand and generate natural text, while also exhibiting a certain level of logical thinking and reasoning ability.

[0050] The technical solution of this application embodiment can be applied to scenarios in large language models where user-input query information is analyzed to obtain corresponding response information. For example, if a user wants to know today's weather, they input the text query "What's the weather like today?" into the Artificial Intelligence (AI) assistant interface.

[0051] The information processing method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0052] Figure 1 This is a flowchart illustrating an information processing method provided in an embodiment of this application. The subject executing this information processing method can be an electronic device, which may be, but is not limited to, a personal computer (PC), a smartphone, a tablet computer, or a personal digital assistant (PDA).

[0053] like Figure 1 As shown, the information processing method provided in this application embodiment may include steps 110-130.

[0054] Step 110: Input the first hidden state information of each word in the first information into the first model, generate the first key-value pair through the network layer of the first model, and obtain the first word based on the first key-value pair.

[0055] The first information can be a query input by the user, such as the query the user wants to ask. This first information can be in the form of at least one of text, speech, etc. If the first information is speech, it needs to be converted into text, and then the first hidden state information of each word in the converted text is input into the first model. Taking text as an example, this first information can be as described in the example above, referencing... Figure 2 The user enters the text query "How is the weather today?" into the AI ​​assistant interface 21 22.

[0056] It should be noted that the aforementioned AI assistant interface can be displayed within an operating system-managed interface using component services provided by the operating system. Specifically, the operating system can be a Linux system. The component services can be, but are not limited to, built using the Spring framework, Spring Boot framework, or ToySpring framework.

[0057] For each word in the first information, the first hidden state information of that word can be a matrix obtained after embedding lookup and position encoding. Specifically, the method for determining the first hidden state information of each word will be described in detail in later embodiments.

[0058] The first model mentioned above can be a pre-trained model used to process the first hidden state information corresponding to each word in the first information. This first model can be, for example, a Decoder-only Transformer architecture model, or a large language model.

[0059] The aforementioned first key-value pair can be the key-value pair of each word obtained by processing the first hidden state information of each word in the first information based on the network layer in the first model. That is, for each word in the first information, the network layer of the first model will generate a key vector, a value vector, and a query vector corresponding to each word. For a word, its corresponding first key-value pair is the key vector and the value vector of that word.

[0060] For a given word, after obtaining its corresponding key vector, value vector, and query vector, self-attention can be calculated. Specifically, a dot product can be performed on the query vector and key vector of the word, and the hidden state information of the word can be obtained based on the result of the dot product calculation. Specifically, how the network layer in the first model processes the first hidden state information of each word in the first information to obtain the first key-value pair corresponding to each word will be described in detail in later embodiments.

[0061] The first token can be the starting token in the second information, where the second information can be the response information of the first information. In the above example, taking the first information as "What's the weather like today", if the response information output by the first model is "The weather is great today", then the first token is the starting token "今" in "The weather is great today".

[0062] In some embodiments of the present application, the first token can be determined based on the first key-value pair generated by the last network layer in the first model. Specifically, the first token can be obtained through the following method: First, obtain the first key-value pair generated by the last network layer of the first model. Specifically, obtain the first key-value pair of the last token in the first information generated by the last network layer of the first model. For example, the first key-value pair of the token "样" in the above example. The first key-value pair of the token "样" represents the comprehensive semantic representation of "what should be said next" formed by the first model after deeply understanding the entire first information "What's the weather like today".

[0063] Then, process the first key-value pair of the last token in the first information through the learning weight matrix corresponding to the linear output layer in the first model to obtain a vector with a dimension of 1 row and V columns, where V is the number of all possible tokens in the vocabulary of the first model. The vocabulary of the first model contains tens of thousands of tokens. Each element in this 1-row V-column vector corresponds to the raw score of a specific token in the vocabulary. Apply the Softmax function to the above 1-row V-column vector to convert the raw score represented by each element in the 1-row V-column vector into a probability distribution, and obtain a probability distribution vector, which is also a 1-row V-column vector. The value at each position in the probability distribution vector represents the possibility of the corresponding token being the starting token in the second information, and the sum of all values in the probability distribution vector is 1.

[0064] In the above example, assume that the vocabulary includes three tokens: "今", "天", and "是". The probability that the token "今" in the vocabulary is the starting token in the second information is 0.8, the probability that the token "天" in the vocabulary is the starting token in the second information is 0.2, and the probability that the token "是" in the vocabulary is the starting token in the second information is 0.05. Then, it can be determined that the token "今" in the vocabulary is the starting token in the second information.

[0065] In some embodiments of the present application, before step 110, the above-mentioned method may further include: Tokenize the first information to obtain P tokens; For each of the P tokens, look up the embedding vector corresponding to the token in the preset embedding table to obtain the embedding vector corresponding to each token; According to the position information of P tokens in the first piece of information, perform position encoding on each of the P tokens to obtain the position vector of each token; According to the embedding vector and position vector corresponding to each token, obtain the first hidden state information corresponding to each token.

[0066] Among them, the preset embedding table can be a pre-trained semantic embedding table, which contains semantic feature vectors of each token in different dimensions. For example, for the token "今", the preset embedding table contains semantic feature vectors of this token "今" in different dimensions. For example, the preset embedding table contains semantic feature vectors of this token "今" in different dimensions such as the sentiment dimension, the domain dimension, and the词性 dimension. The above sentiment dimension can be the sentiment represented by the token, such as whether the token represents a positive sentiment or a negative sentiment. The domain dimension can be the domain represented by the token, such as whether the token is in the literature field or the financial field. The词性 dimension can be the词性 represented by the token, such as whether the token is a noun or a verb, etc.

[0067] For a token, its corresponding embedding vector can be obtained by concatenating the semantic feature vectors of this token in different dimensions found in the preset embedding table. For example, in the above example, the dimension of the token in the preset embedding table is 2560, that is, for a certain token, the preset embedding table contains semantic feature vectors of this token in 2560 dimensions. For example, for the token "今" in "今天天气怎么样", the semantic feature vectors of it in 2560 dimensions can be obtained respectively, and the semantic feature vectors of it in 2560 dimensions are concatenated to obtain a vector with 1 row and 2560 columns. This vector with 1 row and 2560 columns is the embedding vector of the token "今".

[0068] For a token, its corresponding position vector can be a vector generated according to the ordinal number of this token in its corresponding sequence. For example, in the above example, for the token "今" in "今天天气怎么样", it is in the first position in "今天天气怎么样", then according to the position of the token "今" in "今天天气怎么样", perform position encoding to obtain the position vector of the token "今".

[0069] In some embodiments of the present application, the first piece of information can be tokenized first to obtain P tokens, and then for each of the P tokens, look up the embedding vector corresponding to the token in the preset embedding table to obtain the embedding vector corresponding to each token, perform position encoding on the P tokens to obtain the position vector of each token, and according to the embedding vector and position vector corresponding to each token, obtain the first hidden state information corresponding to the first piece of information. Specifically, it can be to add the embedding vector and position vector corresponding to each token to obtain the first hidden state information corresponding to each token.

[0070] It should be noted that the dimension of the position vector of a token is the same as the dimension of the embedding vector, so that the position vector and the embedding vector of the token can be added to obtain the first hidden state information corresponding to the token. Specifically, the vector elements at the same position in the position vector and the embedding vector can be added. For example, the element in the first row and first column of the position vector is added to the element in the first row and first column of the embedding vector to obtain the element in the first row and first column of the first hidden state information of the token.

[0071] Continuing to refer to the above example, taking the first information as "What's the weather like today" and the dimension of the embedding vector in the preset embedding table as 2560, the first information "What's the weather like today" is tokenized to obtain 7 tokens, which are the tokens "今", "天", "天", "气", "怎", "么", and "样". Then, for each of the tokens "今", "天", "天", "气", "怎", "么", and "样", such as for the token "今", the corresponding 2560-dimensional embedding vector is found from the preset embedding table to obtain the embedding vector of the token "今", which is a 1-row and 2560-column vector. Correspondingly, for the other tokens in the first information "What's the weather like today" except the token "今": the tokens "天", "天", "气", "怎", "么", and "样" also use the same method as the token "今" to obtain the embedding vectors corresponding to the tokens "天", "天", "气", "怎", "么", and "样" respectively.

[0072] Then, according to the ordinal position of the token "今" in the first information "What's the weather like today", the position vector of the token "今" is generated, which is also a 1-row and 2560-column vector. Correspondingly, for the other tokens in the first information "What's the weather like today" except the token "今": the tokens "天", "天", "气", "怎", "么", and "样" also use the same method as the token "今" to obtain the position vectors corresponding to the tokens "天", "天", "气", "怎", "么", and "样" respectively.

[0073] Then, add the embedding vector corresponding to the token "今" and the position vector to obtain the hidden state information of the token "今". For example, add the element in the first row and first column of the embedding vector corresponding to the token "今" to the element in the first row and first column of the position vector corresponding to the token "今" to obtain the element in the first row and first column of the first hidden state information of the token "今". Add the element in the first row and second column of the embedding vector corresponding to the token "今" to the element in the first row and second column of the position vector corresponding to the token "今" to obtain the element in the first row and second column of the first hidden state information of the token "今", and so on, until adding the element in the first row and 2560th column of the embedding vector corresponding to the token "今" to the element in the first row and 2560th column of the position vector corresponding to the token "今" to obtain the element in the first row and 2560th column of the first hidden state information of the token "今".

[0074] In some embodiments of the present application, when inputting the first hidden state information of each token in the first information into the first model, it can be inputting the first hidden state information of each token in the first information into the first model in sequence. For example, in the above example, first input the first hidden state information corresponding to the first token "今" in the first information "今天天气怎么样" into the first model, and then input the first hidden state information corresponding to the second token "天" in the first information "今天天气怎么样" into the first model, and so on, until after inputting the first hidden state information corresponding to the last token "样" in the first information "今天天气怎么样" into the first model, generate the first key-value pair through the network layer of the first model.

[0075] In some embodiments of the present application, after generating the hidden state information of each token in the first information through the network layer of the first model, the key vector and value vector in the hidden state information of each token can be cached, that is, the first key-value pair of each token is cached, so as to facilitate the subsequent network layer to calculate the first key-value pair of each token.

[0076] Step 120: Through the network layer of the first model, generate the second key-value pair based on the first token and the first key-value pair generated by each network layer, and obtain the second information based on the second key-value pair; wherein, the types of the network layer of the first model include a reference layer, a prediction layer, and a shared layer; the key-value pair generated by the prediction layer is obtained based on the key-value pair generated by the first reference layer; the first reference layer is the reference layer ranked first and closest to the prediction layer; the key-value pair generated by the shared layer is the key-value pair generated by the previous network layer adjacent to the shared layer; the first information is an inquiry information, and the second information is a reply information of the first information.

[0077] Specifically, for each network layer in the first model, the second key-value pair generated by the network layer can be the key-value pair of each word in the second information generated by the network layer through the first word and the first key-value pair generated by the network layer.

[0078] The second information can be a response to the first information output by the first model. For example, in the example above, refer to... Figure 2 Taking the first information as "How is the weather today?" as an example, if the first information "How is the weather today?" is processed by the first model and the reply information obtained is "The weather is very good today", then the information "The weather is very good today" 23 displayed in the AI ​​assistant interface 21 is the second information.

[0079] In some embodiments of this application, a second key-value pair can be generated through the network layer of the first model, based on the first word and the first key-value pair generated by each network layer, and second information can be obtained based on the second key-value pair. The specific process of generating the second key-value pair and the process of obtaining the second information based on the second key-value pair will be described in detail in the following embodiments.

[0080] It should be noted that step 120 above is the decoding stage of the first model. This decoding stage processes the first lexical unit generated in the pre-filling stage to predict subsequent lexical units, thereby obtaining the second information.

[0081] In some embodiments of this application, in steps 110 and 120 above, the network layer type of the first model may include a base layer, a prediction layer, and a shared layer. The key-value pairs generated by the prediction layer are obtained based on the hidden state information generated by the first base layer, where the first base layer is the one ranked first and closest to the prediction layer. The key-value pairs generated by the shared layer are the key-value pairs generated by the preceding network layer adjacent to the shared layer. That is, in the process of generating the first key-value pairs, the network layer type of the first model includes a base layer, a prediction layer, and a shared layer. The first key-value pairs generated by the prediction layer are obtained based on the first key-value pairs generated by the first base layer, where the first base layer is the one ranked first and closest to the prediction layer. The first key-value pairs generated by the shared layer are the first key-value pairs generated by the preceding network layer adjacent to the shared layer. In the process of generating the second key-value pair, the network layers of the first model also include a base layer, a prediction layer, and a shared layer. The second key-value pair generated by the prediction layer is based on the second key-value pair generated by the first base layer. Here, the first base layer is the base layer that is closest to the prediction layer in the order of the first base layer. The second key-value pair generated by the shared layer is the second key-value pair generated by the network layer that is adjacent to the shared layer.

[0082] It should be noted that the network layers of the first model described above are arranged in a continuous topology, such as... Figure 3As shown, taking the first model as an example with a total of 32 network layers, if the 24th layer is the base layer, then any network layer in the 25th-32nd layers can be a prediction layer or a shared layer. The specifics of whether any network layer in the 25th-32nd layers is a prediction layer or a shared layer will be described in detail in the later embodiments.

[0083] Continue with Figure 3 The network layer structure shown is in Figure 3 In this example, taking layer 24 as the base layer, layers 25 and 29 as prediction layers, and layers 26-28 and 30-32 as shared layers, layer 25 is the prediction layer. The base layer preceding and closest to layer 25 is layer 24. Therefore, the key-value pairs generated by layer 25 are based on the hidden state information generated by layer 24. Figure 3 The 29th layer of the network is also a prediction layer. The reference layer that is closest to the 29th layer is the 24th layer. Therefore, the key-value pairs generated by the 29th layer are also obtained based on the hidden state information generated by the 24th layer.

[0084] exist Figure 3 In this model, layer 26 is a shared layer, and the preceding layer adjacent to layer 26 is layer 25. Therefore, the key-value pairs generated by layer 26 are the same as those generated by layer 25. Layer 27 is also a shared layer, and the preceding layer adjacent to layer 27 is layer 26. Therefore, the key-value pairs generated by layer 27 are the same as those generated by layer 26. Correspondingly, for layer 28, the key-value pairs generated are the same as those generated by layer 27; for layer 30, the key-value pairs generated are the same as those generated by layer 29; for layer 31, the key-value pairs generated are the same as those generated by layer 30; and for layer 32, the key-value pairs generated are the same as those generated by layer 31.

[0085] Thus, in response to the problems of high generation latency of starting words and large memory consumption of key-value pair caching in large language models when processing long sequence input information in related technologies, in this embodiment, for any network layer in the first model, when generating key-value pairs, the key-value pairs generated by the prediction layer can be predicted based on the hidden state information generated by the baseline layer that is ranked first and closest to the prediction layer. The prediction layer does not need to perform the hidden state information generation process, but only needs to calculate the key-value pairs, thereby reducing the amount of computation. Furthermore, the key-value pairs generated by the shared layer are those generated by the preceding network layer adjacent to the shared layer. Thus, the shared layer can directly reuse the key-value pairs generated by the preceding network layer without performing the key-value pair generation process, improving the computation speed of key-value pairs in the pre-filling and decoding stages. This, in turn, improves the efficiency of generating starting words in the response information, thereby improving the efficiency of obtaining the response information. In other words, by allowing some deep network layers to bypass the complete self-attention and pre-feedback computations, and employing low-cost key-value pair caching prediction or zero-cost key-value pair caching sharing, the total computational load required for processing long input sequences is significantly reduced, improving the efficiency of generating starting words in the response information of the query. Moreover, for any network layer in the first model, after generating key-value pairs, the network layer can cache the key-value pairs it generates. For the shared layer, it can directly reuse the key-value pairs of the preceding network layer, thereby reducing the memory footprint of the key-value pair cache in electronic devices. Thus, the solution of this embodiment significantly reduces the computational complexity and memory requirements in the response information generation process.

[0086] In some embodiments of this application, in order to determine the function of each network layer in the first model, the method described above may further include, prior to step 110: The second hidden state information of each word element in the third information is input into the first model. The third hidden state information is generated through the network layer of the first model, and the second word element is obtained based on the third hidden state information. Through the network layers of the first model, based on the second output lexical units and the third hidden state information generated by each network layer, the fourth hidden state information is generated, and based on the fourth hidden state information, the fourth information is obtained. Based on the hidden state information generated by each network layer, the network layer type of each network layer is determined; wherein, the network layer type of any network layer in the first model is a normal layer, a baseline layer, a prediction layer, or a shared layer.

[0087] The third piece of information can be a query that is different from the first piece of information. For example, the third piece of information can be the query "What is the UV intensity today?"

[0088] For each word in the third information, the second hidden state information of that word can be a matrix obtained after embedding lookup and position encoding. The method for determining the second hidden state information of each word in the third information is the same as the method for determining the first hidden state information of each word in the first information in the above embodiment, and will not be repeated here.

[0089] For any network layer in the first model, the network layer processes each word in the third information to generate the third hidden state information for each word.

[0090] The second lexical unit mentioned above is the starting lexical unit of the fourth information, which is the response information to the third information. The process of generating the second lexical unit based on the third hidden state information is the same as the process of generating the first lexical unit based on the first key-value pair in the above embodiments, and will not be repeated here.

[0091] For each network layer in the first model, the fourth hidden state information generated by the network layer can be the hidden state information of each word in the second information generated by the network layer through the second word and the third hidden state information generated by the network layer.

[0092] In some embodiments of this application, the second hidden state information of each word in the third information can be input into the first model, and the third hidden state information can be generated through the network layer of the first model. Based on the third hidden state information, the second word can be obtained. Then, based on the second output word and the third hidden state information generated by each network layer, the fourth hidden state information can be generated through the network layer of the first model. Based on the fourth hidden state information, the fourth information can be obtained.

[0093] Continuing with the example above, with the third piece of information being "What is the UV intensity today?" and the fourth piece of information being "Today's UV level is low, you can go out with peace of mind", the number of network layers in the first model is... Figure 3 Taking the 32-layer example, the second hidden state information of each word element in the third information "What is the UV intensity today?" is input into the first model. Through each network layer of the first model, the third hidden state information of each word element in the third information "What is the UV intensity today?" can be generated. Then, based on the third hidden state information of each word element in the third information "What is the UV intensity today?" generated by the 32nd layer network layer, the starting word element "today" in the fourth information "The UV level is low today, so you can travel with peace of mind" can be obtained.

[0094] Then, according to the starting token "today" in the fourth piece of information "Today's UV index is at a low level, so you can travel with confidence" and the third hidden state information of each token in the third piece of information "What is the UV intensity today" generated by each network layer, the fourth hidden state information of the starting token "today" can be generated. Furthermore, based on the fourth hidden state information of the starting token "today" generated by each network layer, the fourth piece of information "Today's UV index is at a low level, so you can travel with confidence" can be obtained.

[0095] In some embodiments of the present application, after obtaining the hidden state information generated by each network layer, the network layer type of each network layer can be determined according to the hidden state information generated by each network layer. That is, for each network layer in the first model, whether it is a normal layer, a reference layer, a prediction layer, or a shared layer.

[0096] It should be noted that in the process of generating the third hidden state information through the network layer of the first model and obtaining the second token based on the third hidden state information, the determined network layer type of each network layer is not exactly the same as that in the process of generating the fourth hidden state information through the network layer of the first model based on the second token and the third hidden state information generated by each network layer, and obtaining the fourth piece of information based on the fourth hidden state information.

[0097] Continue to refer to Figure 3 , in Figure 3 In the network layer structure shown, in the process of generating the third hidden state information through the network layer of the first model and obtaining the second token based on the third hidden state information, the determined network layer type of the network layer may be that the 24th layer is a reference layer, the 25th and 29th network layers are prediction layers, and the 26th - 28th network layers and the 30th - 32nd network layers are shared layers. However, in the process of generating the fourth hidden state information through the network layer of the first model based on the second token and the third hidden state information generated by each network layer, and obtaining the fourth piece of information based on the fourth hidden state information, the determined network layer type of each network layer may be as shown in Figure 4 : the 24th layer is a reference layer, the 25th - 27th network layers are prediction layers, and the 28th - 32nd network layers are shared layers.

[0098] Although Figure 3 and Figure 4While the network types of each network layer are not entirely identical, the method for determining the network layer type of each network layer during the process of generating the third hidden state information through the network layers of the first model and obtaining the second word based on the third hidden state information is the same as the method for determining the network layer type of each network layer during the process of generating the fourth hidden state information based on the second word and the third hidden state information generated by each network layer through the network layers of the first model and obtaining the fourth information based on the fourth hidden state information. The specific method for determining the network layer type of each network layer based on the hidden state information generated by each network layer will be described in detail in later embodiments.

[0099] In the embodiments of this application, before using the first model, the network layer type of each network layer in the first model is objectively determined in advance using third information that is different from the first information and the corresponding response information, rather than determining the network layer type of each network layer based on user-defined criteria. This improves the objectivity of determining the network layer type of each network layer in the first model. Furthermore, pre-determining the network layer type of each network layer in the first model avoids the problems of low efficiency in obtaining the first word element and low efficiency in obtaining the second information caused by determining the network layer type of each network layer in the first model during actual use.

[0100] In some embodiments of this application, in order to accurately determine the network layer type of each network layer, the step of determining the network layer type of each network layer based on the hidden state information generated by each network layer may specifically include: Based on the hidden state information generated by each network layer, determine the average similarity value of the hidden state information of each network layer; Network layers whose average similarity value of hidden state information is less than the first threshold are defined as ordinary layers; N of the M network layers are selected as the baseline layer; Q out of the M network layers are designated as prediction layers or shared layers.

[0101] In this context, for any given network layer, the average similarity value of the hidden state information corresponding to that network layer can be determined based on the hidden state information generated by that network layer and the hidden state information generated by the network layers located after that network layer in the first model. The specific method for determining the average similarity value of the hidden state information of each network layer will be described in detail in the following embodiments.

[0102] The first threshold can be a pre-set threshold for the average similarity value of the hidden state information. The value of the first threshold can be greater than or equal to 0.85. For example, the first threshold can be 0.85 or 0.9. The specific value of the first threshold can be set by the user according to their needs, and is not limited in this embodiment.

[0103] The M network layers can be network layers whose average hidden state information similarity value is greater than a first threshold; that is, the average hidden state information similarity value of the M network layers is greater than the first threshold. The N network layers can be the network layers ranked first in the first model. M and N are both positive integers, and N < M.

[0104] The Q network layers can be any of the M network layers excluding the base layer; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

[0105] In some embodiments of this application, the average similarity value of the hidden state information of each network layer can be determined first based on the hidden state information generated by each network layer. Then, the network layer whose average similarity value of the hidden state information is less than a first threshold is determined as a normal layer. The N network layers ranked first among the M network layers whose average similarity value of the hidden state information is greater than or equal to the first threshold are determined as the reference layer. The Q network layers other than the reference layer among the M network layers are determined as prediction layers or shared layers.

[0106] In some embodiments of this application, the value of N can be a positive integer, but N < M, because the base layer generates hidden state information in the same way as the ordinary layer, and both need to perform the complete hidden state information generation process. Therefore, in order to save computing resources, in this embodiment of the application, N can be set to 1, that is, the base layer has only 1 layer.

[0107] Continue to refer to Figure 2 Taking a first threshold of 0.85 and N=1 as an example, if it is determined that the average similarity value of the hidden state information of layers 1-23 is less than 0.85, and the average similarity value of the hidden state information of layers 24-32 is greater than 0.85, i.e., M=9, then... Figure 3 As shown, layers 1-23 are designated as ordinary layers, and the first layer in layers 24-32 is designated as the baseline layer (layer 24). Any layer in layers 25-32 is designated as either a prediction layer or a shared layer. The specific layer type (i.e., layer 25-32) is determined as follows: Figure 3 The confirmed result shown is still as follows: Figure 4 The determination results shown can be determined according to the actual situation in the later embodiments.

[0108] In the embodiments of this application, since the hidden state information of the shallow network layers at the beginning of the sequence is more inclined to capture basic, general grammatical and semantic features, their semantic representations are relatively stable and reliable. Choosing these network layers as the base layers means that the "source data" used to predict the deep key-value pair cache is of higher quality and more fundamental, reducing prediction errors introduced by the abstract or unstable representations of the base layers themselves. On the other hand, the hidden state information of the deep network layers at the end of the sequence is more inclined to high-level, abstract semantic reasoning. The semantic representations in these deep network layers are redundant. Therefore, predicting or sharing this part is the most cost-effective. Choosing these network layers as prediction layers or sharing layers is equivalent to accurately optimizing regions with high redundancy and high computational cost, avoiding potentially risky approximate calculations in the key shallow feature extraction stage. The prediction layer undertakes the main task of saving computation by bypassing the complete self-attention calculation through prediction, significantly reducing latency. The sharing layer undertakes the main task of saving memory by directly reusing the key-value pair cache of the prediction layer, without performing the complete self-attention calculation or adding storage, achieving dual "zero-cost" optimization of computation and memory.

[0109] In some embodiments of this application, determining the average similarity value of the hidden state information of each network layer based on the hidden state information generated by each network layer may specifically include: Calculate the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information to obtain P first similarity values; When P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; Wherein, the g-th network layer can be any network layer in the first model, for example, the g-th network layer can be as follows: Figure 3 The 29th network layer in the system.

[0110] The fifth hidden state information can be the hidden state information generated by the g-th network layer. The P sixth hidden state information are the hidden state information generated by P network layers, where P network layers are the P network layers following the g-th network layer in the first model, and g and P are positive integers. (Continue to refer to...) Figure 3 In the g-th network layer is Figure 3 In the case of the 29th network layer, P network layers are Figure 3 In the 30th-32nd layers of the network, the P sixth hidden state information are the hidden state information generated by the 30th-32nd layers of the network respectively.

[0111] For any one of the P first similarities, the first similarity can be the similarity between the fifth hidden state information of the g-th network layer and any one of the P sixth hidden state information.

[0112] In some embodiments of this application, after generating hidden state information in each network layer, the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information can be calculated to obtain P first similarity values. Then, when P equals 1, the first similarity is directly used as the average similarity value of the hidden state information of the g-th network layer.

[0113] Continue to refer to Figure 3 In the g-th network layer is Figure 3 In the case of the 31st network layer, P network layers are Figure 3 If the value of P is 1 for the 32nd network layer, then the first similarity between the hidden state information generated by the 31st network layer and the hidden state information generated by the 32nd network layer can be calculated. This first similarity can be determined as the average similarity value of the hidden state information of the 31st network layer.

[0114] When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer. (Continue to refer to...) Figure 3 In the g-th network layer is Figure 3 In the case of the 29th network layer, P network layers are Figure 3 In the network layers 30-32, where P is 3, the first similarity between the hidden state information generated by the 29th network layer and the hidden state information generated by the 30th network layer is calculated. The first similarity between the hidden state information generated by the 29th network layer and the hidden state information generated by the 31st network layer is also calculated. The first similarity between the hidden state information generated by the 29th network layer and the hidden state information generated by the 32nd network layer is also calculated. This yields three first similarity values. The average of these three first similarity values ​​is then taken as the average similarity value of the hidden state information of the 29th network layer.

[0115] That is, when calculating the average similarity value of the hidden state information of each network layer, it is calculated according to the following formula (1).

[0116] In the above formula (1), This represents the total number of network layers in the first model. For the current network layer, This is the hidden state information of the current network layer, specifically the fifth hidden state information. For the first The hidden state information of a network layer, specifically the sixth hidden state information of any one of the P network layers. , For calculation and The similarity between two pieces of information.

[0117] For the first model, the average similarity value of the hidden state information corresponding to each network layer can be obtained according to the above formula (1). By fitting the average similarity value of the hidden state information corresponding to each network layer, the following can be obtained: Figure 5 The diagram shown illustrates the average similarity value of the hidden state information for each network layer. Figure 5 It can be seen that the later the network layer in the first model, the higher the average similarity value of its hidden state information. This indicates that the semantic information obtained between the later network layers in the first model is also more similar. Thus, the network layer type in the first model can be determined based on this principle.

[0118] In the embodiments of this application, the network layer type is determined by the average similarity value of the hidden state information corresponding to each network layer. This can accurately quantify the redundancy of the semantic information represented within the first model. Similarity analysis can accurately identify which network layers in the first model have highly redundant outputs and which have unique and necessary outputs, providing strong support for determining the network layer type in the first model. Furthermore, based on the determined network layer type, the hidden state information of each word in the first information can be calculated and stored, making the computation and storage resource investment of the first model more "cost-effective" and avoiding waste of computation and storage resources.

[0119] In some embodiments of this application, in order to achieve the optimal allocation of computation and storage for the first model, determining Q network layers out of the M network layers as prediction layers or shared layers may specifically include: The first contribution of the kth network layer is obtained based on the first and second learning weights corresponding to the kth network layer, the hidden state information generated by the (k-1)th network layer, the sequence length of the third information, and the embedding vector of the third information. If the first contribution is greater than the second threshold, the k-th network layer is determined as the prediction layer; If the first contribution is less than or equal to the second threshold, the k-th network layer is determined as a shared layer.

[0120] Here, the first learned weight corresponding to the k-th network layer is the learned weight used by the k-th network layer when generating the key vector in the hidden state information. The second learned weight corresponding to the k-th network layer is the learned weight used by the k-th network layer when generating the value vector in the hidden state information. The k-th network layer mentioned above can be any one of the Q network layers.

[0121] The sequence length of the third information can be the length of the text sequence of the third information, and the sequence length of the third information can be the number of third information tokens. For example, in the example above, if the third information is "what is the UV intensity today", then the sequence length of the third information is 10.

[0122] The embedding vector of the third information can be obtained by concatenating the embedding vectors of each word in the third information. The method for determining the embedding vector of each word in the third information is the same as that for determining the embedding vector of each word in the first information in the above embodiment, and will not be repeated here.

[0123] The aforementioned first contribution can be a probability value used to indicate that the k-th network layer is a prediction layer. The second threshold can be a pre-set threshold for the first contribution, and the value of the second threshold can be greater than or equal to 0.6. For example, the value of the second threshold can be 0.6 or 0.7. The specific value of the second threshold can be set by the user according to their needs, and is not limited in this embodiment.

[0124] In some embodiments of this application, after determining any one of the Q network layers as a prediction layer or a sharing layer, the specific network layers determined as prediction layers and which network layers are determined as sharing layers can be determined according to the learnable weights corresponding to the query vector, key vector and value vector of the k-th network layer that is the prediction layer or the sharing layer, as well as the sequence length of the hidden state information and the third information generated by the (k-1)-th network layer, according to the following formula (2), the first contribution of the k-th network layer can be obtained.

[0125] In the above formula (2), Let be the learned weights of the query vector, the key vector, and the value vector of the k-th network layer, respectively. The first learning weight, the second learning weight, and the third learning weight are, , here Let be the dimension of the embedding vector for each word in the third information. It is the Sigmoid activation function. The bias terms for the first, second, and third learned weights corresponding to the k-th network layer are used to shift the input of the sigmoid activation function so that the bias term is independent of the current input. Under the premise of gating output Provides an initial, adjustable baseline tendency. Let be the embedding vector of the third information, and the length of the embedding vector of the third information is... , here The sequence length of the third information. The hidden state information generated for the (k-1)th network layer.

[0126] After obtaining the first contribution of the kth network layer according to the above formula (2), if the first contribution of the kth network layer is greater than the second threshold, the kth network layer is determined as the prediction layer; if the first contribution of the kth network layer is less than or equal to the second threshold, the kth network layer is determined as the sharing layer.

[0127] In the embodiments of this application, the k-th network layer is determined as either a prediction layer or a shared layer by calculating its first contribution. This differs from a blanket approach of designating the first few layers of the Q network layers as prediction layers and the last few as shared layers. Instead, the first layer dynamically determines whether any one of the Q network layers is a prediction layer or a shared layer based on the third information input into the first model. In other words, during inference, the first model dynamically decides for each network layer whether to execute a shared path (low computational and memory overhead) or a prediction path (high computational and memory overhead) based on specific factors such as the sequence length of the input data. This adaptive capability allows the first model to dynamically adjust its computational resource allocation according to task complexity, achieving optimal allocation of computation and storage. This ensures maximum memory efficiency in simple scenarios and maintains higher performance in complex scenarios, achieving a balance between efficiency and effectiveness in information processing.

[0128] In some embodiments of this application, generating the first key-value pair through the network layer of the first model may specifically include: Based on the first and second learning weights corresponding to the i-th network layer in the first model, and the first key-value pair generated by the (i-1)-th network layer, the first key-value pair of the i-th network layer is obtained; Based on the first and second learning weights corresponding to the j-th network layer in the first model, and the first key-value pair generated by the second baseline layer, the first key-value pair of the j-th network layer is obtained.

[0129] Here, the i-th network layer is either a base layer or a regular layer; i is a positive integer and i≥1. When i=1, the first key-value pair generated by the (i-1)-th network layer is the key vector and value vector in the first hidden state information.

[0130] The first learned weight corresponding to the i-th network layer is the learned weight used by the i-th network layer when generating the key vector in the hidden state information. The second learned weight corresponding to the i-th network layer is the learned weight used by the i-th network layer when generating the value vector in the hidden state information.

[0131] The j-th network layer mentioned above is the prediction layer. The first learning weight corresponding to the j-th network layer is the learning weight used by the j-th network layer when generating the key vector in the hidden state information. The second learning weight corresponding to the j-th network layer is the learning weight used by the j-th network layer when generating the value vector in the hidden state information. The second baseline layer is the baseline layer that is ranked first in the first model and is closest to the j-th network layer, where j is a positive integer.

[0132] In some embodiments of this application, for the base layer and the normal layer, a complete self-attention calculation is performed. That is, for the i-th network layer in the base layer and the normal layer, according to the first learning weight corresponding to the key vector, the second learning weight corresponding to the value vector, and the third learning weight corresponding to the query vector of the i-th network layer, as well as the query vector, key vector, and value vector generated by the (i-1)-th network layer, the query vector, key vector, and value vector of the i-th network layer can be obtained according to the following formula (3). The key vector and value vector generated by the i-th network layer are the first key-value pair corresponding to the i-th network layer.

[0133] In the above formula (3), The third learned weight corresponding to the i-th network layer The first learned weight corresponds to the i-th network layer. The second learned weight corresponds to the i-th network layer. The hidden state information generated for the (i-1)th network layer for The matrix, The length of the first information sequence. Let be the dimension of the embedding vector for each word in the first piece of information. The query vector generated for the i-th network layer. The key vector generated for the i-th network layer. The value vector generated for the i-th network layer.

[0134] Continuing with the example above, with the first piece of information being "How's the weather today?", the preset dimension of the embedding vector in the embedding table is 2560, and the dimensions of the normal layer and the baseline layer are... Figure 2Taking layers 1-23 of the first information as an example, the first hidden state information of each word in the first information is a vector with 1 row and 2560 columns. Then, the first hidden state information of each word in the first information can be input into the first network layer of the first model. Then, based on the first learning weight, second learning weight and third learning weight corresponding to the first network layer, as well as the query vector, key vector and value vector of each word in the first information, the query vector, key vector and value vector of each word in the first information can be obtained.

[0135] Then, the query vector, key vector, and value vector of each word in the first information generated by the first network layer are input into the second network layer. Based on the first learning weight, second learning weight, and third learning weight corresponding to the second network layer, as well as the query vector, key vector, and value vector of each word in the first information generated by the first network layer, the query vector, key vector, and value vector of each word in the first information can be obtained.

[0136] Then, the query vector, key vector, and value vector of each word in the first information generated by the second network layer are input into the third network layer. Based on the first, second, and third learning weights corresponding to the third network layer, and the hidden state information of each word in the first information generated by the second network layer, the query vector, key vector, and value vector of each word in the first information can be obtained. This process is repeated until the query vector, key vector, and value vector of each word in the first information are generated through the 23rd network layer.

[0137] It should be noted that after the i-th network layer generates the query vector, key vector, and value vector for each word in the first information, it is also necessary to perform a self-attention calculation process of the dot product of the query vector and the key vector. Then, the i-th network layer generates the hidden state information corresponding to each word and outputs the hidden state information to the (i+1)-th network layer.

[0138] It should be noted that the hidden state information of each word in the first information obtained by the later network layer has more complex semantic features than the hidden state information of each word in the first information obtained by the previous network layer.

[0139] For the prediction layer, for example, for the j-th network layer, the first key-value pair of the j-th network layer can be obtained according to the first learning weight, the second learning weight and the third learning weight corresponding to the j-th network layer in the first model, as well as the key vector and value vector generated by the reference layer that is closest to the j-th network layer before the j-th network layer, according to the following formula (4). That is to say, for the prediction layer, its key vector and value vector are not necessarily determined by the key vector and value vector generated by the previous network layer, but are obtained by the key vector and value vector generated by the reference layer that is closest to the prediction layer before the prediction layer.

[0140] In the above formula (4), The first learned weight corresponds to the j-th network layer. The second learned weight corresponds to the j-th network layer. Hidden state information generated for the second baseline layer. for The matrix, The length of the first information sequence. Let be the dimension of the embedding vector for each word in the first piece of information. The key vector generated for the j-th network layer. The value vector generated for the j-th network layer.

[0141] Continue to refer to Figure 3 ,exist Figure 3 In the above formula (2), the 25th and 29th network layers are determined as prediction layers, and the 26th-28th and 30th-32nd network layers are as shared layers. Then, for the 29th network layer, its corresponding key-value pair is no longer determined by the hidden state information generated by the 28th network layer, but is determined based on the hidden state information generated by the reference layer that is closest to the prediction layer before the prediction layer, that is, based on the hidden state information generated by the 24th network layer.

[0142] For a shared layer, it can directly reuse the key-value pair of the preceding network layer adjacent to it. That is, for the h-th network layer, which is a shared layer, its corresponding first key-value pair is obtained according to the following formula (5).

[0143] In the above formula (5), The key vector of the preceding network layer that is adjacent to the h-th network layer. Let h be the value vector of the preceding network layer adjacent to the h-th network layer. The key vector generated for the h-th network layer. The value vector generated for the h-th network layer.

[0144] Continue to refer to Figure 3 ,exist Figure 3 In the above formula (2), the 25th and 29th network layers are determined to be prediction layers, and the 26th-28th and 30th-32nd network layers are shared layers. Therefore, for the 26th network layer, the corresponding key-value pairs can be directly reused from the key-value pairs generated by the 25th network layer.

[0145] In an embodiment of the present application, for the key-value pairs in the ordinary layer and the reference layer, a complete self-attention calculation needs to be performed. However, for the prediction layer, a complete self-attention calculation may not be executed. Instead, it can be directly calculated based on the hidden state information of the reference layer closest to the prediction layer before the prediction layer, without performing the dot product calculation process between the query vector and the key vector. That is, in the embodiment of the present application, each network layer in the first model is not allowed to independently execute the self-attention calculation. Instead, by utilizing the characteristic that the deep representation in the first model has redundancy, some deep layers directly predict and generate the key-value pairs they need from the hidden states already calculated by the shallow layers, and cache the generated key-value pairs. This mechanism bypasses the square complexity calculation of some layers, reduces the calculation latency from the source, saves the computing resources of the electronic device, and thus improves the determination efficiency of the first token. In addition, allowing some deep network layers to directly reuse the key-value pairs in the shallow network layers reduces the memory occupancy from the source.

[0146] In some embodiments of the present application, step 120 may specifically include: Through each network layer of the first model, based on the first token and the first key-value pairs generated by each network layer, generate the second key-value pairs of the first token; Through the first model, obtain the third token according to the second key-value pairs of the first token generated by the last network layer in the first model; In the case where the third token is the same as the reference end token, determine the first token as the second information.

[0147] Among them, the third token is the second token in the second information. For example, in the above example, when the user inputs the first information "What's the weather like today" and the reply information obtained is "The weather is fine today", then "今" in "The weather is fine today" is the first token, and the third token is "天" which is the second token in "The weather is fine today".

[0148] The above-mentioned third token can be the token with the largest probability value after the first token, that is, after obtaining the probability value distribution of which token is the next output token according to the first token, select the token with the largest probability value as the third token.

[0149] The reference end terminator can be pre-defined and used to represent the end of the response to the first information. For example, in the example above, for the first information "How's the weather today?", the response is "The weather is very good today." After "good" in the response, the hidden state information corresponding to "good" is input into the first model, resulting in the terminator following "good," such as a period. Because a complete response to the first information "How's the weather today?" has been obtained after "good," no other terminator will appear after "good." Therefore, the period after "good" is the reference end terminator. The above reference end terminator may not be displayed. Figure 2 In the AI ​​assistant interface shown, the reference ending term can be any of the following: character, number, text, etc., and is not limited in this embodiment.

[0150] In some embodiments of this application, after obtaining the first word element, the network layer type of any network layer in the first model can be divided into a normal layer, a base layer, a prediction layer, or a shared layer according to the above formulas (1) and (2). Then, based on the normal layer and the base layer, the first key-value pair of the network layer is generated according to the above formula (3). For the prediction layer, the first key-value pair corresponding to the prediction layer is generated according to the above formula (4). For the shared layer, the first key-value pair corresponding to the shared layer is generated according to the above formula (5). Then, the first word element is obtained according to the hidden state information of the last network layer.

[0151] Then, the embedding vector and position vector of the first word element are determined to obtain the hidden state information corresponding to the first word element. The hidden state information corresponding to the first word element is then input into the first model. Then, according to the method of determining the first key-value pair described above, the second key-value pair of the first word element is determined through the network layer in the first model. Based on the second key-value pair of the first word element generated by the last network layer in the first model, the hidden state information of the first word element is obtained. Then, based on the hidden state information of the first word element, the third word element is obtained.

[0152] If the third word is a reference ending word, it means that the response information of the first information has been determined and completed, and the first word can be determined as the second information.

[0153] Continuing with the example above, taking "How's the weather today?" as the first piece of information, the first hidden state information of each word element in the first piece of information "How's the weather today?" is input into the first model to obtain the first word element "good". Then, the embedding vector and position vector of the first word element "good" are determined to obtain the hidden state information corresponding to the first word element "good". The hidden state information corresponding to the first word element "good" is input into the first model. By processing the network layers in the first model, the second key-value pair of the first word element "good" is generated. Then, based on the hidden state information of the last network layer in the first model, the third word element is obtained as the punctuation mark period. The punctuation mark period is the reference end word element, so the second piece of information can be determined to be "good".

[0154] In the embodiments of this application, after obtaining the first word element, the first word element is not compared with the reference end word element. Instead, the third word element obtained based on the first word element is compared with the reference end word element. This is because after the first model generates the first word element, the first model has no self-generated context to rely on and must rely purely on the first information for context understanding. This may lead to inaccurate context understanding. By comparing the third word element obtained based on the first word element with the reference end word element, the third word element and the first word element can have a better context understanding. Thus, the second information obtained is more consistent with the first information.

[0155] In some embodiments of this application, after obtaining the third lexical unit, the method described above may further include: The first key-value pair generated by each network layer and the second key-value pair of the first word generated by the network layer are combined to obtain the combined key-value pair of the network layer. The synthesized key-value pair generated in the h-th iteration of the network layer is determined as the first key-value pair of the network layer in the (h+1)-th iteration. If the third lexical unit is different from the reference end lexical unit, the third lexical unit is input into the first model. Through the network layer of the first model, a second key-value pair is generated based on the third lexical unit and the first key-value pair generated by each network layer at the (h+1)th iteration. Based on the second key-value pair, the fourth lexical unit is obtained.

[0156] In this model, for any network layer, the synthesized key-value pair can be obtained by synthesizing the second key-value pair generated by the network layer with the previously generated first key-value pair. For example, in the example above, taking the first network layer in the first model as an example, when processing the first hidden state information of each word in the first information "How is the weather today", the first network layer generates the first key-value pair of each word in the first information "How is the weather today", that is, the first network layer generates 7 first key-value pairs.

[0157] When the first network layer processes the first token "今" in the second piece of information "今天天气很好", it generates the second key-value pair of the first token "今". Then, the second key-value pair of the first token "今" can be combined with the previously generated 7 first key-value pairs to obtain 8 key-value pairs. Specifically, these 8 key-value pairs can be formed into a matrix, and this matrix is the combined key-value pair. Specifically, when combining the second key-value pair of the first token "今" with the previously generated 7 first key-value pairs, one key-value pair can be one row. Therefore, the combined key-value pair is a matrix with 8 rows and n columns, where n is consistent with the dimension of the embedding vector of each token in the first piece of information "今天天气怎么样".

[0158] The above fourth token is the third token in the second piece of information, that is, the fourth token is a token following the third token in the second piece of information. In the above example, taking the second piece of information as "今天天气很好", the fourth token is the third token "天" in the second piece of information "今天天气很好".

[0159] In some embodiments of the present application, after obtaining the third token according to the first token, during the process of generating the third token, the first key-value pairs generated by each network layer and the second key-value pair of the first token generated by the network layer can be combined to obtain the combined key-value pair of the network layer. Then, the combined key-value pair generated by the h-th iteration of the network layer is determined as the first key-value pair of the network layer at the (h + 1)-th iteration. When the third token is different from the reference end token, the third token is input into the first model. Through the network layer of the first model, based on the third token and the first key-value pair generated by each network layer at the (h + 1)-th iteration, a second key-value pair is generated, and based on the second key-value pair, a fourth token is obtained.

[0160] Continuing to refer to the above example, taking the second piece of information as "今天天气很好", after obtaining the third token "天" in the second piece of information "今天天气很好", if the third token "天" is different from the reference end token, the punctuation mark period, then the embedding vector and position vector of the third token "天" are determined to obtain the hidden state information corresponding to the third token "天". The hidden state information corresponding to the third token "天" is input into the first model. Through the network layer in the first model, the second key-value pair of the third token "天" is generated. Then, based on the hidden state information of the last network layer in the first model, a fourth token is obtained. For any network layer in the first model, when calculating the second key-value pair of the third token "天" through this network layer, the second key-value pair of the first token "今" generated in this network layer and the first key-value pairs of each token in "今天天气怎么样" generated previously are combined to obtain the combined key-value pair of this network layer, and then the second key-value pair of the third token "天" is calculated according to this combined key-value pair.

[0161] In the embodiments of this application, when the third word element is different from the reference end word element, the third word element is input into the first model. Through the network layer of the first model, a second key-value pair is generated based on the third word element and the first key-value pair generated by each network layer at the (h+1)th iteration. Based on the second key-value pair, the fourth word element is obtained. This process is repeated to continuously update the key-value pairs cached by each network layer. More key-value pairs of historical word elements can be referenced when predicting the next word element. The next word element obtained in this way has a higher contextual understanding with the previous historical word elements, thereby improving the prediction accuracy of the next word element.

[0162] In some embodiments of this application, before using the first model, a model with the same structure as the first model needs to be trained to obtain the first model. That is, before step 110, the method described above may further include: Obtain the hidden state information of each word element in the sample information and the reference information of the sample information; The hidden state information of each word in the sample information is input into the teacher model. The first teacher key-value pair is generated through the network layer of the teacher model, and the first teacher word is obtained based on the first teacher key-value pair. Through the network layers of the teacher model, based on the first teacher lexical units and the first teacher key-value pairs generated by each network layer, the second teacher key-value pairs are generated, and based on the second teacher key-value pairs, the teacher information is obtained; Input the hidden state information of each word in the sample information into the student model, generate the first student key-value pair through the network layer of the student model, and obtain the first student word based on the first student key-value pair; Through the network layers of the student model, based on the first student word and the first student key-value pairs generated by each network layer, the second student key-value pairs are generated, and based on the second student key-value pairs, the student information is obtained. Based on the reference information, teacher information, student information, teacher key-value pairs, and student key-value pairs, the student model is trained to obtain the first model.

[0163] The sample information can be information used to train a model with the same structure as the first model, and the sample information can be query information that is different from the first information mentioned above.

[0164] For any word element in the sample information, the sample hidden state information of that word element can be the hidden state information of that word element. The method for determining the sample hidden state information of that word element is the same as the method for determining the first hidden state information of each word element in the first information in the above embodiment, and will not be repeated here.

[0165] The reference information refers to the sample information and its corresponding response. This can be understood as a perfect response to the sample information that has been pre-set.

[0166] The teacher and student models can have the same model structure as the first model. During the training of the teacher model using sample information, its model parameters cannot be adjusted. During the training of the student model using sample information, its model parameters need to be continuously optimized to approximate the teacher model's parameters.

[0167] The teacher and student models can be understood as follows: the teacher model is a model whose training effect is not significantly different from the first model and can be used directly. While the teacher model's parameters are inferior to the first model, this does not affect its usability. The student model's parameters need to be trained to continuously bring them closer to those of the teacher model.

[0168] The first teacher key-value pair mentioned above is the teacher key-value pair for each word in the sample information obtained based on the teacher model. The method of obtaining the first teacher key-value pair is similar to the method of obtaining the first key-value pair for each word in the first information mentioned above, and will not be repeated here.

[0169] The first teacher lexical unit can be the starting lexical unit in the teacher information obtained based on the teacher model, which is the response information of the sample information obtained based on the teacher model.

[0170] The second teacher key-value pair can be the key-value pair of the first teacher word generated by the network layer of the teacher model. The generation method of the second teacher key-value pair can refer to the generation method of the second key-value pair of the first word in the above embodiment, and will not be repeated here.

[0171] The first student key-value pair is the student key-value pair of each word in the sample information obtained based on the student model. The method of obtaining the first student key-value pair is similar to the method of obtaining the first key-value pair of each word in the first information above, and will not be repeated here.

[0172] The first student word can be the starting word in the student information obtained based on the student model, which is the response information of the sample information obtained based on the student model.

[0173] The second student key-value pair can be the key-value pair of the first student word generated by the network layer of the student model. The generation method of the second student key-value pair can refer to the generation method of the second key-value pair of the first word in the above embodiment, and will not be repeated here.

[0174] Teacher key-value pairs are generated based on the first network layer of the teacher model, where the first network layer can be any network layer in the teacher model. Student key-value pairs are generated based on the second network layer of the student model, where the second network layer can be any network layer in the student model. The first and second network layers correspond to each other; that is, if the first network layer is the first network layer in the teacher model, then the second network layer is the first network layer in the student model. Teacher key-value pairs generated based on the first network layer are the first teacher key-value pairs, and student key-value pairs generated based on the second network layer are the first student key-value pairs.

[0175] In some embodiments of this application, the following can be achieved: The hidden state information of each word in the sample information and the reference information of the sample information are obtained; the hidden state information of each word in the sample information is input into the teacher model; a first teacher key-value pair is generated through the network layer of the teacher model; a first teacher word is obtained based on the first teacher key-value pair; a second teacher key-value pair is generated through the network layer of the teacher model based on the first teacher word and the first teacher key-value pair generated by each network layer; teacher information is obtained based on the second teacher key-value pair; the hidden state information of each word in the sample information is input into the student model; a first student key-value pair is generated through the network layer of the student model; a second student key-value pair is generated through the network layer of the student model based on the first student word and the first student key-value pair generated by each network layer; student information is obtained based on the second student key-value pair; and the student model is trained according to the reference information, teacher information, student information, teacher key-value pair, and student key-value pair to obtain the first model.

[0176] In the embodiments of this application, by using the trained model with frozen parameters as the teacher model and the untrained model as the student model, the student model is trained using sample information, allowing the output of the student model to continuously approach that of the teacher model, thus obtaining the first model. This avoids the problem of training the student model without a reference, which would lead to inaccurate response information if the student model is used directly. By continuously optimizing the student model, making it increasingly closer to the trained teacher model with frozen parameters, the training effect of the student model has a reference, thus ensuring that the trained student model is consistent with the teacher model and improving the reliability of model training.

[0177] In some embodiments of this application, the step of training the student model based on reference information, teacher information, student information, teacher key-value pairs, and student key-value pairs to obtain a first model may specifically include: Based on the teacher and student information, the first loss function value is obtained; The second loss function value is obtained based on the teacher key-value pairs and student key-value pairs; Based on student information and reference information, the value of the third loss function is obtained; The student model is trained based on the first loss function value, the second loss function value, and the third loss function value to obtain the first model.

[0178] The first loss function value can be obtained based on teacher and student information.

[0179] The second loss function value can be obtained based on the teacher key-value pairs and student key-value pairs.

[0180] The third loss function value can be obtained based on student information and reference information.

[0181] In some embodiments of this application, the first loss function value can be obtained according to the teacher information obtained from the teacher model and the student information obtained from the student model, according to the following formula (6).

[0182] In the above formula (6), The first loss function value, Divergence is used to measure the difference between two probability distributions. The teacher information obtained from the teacher model. Teacher information obtained from the student model The maximum value in the probability distribution For each word element in the teacher information of the teacher model, The student model is based on each lexical element in the student information.

[0183] In some embodiments of this application, the second loss function value is obtained according to the teacher key-value pairs generated by each network layer in the teacher model and the student key-value pairs generated by each network layer in the student model, according to the following formula (7).

[0184] In the above formula (7), This is the value of the second loss function. The key vector generated for the i-th network layer of the teacher model. The key vector generated for the i-th network layer of the student model. The value vector generated for the teacher model. The value vector generated for the student model. This represents the mean square error.

[0185] In some embodiments of this application, a third loss function value is obtained based on student information and reference information obtained from the student model.

[0186] Then, based on the first loss function value, the second loss function value, and the third loss function value, the student model is trained according to the following formula (8) to obtain the first model.

[0187] In the above formula (8), The value of the third loss function. and To pre-set the weights of each loss term, and The value range is between 0 and 1.

[0188] In the embodiments of this application, the student information output by the student model is directly optimized through the first loss function value to align with the teacher information output by the teacher model. This ensures that the student information generated by the student model is aligned with the teacher information output by the teacher model, guaranteeing the correctness of the features represented internally by the student model. The second loss function value requires the student key-value pairs generated by the student model to be as close as possible to the teacher key-value pairs generated by the teacher model. Since key-value pairs are the core of the attention mechanism and directly relate to the model's ability to understand context, forcing alignment of key-value pairs is equivalent to providing guidance on the model's "thinking process" or "intermediate steps." This ensures that even if the student model skips certain attention mechanism calculations, the intermediate representations it generates are still correct and effective. This fundamentally compensates for the accuracy loss that may result from key-value pair prediction and sharing, ensuring that the semantic space of the key-value pairs predicted by the student model is aligned with that of the key-value pairs generated by the teacher model. This allows the student model to retain the capabilities of the teacher model to the maximum extent possible, ensuring that the training of the student model does not sacrifice the core performance of the model. The third loss function value ensures that the student model's autoregressive language modeling ability is not compromised, thus guaranteeing that the student model's basic capabilities are not unbalanced and preventing the student model from losing its autoregressive computational ability in the process of excessively imitating the teacher model.

[0189] The information processing method provided in this application can be executed by an information processing device. This application uses an information processing device executing the information processing method as an example to illustrate the information processing device provided in this application.

[0190] Figure 6 This is a schematic diagram illustrating the structure of an information processing apparatus according to an exemplary embodiment. For example... Figure 6 As shown, the information processing device 500 may include: The information input module 510 is used to input the first hidden state information of each word element in the first information into the first model, generate the first key-value pair through the network layer of the first model, and obtain the first word element based on the first key-value pair; The generation module 520 is used to generate a second key-value pair based on the first word and the first key-value pair generated by each network layer through the network layers of the first model, and to obtain second information based on the second key-value pair; wherein, the first word is the starting word of the second information; the network layers of the first model include a base layer, a prediction layer, and a shared layer; the key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer; the first base layer is the base layer that is ranked first and is closest to the prediction layer; the key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer; the first information is query information, and the second information is the response information to the first information.

[0191] In the embodiments of this application, by inputting the first hidden state information of each word in the first information into the first model, generating the first key-value pair through the network layer of the first model, and obtaining the first word based on the first key-value pair, and then generating the second key-value pair based on the first word and the first key-value pair generated by each network layer through the network layer of the first model, and obtaining the second information based on the second key-value pair, wherein the network layer of the first model includes a base layer, a prediction layer and a shared layer, and the key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer which is ranked first and closest to the prediction layer, and the key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer. In this way, during the processing of the first information through the network layer of the first model, the key-value pair generated by the prediction layer can be predicted based on the hidden state information generated by the base layer which is ranked first and closest to the prediction layer. The prediction layer does not need to perform the hidden state information generation process, but only needs to calculate the key-value pair, thus reducing the amount of computation. Furthermore, the key-value pairs generated by the shared layer are the same as those generated by the preceding network layer adjacent to the shared layer. In this way, the shared layer does not need to perform the key-value pair generation process and can directly reuse the key-value pairs generated by the preceding network layer, thereby improving the calculation speed of key-value pairs and thus improving the efficiency of obtaining response information.

[0192] In some embodiments of this application, the information input module is further configured to input the second hidden state information of each word element in the third information into the first model, generate the third hidden state information through the network layer of the first model, and obtain the second word element based on the third hidden state information; wherein, the third information is query information, and the third information is different from the first information; The generation module is further configured to generate fourth hidden state information based on the second lexical unit and the third hidden state information generated by each network layer through the network layer of the first model, and obtain fourth information based on the fourth hidden state information; wherein, the second lexical unit is the starting lexical unit of the fourth information; The device further includes: a first determining module, used to determine the network layer type of each network layer based on the hidden state information generated by each network layer; wherein, the network layer type of any network layer in the first model is a normal layer, a base layer, a prediction layer, or a shared layer.

[0193] In some embodiments of this application, the first determining module is specifically used for: Based on the hidden state information generated by each network layer, determine the average similarity value of the hidden state information of each network layer; Network layers whose average similarity value of the hidden state information is less than a first threshold are defined as ordinary layers. N network layers out of M network layers are determined as the baseline layers; wherein, the average similarity value of the hidden state information of the M network layers is greater than a first threshold; the N network layers are the network layers ranked first in the first model; M and N are both positive integers, and N < M; Q of the M network layers are determined as prediction layers or shared layers; the Q network layers are the network layers other than the reference layer among the M network layers; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

[0194] In some embodiments of this application, the first determining module is specifically used for: Calculate the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information to obtain P first similarity values; wherein, the fifth hidden state information is the hidden state information generated by the g-th network layer, the P sixth hidden state information is the hidden state information generated by P network layers, and the P network layers are the P network layers in the first model located after the g-th network layer, where g and P are positive integers; When P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; Wherein, the g-th network layer is any network layer in the first model.

[0195] In some embodiments of this application, the information input module is specifically used for: Based on the first and second learning weights corresponding to the i-th network layer in the first model, and the first key-value pair generated by the (i-1)-th network layer, the first key-value pair of the i-th network layer is obtained; wherein, the i-th network layer is a base layer or a normal layer; i is a positive integer and i≥1; when i=1, the first key-value pair generated by the (i-1)-th network layer is the key vector and value vector in the first hidden state information; Based on the first learning weight and the second learning weight corresponding to the j-th network layer in the first model, and the first key-value pair generated by the second reference layer, the first key-value pair of the j-th network layer is obtained; the second reference layer is the reference layer that is closest to the j-th network layer in the sorted order, and the j-th network layer is the prediction layer; j is a positive integer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information.

[0196] In some embodiments of this application, the first determining module is specifically used for: Based on the first, second, and third learning weights corresponding to the k-th network layer, the hidden state information generated by the (k-1)-th network layer, the sequence length of the third information, and the embedding vector of the third information, the first contribution of the k-th network layer is obtained; wherein, the k-th network layer is any one of the Q network layers; the first contribution is used to indicate the probability value that the k-th network layer is a prediction layer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information; and the third learning weight is the learning weight of the query vector in the hidden state information. If the first contribution is greater than the second threshold, the kth network layer is determined as the prediction layer; If the first contribution is less than or equal to the second threshold, the kth network layer is determined as a shared layer.

[0197] In some embodiments of this application, the generation module is specifically used for: Through each network layer of the first model, a second key-value pair of the first word is generated based on the first word and the first key-value pair generated by each network layer; Using the first model, a third word is obtained based on the second key-value pair of the first word generated by the last network layer in the first model; wherein, the third word is the second word in the second information; If the third word is the same as the reference end word, the first word is identified as the second information.

[0198] In some embodiments of this application, the apparatus further includes: A synthesis module is used to synthesize the first key-value pair generated by each network layer and the second key-value pair of the first word generated by the network layer to obtain the synthesized key-value pair of the network layer. The second determining module is used to determine the synthesized key-value pair generated by the h-th iteration of the network layer as the first key-value pair of the network layer at the (h+1)-th iteration. The generation module is further configured to, when the third lexical element is different from the reference end lexical element, input the third lexical element into the first model, generate a second key-value pair through the network layer of the first model based on the third lexical element and the first key-value pair generated by each network layer at the (h+1)th iteration, and obtain a fourth lexical element based on the second key-value pair; the fourth lexical element is the third lexical element in the second information.

[0199] In some embodiments of this application, the apparatus further includes: The acquisition module is used to acquire the hidden state information of each word in the sample information and the reference information of the sample information; the reference information is the reference response information of the sample information. The information input module is also used to input the sample latent state information of each word in the sample information into the teacher model, generate the first teacher key-value pair through the network layer of the teacher model, and obtain the first teacher word based on the first teacher key-value pair; The generation module is further configured to generate a second teacher key-value pair based on the first teacher lexical and the first teacher key-value pair generated by each network layer through the network layer of the teacher model, and obtain teacher information based on the second teacher key-value pair; The information input module is also used to input the sample hidden state information of each word in the sample information into the student model, generate the first student key-value pair through the network layer of the student model, and obtain the first student word based on the first student key-value pair; The generation module is further configured to generate a second student key-value pair based on the first student word and the first student key-value pair generated by each network layer through the network layer of the student model, and obtain student information based on the second student key-value pair; The model training module is used to train the student model based on the reference information, the teacher information, the student information, the teacher key-value pairs, and the student key-value pairs to obtain a first model; wherein the teacher model and the student model have the same model structure; the teacher key-value pairs are generated based on the first network layer of the teacher model, and the student key-value pairs are generated based on the second network layer of the student model, and the first network layer and the second network layer correspond to each other.

[0200] In some embodiments of this application, the model training module is specifically used for: Based on the teacher information and the student information, the first loss function value is obtained; The second loss function value is obtained based on the teacher key-value pairs and the student key-value pairs; Based on the student information and the reference information, the third loss function value is obtained; The student model is trained based on the first loss function value, the second loss function value, and the third loss function value to obtain the first model.

[0201] The information processing device in this application embodiment can be an electronic device or a component within an electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal or other devices besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television set (TV), ATM, or self-service machine, etc. This application embodiment does not specifically limit the device.

[0202] The information processing device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0203] The information processing device provided in this application embodiment can achieve... Figure 1 The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0204] Optionally, such as Figure 7As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described information processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0205] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0206] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0207] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0208] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 8 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0209] The processor 710 is configured to input the first hidden state information of each word in the first information into a first model, generate a first key-value pair through the network layers of the first model, and obtain a first word based on the first key-value pair; generate a second key-value pair through the network layers of the first model based on the first word and the first key-value pair generated by each network layer, and obtain second information based on the second key-value pair; wherein the first word is the starting word of the second information; the network layers of the first model include a base layer, a prediction layer, and a shared layer; the key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer; the first base layer is the base layer that is ranked first and is closest to the prediction layer; the key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer; the first information is query information, and the second information is the response information to the first information.

[0210] Thus, by inputting the first hidden state information of each word in the first information into the first model, the network layers of the first model generate the first key-value pair, and the first word is obtained based on the first key-value pair. Then, the network layers of the first model generate the second key-value pair based on the first word and the first key-value pair generated by each network layer, and the second information is obtained based on the second key-value pair. The network layers of the first model include a base layer, a prediction layer, and a shared layer. The key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer, which is the first layer in the order and closest to the prediction layer. The key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer. In this way, during the processing of the first information by the network layers of the first model, the key-value pair generated by the prediction layer can be predicted based on the hidden state information generated by the first base layer, which is the first layer in the order and closest to the prediction layer. The prediction layer does not need to perform the hidden state information generation process, but only needs to calculate the key-value pair, thus reducing the amount of computation. Furthermore, the key-value pairs generated by the shared layer are the same as those generated by the preceding network layer adjacent to the shared layer. In this way, the shared layer does not need to perform the key-value pair generation process and can directly reuse the key-value pairs generated by the preceding network layer, thereby improving the calculation speed of key-value pairs and thus improving the efficiency of obtaining response information.

[0211] Optionally, the processor 710 is further configured to input the second hidden state information of each word in the third information into the first model, generate third hidden state information through the network layer of the first model, and obtain the second word based on the third hidden state information; wherein the third information is query information and is different from the first information; generate fourth hidden state information through the network layer of the first model based on the second word and the third hidden state information generated by each network layer, and obtain fourth information based on the fourth hidden state information; wherein the second word is the starting word of the fourth information; determine the network layer type of each network layer according to the hidden state information generated by each network layer; wherein the network layer type of any network layer in the first model is a normal layer, a base layer, a prediction layer, or a shared layer.

[0212] Optionally, the processor 710 is further configured to: determine the average similarity value of the hidden state information of each network layer based on the hidden state information generated by each network layer; determine the network layer whose average similarity value of the hidden state information is less than a first threshold as a normal layer; determine N network layers out of M network layers as a reference layer; wherein the average similarity value of the hidden state information of the M network layers is greater than the first threshold; the N network layers are the network layers ranked first in the first model; M and N are both positive integers, N < M; determine Q network layers out of the M network layers as prediction layers or shared layers; the Q network layers are the network layers other than the reference layer among the M network layers; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

[0213] Optionally, the processor 710 is further configured to calculate a first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information, to obtain P first similarity values; wherein the fifth hidden state information is the hidden state information generated by the g-th network layer, the P sixth hidden state information is the hidden state information generated by P network layers, the P network layers are the P network layers in the first model located after the g-th network layer, and g and P are positive integers; when P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; when P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; wherein the g-th network layer is any network layer in the first model.

[0214] Optionally, the processor 710 is further configured to obtain the first key-value pair of the i-th network layer based on the first learning weight and the second learning weight corresponding to the i-th network layer in the first model, and the first key-value pair generated by the (i-1)-th network layer; wherein the i-th network layer is a base layer or a normal layer; i is a positive integer and i≥1; when i=1, the first key-value pair generated by the (i-1)-th network layer is the key vector and value vector in the first hidden state information; and to obtain the first key-value pair of the j-th network layer based on the first learning weight and the second learning weight corresponding to the j-th network layer in the first model, and the first key-value pair generated by the second base layer; the second base layer is the base layer that is closest to the j-th network layer in the order of priority, and the j-th network layer is the prediction layer; j is a positive integer; the first learning weight is the learning weight of the key vector in the hidden state information; and the second learning weight is the learning weight of the value vector in the hidden state information.

[0215] Optionally, the processor 710 is further configured to obtain a first contribution of the k-th network layer based on the first learning weight, the second learning weight, and the third learning weight corresponding to the k-th network layer, the hidden state information generated by the (k-1)-th network layer, the sequence length of the third information, and the embedding vector of the third information; wherein the k-th network layer is any one of the Q network layers; the first contribution is used to indicate the probability value of the k-th network layer being a prediction layer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information; and the third learning weight is the learning weight of the query vector in the hidden state information; if the first contribution is greater than a second threshold, the k-th network layer is determined to be a prediction layer; if the first contribution is less than or equal to the second threshold, the k-th network layer is determined to be a sharing layer.

[0216] Optionally, the processor 710 is further configured to generate a second key-value pair of the first word based on the first word and the first key-value pair generated by each network layer of the first model; and to obtain a third word based on the second key-value pair of the first word generated by the last network layer in the first model; wherein the third word is the second word in the second information; and if the third word is the same as the reference end word, the first word is determined as the second information.

[0217] Optionally, the processor 710 is further configured to synthesize the first key-value pair generated by each network layer and the second key-value pair of the first word generated by the network layer to obtain the synthesized key-value pair of the network layer; determine the synthesized key-value pair generated by the network layer in the h-th iteration as the first key-value pair of the network layer in the h+1-th iteration; if the third word is different from the reference end word, input the third word into the first model, generate the second key-value pair through the network layer of the first model based on the third word and the first key-value pair generated by each network layer in the h+1-th iteration, and obtain the fourth word based on the second key-value pair; the fourth word is the third word in the second information.

[0218] Optionally, the processor 710 is further configured to acquire the sample latent state information of each word in the sample information and the reference information of the sample information; the reference information is the reference response information of the sample information; input the sample latent state information of each word in the sample information into a teacher model, generate a first teacher key-value pair through the network layer of the teacher model, and obtain a first teacher word based on the first teacher key-value pair; generate a second teacher key-value pair through the network layer of the teacher model based on the first teacher word and the first teacher key-value pair generated by each network layer, and obtain teacher information based on the second teacher key-value pair; input the sample latent state information of each word in the sample information into a student model, and obtain the sample latent state information of each word in the sample information through the network layer of the student model. A first student key-value pair is generated, and a first student word is obtained based on the first student key-value pair; a second student key-value pair is generated through the network layers of the student model, based on the first student word and the first student key-value pair generated by each network layer, and student information is obtained based on the second student key-value pair; the student model is trained according to the reference information, the teacher information, the student information, the teacher key-value pair, and the student key-value pair to obtain a first model; wherein, the model structure of the teacher model and the student model is the same; the teacher key-value pair is generated based on the first network layer of the teacher model, and the student key-value pair is generated based on the second network layer of the student model, and the first network layer and the second network layer correspond to each other.

[0219] Optionally, the processor 710 is further configured to obtain a first loss function value based on the teacher information and the student information; obtain a second loss function value based on the teacher key-value pair and the student key-value pair; obtain a third loss function value based on the student information and the reference information; and train the student model based on the first loss function value, the second loss function value, and the third loss function value to obtain a first model.

[0220] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a color camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, and joysticks, which will not be described in detail here.

[0221] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 709 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0222] Processor 710 may include one or more processing units; optionally, processor 710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 710.

[0223] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described information processing method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here.

[0224] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0225] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above information processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0226] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0227] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the information processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0228] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0229] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0230] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An information processing method, characterized in that, The method includes: The first hidden state information of each word element in the first information is input into the first model. The first key-value pair is generated through the network layer of the first model, and the first word element is obtained based on the first key-value pair. Through the network layers of the first model, a second key-value pair is generated based on the first word and the first key-value pair generated by each network layer, and the second information is obtained based on the second key-value pair; Wherein, the first word is the starting word of the second information; the network layer of the first model includes a base layer, a prediction layer, and a shared layer; the key-value pairs generated by the prediction layer are obtained based on the hidden state information generated by the first base layer; the first base layer is the base layer that is ranked first and is closest to the prediction layer; the key-value pairs generated by the shared layer are the key-value pairs generated by the network layer adjacent to the shared layer; the first information is query information, and the second information is the response information to the first information.

2. The method according to claim 1, characterized in that, Before inputting the first hidden state information of each lexical in the first information into the first model, generating the first key-value pair through the network layer of the first model, and obtaining the first lexical based on the first key-value pair, the method further includes: The second hidden state information of each word element in the third information is input into the first model. The third hidden state information is generated through the network layer of the first model, and the second word element is obtained based on the third hidden state information. The third information is query information and is different from the first information. Through the network layers of the first model, based on the second lexical unit and the third hidden state information generated by each network layer, fourth hidden state information is generated, and based on the fourth hidden state information, fourth information is obtained; wherein, the second lexical unit is the starting lexical unit of the fourth information; The network layer type of each network layer is determined based on the hidden state information generated by each network layer. In the first model, any network layer can be a normal layer, a baseline layer, a prediction layer, or a shared layer.

3. The method according to claim 2, characterized in that, The step of determining the network layer type of each network layer based on the hidden state information generated by each network layer includes: Based on the hidden state information generated by each network layer, determine the average similarity value of the hidden state information of each network layer; Network layers whose average similarity value of the hidden state information is less than a first threshold are defined as ordinary layers. N network layers out of M network layers are determined as the baseline layers; wherein, the average similarity value of the hidden state information of the M network layers is greater than a first threshold; the N network layers are the network layers ranked first in the first model; M and N are both positive integers, and N < M; Q of the M network layers are determined as prediction layers or shared layers; the Q network layers are the network layers other than the reference layer among the M network layers; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

4. The method according to claim 3, characterized in that, The step of determining the average similarity value of the hidden state information of each network layer based on the hidden state information generated by each network layer includes: Calculate the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information to obtain P first similarity values; wherein, the fifth hidden state information is the hidden state information generated by the g-th network layer, the P sixth hidden state information is the hidden state information generated by P network layers, and the P network layers are the P network layers in the first model located after the g-th network layer, where g and P are positive integers; When P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; Wherein, the g-th network layer is any network layer in the first model.

5. The method according to claim 2, characterized in that, The step of generating the first key-value pair through the network layer of the first model includes: Based on the first and second learning weights corresponding to the i-th network layer in the first model, and the first key-value pair generated by the (i-1)-th network layer, the first key-value pair of the i-th network layer is obtained; wherein, the i-th network layer is a base layer or a normal layer; i is a positive integer and i≥1; when i=1, the first key-value pair generated by the (i-1)-th network layer is the key vector and value vector in the first hidden state information; Based on the first learning weight and the second learning weight corresponding to the j-th network layer in the first model, and the first key-value pair generated by the second reference layer, the first key-value pair of the j-th network layer is obtained; the second reference layer is the reference layer that is closest to the j-th network layer in the sorted order, and the j-th network layer is the prediction layer; j is a positive integer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information.

6. The method according to claim 3, characterized in that, The step of determining Q network layers out of the M network layers as prediction layers or shared layers includes: Based on the first, second, and third learning weights corresponding to the k-th network layer, the hidden state information generated by the (k-1)-th network layer, the sequence length of the third information, and the embedding vector of the third information, the first contribution of the k-th network layer is obtained; wherein, the k-th network layer is any one of the Q network layers; the first contribution is used to indicate the probability value that the k-th network layer is a prediction layer; the first learning weight is the learning weight of the key vector in the hidden state information; the second learning weight is the learning weight of the value vector in the hidden state information; and the third learning weight is the learning weight of the query vector in the hidden state information. If the first contribution is greater than the second threshold, the kth network layer is determined as the prediction layer; If the first contribution is less than or equal to the second threshold, the kth network layer is determined as a shared layer.

7. The method according to claim 1, characterized in that, The step of generating a second key-value pair based on the first word and the first key-value pair generated by each network layer of the first model, and obtaining second information based on the second key-value pair, includes: Through each network layer of the first model, a second key-value pair of the first word is generated based on the first word and the first key-value pair generated by each network layer; Using the first model, a third word is obtained based on the second key-value pair of the first word generated by the last network layer in the first model; wherein, the third word is the second word in the second information; If the third word is the same as the reference end word, the first word is identified as the second information.

8. The method according to claim 7, characterized in that, After obtaining the third lexical unit, the method further includes: The first key-value pair generated by each network layer and the second key-value pair of the first word generated by the network layer are combined to obtain the combined key-value pair of the network layer. The synthesized key-value pair generated in the h-th iteration of the network layer is determined as the first key-value pair of the network layer in the (h+1)-th iteration. If the third lexical unit is different from the reference end lexical unit, the third lexical unit is input into the first model. Through the network layer of the first model, a second key-value pair is generated based on the third lexical unit and the first key-value pair generated by each network layer at the (h+1)th iteration. Based on the second key-value pair, a fourth lexical unit is obtained. The fourth lexical unit is the third lexical unit in the second information.

9. The method according to claim 1, characterized in that, Before inputting the first hidden state information of each lexical in the first information into the first model, generating the first key-value pair through the network layer of the first model, and obtaining the first lexical based on the first key-value pair, the method further includes: Obtain the hidden state information of each word element in the sample information and the reference information of the sample information; the reference information is the reference response information of the sample information; The hidden state information of each word in the sample information is input into the teacher model. The first teacher key-value pair is generated through the network layer of the teacher model, and the first teacher word is obtained based on the first teacher key-value pair. Through the network layers of the teacher model, a second teacher key-value pair is generated based on the first teacher lexical unit and the first teacher key-value pair generated by each network layer, and teacher information is obtained based on the second teacher key-value pair; The hidden state information of each word in the sample information is input into the student model. The first student key-value pair is generated through the network layer of the student model, and the first student word is obtained based on the first student key-value pair. Through the network layers of the student model, based on the first student word and the first student key-value pairs generated by each network layer, a second student key-value pair is generated, and based on the second student key-value pair, student information is obtained; The student model is trained based on the reference information, the teacher information, the student information, the teacher key-value pairs, and the student key-value pairs to obtain the first model; The teacher model and the student model have the same model structure; the teacher key-value pairs are generated based on the first network layer of the teacher model, and the student key-value pairs are generated based on the second network layer of the student model, with the first network layer and the second network layer corresponding to each other.

10. The method according to claim 9, characterized in that, The step of training the student model based on the reference information, the teacher information, the student information, teacher key-value pairs, and student key-value pairs to obtain a first model includes: Based on the teacher information and the student information, the first loss function value is obtained; The second loss function value is obtained based on the teacher key-value pairs and the student key-value pairs; Based on the student information and the reference information, the third loss function value is obtained; The student model is trained based on the first loss function value, the second loss function value, and the third loss function value to obtain the first model.

11. An information processing device, characterized in that, The device includes: The information input module is used to input the first hidden state information of each word element in the first information into the first model, generate the first key-value pair through the network layer of the first model, and obtain the first word element based on the first key-value pair; The generation module is used to generate a second key-value pair based on the first word and the first key-value pair generated by each network layer of the first model, and to obtain second information based on the second key-value pair; wherein, the first word is the starting word of the second information; the network layers of the first model include a base layer, a prediction layer, and a shared layer; the key-value pair generated by the prediction layer is obtained based on the hidden state information generated by the first base layer; the first base layer is the base layer that is ranked first and is closest to the prediction layer; the key-value pair generated by the shared layer is the key-value pair generated by the network layer adjacent to the shared layer; the first information is query information, and the second information is the response information to the first information.

12. The apparatus according to claim 11, characterized in that, The information input module is further configured to input the second hidden state information of each word element in the third information into the first model, generate the third hidden state information through the network layer of the first model, and obtain the second word element based on the third hidden state information; wherein, the third information is query information, and the third information is different from the first information; The generation module is further configured to generate fourth hidden state information based on the second lexical unit and the third hidden state information generated by each network layer through the network layer of the first model, and obtain fourth information based on the fourth hidden state information; wherein, the second lexical unit is the starting lexical unit of the fourth information; The device further includes: a first determining module, used to determine the network layer type of each network layer based on the hidden state information generated by each network layer; wherein, the network layer type of any network layer in the first model is a normal layer, a base layer, a prediction layer, or a shared layer.

13. The apparatus according to claim 12, characterized in that, The first determining module is specifically used for: Based on the hidden state information generated by each network layer, determine the average similarity value of the hidden state information of each network layer; Network layers whose average similarity value of the hidden state information is less than a first threshold are defined as ordinary layers. N network layers out of M network layers are determined as the baseline layers; wherein, the average similarity value of the hidden state information of the M network layers is greater than a first threshold; the N network layers are the network layers ranked first in the first model; M and N are both positive integers, and N < M; Q of the M network layers are determined as prediction layers or shared layers; the Q network layers are the network layers other than the reference layer among the M network layers; Q < M; Q is a positive integer and Q ≥ 1; M = N + Q.

14. The apparatus according to claim 13, characterized in that, The first determining module is specifically used for: Calculate the first similarity between the fifth hidden state information of the g-th network layer and each of the P sixth hidden state information to obtain P first similarity values; wherein, the fifth hidden state information is the hidden state information generated by the g-th network layer, the P sixth hidden state information is the hidden state information generated by P network layers, and the P network layers are the P network layers in the first model located after the g-th network layer, where g and P are positive integers; When P equals 1, the first similarity value is determined as the average similarity value of the hidden state information of the g-th network layer; When P is greater than or equal to 2, the average of the P first similarity values ​​is determined as the average similarity value of the hidden state information of the g-th network layer; Wherein, the g-th network layer is any network layer in the first model.

15. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the information processing method as described in any one of claims 1-10.

Images