Text data processing method, device and electronic equipment

Abstract

This disclosure provides a text data processing method, a training method for a large language model, an apparatus, and an electronic device, relating to the field of data processing technology, particularly artificial intelligence and large model technology. The implementation scheme is as follows: Determine the relevance vector between the current word element's contextual hidden state in the current layer of the large language model and the dictionary basis vectors of the current layer; perform element-wise multiplication of the relevance vector and the intermediate hidden encoding of the current word element in the current layer to obtain a gating vector; use the gating vector to linearly combine the dictionary basis vectors of the current layer to obtain the memory embedding residual for the current word element; inject the memory embedding residual into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model; and use the large language model to output text prediction results based on the contextual hidden state injected with the memory embedding residual.

Description

Technical Field

[0001] This disclosure relates to the field of data processing technology, and in particular to artificial intelligence and large model technology, specifically to a text data processing method, a training method for a large language model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graph technologies.

[0003] In the field of memory embedding vocabularies for large language models, the adaptive integration of retrieved memory embeddings into the backbone network computation is crucial for enhancing the model's knowledge retention and complex reasoning capabilities. The methods described in this section are not necessarily those previously conceived or employed. Unless otherwise stated, no method described in this section should be assumed to be prior art simply because it is included here. Similarly, unless otherwise stated, the issues mentioned in this section should not be considered to be recognized in any prior art. Summary of the Invention

[0004] This disclosure provides a text data processing method, a method for training a large language model, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a text data processing method is provided, comprising: determining a relevance vector between the contextual hidden state of the current word element in the current layer of a large language model and the dictionary basis vectors of the current layer; performing element-wise multiplication of the relevance vector and the intermediate hidden encoding of the current word element in the current layer to obtain a gating vector; using the gating vector to linearly combine the dictionary basis vectors of the current layer to obtain a memory embedding residual for the current word element in the current layer; injecting the memory embedding residual into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model; and using the large language model to output a text prediction result based on the contextual hidden state injected with the memory embedding residual.

[0006] According to another aspect of this disclosure, a training method for a large language model is provided, comprising: acquiring a training dataset, wherein the training dataset includes sample text input and real text output corresponding to the sample text input; processing the sample text input using the current parameters of the large language model to obtain a predicted text output for the sample text; comprising: determining a relevance vector between the contextual hidden state of the current layer of the large language model and the dictionary basis vectors of the current layer for the current word in the sample text input; performing element-wise multiplication of the relevance vector and the intermediate hidden encoding of the current word in the current layer to obtain a gating vector; linearly combining the dictionary basis vectors of the current layer using the gating vector to obtain a memory embedding residual for the current word in the current layer; injecting the memory embedding residual into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model; outputting the predicted text output using the large language model based on the contextual hidden state injected with the memory embedding residual; and adjusting the current parameters of the large language model by optimizing a loss function that includes at least the difference between the predicted text output and the real text output.

[0007] According to another aspect of this disclosure, a text data processing apparatus is provided, comprising: a relevance determination unit configured to determine a relevance vector between the contextual latent state of a current word element in a current layer of a large language model and the dictionary basis vectors of the current layer; a gating determination unit configured to perform element-wise multiplication of the relevance vector and the intermediate latent encoding of the current word element in the current layer to obtain a gating vector; a memory embedding residual determination unit configured to linearly combine the dictionary basis vectors of the current layer using the gating vector to obtain a memory embedding residual for the current word element in the current layer; an injection unit configured to inject the memory embedding residual into the contextual latent state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model; and a prediction unit configured to output a text prediction result based on the contextual latent state injected with the memory embedding residual using the large language model.

[0008] According to another aspect of this disclosure, a training apparatus for a large language model is provided, comprising: a training data acquisition unit configured to acquire a training dataset, wherein the training dataset includes sample text input and real text output corresponding to the sample text input; and a prediction unit configured to process the sample text input using current parameters of the large language model to obtain a predicted text output for the sample text, including: determining a relevance vector between the contextual hidden state of the current word in the sample text input and the dictionary basis vectors of the current layer of the large language model, and setting the relevance vector and the current word in the middle of the current layer... The implicit encoding is element-wise multiplied to obtain a gating vector. The gating vector is then used to linearly combine the dictionary base vectors of the current layer to obtain the memory embedding residual for the current word. The memory embedding residual is injected into the contextual hidden state to embed the semantics of the dictionary base vectors of the current layer into the context-aware process of the large language model. The large language model outputs the predicted text output based on the contextual hidden state injected with the memory embedding residual. The training unit is configured to adjust the current parameters of the large language model by optimizing a loss function that includes at least the difference between the predicted text output and the real text output.

[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform a method according to an embodiment of this disclosure.

[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to an embodiment of this disclosure.

[0011] According to another aspect of this disclosure, a computer program product is provided, including a computer program, wherein the computer program, when executed by a processor, implements a method according to an embodiment of this disclosure.

[0012] According to one or more embodiments of this disclosure, the semantics of dictionary basis vectors can be adaptively incorporated into the context-aware process of a large language model, thereby improving the accuracy of text prediction results.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 A schematic diagram of an exemplary system in which the various methods described herein can be implemented according to embodiments of the present disclosure is shown.

[0016] Figure 2 A flowchart illustrating an exemplary process of a text data processing method according to an embodiment of the present disclosure is shown.

[0017] Figure 3 An exemplary flowchart of a method for training a large language model according to embodiments of the present disclosure is shown.

[0018] Figure 4 A schematic diagram of the overall architecture of a basis vector level gating algorithm in a text data processing method according to an embodiment of the present disclosure is shown.

[0019] Figure 5 An exemplary block diagram of a text data processing apparatus according to an embodiment of the present disclosure is shown.

[0020] Figure 6 An exemplary block diagram of a training apparatus for a large language model according to an embodiment of the present disclosure is shown.

[0021] Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0022] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0023] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0024] The terminology used in the description of the various examples described in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0025] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0026] Figure 1 A schematic diagram of an exemplary system 100 in which the various methods and apparatus described herein can be implemented according to embodiments of this disclosure is shown. Reference Figure 1 The system 100 includes one or more client devices 101, 102, 103, 104, 105 and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120. The client devices 101, 102, 103, 104, 105 and 106 can be configured to execute one or more applications.

[0027] In embodiments of this disclosure, server 120 may run one or more services or software applications that enable the execution of methods for processing text data according to embodiments of this disclosure.

[0028] In some embodiments, server 120 may also provide other services or software applications, which may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, such as to users of client devices 101, 102, 103, 104, 105 and / or 106 under a Software as a Service (SaaS) model.

[0029] exist Figure 1 In the configuration shown, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or combinations thereof that can be executed by one or more processors. Users operating client devices 101, 102, 103, 104, 105, and / or 106 can sequentially interact with server 120 using one or more client applications to utilize the services provided by these components. It should be understood that various different system configurations are possible and may differ from system 100. Therefore, Figure 1 This is an example of a system used to implement the various methods described herein, and is not intended to be limiting.

[0030] Users can use client devices 101, 102, 103, 104, 105, and / or 106 to receive text input and display the text output results predicted by the large model. The client devices can provide an interface that allows users to interact with the client devices. The client devices can also output information to the user through this interface. Although... Figure 1 Only six client devices are described, but those skilled in the art will understand that this disclosure can support any number of client devices.

[0031] Client devices 101, 102, 103, 104, 105, and / or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors, or other sensing devices. These computer devices can run various types and versions of software applications and operating systems, such as Microsoft Windows, Apple iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as Google Chrome OS); or include various mobile operating systems, such as Microsoft Windows Mobile OS, iOS, Windows Phone, and Android. Portable handheld devices may include cellular phones, smartphones, tablets, personal digital assistants (PDAs), etc. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, internet-enabled gaming devices, etc. Client devices are capable of executing various applications, such as various internet-related applications, communication applications (such as email applications), short message service (SMS) applications, and can use various communication protocols.

[0032] Network 110 can be any type of network well known to those skilled in the art, and can support data communication using any of a variety of available protocols (including but not limited to TCP / IP, SNA, IPX, etc.). By way of example only, one or more networks 110 can be a local area network (LAN), an Ethernet-based network, a token ring network, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, a blockchain network, a public switched telephone network (PSTN), an infrared network, a wireless network (e.g., Bluetooth, WIFI), and / or any combination of these and / or other networks.

[0033] Server 120 may include one or more general-purpose computers, special-purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-range servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and / or combination. Server 120 may include one or more virtual machines running a virtual operating system, or other computing architectures involving virtualization (e.g., one or more flexible pools of logical storage devices that can be virtualized to maintain virtual storage devices for servers). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

[0034] The computing unit in server 120 can run one or more operating systems, including any of the aforementioned operating systems and any commercially available server operating system. Server 120 can also run any of a variety of additional server applications and / or middleware applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, etc.

[0035] In some implementations, server 120 may include one or more applications to analyze and merge data feeds and / or event updates received from users of client devices 101, 102, 103, 104, 105 and / or 106. Server 120 may also include one or more applications to display data feeds and / or real-time events via one or more display devices of client devices 101, 102, 103, 104, 105 and / or 106.

[0036] In some implementations, server 120 can be a server for a distributed system or a server integrated with blockchain. Server 120 can also be a cloud server, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology. A cloud server is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability.

[0037] System 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as audio files and video files. Databases 130 may reside in various locations. For example, a database used by server 120 may be local to server 120, or it may be located away from server 120 and may communicate with server 120 via a network-based or dedicated connection. Databases 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database. One or more of these databases may store, update, and retrieve data from and from the databases in response to commands.

[0038] In some embodiments, one or more of the databases 130 may also be used by an application to store application data. The databases used by the application may be of different types, such as key-value stores, object stores, or regular stores supported by a file system.

[0039] Figure 1 The system 100 can be configured and operated in various ways to enable the application of the various methods and apparatus described in this disclosure.

[0040] In related technologies, a scalar-level gating mechanism is used to implement the memory embedding mechanism for large language models. The characteristics of its gating design are: the entire embedding vector shares a scalar gating value, and the gating parameters are independently mounted in the form of residual bypasses, without modifying the structure of the backbone network itself.

[0041] In existing memory-embedded vocabulary construction and training methods, the storage and learning of embedding slots for each word group are independent, resulting in a lack of parameter sharing and combined expression capabilities among different word groups, leading to low utilization of vocabulary parameters. Existing gating methods responsible for fusing memory embeddings with the backbone network only use scalar values ​​to uniformly control the injection intensity of the entire embedding vector, failing to differentiate the modulation of different semantic dimensions that need to be activated for the same word group in different contexts. Context-independent semantic components are proportionally modulated, thus introducing noise into the contextual reasoning process.

[0042] The aforementioned issues limit the efficiency and semantic accuracy of large language models in retrieving local static knowledge in long-context language modeling tasks. A key requirement for improving the capabilities of large language models' memory embedding vocabulary modules is to flexibly and accurately retrieve memory embeddings with manageable parameter counts and computational overhead, adapt them to contextual changes, and integrate them into the backbone network's computational process.

[0043] To improve the retrieval efficiency and semantic accuracy of local static knowledge in large language models, this disclosure provides a new text data processing method.

[0044] Figure 2 A flowchart illustrating an exemplary process of a text data processing method according to an embodiment of the present disclosure is shown.

[0045] In step S202, the correlation vector between the current word element and the contextual hidden state of the current layer of the large language model and the dictionary basis vector of the current layer is determined.

[0046] In step S204, the relevance vector and the intermediate hidden code of the current word in the current layer are multiplied element-wise to obtain the gating vector.

[0047] In step S206, the dictionary basis vectors of the current layer are linearly combined using the gate vectors to obtain the memory embedding residuals of the current layer for the current word.

[0048] In step S208, the memory embedding residual is injected into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model.

[0049] In step S210, the text prediction result is output using a large language model based on the contextual latent state injected with memory embedding residuals.

[0050] The text data processing method provided by the embodiments of this disclosure improves the accuracy of text prediction results by performing element-wise multiplication gating on the intermediate hidden encoding based on the correlation vector between the contextual hidden state and the dictionary basis vector, and injecting the gated memory into the embedded residual into the contextual hidden state.

[0051] The principles of this disclosure will now be described in detail.

[0052] In step S202, the correlation vector between the current word element and the contextual hidden state of the current layer of the large language model and the dictionary basis vector of the current layer is determined.

[0053] The current token can refer to the smallest language unit (token) that the large language model is currently processing during inference. It is the basic granularity at which the model gradually generates or understands text.

[0054] The current layer in a large language model can refer to a specific network layer in the model's architecture (such as a Transformer-based model or any other suitable model for text processing) that is performing forward computation. For example, each layer in a large language model maintains an independent hidden state representation.

[0055] The contextual hidden state refers to the vector representation of the current word in the current layer. It encodes the semantic information of the word in the current context and is the core feature that is passed down layer by layer in the backbone network of a large language model.

[0056] Dictionary basis vectors refer to a set of pre-trained atomic semantic vectors in a memory embedding vocabulary, which together form a dictionary matrix D. Each basis vector carries an atomic semantic pattern. The information in the dictionary matrix D can be represented by linear combinations of dictionary basis vectors. Different semantic information can be represented by different linear combinations of basis vectors, thereby enabling differentiated injection of different semantics in the memory embedding mechanism.

[0057] The relevance vector obtained in step S202 has the same dimension as the number of dictionary basis vectors, and each component of the relevance vector represents the semantic relevance between the corresponding dictionary basis vector dimension and the current context. Using this method, by clearly defining the dimensional structure and semantic meaning of the relevance vector, the executability of the basis vector-level gating mechanism is ensured, enabling differentiated and precise modulation of each semantic component of the memory embedding, and avoiding semantic noise introduced by scalar gating.

[0058] The vector similarity between the contextual hidden state and the dictionary basis vectors of the current layer can be calculated in any suitable way.

[0059] In some embodiments, step S202 may include: multiplying the dictionary basis vectors by the key projection matrix to obtain the key space representation of the dictionary basis vectors; normalizing the contextual hidden state and the key space representation respectively; calculating the inner product similarity between the normalized contextual hidden state and the normalized key space representation; and applying an activation function to the inner product similarity to obtain the relevance vector. In some examples, the normalization process can be root mean square normalization. Root mean square normalization scales the vector by calculating the root mean square of each element, so that the inner product similarity calculation is not affected by the vector magnitude and only reflects the semantic relevance in the direction. In some examples, the activation function can be the element-wise sigmoid function. The sigmoid function independently maps each inner product similarity component to the (0, 1) interval, making each component of the relevance vector a continuous soft relevance score, which facilitates subsequent element-wise multiplication with the intermediate hidden code.

[0060] It is understood that the implementation of step S202 is not limited to this without departing from the principles of this disclosure. For example, cosine similarity can be used to determine the vector similarity between the contextual hidden state and the key space representation. If the vector dimensions of the contextual hidden state and the dictionary basis vectors match, the step of mapping the dictionary basis vectors to the key space can be omitted. Alternatively, layer normalization can be used instead of root mean square normalization. In some alternative embodiments, the contextual hidden state and the dictionary basis vectors can be projected to multiple subspaces, and relevance scores can be calculated separately in each subspace before concatenation or weighted summation. This allows the relevance vector to capture the association between the contextual hidden state and the dictionary basis vectors from multiple semantic perspectives, further improving the fineness of gating.

[0061] In step S204, the relevance vector and the intermediate hidden code of the current word in the current layer are multiplied element-wise to obtain the gating vector.

[0062] Intermediate-state implicit encoding refers to a vector representation of the current word at the current layer, denoted as... Its dimension is consistent with the number of dictionary basis vectors. The intermediate state implicit code can be pre-stored by a hash lookup table or any other suitable mapping method. During inference, it can be directly retrieved by looking up the table using a hash function mapping. Such an intermediate state implicit code carries the original semantic activation strength of the current word in the memory embedding vocabulary, reflecting the "static" representation of the word itself in the dictionary space.

[0063] The element-wise multiplication in step S204 can refer to multiplying two vectors of the same dimension independently, component by component, to obtain an output vector with the same dimension as the input. In the embodiments of this disclosure, the relevance vector represents the semantic relevance of the context to each basis vector, and is obtained by combining it with the intermediate latent encoding representing the activation intensity of the lexical itself to each basis vector. Element-wise multiplication allows the activation intensity of each dictionary basis vector dimension to be dynamically modulated by context relevance.

[0064] In some embodiments, the intermediate state implicit encoding of the current word in the current layer includes a first implicit encoding obtained by mapping the current word using a hash function. Using the above method, the implicit encoding of the word corresponding to this layer can be directly obtained through hash mapping, improving computational efficiency.

[0065] In some embodiments, in addition to mapping the current word to obtain the first implicit code, the intermediate implicit code may also include a second implicit code obtained by mapping a phrase consisting of the current word and at least one word preceding the current word through a hash function. Using the above method, the intermediate implicit code can simultaneously contain the first implicit code (word word hash) and the second implicit code (phrase hash). The second implicit code takes the phrase consisting of the current word and its preceding word as input, capturing the n-gram semantic pattern of the local context. By utilizing both types of implicit codes to construct the intermediate implicit code, the gating vector obtained in step S204 can simultaneously fuse word word semantics and short-range context semantics, enhancing the expressive power of the gating vector.

[0066] In step S206, the dictionary basis vectors of the current layer are linearly combined using the gate vectors to obtain the memory embedding residuals of the current layer for the current word.

[0067] In some embodiments, a weighted summation operation can be performed on the dictionary basis vector set using the components of the gated vector as coefficients. For example, this can be achieved by calculating the product of the transpose of the dictionary matrix and the gated vector (i.e.,...). g), where g is the gate vector, is used to superimpose the basis vectors into a single vector according to the gate weights. Using the above method, multiple discrete atomic semantic vectors can be fused into a continuous memory representation, which can then be injected into the backbone network of a large language model in subsequent steps.

[0068] In some embodiments, step S206 may include: calculating the product of the transpose of the shared dictionary matrix and the gate vector to obtain a linear combination representation, and transforming the linear combination representation through a value projection matrix to obtain a memory embedding residual. Using the above method, the result obtained from the linear combination can be mapped to a semantic space with the same dimension as the current word's contextual hidden state in the current layer, thereby decoupling the constraints between the dimension of the dictionary basis vectors and the dimension of the hidden state, and avoiding the semantic misalignment problem caused by directly injecting dictionary space vectors into the hidden state space.

[0069] In step S208, the memory embedding residual is injected into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model.

[0070] The context-aware process refers to the complete computational flow of a large language model in which the semantic understanding and representation of the current word are updated at the current layer. The context-aware process takes the contextual hidden state as its core carrier and refines the semantic representation of the word layer by layer through modules such as attention mechanisms and feedforward networks.

[0071] After step S208 injects the memory embedding residual into the contextual hidden state, the subsequent context-aware process will continue on the basis of the hidden state that has been incorporated into the semantics of the dictionary basis vectors, thereby realizing the continuous influence of the semantics of the dictionary basis vectors on the entire context-aware process.

[0072] The injection operation in step S208 allows an additional component to be superimposed on the contextual hidden state obtained in the backbone network, thereby obtaining semantic enhancement from the dictionary basis vectors while maintaining the integrity of the contextual representation. In some embodiments, the memory is embedded into the hidden state of the large language model through a residual bypass, thus realizing the embedding of the memory into the contextual hidden state. Using the above method, the memory embedding module is mounted in a bypass form, without modifying the structure and parameters of the backbone network itself, and retaining the original context-aware capability of the backbone network.

[0073] Without departing from the principles of this disclosure, those skilled in the art can implement memory embedding residual injection using various suitable methods. For example, the memory embedding residual can be injected into an intermediate hidden state after the output of the attention sublayer of the current layer and before the input of the feedforward network sublayer, or it can be injected into the final output hidden state of the current layer. For example, before injecting the memory embedding residual into the context hidden state, the memory embedding residual obtained in step S206 can be scaled, normalized, or subjected to threshold-based filtering, thereby improving the stability of the injected semantics.

[0074] In step S210, the text prediction result is output using a large language model based on the contextual latent state injected with memory embedding residuals.

[0075] After step S208, where the memory is embedded into the residual and injected into the contextual hidden state, the contextual hidden state has incorporated the static semantic information carried by the dictionary basis vectors. Compared to the original contextual hidden state, it has been enhanced in terms of the semantic dimension corresponding to the dictionary basis vectors, thus enabling the large language model to more accurately perceive the memory content related to the semantics of the current word in subsequent calculations.

[0076] Step S210 uses the enhanced contextual hidden state as input to drive subsequent computations of the large language model (including the attention mechanism and feedforward network of the remaining layers). Finally, the hidden state is mapped to the vocabulary space via the language model head (LM Head), outputting the text prediction result for the current word position. The text prediction result can represent the model's prediction output for the next word under the current context, i.e., the probability distribution of the next word.

[0077] It is understood that the implementation of step S210 is not limited to this without departing from the principles of this disclosure. For example, the text prediction result can be a single-step prediction output for the current word position, or it can be an autoregressive continuous decoding output for multiple subsequent word positions; the language model head can share the word embedding matrix with the backbone network of the large language model, or it can use an independent linear projection matrix; the output space can be a probability distribution on the complete vocabulary, or it can be further filtered and generated during the inference stage using decoding strategies such as Top-K sampling, Top-P (kernel sampling), or beam search. A suitable output structure can be selected for the large language model according to the actual situation.

[0078] Figure 2 This paper describes the process of injecting memory embedding residuals into a single network layer in a large language model. It is understood that when the structure of a large language model includes multiple intermediate network layers with output context hidden states, those skilled in the art can select one or more of these intermediate network layers to perform the aforementioned memory embedding residual injection operation, thereby enhancing the model's performance. The selected intermediate network layers can also be discontinuous, and their specific form is not limited here.

[0079] Furthermore, when selecting multiple intermediate network layers for memory embedding residual injection, the corresponding dictionary basis vectors and hidden coding parameters of each intermediate network layer can be the same or different. Configuring the same shared parameters for different intermediate network layers can reduce the number of model parameters and the computational resources required for model inference and training. Configuring parameters independently for different intermediate network layers allows the memory embedding residual injection of each layer to adaptively capture semantic abstract features at different levels, thereby improving the model's ability to express complex semantic relationships and the accuracy of text prediction.

[0080] Figure 3 An exemplary flowchart of a method for training a large language model according to embodiments of the present disclosure is shown.

[0081] In step S302, a training dataset can be obtained, which includes sample text input and real text output corresponding to the sample text input.

[0082] A training dataset is a collection of supervised corpora used to tune the parameters of a large language model. Each sample consists of sample text input (i.e., the text sequence input to the large language model, such as natural language sentences, document fragments, etc.) and real text output (i.e., the labeled target sequence corresponding to the sample text input, such as the real label or target text sequence of a word below). Training datasets can come from large-scale pre-training corpora or fine-tuning datasets built for specific tasks.

[0083] In step S304, the current parameters of the large language model can be used to process the sample text input to obtain the predicted text output for the sample text.

[0084] Specifically, the processing procedure of step S304 is similar to... Figure 2 The text data processing method shown is consistent, that is, for the current word in the sample text input, the operations described in steps S202 to S210 are sequentially performed, including determining the relevance vector, calculating the gating vector, generating the memory embedding residual, injecting the residual, and predicting the text output, to finally obtain the predicted text output. The current parameters refer to the parameter state held by the large language model in the current training iteration, including the backbone network parameters, the shared dictionary matrix D, the gating projection matrices K and V, and the intermediate hidden codes in the hash lookup table. The above parameters participate in gradient updates together with the backbone network parameters of the large language model during end-to-end training.

[0085] In step S306, the current parameters of the large language model are adjusted by optimizing a loss function that includes at least the difference between the predicted text output and the real text output.

[0086] The loss function is an objective function that measures the degree of difference between the predicted text output and the real text output. Its optimization result directly determines the direction and magnitude of parameter updates for the large language model. In the embodiments of this disclosure, the loss function includes at least a term representing the difference between the predicted and real text outputs, such as cross-entropy loss, which measures the deviation between the model's predicted probability distribution at each lexical position and the true label. By optimizing the loss function through backpropagation and gradient descent, the current parameters of the large language model can be adjusted, allowing the model's predicted text output to gradually approach the real text output, thereby improving the model's language modeling ability.

[0087] In some embodiments, the loss function may further include an L1 loss term for intermediate-state implicit coding. Specifically, the loss function consists of a difference term between the predicted text output and the true text output (such as cross-entropy loss) and intermediate-state implicit coding. The L1 norm regularization term is constructed by weighted summation. The L1 loss term imposes sparsity constraints on the intermediate implicit encoding, thereby promoting... Most components in the sparse latent code approach zero, retaining only a few non-zero activation components that are highly relevant to the semantics of the current lexical unit. This naturally shapes the sparse structure of the sparse latent code during training. This sparsity constraint helps improve the discriminativeness of the gating vectors, enabling basis vector-level gating to more accurately select dictionary basis vectors relevant to the current context. Furthermore, it reduces the computational overhead during inference because the sparse latent code... This makes a large number of components in the gated vector close to zero, and the corresponding dictionary basis vectors contribute very little to the linear combination. Inference can be accelerated through sparse computation optimization.

[0088] The method for training a large language model using the embodiments of this disclosure achieves end-to-end joint optimization of the language modeling loss and the L1 sparse regularization loss of the intermediate implicit coding. During the training phase, the backbone network parameters, shared dictionary matrix, gated projection matrix and intermediate implicit coding are simultaneously optimized. This achieves deep adaptation between the basis vector level gating mechanism and the backbone network of the large language model, effectively improving the language modeling capability and semantic accuracy of memory embedding of the large language model without modifying the backbone network structure.

[0089] Figure 4 A schematic diagram of the overall architecture of a basis vector level gating algorithm in a text data processing method according to an embodiment of the present disclosure is shown. Figure 4 As shown, the module used to perform memory embedding residual injection is mounted in the shallow layer of the backbone network. The overall computation process can be divided into three stages: implicit coding lookup, gated computation, and residual synthesis.

[0090] During the implicit coding lookup phase, for the term T at the current position t The input is constructed using three granularities: single token, 2-gram, and 3-gram. The input is based on the word unit T. t As input, with (T) t 1,T t ) as 2-gram input, with (T t 2,T t 1,T t The three granularities of words are used as 3-gram input. Each word is mapped using a hash function to obtain its corresponding intermediate hidden coding component. The hash lookup results of each granularity are then concatenated using a concat operation to obtain the intermediate hidden coding of the current word in the current layer. This is sparse implicit coding. Intermediate state implicit coding. Pre-stored in a hash lookup table, only a forward lookup needs to be performed during inference, without the need for gradient updates.

[0091] Understandable, Figure 4 The document illustrates implicit lookup methods at three granularities: token, 2-gram, and 3-gram. Without departing from the principles of this disclosure, the granularity of implicit lookup is not limited to these; it may also include 4-gram and higher-order n-grams, or use only one or a combination of two of these granularities to achieve a balance between parameter size and semantic coverage.

[0092] During the gated computation phase, the contextual hidden state h of the l-th layer of the backbone network... t,l Together with the dictionary matrix D, they participate in the gating computation to determine the correlation vector α between the contextual hidden state and the dictionary basis vectors. t This leads to the gate vector Gate( The formula for calculating the gate vector can be expressed as:

[0093] Here, ⊙ denotes element-wise multiplication. Specifically, each row of dictionary D (i.e., each dictionary basis vector) is first multiplied by the key projection matrix K to obtain the representation DK of each basis vector in the key space. Then, the hidden context state h of the current layer is... t,l After RMSNorm normalization, the result is compared with the normalized DK. The inner product similarity is calculated for each row, and then activated by the element-wise Sigmoid activation function σ( After activation, d is obtained. dict 1-dimensional relevance vector α t Relevance vector α t Each component represents the semantic relevance of the corresponding dictionary basis vector dimension to the current context, and its dimension is consistent with the number of dictionary basis vectors.

[0094] Subsequently, the relevance vector α t With sparse implicit coding Perform element-wise multiplication (i.e.) Figure 4 In (operation), to obtain the gate vector Gate( The gating vector integrates the static semantic activation strength of the lexical unit in the dictionary space (by...). Provided) and the degree of relevance to the dynamic semantics of the context (by α) t It provides differential modulation at the basis vector level. Compared to traditional scalar-level gating that uniformly scales the entire memory embedding vector, basis vector-level gating calculates the relevance of the retrieved content to the context and the final activation weights dimension by dimension, thereby differentially and specifically enhancing or suppressing different semantic components of the memory content, thus overcoming the semantic noise problem introduced by the uniform scaling of scalar-level gating.

[0095] In the residual synthesis stage, the gate vector Gate( The dictionary basis vectors in dictionary D are linearly combined and then value-projected to obtain the final memory embedding residuals, which are then injected into the contextual hidden states of the backbone network via residual bypass. That is, .in Identify the injected contextual hidden state.

[0096] Specifically, the transpose of the shared dictionary matrix D and the gate vector Gate are calculated. The product of D) Gate This yields a linear combination representation; then, the linear combination representation is linearly transformed using the value projection matrix V to obtain the memory embedding residual for the current word in the current layer. Finally, the residual is bypassed (…). Figure 4 The left-hand ⊕ operation embeds the memory into the residual injection back into the contextual hidden state h of the backbone network. t,l This involves embedding the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model.

[0097] because Figure 4 The modules shown are mounted on the backbone network using a residual bypass method, without modifying the backbone network's feedforward network or other structures. Therefore, they can be incrementally mounted onto a regular backbone network and then directly fine-tuned or deployed, supporting incremental mounting of new memory embedding vocabularies. Furthermore, the dictionary matrix D, gated projection matrices K and V, and intermediate state implicit codes in the hash lookup table are also included. All parameters can be incorporated into the existing large language model pre-training process for end-to-end training, fully adapting to the existing training paradigm of large language models.

[0098] remove Figure 4 In addition to the mounting modes shown, you can also mount the other modes according to the actual situation. Figure 4 The bypass modules shown can be integrated into the model in other ways. For example, the bypass modules described above can be used to replace the up-projection computation branch of a SwiGLU-style feedforward network layer, thereby introducing a basis vector-level gating algorithm for memory embedding of different network structures.

[0099] Therefore, the method provided by the embodiments of this disclosure simultaneously considers the shared utilization rate of embedding parameters and the mixed semantic properties of memory content. Based on the two-layer decomposition structure of the shared dictionary by implicitly encoding the memory embedding, a basis vector-level context gating algorithm is designed to independently calculate the backbone integration coefficient for each activated basis vector in the shared dictionary. This algorithm can be flexibly applied to the memory embedding vocabulary module of large language models with different architectures, thereby improving the overall language modeling capability of the model.

[0100] Figure 5 An exemplary block diagram of a text data processing apparatus according to an embodiment of the present disclosure is shown.

[0101] like Figure 5 As shown, the device 500 includes a correlation determination unit 510, a gating determination unit 520, a memory embedding residual determination unit 530, an injection unit 540, and a prediction unit 550.

[0102] The relevance determination unit 510 can be configured to determine the relevance vector between the current word element in the contextual hidden state of the current layer of the large language model and the dictionary basis vector of the current layer.

[0103] The gating determination unit 520 can be configured to perform element-wise multiplication of the relevance vector and the intermediate hidden encoding of the current word in the current layer to obtain the gating vector.

[0104] The memory embedding residual determination unit 530 can be configured to use a gating vector to linearly combine the dictionary basis vectors of the current layer to obtain the memory embedding residual of the current layer for the current word.

[0105] The injection unit 540 can be configured to inject memory into the residual injection contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model.

[0106] Prediction unit 550 can be configured to output text prediction results based on the contextual latent state of a large language model with injected memory embedding residuals.

[0107] In some embodiments, the intermediate state implicit encoding of the current term in the current layer includes a first implicit encoding obtained by mapping the current term through a hash function.

[0108] In some embodiments, the intermediate state implicit encoding of the current word element in the current layer further includes a second implicit encoding obtained by mapping a word group consisting of the current word element and at least one word element preceding the current word element through a hash function.

[0109] In some embodiments, the relevance determination unit is configured to: multiply the dictionary basis vectors by the key projection matrix to obtain the key space representation of the dictionary basis vectors; normalize the contextual hidden state and the key space representation respectively; calculate the inner product similarity between the normalized contextual hidden state and the normalized key space representation; and apply an activation function to the inner product similarity to obtain the relevance vector.

[0110] In some embodiments, the normalization process is root mean square normalization.

[0111] In some embodiments, the activation function is the element-wise sigmoid function.

[0112] In some embodiments, the dimension of the relevance vector is the same as the number of dictionary basis vectors, and each component of the relevance vector represents the degree of semantic relevance between the corresponding dictionary basis vector dimension and the current context.

[0113] In some embodiments, the dictionary basis vectors constitute a dictionary matrix.

[0114] In some embodiments, the memory embedding residual determination unit is configured to: calculate the product of the transpose of the shared dictionary matrix and the gate vector to obtain a linear combination representation; and transform the linear combination representation through a value projection matrix to obtain the memory embedding residual.

[0115] In some embodiments, memory is embedded into the hidden state of the large language model by injecting the residual back into the hidden state of the residual through residual bypass.

[0116] It should be understood that Figure 5 The various modules or units of the apparatus 500 shown can be connected to the reference. Figure 2 The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for method 200 also apply to apparatus 500 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0117] Figure 6 An exemplary block diagram of a training apparatus for a large language model according to an embodiment of the present disclosure is shown.

[0118] like Figure 6 As shown, the device 600 includes a training data acquisition unit 610, a prediction unit 620, and a training unit 630.

[0119] The training data acquisition unit 610 is configured to acquire a training dataset, wherein the training dataset includes sample text input and real text output corresponding to the sample text input.

[0120] The prediction unit 620 is configured to process the sample text input using the current parameters of the large language model to obtain a predicted text output for the sample text. This includes: determining the relevance vector between the current word in the sample text input and the contextual hidden state of the current layer of the large language model and the dictionary basis vector of the current layer; multiplying the relevance vector and the intermediate hidden code of the current word in the current layer element-wise to obtain a gating vector; using the gating vector to linearly combine the dictionary basis vector of the current layer to obtain the memory embedding residual of the current layer for the current word; injecting the memory embedding residual into the contextual hidden state to embed the semantics of the dictionary basis vector of the current layer into the context-aware process of the large language model; and using the large language model to output a predicted text output based on the contextual hidden state injected with the memory embedding residual.

[0121] Training unit 630 is configured to tune the current parameters of the large language model by optimizing a loss function that includes at least the difference between the predicted text output and the real text output.

[0122] In some embodiments, the loss function further includes an L1 loss term for intermediate state implicit coding.

[0123] It should be understood that Figure 6 The various modules or units of the device 600 shown can be connected to the reference. Figure 3 The steps in method 300 described correspond to each other. Therefore, the operations, features, and advantages described above for method 300 also apply to apparatus 600 and its included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0124] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the units discussed in this article can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit.

[0125] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0126] According to embodiments of this disclosure, an electronic device, a readable storage medium, and a computer program product are also provided.

[0127] An electronic device provided according to embodiments of the present disclosure may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to implement the method according to embodiments of the present disclosure.

[0128] An electronic device provided according to embodiments of the present disclosure may include a transient or non-transient computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform a method according to embodiments of the present disclosure.

[0129] A computer program product provided according to embodiments of the present disclosure may include a computer program, wherein the computer program, when executed by a processor, implements a method according to embodiments of the present disclosure.

[0130] refer to Figure 7The present invention describes a structural block diagram of an electronic device 700 that can serve as a server or client of the present disclosure, which is an example of a hardware device that can be applied to various aspects of the present disclosure. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0131] like Figure 7 As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 may also store various programs and data required for the operation of the electronic device 700. The computing unit 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0132] Multiple components in electronic device 700 are connected to I / O interface 705, including: input unit 706, output unit 707, storage unit 708, and communication unit 709. Input unit 706 can be any type of device capable of inputting information to electronic device 700. Input unit 706 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device, and may include, but is not limited to, a mouse, keyboard, touchscreen, trackpad, trackball, joystick, microphone, and / or remote control. Output unit 707 can be any type of device capable of presenting information, and may include, but is not limited to, a monitor, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 708 may include, but is not limited to, disk and optical disk. Communication unit 709 allows electronic device 700 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0133] The computing unit 701 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs the various methods and processes described above, such as methods 200 and 300. For example, in some embodiments, methods 200 and 300 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by the computing unit 701, one or more steps of methods 200 and 300 described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform methods 200, 300 by any other suitable means (e.g., by means of firmware).

[0134] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0136] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0138] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0139] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0140] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0141] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. A text data processing method, comprising: Determine the correlation vector between the contextual hidden state of the current word element in the current layer of the large language model and the dictionary basis vector of the current layer; The relevance vector and the intermediate hidden code of the current word in the current layer are multiplied element-wise to obtain the gating vector; The gating vector is used to linearly combine the dictionary basis vectors of the current layer to obtain the memory embedding residual of the current layer for the current word; The memory embedding residual is injected into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model; The large language model is used to output text prediction results based on the contextual latent state injected with the memory embedding residuals.

2. The method according to claim 1, wherein, The intermediate state implicit encoding of the current word in the current layer includes a first implicit encoding obtained by mapping the current word through a hash function.

3. The method according to claim 2, wherein, The intermediate state implicit encoding of the current word element in the current layer also includes a second implicit encoding obtained by mapping a word group consisting of the current word element and at least one word element preceding the current word element through a hash function.

4. The method according to claim 1, wherein, Determining the relevance vector between the contextual hidden state of the current word element in the current layer of the large language model and the dictionary basis vector of the current layer includes: Multiply the dictionary basis vectors by the key projection matrix to obtain the key space representation of the dictionary basis vectors; The context hidden state and the key space representation are normalized respectively; Calculate the inner product similarity between the normalized contextual hidden state and the normalized key space representation; An activation function is applied to the inner product similarity to obtain the relevance vector.

5. The method according to claim 4, wherein, The normalization process is root mean square normalization.

6. The method according to claim 4, wherein, The activation function is the element-wise Sigmoid function.

7. The method according to claim 1, wherein, The dimension of the relevance vector is the same as the number of dictionary base vectors, and each component of the relevance vector represents the semantic relevance of the corresponding dictionary base vector dimension to the current context.

8. The method according to claim 1, wherein, The dictionary basis vectors form the dictionary matrix.

9. The method according to claim 8, wherein, The step of linearly combining the dictionary basis vectors of the current layer using the gate vector to obtain the memory embedding residual of the current layer for the current word includes: Calculate the product of the transpose of the dictionary matrix and the gate vector to obtain a linear combination representation; The linear combination representation is transformed by a value projection matrix to obtain the memory embedding residual.

10. The method according to claim 1, wherein, By using residual bypass, the memory is embedded into the hidden state of the large language model and injected back into the residual, thus embedding the memory into the contextual hidden state.

11. A training method for large language models, comprising: Obtain a training dataset, wherein the training dataset includes sample text input and real text output corresponding to the sample text input. The sample text input is processed using the current parameters of the large language model to obtain a predicted text output for the sample text, including: Determine the relevance vector between the current word element in the sample text input and the contextual hidden state of the current layer of the large language model and the dictionary basis vector of the current layer. The relevance vector and the intermediate hidden code of the current word in the current layer are multiplied element-wise to obtain the gating vector. The gating vector is used to linearly combine the dictionary basis vectors of the current layer to obtain the memory embedding residual of the current layer for the current word. The memory embedding residual is injected into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model. The predicted text is output using the large language model based on the contextual hidden state injected with the memory embedding residuals; The current parameters of the large language model are adjusted by optimizing a loss function that includes at least the difference between the predicted text output and the real text output.

12. The training method as described in claim 11, wherein, The loss function also includes the L1 loss term of the intermediate state implicit encoding.

13. A text data processing apparatus, comprising: The relevance determination unit is configured to determine the relevance vector between the contextual hidden state of the current word element in the current layer of the large language model and the dictionary basis vector of the current layer. The gating determination unit is configured to perform element-wise multiplication of the relevance vector and the intermediate hidden code of the current word in the current layer to obtain the gating vector; The memory embedding residual determination unit is configured to use the gate vector to linearly combine the dictionary basis vectors of the current layer to obtain the memory embedding residual of the current layer for the current word; The injection unit is configured to inject the memory embedding residual into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model; The prediction unit is configured to output a text prediction result based on the contextual latent state injected with the memory embedding residual using the large language model.

14. A training device for a large language model, comprising: The training data acquisition unit is configured to acquire a training dataset, wherein the training dataset includes sample text input and real text output corresponding to the sample text input. A prediction unit, configured to process the sample text input using the current parameters of the large language model to obtain a predicted text output for the sample text, includes: Determine the relevance vector between the current word element in the sample text input and the contextual hidden state of the current layer of the large language model and the dictionary basis vector of the current layer. The relevance vector and the intermediate hidden code of the current word in the current layer are multiplied element-wise to obtain the gating vector. The gating vector is used to linearly combine the dictionary basis vectors of the current layer to obtain the memory embedding residual of the current layer for the current word. The memory embedding residual is injected into the contextual hidden state to embed the semantics of the dictionary basis vectors of the current layer into the context-aware process of the large language model. The predicted text is output using the large language model based on the contextual hidden state injected with the memory embedding residuals; The training unit is configured to adjust the current parameters of the large language model by optimizing a loss function that includes at least the difference between the predicted text output and the real text output.

15. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-12.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-12.

17. A computer program product comprising a computer program, wherein, When the computer program is executed by a processor, it implements the method of any one of claims 1-12.

Images