Training method, device, computer equipment and readable storage medium for large language model based on trainable residual connection and dual-scale convolutional Transformer

Abstract

The present invention discloses a training method, apparatus, computer device, and readable storage medium for a large language model based on trainable residual connections and a two-scale convolutional Transformer. The method comprises: first obtaining a base model based on a multi-layer Transformer architecture, wherein each layer contains self-attention and feedforward networks and is embedded with a two-scale convolutional module. The outputs are fused as the output of the layer, and a trainable weight matrix is ​​configured between the input and output of each layer to adjust the residual connection strength. A training set is constructed by obtaining preprocessed sample documents. Based on this, the base model is trained to preset conditions to obtain a large language model that integrates trainable residual connections and two-scale convolutions, which can improve model performance and generalization ability.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method, apparatus, computer equipment, and readable storage medium for a large language model based on trainable residual connections and a dual-scale convolutional Transformer. Background Art

[0002] With the development of artificial intelligence, language models have evolved from basic models to neural network models. Large language models such as ChatGPT have demonstrated remarkable capabilities, leveraging massive datasets and achieving success in multiple fields, reshaping human-computer interaction. However, despite its success, the Transformer model still faces challenges. Its emphasis on capturing global information while neglecting local information capture, and its reliance on residual connections, however, increases the difficulty of information transmission as the number of network layers increases, leading to significant bottlenecks when processing long sequences of data. Summary of the Invention

[0003] The object of the present invention is to provide a training method, apparatus, computer equipment and readable storage medium for a large language model based on trainable residual connections and a dual-scale convolutional Transformer.

[0004] In a first aspect, an embodiment of the present invention provides a method for training a large language model based on a trainable residual connection and a two-scale convolutional Transformer, comprising:

[0005] Obtaining a base model based on a multi-layer Transformer architecture, wherein the base model includes multiple cascaded Transformer decoder layers, each Transformer decoder layer includes a self-attention mechanism and a feedforward neural network, each Transformer decoder layer is embedded with a two-scale convolution module, the two-scale convolution modules of different scales are used to capture different local features, the outputs of the two-scale convolution modules are fused as the output of the corresponding Transformer decoder layer, and a trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer, and the trainable weight matrix is ​​used to adjust the strength of the residual connection between each Transformer decoder layer;

[0006] Obtain preprocessed sample document data to build a training set;

[0007] The basic model based on the multi-layer Transformer architecture is trained based on the training set until a preset training termination condition is reached, thereby obtaining a trained large language model that integrates trainable residual connections and dual-scale convolutional Transformers.

[0008] In a possible implementation, the training of the basic model based on the multi-layer Transformer architecture based on the training set includes:

[0009] Performing unsupervised learning based on the training set to complete pre-training of the basic model based on the multi-layer Transformer architecture to learn the statistical characteristics and contextual relationships of the language;

[0010] Fine-tune the pre-trained base model based on the multi-layer Transformer architecture according to the preset downstream tasks;

[0011] The basic model based on the multi-layer Transformer architecture after supervised fine-tuning is combined with the reward model to perform reinforcement learning through the preset strategy optimization algorithm.

[0012] In a possible implementation, performing unsupervised learning based on the training set includes:

[0013] The text data included in the training set is converted into a plurality of digital sequences through a word segmenter; the numbers included in the digital sequences are index numbers of the text data in the dictionary;

[0014] Obtaining multiple high-dimensional vectors of high dimension by embedding the multiple digital sequences;

[0015] Performing inference operations according to the multiple high-dimensional directions to obtain index sequences corresponding to the multiple digital sequences;

[0016] The word segmenter is used in combination with the index sequence to perform text restoration to obtain the target natural language.

[0017] In one possible implementation, performing an inference operation based on the multiple high-dimensional vectors to obtain index sequences corresponding to the multiple digital sequences includes:

[0018] Normalize the high-dimensional vectors corresponding to each layer of the Transformer architecture, and process the normalized original features through a linear layer;

[0019] The processed original features are divided into blocks on the time axis to obtain the first features; the block operation adopts an overlap factor of 50%;

[0020] A multi-scale Transformer using a trainable residual connection structure and adding a convolutional module uses an intra-frame Transformer and an inter-frame Transformer to learn the short-term and long-term dependencies of the first features respectively to obtain a second feature, and the intra-frame Transformer and the inter-frame Transformer have the same structure;

[0021] The second feature generated by the multi-scale Transformer is then passed through a PReLU activation function and a linear layer to obtain the third feature;

[0022] Performing an overlap-add operation on the third feature to obtain a fourth feature; the overlap rate of the overlap-add operation is 0.5;

[0023] Processing the fourth feature through a ReLU activation function and a feedforward network layer, using nonlinear transformation to extract and learn the feature representation of each token vocabulary;

[0024] When reaching the last layer of the Transformer architecture, the SoftMax function is used to obtain the sampling probability of all digital sequences, and the next digital sequence is sampled according to the preset sampling algorithm;

[0025] Repeat the steps of normalizing the high-dimensional vector corresponding to each layer of the Transformer architecture and processing the normalized original features through a linear layer, until the fourth feature is processed through a ReLU activation function and a feedforward network layer, and a nonlinear transformation is used to extract and learn the feature representation of each token vocabulary, until the preset number of cycles is reached, and then the index sequence is generated.

[0026] In a possible implementation, the multi-scale Transformer using a trainable residual connection structure and adding a convolution module uses an intra-frame Transformer and an inter-frame Transformer to respectively learn the short-term and long-term dependencies of the first feature to obtain the second feature, including:

[0027] Assigning a unique representation to each position of the first feature using absolute position encoding to obtain an intermediate first feature;

[0028] Performing a linear transformation of weights and biases on the intermediate first feature through a feedforward neural network, and introducing a nonlinear transformation through a ReLU function to obtain an intermediate second feature;

[0029] The intermediate second feature is processed through a multi-head attention mechanism to obtain an intermediate third feature;

[0030] Processing the intermediate third feature through a dual-scale convolution module to obtain an intermediate fourth feature and an intermediate fifth feature;

[0031] Processing the intermediate fourth feature and the intermediate fifth feature through the Mish activation function to obtain the intermediate sixth feature and the intermediate seventh feature;

[0032] splicing the intermediate sixth feature and the intermediate seventh feature to obtain an intermediate eighth feature;

[0033] The intermediate eighth feature is subjected to adaptive average pooling to obtain an intermediate ninth feature, and the intermediate ninth feature is processed by the feedforward neural network to obtain the second feature.

[0034] In a possible implementation, processing the intermediate ninth feature through the feedforward neural network to obtain the second feature includes:

[0035] The intermediate ninth feature in the intra-frame Transformer and the inter-frame Transformer is added to the input of each feedforward neural network layer through a trainable residual connection and then subjected to a layer normalization operation to obtain the second feature.

[0036] In one possible implementation, the trainable residual connection is represented by the formula: y=F(W*x);

[0037] Where x represents the input, W represents the trainable weight matrix; F(x) represents the nonlinear mapping function from input to output, and y represents the output.

[0038] In a second aspect, an embodiment of the present invention provides a training device for a large language model based on a trainable residual connection and a two-scale convolutional Transformer, comprising:

[0039] An acquisition module is used to obtain a basic model based on a multi-layer Transformer architecture, wherein the basic model includes multiple cascaded Transformer decoder layers, each of which includes a self-attention mechanism and a feedforward neural network, and each of which is embedded with a two-scale convolution module. The two-scale convolution modules of different scales are used to capture different local features. The outputs of the two-scale convolution modules are fused as the output of the corresponding Transformer decoder layer. A trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer, and the trainable weight matrix is ​​used to adjust the strength of the residual connection between each Transformer decoder layer; obtaining preprocessed sample document data to construct a training set;

[0040] A training module is used to train the basic model based on the multi-layer Transformer architecture based on the training set until a preset training termination condition is reached, thereby obtaining a trained large language model that integrates trainable residual connections and dual-scale convolutional Transformers.

[0041] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method described in the first aspect.

[0042] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, which includes a computer program, and when the computer program is running, it controls the computer device where the readable storage medium is located to execute the method described in the first aspect. Compared with the prior art, the beneficial effects provided by the present invention include: using a training method, device, computer device and readable storage medium for a large language model based on a trainable residual connection and a two-scale convolutional Transformer disclosed in the present invention, including: first obtaining a basic model based on a multi-layer Transformer architecture, each layer of which contains self-attention and a feedforward network and is embedded with a two-scale convolution module, and the output is fused as the output of the layer, and a trainable weight matrix is ​​configured between the input and output of each layer to adjust the residual connection strength. Preprocessed sample documents are obtained to construct a training set, and based on this, the basic model is trained to preset conditions to obtain a large language model that integrates trainable residual connections and two-scale convolutions, which can improve model performance and generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly describes the drawings required for use in the embodiments. It should be understood that the following drawings illustrate only certain embodiments of the present invention and should not be construed as limiting the scope of the present invention. Those skilled in the art can, without inventive effort, derive other relevant drawings from these drawings.

[0044] Figure 1 A schematic flow chart of the steps of a large language model training method based on trainable residual connections and a dual-scale convolutional Transformer provided in an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the architecture of a large language model based on trainable residual connections and dual-scale convolutional Transformer provided by an embodiment of the present invention;

[0046] Figure 3 Schematic diagram of the internal structure of the TransformerX and dual-scale convolution multi-scale Transformer modules provided by the embodiments of the present invention;

[0047] Figure 4 A schematic diagram of the internal structure of a dual-scale convolutional Transformer with a trainable residual connection structure provided by an embodiment of the present invention;

[0048] Figure 5Schematic diagram of the internal structure of a dual-scale convolution module and a single convolution module provided in an embodiment of the present invention;

[0049] Figure 6 A schematic block diagram of the structure of a large language model training device based on trainable residual connections and dual-scale convolutional Transformer provided by an embodiment of the present invention;

[0050] Figure 7 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0051] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the embodiments of the present invention. It should be understood that the described embodiments are only a portion of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations.

[0052] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0053] In order to solve the technical problems in the above background technology, Figure 1 A flow chart of a method for training a large language model based on trainable residual connections and a two-scale convolutional transformer is provided in an embodiment of the present disclosure. The following is a detailed introduction to the method for training a large language model based on trainable residual connections and a two-scale convolutional transformer.

[0054] Step S201: Obtain a basic model based on a multi-layer Transformer architecture, wherein the basic model includes multiple cascaded Transformer decoder layers, each Transformer decoder layer includes a self-attention mechanism and a feedforward neural network, and each Transformer decoder layer is embedded with a two-scale convolution module. The two-scale convolution modules of different scales are used to capture different local features. The outputs of the two-scale convolution modules are fused as the output of the corresponding Transformer decoder layer. A trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer, and the trainable weight matrix is ​​used to adjust the strength of the residual connection between each Transformer decoder layer.

[0055] Step S202: obtaining pre-processed sample document data to construct a training set;

[0056] Step S203: Train the basic model based on the multi-layer Transformer architecture based on the training set until a preset training termination condition is reached, thereby obtaining a trained large language model that integrates trainable residual connections and dual-scale convolutional Transformers.

[0057] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0058] Get the base model based on a multi-layer Transformer architecture:

[0059] The server receives a natural language processing task and needs to build a large language model. First, the server creates a base model based on a multi-layer Transformer architecture. For example, this base model consists of 12 Transformer decoder layers, each of which includes a self-attention mechanism and a feedforward neural network. A two-scale convolution module is embedded in each Transformer decoder layer. One of these two-scale convolution modules has kernel sizes of 3 and 7, respectively, to capture local features at different scales. The outputs of these two convolution kernels are concatenated and fused to form the output of the Transformer decoder layer. Furthermore, a trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer to adjust the strength of the residual connection. As training progresses, this weight matrix is ​​continuously learned and adjusted to better adapt to different language tasks and data.

[0060] Get preprocessed sample document data to build a training set:

[0061] The server collects a large amount of text data from the Internet, including news articles, academic papers, novels, etc. Then, this data is preprocessed to construct a training set.

[0062] First prepare the document:

[0063] Read data from WARC files, avoiding the complexity of extracting content from HTML and reducing the interference of irrelevant information.

[0064] Perform preliminary filtering of URLs to exclude fraudulent and illegal websites, based on a domain blocklist and URL scoring based on a specific wordlist, while excluding URLs from high-quality text datasets such as Wikipedia and arXiv.

[0065] Use the Trafilatura library and regular expressions to extract the main content from an HTML page, ignoring menus, headers, footers, advertisements, etc., limiting new lines to two consecutive lines and removing all URL links.

[0066] Use language classifiers such as fastText to identify the language, select texts in the corresponding language, and delete documents with language scores below a threshold.

[0067] Then filter:

[0068] A heuristic approach is used to formulate rules to remove documents with excessive line, paragraph, or n-gram duplication to reduce costs and improve efficiency.

[0069] It primarily retains natural language documents written by humans, removes machine-generated spam, and uses quality filtering heuristics to remove outliers based on criteria such as document length and symbol-to-word ratio.

[0070] Use linear correction filters to remove content irrelevant to the main text (such as likes, navigation buttons, etc.) to ensure data quality.

[0071] Finally, perform deduplication:

[0072] The MinHash algorithm is used to calculate the approximate similarity between documents and remove document pairs with high overlap.

[0073] Use the suffix array to find exact matches between strings, removing paragraphs that repeat more than k consecutive tokens.

[0074] Removed duplicated URLs in cross-CC dumps to ensure data uniqueness.

[0075] After these preprocessing steps, a training set containing a large amount of high-quality text data was finally constructed for subsequent model training.

[0076] Train the basic model based on the training set:

[0077] The server inputs the constructed training set into the basic model based on the multi-layer Transformer architecture for training.

[0078] During the pre-training phase, the model performs unsupervised learning on large amounts of text data, aiming to learn the statistical properties and contextual relationships of language. For example, the model is trained by predicting the next word, completing blanks, or other language tasks to capture the grammatical, semantic, and contextual information of the language. The server continuously adjusts the model's parameters to better understand the patterns and regularities of the language.

[0079] After pre-training is complete, the supervised fine-tuning phase begins. Labeled data is used to fine-tune the model for specific downstream tasks, such as text classification, question answering, and dialogue generation. The server adjusts the model's parameters based on this labeled data to optimize performance on the specific task.

[0080] Next, the reward model phase begins. Multiple responses generated by the same prompt are manually ranked to construct a training sample. The server compares the quality of different responses to create a sample set with relative merits. This ranking information is used to train the reward model, enabling it to predict the quality scores of different responses based on human feedback.

[0081] Finally, the reinforcement learning phase begins. A reward model is used to score the results generated by the supervised fine-tuning model. These scores serve as feedback signals to train the model using a policy optimization algorithm (such as PPO). The goal is to maximize the reward for the model's generated responses. By continuously adjusting the generation strategy, the model can produce higher-quality results for specific tasks.

[0082] During training, the server continuously iteratively updates the model parameters until it reaches the preset training termination criteria, such as reaching a certain accuracy rate or convergence of the loss value. At this point, the server obtains a trained large language model that integrates trainable residual connections and dual-scale convolutional Transformers.

[0083] For example, after millions of training iterations, the model's accuracy on some common natural language processing tasks reaches over 90%, and the loss value stabilizes at a low level. At this point, the model can be considered to have been trained and can be used in actual natural language processing tasks.

[0084] In an embodiment of the present invention, the training of the basic model based on the multi-layer Transformer architecture based on the training set can be implemented through the following examples.

[0085] Performing unsupervised learning based on the training set to complete pre-training of the basic model based on the multi-layer Transformer architecture to learn the statistical characteristics and contextual relationships of the language;

[0086] Fine-tune the pre-trained base model based on the multi-layer Transformer architecture according to the preset downstream tasks;

[0087] The basic model based on the multi-layer Transformer architecture after supervised fine-tuning is combined with the reward model to perform reinforcement learning through the preset strategy optimization algorithm.

[0088] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0089] Unsupervised learning stage (pre-training of the basic model to learn language statistical characteristics and contextual relationships):

[0090] The server receives a large amount of unlabeled text data, which covers a variety of topics and styles.

[0091] During this phase, the server begins unsupervised learning of a basic model based on a multi-layer Transformer architecture. For example, the server will ask the model to process a news article. The model uses a self-attention mechanism to analyze the relationship between each word in the article and other words, gradually understanding the meaning and role of each word in the context.

[0092] For example, in the sentence "White clouds float in the sky, and a gentle breeze blows across the earth," the basic model's self-attention mechanism focuses on the connections between words like "sky," "clouds," "breeze," and "earth." It discovers that "sky" and "clouds" often appear together, and that "breeze" and "earth" also have a certain connection. By continuously processing large amounts of text like this, the model gradually learns statistical characteristics of language, such as word co-occurrence patterns and grammatical structure.

[0093] As the model processes more text, it begins to capture more complex contextual relationships. For example, the word "floating" can have different meanings in different contexts, and its usage and semantics can differ when describing clouds versus other objects. By observing and learning from a large amount of text, the model can accurately understand these differences.

[0094] During this process, the server continuously adjusts the model's parameters, enabling it to more accurately learn the statistical characteristics and contextual relationships of language. For example, the model assigns higher weights to frequently occurring word combinations, enabling it to better handle these combinations in subsequent tasks. After a long period of unsupervised learning, the basic model has initially grasped the fundamental laws and patterns of language, laying a solid foundation for subsequent tasks.

[0095] Supervised fine-tuning phase (fine-tuning the pre-trained basic model according to the preset downstream tasks):

[0096] Once the pre-training phase is complete, the server begins supervised fine-tuning of the base model for specific downstream tasks.

[0097] Assume that the preset downstream task is text classification, such as classifying news articles into different categories such as politics, technology, and entertainment. The server will select a portion of news text with clear category labels as fine-tuning data.

[0098] For a specific political news article, the server feeds it into a pre-trained base model. The model encodes and processes the text based on its previously learned linguistic knowledge. The server then calculates the difference between the model output and the true classification label, or loss, based on the text's actual classification label.

[0099] Using the backpropagation algorithm, the server propagates this loss back into the model's various parameters, adjusting them to bring the model's output closer to the true classification label. For example, if the model initially misclassifies a piece of political news as technology, the server adjusts parameters related to vocabulary understanding and semantic analysis to prioritize the characteristics of political text.

[0100] During this process, the server performs fine-tuning on a large number of texts with classification labels, continuously optimizing the model's performance on specific downstream tasks. After multiple iterations and adjustments, the model gradually adapts to the specific text classification task, and its accuracy on that task gradually improves.

[0101] Reinforcement learning phase (the basic model after supervision fine-tuning is combined with the reward model to perform reinforcement learning through a preset strategy optimization algorithm):

[0102] After supervised fine-tuning is completed, the server introduces the reward model and performs reinforcement learning in combination with the preset policy optimization algorithm.

[0103] First, multiple responses generated by the same prompt were manually ranked to construct training samples. For example, for the question "How can I improve my learning efficiency?", the model generated multiple different responses, such as "Manage your time properly," "Do more exercises," and "Maintain a good learning environment." Humans then ranked these responses based on their own judgment to determine which ones were most effective.

[0104] The server then feeds these ranked samples into a reinforcement learning process. For each generated answer, the reward model assigns a reward score based on the human ranking results. For example, the answer "Manage your time properly" is considered good and will be given a higher score by the reward model, while the answer "Eating more chocolate can improve learning efficiency" might receive a lower score.

[0105] Next, the server uses a policy optimization algorithm (such as PPO) to adjust the model's generation strategy. Based on the reward scores assigned by the reward model, the PPO algorithm attempts to adjust the model's response generation strategy, making it more likely to generate responses with higher reward scores. For example, if the model finds that generating the response "Schedule time properly" earns a higher reward, it will be more likely to generate similar responses in subsequent generation processes.

[0106] During the reinforcement learning process, the server continuously instructs the model to generate answers and adjusts them based on feedback from the reward model until the model achieves optimal results. For example, after multiple rounds of reinforcement learning, the quality of the answers generated by the model has significantly improved, more accurately meeting user needs and providing more reasonable responses to a variety of questions.

[0107] Through these three stages of training, namely unsupervised learning, supervised fine-tuning, and reinforcement learning, the server successfully trained the basic model based on the multi-layer Transformer architecture, enabling it to perform well in various natural language processing tasks and provide users with better language processing services.

[0108] In the embodiment of the present invention, the unsupervised learning based on the training set can be implemented through the following examples.

[0109] The text data included in the training set is converted into a plurality of digital sequences through a word segmenter; the numbers included in the digital sequences are index numbers of the text data in the dictionary;

[0110] Obtaining multiple high-dimensional vectors of high dimension by embedding the multiple digital sequences;

[0111] Performing inference operations according to the multiple high-dimensional directions to obtain index sequences corresponding to the multiple digital sequences;

[0112] The word segmenter is used in combination with the index sequence to perform text restoration to obtain the target natural language.

[0113] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0114] Convert the training set text data into multiple digital sequences through the word segmenter:

[0115] The server receives a training set containing a large amount of text covering a variety of fields and styles.

[0116] For example, consider a news article about technological development, such as "In recent years, artificial intelligence technology has achieved tremendous breakthroughs, and a variety of intelligent devices have emerged, bringing numerous conveniences to people's lives." The server launches a word segmenter, which acts like a pair of linguistic scissors, slicing the article into individual words according to specific rules. For example, it would segment "in recent years" into "recent years" and "coming," "artificial intelligence technology" into "artificial," "intelligence," and "technology," and so on.

[0117] After the word segmenter has processed the entire article, each word is segmented individually. The server then assigns each word an index number within a dictionary. This dictionary acts like a "dictionary" of the language, recording all possible words and their corresponding numbers. For example, the word "artificial" is assigned number 1001, "intelligence" is assigned number 1002, "technology" is assigned number 1003, and so on. In this way, the entire article is converted into a series of numbers, each number representing a word's position in the dictionary.

[0118] Multiple digital sequences are embedded to obtain multiple high-dimensional vectors of high dimensions:

[0119] After obtaining the text sequence composed of numbers, the server begins to use embedding technology to convert these number sequences into high-dimensional vectors.

[0120] Embedding is like giving each word a unique "fingerprint," a high-dimensional vector. For the number sequence [1001, 1002, 1003] in the previous example, the embedding process will generate a high-dimensional vector for the number 1001 corresponding to "artificial," such as [0.2, 0.3, 0.1, 0.4, 0.7, ...] (this is just an example; the actual vector dimensions and values ​​will vary depending on the model settings). It will also generate another high-dimensional vector for the number 1002 corresponding to "intelligent," such as [0.1, 0.8, 0.2, 0.3, 0.6, ...], and for the number 1003 corresponding to "technology," it will generate [0.3, 0.4, 0.7, 0.1, 0.9, ...].

[0121] In this way, the original sequence of numbers is converted into a series of high-dimensional vectors, each of which contains rich semantic information about the corresponding word. These high-dimensional vectors not only retain the basic meaning of the word, but also capture the semantic relationship and contextual information between words.

[0122] Perform inference operations based on multiple high-dimensional vectors to obtain index sequences corresponding to multiple digital sequences:

[0123] The server uses its internal multi-layer Transformer architecture to perform complex inference operations on these high-dimensional vectors.

[0124] To illustrate with a simple example, suppose a layer in the Transformer architecture receives the three high-dimensional vectors generated previously: [0.2, 0.3, 0.1, 0.4, 0.7, ...], [0.1, 0.8, 0.2, 0.3, 0.6, ...], and [0.3, 0.4, 0.7, 0.1, 0.9, ...]. Through the self-attention mechanism, this layer will pay attention to the relationship between each vector and the other vectors, trying to understand their role and relationship with each other in the entire text.

[0125] These relationships are then further processed and calculated through a feedforward neural network, transforming and combining vectors according to certain rules and algorithms. After this series of inference operations, the layer outputs a new vector representation.

[0126] As the multi-layer Transformer architecture continues processing, it ultimately produces a high-dimensional vector representing the entire text. Then, through specific decoding operations, this high-dimensional vector is converted into a corresponding sequence of numbers, resulting in a sequence of indices corresponding to the multiple sequences of numbers. This index sequence acts as an encoded representation of the original text, encompassing both its semantic and structural information.

[0127] Use the word segmenter combined with the index sequence to restore the text and obtain the target natural language:

[0128] After the server obtains the index sequence corresponding to the digital sequence, it will use the word segmenter again to restore the text.

[0129] The word breaker converts each index number back to its corresponding word based on the previously established dictionary. For example, if the index number of a certain position in the sequence is 1001, the word breaker will search the dictionary to find the word "artificial" corresponding to the number 1001, and then add the word "artificial" to the restored text.

[0130] In this way, each number in the index sequence is converted into a word, and the entire index sequence is ultimately restored to the target natural language. For example, in the previous example, after the word segmenter's restoration, the index sequence might be restored to "In recent years, artificial intelligence technology has made tremendous breakthroughs, and various smart devices have continued to emerge, bringing numerous conveniences to people's lives." In this way, the server successfully restores the number sequence to the original natural language text, completing the conversion process from training data to understandable natural language.

[0131] In the embodiment of the present invention, the inference operation is performed based on the multiple high-dimensional vectors to obtain the index sequences corresponding to the multiple digital sequences, which can be implemented through the following examples.

[0132] Normalize the high-dimensional vectors corresponding to each layer of the Transformer architecture, and process the normalized original features through a linear layer;

[0133] The processed original features are divided into blocks on the time axis to obtain the first features; the block operation adopts an overlap factor of 50%;

[0134] A multi-scale Transformer using a trainable residual connection structure and adding a convolutional module uses an intra-frame Transformer and an inter-frame Transformer to learn the short-term and long-term dependencies of the first features respectively to obtain a second feature, and the intra-frame Transformer and the inter-frame Transformer have the same structure;

[0135] The second feature generated by the multi-scale Transformer is then passed through a PReLU activation function and a linear layer to obtain the third feature;

[0136] Performing an overlap-add operation on the third feature to obtain a fourth feature; the overlap rate of the overlap-add operation is 0.5;

[0137] Processing the fourth feature through a ReLU activation function and a feedforward network layer, using nonlinear transformation to extract and learn the feature representation of each token vocabulary;

[0138] When reaching the last layer of the Transformer architecture, the SoftMax function is used to obtain the sampling probability of all digital sequences, and the next digital sequence is sampled according to the preset sampling algorithm;

[0139] Repeat the steps of normalizing the high-dimensional vector corresponding to each layer of the Transformer architecture and processing the normalized original features through a linear layer, until the fourth feature is processed through a ReLU activation function and a feedforward network layer, and a nonlinear transformation is used to extract and learn the feature representation of each token vocabulary, until the preset number of cycles is reached, and then the index sequence is generated.

[0140] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0141] Normalize the high-dimensional vectors corresponding to each layer of the Transformer architecture, and process the normalized original features through the linear layer:

[0142] The server receives multiple high-dimensional vectors after embedding processing. These high-dimensional vectors represent the characteristics of each word or element in the text.

[0143] For example, in an article about a historical event, one of the high-dimensional vectors might represent the characteristics of the "Battle of Red Cliffs." The server first normalizes this high-dimensional vector.

[0144] The purpose of normalization is to make each high-dimensional vector have a similar distribution at a specific scale, preventing excessively large or small values ​​from adversely affecting subsequent operations. By calculating the mean and standard deviation of the high-dimensional vector, then subtracting the mean from each element and dividing it by the standard deviation, the vector's mean becomes 0 and its variance becomes 1.

[0145] Next, the server uses a linear layer to process the normalized original features. The linear layer acts like a filter, performing a linear transformation on the features by performing matrix multiplication and addition operations on the original features. For example, for the normalized high-dimensional vector of "The Battle of Red Cliffs," the linear layer's weight matrix might be a matrix of size [original feature dimension, new feature dimension]. Through matrix multiplication and addition operations, the original features are mapped to a new feature space, resulting in the processed features.

[0146] In this way, each high-dimensional vector has been processed by normalization and linear layers, preparing for subsequent feature learning and relationship extraction.

[0147] The processed original features are divided into blocks on the time axis to obtain the first feature; the block operation adopts a 50% overlap factor:

[0148] The server divides the original features processed by the linear layer into blocks on the time axis. Taking the processed features of "The Battle of Red Cliffs" as an example, the time axis can be regarded as the sequence position of the text.

[0149] The block splitting operation uses a 50% overlap factor, meaning each block overlaps by half. For example, if a 100-length feature sequence is split into 60-block blocks, there will be 30 elements of overlap between adjacent blocks. This approach allows for better capture of local features in the text and sequence continuity.

[0150] Through the block operation, a series of first features are obtained, each of which represents the local information of the original feature at different time segments. These first features retain some information of the original features while having a certain degree of independence, which facilitates subsequent processing and analysis.

[0151] The multi-scale Transformer with a trainable residual connection structure and a convolutional module uses intra-frame Transformer and inter-frame Transformer to learn the short-term and long-term dependencies of the first feature respectively to obtain the second feature. The intra-frame Transformer and inter-frame Transformer have the same structure:

[0152] For the first feature obtained from each block, the server uses intra-frame Transformer and inter-frame Transformer to learn its short-term and long-term dependencies respectively.

[0153] The intra-frame Transformer focuses on local dependencies within the current time segment. Using mechanisms such as self-attention, it analyzes the relationships between elements within the current block, capturing local semantic and grammatical information. For example, in the first feature of "The Battle of Red Cliffs," the intra-frame Transformer focuses on the relationship between elements such as "Red Cliffs" and "Battle," as well as their context within the current block.

[0154] The Inter-frame Transformer focuses on learning long-term dependencies between different time segments. It can capture the connections between different parts of the entire text sequence across time intervals. For example, in the entire text sequence about the "Battle of Red Cliffs", the Inter-frame Transformer can learn the relationship between the "Battle of Red Cliffs" and previous historical events and their subsequent historical impact.

[0155] Through the synergy of these two Transformers, the first feature is processed from different angles to obtain a richer and more comprehensive second feature.

[0156] The second feature generated by the multi-scale Transformer is converted into the third feature through the PReLU activation function and the linear layer:

[0157] After the second feature is processed by the intra-frame and inter-frame Transformer, the server uses the PReLU activation function to perform nonlinear transformation on it.

[0158] The PReLU activation function is linear in the positive region and has a slope in the negative region, introducing nonlinear characteristics and enabling the model to learn more complex functional relationships. The PReLU activation function further activates and enhances the information in the second feature, highlighting important features and suppressing less important information.

[0159] Then, the second feature after PReLU activation is processed by the linear layer. Similar to the previous linear layer operation, the feature is linearly transformed and mapped to obtain the third feature.

[0160] The fourth feature is obtained by performing an overlap-add operation on the third feature; the overlap ratio of the overlap-add operation is 0.5:

[0161] The server performs overlap-addition operations on the third feature after PReLU activation and linear layer processing.

[0162] Since the previous block operation uses a 50% overlap factor, half of the elements between two adjacent third feature blocks overlap during overlap-add. By adding these overlapping parts, more information can be retained while avoiding excessive duplication of information.

[0163] In this way, a fourth feature with richer information is obtained, which integrates the features of multiple time segments and reflects the overall characteristics of the text more comprehensively.

[0164] The fourth feature is processed through the ReLU activation function and the feedforward network layer, and the feature representation of each token vocabulary is extracted and learned using nonlinear transformation:

[0165] The server uses the ReLU activation function to process the fourth feature. The ReLU activation function is linear in the part greater than 0, which can maintain the transmission of information. At the same time, it changes the value to 0 in the part less than 0, which plays a sparse role.

[0166] Through the processing of the ReLU activation function, some unimportant or irrelevant information in the fourth feature is suppressed, while important information is retained and enhanced.

[0167] The ReLU-activated fourth feature is then further processed by the feedforward layer. This fully connected neural network layer performs matrix multiplication and addition operations on the fourth feature, performing nonlinear transformations and learning on the feature to extract the feature representation of each token.

[0168] In this process, the model continuously learns and adjusts parameters to better extract and represent the features of each token vocabulary, thereby providing a more accurate basis for subsequent generation and reasoning tasks.

[0169] When reaching the last layer of the Transformer architecture, the SoftMax function is used to obtain the sampling probability of all digital sequences, and the next digital sequence is sampled according to the preset sampling algorithm:

[0170] As the multi-layer Transformer architecture processes, it gradually approaches the last layer. When it reaches the last layer, the server uses the SoftMax function to calculate the probability distribution of each possible sequence of numbers.

[0171] The SoftMax function converts the input numerical value into a probability value so that the sum of all probabilities is 1. Each digital sequence corresponds to a probability value, and the size of the probability value reflects the possibility of the occurrence of the digital sequence.

[0172] Then, according to a preset sampling algorithm, samples are taken from these probability distributions to obtain the next number sequence. The sampling algorithm can be different methods such as random sampling and greedy sampling, and the selection depends on the specific task and requirements.

[0173] In this way, the server gradually generates a digital sequence, continuously performing feature learning and sequence generation until it reaches the preset number of cycles and finally generates a complete index sequence.

[0174] Throughout the entire process, the server continuously performs various operations and processing on high-dimensional vectors. Through the synergistic effect of the multi-layer Transformer architecture, it gradually extracts and learns the features of the text, and ultimately generates an index sequence with semantic and structural information, providing an important foundation for subsequent natural language processing tasks.

[0175] In an embodiment of the present invention, the multi-scale Transformer using a trainable residual connection structure and adding a convolution module uses an intra-frame Transformer and an inter-frame Transformer to respectively learn the short-term and long-term dependencies of the first feature to obtain the second feature, which can be implemented through the following examples.

[0176] Assigning a unique representation to each position of the first feature using absolute position encoding to obtain an intermediate first feature;

[0177] Performing a linear transformation of weights and biases on the intermediate first feature through a feedforward neural network, and introducing a nonlinear transformation through a ReLU function to obtain an intermediate second feature;

[0178] The intermediate second feature is processed through a multi-head attention mechanism to obtain an intermediate third feature;

[0179] Processing the intermediate third feature through a dual-scale convolution module to obtain an intermediate fourth feature and an intermediate fifth feature;

[0180] Processing the intermediate fourth feature and the intermediate fifth feature through the Mish activation function to obtain the intermediate sixth feature and the intermediate seventh feature;

[0181] splicing the intermediate sixth feature and the intermediate seventh feature to obtain an intermediate eighth feature;

[0182] The intermediate eighth feature is subjected to adaptive average pooling to obtain an intermediate ninth feature, and the intermediate ninth feature is processed by the feedforward neural network to obtain the second feature.

[0183] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0184] Absolute position encoding is used to assign a unique representation to each position of the first feature to obtain the intermediate first feature:

[0185] The server gets the first feature which contains various information, like a box filled with various objects, each position has its specific element.

[0186] To help the model better understand this positional information, we introduce absolute position encoding. For example, let's take a text describing a natural landscape and assume that the first feature represents the characteristics of each word in the sentence, such as the position of words like "mountains," "lakes," and "forests" in the sequence.

[0187] Absolute position encoding is like attaching a unique label to each position, which is implemented by a specific mathematical formula. For example, for the i-th position in the sequence, the calculation formula for its absolute position encoding may be:

[0188] [Pos_i=[sin(2πi / 10000^{2j / d}), cos(2πi / 10000^{2j / d})]]

[0189] Here, d is the model's dimensionality, and j controls the periodicity of the sine and cosine functions. Each position is thus assigned a two-dimensional vector containing sine and cosine values ​​as its unique representation, the intermediate first feature. This allows the model to leverage positional information to better understand the order and structural relationships of words in a text, as if assigning specific "coordinates" to each word's position in the text.

[0190] The feedforward neural network is used to perform a linear transformation of the weights and biases of the intermediate first feature, and a nonlinear transformation is introduced through the ReLU function to obtain the intermediate second feature:

[0191] The server feeds the intermediate first feature with absolute position encoding into the feedforward neural network.

[0192] A feedforward neural network is like a complex processing machine, performing a series of operations on the intermediate first feature. First, the intermediate first feature is multiplied by the weight matrix through matrix multiplication, and then the bias vector is added. This is the linear transformation process of the weights and biases. For example, the intermediate first feature is a matrix with a dimension of [100, 512] (100 represents the number of words, 512 represents the feature dimension), the weight matrix is ​​[512, 1024], and the bias vector is

[1024] . After multiplication and addition, a new intermediate feature with a dimension of [100, 1024] is obtained.

[0193] Next, a nonlinear transformation is introduced using the ReLU function. The ReLU function converts negative values ​​in the input to 0, while leaving positive values ​​unchanged. This allows the model to learn more complex nonlinear relationships and avoid issues like vanishing gradients. For example, for each element in the intermediate feature, if its value is less than 0, it is converted to 0, while values ​​greater than or equal to 0 remain unchanged. After processing with the ReLU function, the intermediate second feature is obtained. This retains the important information in the intermediate first feature while introducing nonlinear factors, enabling the model to process text data more flexibly.

[0194] The second middle feature is processed by the multi-head attention mechanism to obtain the third middle feature:

[0195] The server inputs the intermediate second feature into the multi-head attention mechanism.

[0196] The multi-head attention mechanism is like multiple "little eyes" simultaneously observing the intermediate second feature, capturing information from different perspectives. First, the intermediate second feature is projected into three different representation spaces through linear transformations, resulting in query, key, and value matrices, respectively. For example, the intermediate second feature is a matrix of dimension [100, 1024]. Through three different linear transformations, the query, key, and value matrices are respectively obtained with dimensions [100, 32*64] (here 64 is the number of heads, and 32 is the dimension of each head).

[0197] The query, key, and value are then divided into 64 heads, each with a dimension of 32. For each head, an attention weight is calculated: the dot product of the query and the key divided by the square root of the sum of the squares of the keys. This weight is then normalized using a softmax function. The attention weight is then multiplied by the value, and the outputs of all heads are concatenated to produce the result of the multi-head attention mechanism, the intermediate third feature. This multi-head attention mechanism allows the model to simultaneously focus on information at different locations, capturing long-range dependencies within the text and ultimately providing a better understanding of its semantics.

[0198] The middle third feature is processed by a dual-scale convolution module to obtain the middle fourth feature and the middle fifth feature:

[0199] The server feeds the intermediate third feature into the dual-scale convolution module.

[0200] The dual-scale convolution module consists of two convolution kernels of different sizes, each used to capture local features at different scales. One convolution kernel has a size of 3, and the other has a size of 11. When the intermediate third feature is input, its dimension is converted from [100, 32*64] to [100, 64, 32] to perform a one-dimensional depthwise convolution operation.

[0201] For a convolution kernel of size 3, the kernel is divided into 64 groups in the depth direction, and a 3-point convolution operation is performed on each group, resulting in an output dimension of [100, 64, 32]. For a convolution kernel of size 11, the kernel is similarly divided into 64 groups in the depth direction, and an 11-point convolution operation is performed, resulting in an output dimension of [100, 64, 32]. This results in two feature maps of different scales: the intermediate fourth feature and the intermediate fifth feature. Through the dual-scale convolution module, the model can simultaneously capture local detailed features and global contextual information, enriching the feature representation.

[0202] The intermediate fourth feature and the intermediate fifth feature are processed by the Mish activation function to obtain the intermediate sixth feature and the intermediate seventh feature:

[0203] The server applies the Mish activation function to the intermediate fourth feature and the intermediate fifth feature respectively.

[0204] The expression of the Mish activation function is (Mish(x) = x*tanh(softplus(x))), where (softplus(x) = ln(1+e^x)). The Mish activation function has a smaller slope in the negative part and a larger slope in the positive part, which can better balance the nonlinear characteristics of the model.

[0205] Each element in the intermediate fourth and fifth features is processed using the Mish activation function, resulting in richer and more nonlinear intermediate sixth and seventh features. The introduction of the Mish activation function enables the model to better mine the potential information in the data, improving its expressive power.

[0206] Concatenate the sixth and seventh middle features to get the eighth middle feature:

[0207] The server concatenates the intermediate sixth feature and the intermediate seventh feature processed by the Mish activation function in the feature dimension.

[0208] Since they have the same dimensions, the concatenation operation simply concatenates the two feature matrices along the channel dimension to obtain an intermediate eighth feature with a larger dimension. In this way, the model integrates the feature information obtained by convolution operations at different scales, enabling it to better process complex text data.

[0209] The eighth feature in the middle is obtained by adaptive average pooling to obtain the ninth feature in the middle, and the ninth feature in the middle is processed by a feedforward neural network to obtain the second feature:

[0210] The server performs an adaptive average pooling operation on the eighth middle feature.

[0211] Adaptive average pooling automatically adjusts the size of the pooling window based on the input features to accommodate features of varying scales. It traverses all channels of the intermediate eighth feature, dividing the feature into fixed-size regions, and then takes the average of the feature values ​​within each region as the pooling result for that channel. This keeps the size of the output intermediate ninth feature fixed while preserving the key information of the input feature.

[0212] Finally, the ninth feature is fed into a feedforward neural network for processing. Similar to the previous steps, it undergoes further transformation and learning through operations such as linear transformations and ReLU functions, ultimately yielding the second feature. This second feature, rich in semantic information and feature representation, is a key result of intra-frame Transformer processing and provides a solid foundation for subsequent processing and analysis.

[0213] In the embodiment of the present invention, the processing of the intermediate ninth feature through the feedforward neural network to obtain the second feature can be implemented through the following example.

[0214] The intermediate ninth feature in the intra-frame Transformer and the inter-frame Transformer is added to the input of each feedforward neural network layer through a trainable residual connection and then subjected to a layer normalization operation to obtain the second feature.

[0215] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0216] In the process of processing the natural language task, the server reaches the step of processing the intermediate ninth feature through a feedforward neural network to obtain the second feature.

[0217] Taking a long text about a historical story as an example, the ninth feature in the middle is like a set of key features extracted from the text after a series of complex operations. It contains various semantic, structural and other information about the story.

[0218] In the intra-frame Transformer, the feedforward neural network of each layer begins processing the ninth feature in the middle. First, the ninth feature in the current layer is fed into the feedforward neural network as input. The feedforward neural network here resembles a complex network structure composed of multiple neurons, with multiple hidden layers and an output layer.

[0219] During the input to the feedforward neural network, the output of the previous layer is added through a trainable residual connection. The purpose of the trainable residual connection is to allow information to flow more smoothly in the network, avoiding information loss or transmission difficulties as the number of network layers increases. For example, the ninth feature in the middle of the input of the current layer is a tensor of shape [B, C, H] (B represents the batch size, C represents the number of channels, and H represents the height or width of the feature, etc.), and the output of the previous layer is also a tensor of the same shape after some processing. These two tensors are added together through a trainable residual connection, that is, the elements in the corresponding positions are added. This makes the input of the current layer contain new information while retaining the information of the previous layer, which facilitates the transmission and fusion of information.

[0220] The result of adding the trainable residual connections then undergoes layer normalization. The goal of layer normalization is to ensure that the mean of each layer's input approaches 0 and the variance approaches 1 after various operations, thereby accelerating training and improving model stability. Specifically, for the tensor added via the trainable residual connections, the mean and standard deviation along each dimension are calculated. The mean is then subtracted from each element and divided by the standard deviation to stabilize the data distribution. After layer normalization, a resized intermediate feature is obtained.

[0221] The same process repeats itself within the inter-frame Transformer. Each layer of the feedforward neural network uses intermediate features, processed through trainable residual connections and layer normalization, as input, and continues a series of operations and processing. As the multiple layers of feedforward neural networks in the intra-frame and inter-frame Transformers process sequentially, the intermediate ninth feature is continuously refined, adjusted, and optimized, gradually incorporating more semantic and structural information.

[0222] After the entire feedforward neural network processing, the final result is the second feature. This second feature is like the essence of the text after deep processing and refinement. It contains the most core semantic and structural information about the entire text, providing strong support for subsequent tasks such as language generation and semantic understanding. For example, when performing text generation tasks, the server can use this second feature to generate new text content related to the original text; when performing semantic understanding tasks, this second feature can help the server more accurately understand the meaning and intent of the text.

[0223] In an embodiment of the present invention, the trainable residual connection is represented by the formula: y=F(W*x);

[0224] Where x represents the input, W represents the trainable weight matrix; in the weight matrix, α close to the current layer is initialized to 1, while the previous and farther layers are initialized to 0; F(x) represents the nonlinear mapping function from input to output, and y represents the output.

[0225] In the embodiment of the present invention, for example, the following is a detailed scenario example:

[0226] Trainable residual connections play a crucial role in the server's construction and training of large language models based on trainable residual connections.

[0227] Taking a specific language translation task as an example, the server receives a text in the source language, such as an English text about a technology product introduction.

[0228] When entering a network structure with trainable residual connections, we first look at the input. The input is the vector representation of the English text to be translated after various preprocessing and feature extraction, denoted as ({X}).

[0229] The trainable weight matrix acts like a regulating valve in this process. It dynamically adjusts the flow of information based on the progress of network training and the relationships between different layers. Taking the first layer of the network as an example, the weight matrix here is denoted as (W_1). The weights (α) close to the current layer are initialized to 1, meaning that at the beginning of the first layer, the input ({X}) is passed almost intact to the output via the residual connection. For layers further ahead, their weights (α) are initialized to 0, indicating that in the early stages, these layers have less influence on the current layer.

[0230] As the network progresses, the value of (α) is adjusted based on the training process. For example, in the second layer, (α) may be adjusted to a value between 0 and 1 based on the information interaction between the second and first layers and their contribution to the overall translation task. If the second layer finds, through interaction with the first layer, that retaining more input information helps improve translation accuracy, then (α) may be closer to 1. Conversely, if it finds that more inter-layer interaction and information fusion are needed, (α) may be adjusted to a value closer to 0, while still retaining some input information.

[0231] By multiplying the input ({X}) by the trainable weight matrix (W_1), that is, (W_1{X}), and then adding it to the input ({X}), that is, ({X}+W_1{X}), we get an intermediate result. This intermediate result is actually the fusion of the input ({X}) after some transformation by the weight matrix (W_1) and the input itself.

[0232] Next, this intermediate result passes through a nonlinear mapping function (f) from input to output. This nonlinear mapping function (f) can be an activation function such as ReLU (rectified linear unit) or Leaky ReLU, which introduces nonlinear factors and enables the model to learn more complex relationships. For example, with the ReLU function, when an element in the intermediate result is greater than 0, it remains unchanged; when an element is less than 0, it is set to 0. This allows the model to filter out important information and suppress unimportant information, thereby better adapting to various situations in language translation tasks.

[0233] After processing by the nonlinear mapping function (f), the final output is obtained, denoted as ({Y}). ({Y}) is the output after the trainable residual connection processing, which contains the information of the input ({X}) as well as the new information and features introduced by the weight matrix and nonlinear mapping function.

[0234] The same trainable residual connection mechanism continues to operate in subsequent network layers. The input of each layer is the output of the previous layer processed by the trainable residual connection. By continuously adjusting the weight matrix and nonlinear mapping function, information is gradually transmitted, integrated, and optimized within the network, ultimately enabling the entire model to accurately perform tasks such as language translation.

[0235] For example, during the translation process, if complex grammatical structures or semantic expressions are encountered, the trainable residual connection mechanism allows the model to better retain and utilize the knowledge of grammar and semantics learned by previous layers, while also adjusting and innovating based on the characteristics of the current layer, thereby more accurately translating the source language text into the target language. When processing different types of scientific, literary, or news texts, the trainable residual connection mechanism can flexibly adjust the flow and processing of information based on the characteristics and needs of the text, giving the model greater generalization and adaptability.

[0236] In order to more clearly describe the solution provided by the embodiment of the present invention, a relatively complete implementation method is provided below. Figure 2 , Figure 2 Schematic diagram of the architecture of a large language model based on trainable residual connections and dual-scale convolutional Transformer provided by an embodiment of the present invention.

[0237] To address the shortcomings of current language models in handling complex language structures, long-range dependencies, and capturing multi-scale semantic features, this paper proposes a revolutionary large language model architecture—"a large language model architecture based on trainable residual connections and two-scale convolutional Transformers." This solution integrates two core technological innovations, resulting in significant improvements in model performance and generalization.

[0238] First, in response to the fixedness and limitations of the residual connection method in the traditional Transformer, this paper pioneered the introduction of a new trainable residual connection structure. This innovative design gives the model the ability to dynamically adjust the information flow path, enabling it to intelligently optimize the hierarchical information transmission process based on the characteristics of the input data and task requirements. This not only significantly enhances the model's ability to capture deep language features, but also promotes the effective fusion of information between different levels, thereby greatly improving the model's expressive power and depth of understanding. Please refer to Figure 4 , Figure 4 A schematic diagram of the internal structure of a dual-scale convolutional Transformer with a trainable residual connection structure provided by an embodiment of the present invention;

[0239] Secondly, to make up for the shortcomings of traditional models in capturing multi-scale semantic features, this paper cleverly embeds a dual-scale convolution module into the Transformer architecture. This design can simultaneously capture local detail features and global context information in language data by processing convolution kernels of different sizes in parallel, achieving a comprehensive and in-depth analysis of language information. This innovation greatly enriches the feature dimensions that the model can extract and significantly improves the model's ability to process complex language structures and understand implicit semantic relationships. Please refer to Figure 3 , Figure 3 Schematic diagram of the internal structure of the TransformerX and multi-scale Transformer modules with dual-scale convolution provided by the embodiments of the present invention.

[0240] Based on these two core technological innovations, the present invention successfully constructed a new component—TransformerX. This component not only inherits the advantages of the Transformer architecture but also significantly improves the model's performance and generalization capabilities for complex language tasks by introducing trainable residual connections and two-scale convolutional modules. Through the introduction of TransformerX, the present invention's large language model architecture has performed well across multiple natural language processing tasks, demonstrating its strong application potential.

[0241] To achieve the above objectives, the present invention adopts a technical solution: a large language model based on trainable residual connections and a dual-scale convolutional Transformer, which specifically includes the following steps:

[0242] To build a large language model, we first designed a base model based on the TransformerX architecture. This model consists of multiple TransformerX decoder layers, each of which incorporates a self-attention mechanism and a feedforward neural network. To enhance the model's ability to capture local features, a dual-scale convolution module is embedded in each TransformerX layer. This dual-scale convolution module consists of two convolution kernels of different sizes, each designed to capture local features at a different scale. Depthwise separable convolution is used to replace traditional standard convolutions, reducing the model's parameter count and computational complexity. The outputs of these two convolution kernels are combined through concatenation or weighted summation to serve as the module's output, which is then passed to subsequent TransformerX layers. In traditional Transformers, residual connections are fixed to maintain information flow. In contrast, this paper proposes a trainable residual connection structure. This structure allows the model to learn how to better adjust the information flow during training, thereby optimizing information transmission efficiency. Specifically, a trainable weight matrix is ​​added between the input and output of each TransformerX layer to adjust the strength of the residual connection.

[0243] Data collection and processing: collect text data from the Internet and other channels to obtain training sets. Specifically,

[0244] S1, document preparation, document preparation includes data reading, URL filtering, text extraction and language identification. The specific process of S1 is:

[0245] S1.1, Data Reading: Text data can be extracted from WET files or WARC files. To avoid the complexity of extracting content from HTML, it is preferable to read data from WARC files to reduce the interference of irrelevant information;

[0246] S1.2, Filtering URLs: Before processing text data, perform preliminary filtering on URLs to exclude fraudulent and illegal websites. Filtering is based on a domain blocklist and URL scoring based on a specific wordlist. URLs from high-quality text datasets (such as Wikipedia and arXiv) are also excluded.

[0247] S1.3, Extract Text: Extract the main content from the HTML page, ignoring menus, headers, footers, and advertisements. Use the Trafilatura library and regular expressions for text extraction, ultimately limiting new lines to two consecutive lines and removing all URL links;

[0248] S1.4, Language Identification: Use language classifiers such as fastText to identify the language, select texts in the corresponding language, and delete documents with language scores below a threshold;

[0249] S2, filtering, the purpose of filtering is to improve the quality of text, remove duplicate paragraphs, irrelevant content and non-natural language, including document-level and line-level filtering. The specific process of S2 is as follows:

[0250] S2.1, Duplicate Document Removal: Using heuristic methods, we formulate rules to remove documents with excessive line, paragraph, or n-gram duplication to reduce costs and improve efficiency;

[0251] S2.2, document filtering: Mainly retain natural language documents written by humans, remove machine-generated spam, use quality filtering heuristics, and remove outliers based on criteria such as document length and symbol-to-word ratio;

[0252] S2.3, row-level filtering: Use linear correction filters to remove content irrelevant to the text (such as likes, navigation buttons, etc.) to ensure data quality;

[0253] S3, deduplication. After filtering, although the data quality is improved, duplicate documents still exist. Combining fuzzy matching and exact matching methods to deduplication, the specific process of S3 is as follows:

[0254] S3.1, fuzzy deduplication: Use the MinHash algorithm to calculate the approximate similarity between documents and delete document pairs with high overlap;

[0255] S3.2, accurate deduplication: Use suffix arrays to find exact matches between strings and remove paragraphs that repeat more than k consecutive tokens;

[0256] S3.3, URL deduplication: removes URLs that are accessed repeatedly in cross-CC dumps to ensure data uniqueness;

[0257] The training process of a large language model. Specifically,

[0258] S1, Pre-training: In this stage, the model performs unsupervised learning on large-scale text data, aiming to learn the statistical properties and contextual relationships of language. The model is trained by predicting the next word, filling in the blank, or other language tasks to capture the grammatical, semantic, and contextual information of the language.

[0259] S2, Supervised Fine-tuning: After pre-training, the model undergoes supervised fine-tuning on specific downstream tasks. This process typically uses labeled data, and the model learns from this data to optimize performance on specific tasks such as text classification, question answering, and dialogue generation.

[0260] S3, Reward Modeling: In this stage, multiple responses generated for the same prompt are manually ranked to construct training samples. By comparing the quality of different responses, a sample set with relative merits is created. This ranking information is used to train a reward model, enabling it to predict the quality scores of different responses based on human feedback. Specifically, the goal of the reward model is to learn a function that assigns a score to each generated response that reflects its performance in a specific task.

[0261] S4, Reinforcement Learning: During the reinforcement learning phase, a reward model is used to score the results generated by the supervised fine-tuning (SFT) model. These scores serve as feedback signals for training the model using policy optimization algorithms (such as Proximal Policy Optimization (PPO). The goal is to maximize the rewards for the model's generated responses. By continuously adjusting the generation strategy, the model produces higher-quality results for specific tasks. Ultimately, after multiple rounds of optimization and training, the final model is obtained, with significantly improved generation capabilities that can better meet user needs and expectations.

[0262] The significant advantage of the present invention lies in its innovative introduction of the TransformerX component within and between frames of traditional large language models. This component, by embedding a dual-scale convolutional module and cleverly replacing traditional convolution with depthwise separable convolution, greatly enhances the model's ability to capture local features. At the same time, this design significantly reduces the model's parameter count and computational overhead, achieving an efficient and lightweight model. Furthermore, TransformerX incorporates additional feedforward neural network structures, advanced activation functions, and layer normalization techniques, all of which further enhance the model's representational capabilities and robustness, making it more adept at handling complex and diverse natural language tasks. Of particular note, the present invention pioneered the use of a trainable residual connection structure, replacing the traditional fixed residual connection approach. This revolutionary design enables the model to autonomously transmit hierarchical information, significantly optimizing the training process and improving the model's optimization effect. Through the integration of this series of innovative designs and technical applications, the present invention demonstrates superior performance on multiple natural language processing tasks without significantly increasing model parameters. This has undoubtedly injected new vitality into the development of natural language processing and pushed the field towards a more efficient and intelligent direction. Figure 5 , Figure 5 Schematic diagram of the internal structure of a dual-scale convolution module and a single convolution module provided in an embodiment of the present invention;

[0263] The core of the large language model architecture of this embodiment lies in two key improvements to the Transformer structure. First, an innovative trainable residual connection structure is proposed to replace the traditional fixed connection method in Transformer. This new structure can autonomously transmit hierarchical information, significantly improving the expressiveness and generalization capabilities of the model. Secondly, a dual-scale convolution module is incorporated to more effectively capture semantic features of different scales and enhance the model's ability to process complex language information. Finally, by integrating these two technologies, a new TransformerX component was launched, and an excellent large language model architecture was built on this basis.

[0264] The above-mentioned solution of the present invention is specifically applied to the following embodiments:

[0265] S1 and S2 in step 3) of this embodiment are specifically:

[0266] S1, the input text is first converted into a sequence of numbers by the tokenizer. These numbers are the index numbers of the words in the dictionary (vocab);

[0267] S2, obtains a high-dimensional vector by embedding the digital sequence;

[0268] S3, these vectors are processed by the decoder for complex reasoning operations to generate the digital index of the next word. The entire index sequence can be generated by repeating the operation multiple times. The specific process of S3 is as follows:

[0269] S3.1, the input code representation h0 is layer normalized, and the specific formula is:

[0270]

[0271] Where x is the input data, μ is the mean of the input data in a certain dimension, and the calculation formula is:

[0272]

[0273] D is the total number of dimensions of the input data, i is a dimension of the input data. σ is the standard deviation of the input data in a certain dimension, and the calculation formula is:

[0274]

[0275] ε is a very small constant used to avoid the denominator being zero. g is a learnable scaling parameter, b is a learnable bias parameter, and ⊙ represents element-wise multiplication.

[0276] Layer normalization normalizes the input data so that the input of each layer is distributed within a range with a mean of 0 and a variance of 1, thereby alleviating the internal covariate shift problem in the neural network;

[0277] S3.2, uses a linear layer to process the layer-normalized feature h0. The linear layer is a commonly used network layer in deep learning, also known as a fully connected layer or affine layer. The specific formula is:

[0278] y=Wx+b,

[0279] Where x is the input vector, W is the weight matrix, b is the bias vector, and y is the output vector. This formula indicates that the input vector x is multiplied by the matrix and the bias term is added to get the output vector y.

[0280] Specifically, if x is a vector of dimension n, then W is a matrix of dimension m×n, where m represents the weight of each neuron for each input feature; b is a vector of dimension m, representing the bias of each neuron; y is an output vector of dimension m, representing the output result of each neuron;

[0281] S3.3, the layer-normalized feature h0 is divided into blocks on the time axis to obtain h1, and the block operation adopts a 50% overlap factor;

[0282] S3.4, a multi-scale Transformer with a trainable residual connection structure and a convolutional module uses intra-frame Transformer and inter-frame Transformer to learn the short-term and long-term dependencies of feature h1 respectively to obtain feature h2. The intra-frame Transformer and inter-frame Transformer have the same structure, specifically:

[0283] S3.4.1, use absolute position encoding to assign a unique representation to each position of the input feature to obtain feature X1. The specific formula of absolute position encoding is:

[0284]

[0285] Among them, pos represents the position index in the sequence, i represents the dimension index in the position encoding vector, and d represents the dimensional size of the model. The sine function formula maps the position index to a small numerical range, and uses the periodic characteristics of the sine function to ensure that the encoding values ​​of adjacent positions are different. Similar to the sine function, the cosine function also has periodic characteristics. Using the cosine function to encode odd dimensions further increases the diversity of encoding. The results of the calculated sine and cosine functions are combined into a position encoding vector. The position encoding vector is added to the audio feature vector in the time window to obtain the final position encoding representation. By introducing absolute position encoding, the model can use information at different positions in the sequence to better understand the structure and sequential relationship of text information;

[0286] 3.4.2, feature X2 is obtained through a feedforward neural network. In a feedforward neural network, the input is first linearly transformed by weights and biases through a linear layer. Then, a nonlinear transformation is introduced through the ReLU function. The ReLU function converts the input value less than 0 to 0 and keeps the input value greater than or equal to 0 unchanged. The mathematical expression is:

[0287] ReLU(x)=max(0,x)

[0288] Using the ReLU activation function in feedforward neural networks can introduce nonlinearity, alleviate the gradient vanishing problem, extract useful features, and improve computational efficiency, thereby helping to improve the performance of speech separation tasks;

[0289] 3.4.3, feature X2 is used to obtain feature X3 through the multi-head self-attention mechanism. The specific steps are as follows:

[0290] First, the input sequence X is projected into three different representation spaces: query, key, and value through linear transformation. This is achieved by the following formula:

[0291] Q=XW q

[0292] K=XW k

[0293] V=XW v

[0294] Where W q 、W k 、W v is a learnable weight matrix used for linear transformation of query, key and value respectively. Next, the obtained Q, K, V are divided into 8 heads (or subspaces), each with a dimension of 32:

[0295]

[0296] Among them, Q i , K i and V i is the query, key, and value of the i-th head. Then, the attention weight of each head is calculated:

[0297]

[0298] Among them, softmax represents the normalization of the attention score, d h is the dimension of each head, which is 32. Divided by The purpose is to scale the attention scores to better handle attention in different dimensions. Finally, the outputs of all heads are concatenated to obtain the result of the multi-head attention mechanism;

[0299] In S3.4.4, feature X3 is converted into features X41 and X42 through a dual-scale convolution module. The two convolution modules have the same internal structure. The specific steps are as follows:

[0300] First, the input tensor x is dimensionally converted from [B, N, L] to [B, L, N], where B represents the batch size, N represents the number of channels, and L represents the sequence length.

[0301] Next, a one-dimensional depthwise convolution is used to convolve the features simultaneously. The number of input and output channels of the convolution layer is 256, one convolution kernel size is 3, the other is 11, the stride is 1, and the padding size is 1. The input and output channels are divided into 256 groups for convolution. The channels within each group are independent of each other and no convolution operation is performed between them.

[0302] Next, a one-dimensional point-by-point convolution is used to convolve the output of the depthwise convolution, which exchanges the position information on each channel. The number of input and output channels of this convolution layer is still 256, the convolution kernel size is 1, the stride is 1, and there is no padding;

[0303] S3.4.5, features X41 and X42 are converted into features X51 and X52 through the Mish activation function. The expression of the Mish activation function is as follows:

[0304] Mish(x)=x·tanh(ln(1+e x ))

[0305] In S3.4.6, features X51 and X52 are concatenated to obtain feature X6. The specific steps are as follows:

[0306] The shape of feature X51 is [B, L1, N]. The shape of feature X52 is [B, L2, N], where B represents the batch size, N represents the number of channels, and L represents the sequence length. The concatenation operation connects the two feature maps in the depth dimension to generate a deeper feature X6 with a shape of [B, L1+L2, N].

[0307] In step S3.4.7, feature X6 is adaptively averaged pooled to obtain feature X7. The steps are as follows: traverse all channels of feature X6, divide feature X6 into fixed-size regions, and take the average of the feature values ​​in each region as the pooling result of the channel, so that the output size remains fixed;

[0308] S3.4.8, feature X7 is processed again by the feedforward neural network;

[0309] In the intra-frame Transformer and inter-frame Transformer, the input of each feedforward neural network layer is added to the output through a trainable residual connection, and then undergoes layer normalization. The convolution module and multi-head attention module also adopt this method, and the input is added to the output through a trainable residual connection, and then undergoes layer normalization.

[0310] The expression of the trainable residual connection is:

[0311] y=F(W*x)

[0312] Here, x represents the input and W represents the trainable weight matrix. In the weight matrix, the alpha close to the current layer is initialized to 1, while the previous and further layers are initialized to 0. After training, the model can autonomously select the information of the required layer. F(x) represents the nonlinear mapping function from input to output, and y represents the output. Through training, the value of W is learned and multiplied by F(x), and then added to the input x, thus realizing the trainable residual connection.

[0313] S3.5, the feature h2 generated by the multi-scale Transformer is activated by the PReLU function and the linear layer to obtain the feature h3. The expression of the PReLU activation function is:

[0314] PReLU(x)=max(0,x)+a·min(0,x)

[0315] S3.6, perform an overlap-add operation on feature h3 to obtain feature h4, with an overlap ratio of 0.5. This step is used to restore the block-wise features to their original length;

[0316] In S3.7, feature h4 is processed through the ReLU activation function and the feedforward network layer, using nonlinear transformations to extract and learn the feature representation of each token vocabulary. This process is repeated through multiple layers until the last layer. Finally, the SoftMax function is used to obtain the sampling probability of all digital sequences, and the next digital sequence is sampled according to a specific sampling algorithm.

[0317] S3.8, after multiple cycles of calculation, the entire index sequence can be generated;

[0318] S4 uses a tokenizer to restore these digital sequences to natural language that humans can understand.

[0319] Through the large language model architecture based on trainable residual connections and dual-scale convolutional Transformer proposed in this invention, significant experimental results have been successfully improved in the field of natural language processing.

[0320] Please refer to Figure 6 , Figure 6 An embodiment of the present invention provides a large language model training device 110 based on a trainable residual connection and a two-scale convolutional Transformer, including:

[0321] Acquisition module 1101 is used to obtain a basic model based on a multi-layer Transformer architecture, wherein the basic model includes multiple cascaded Transformer decoder layers, each of which includes a self-attention mechanism and a feedforward neural network, and each of which is embedded with a two-scale convolution module. The two-scale convolution modules of different scales are used to capture different local features. The outputs of the two-scale convolution modules are fused as the output of the corresponding Transformer decoder layer. A trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer, and the trainable weight matrix is ​​used to adjust the strength of the residual connection between each Transformer decoder layer; obtaining preprocessed sample document data to construct a training set;

[0322] The training module 1102 is used to train the basic model based on the multi-layer Transformer architecture based on the training set until the preset training termination condition is reached, thereby obtaining a large language model that has been trained and integrates the trainable residual connection and the two-scale convolutional Transformer.

[0323] It should be noted that the implementation principles of the aforementioned large language model training device 110 based on trainable residual connections and two-scale convolutional transformers can be referenced to the implementation principles of the aforementioned large language model training method based on trainable residual connections and two-scale convolutional transformers, and will not be repeated here. It should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. Moreover, these modules can be implemented entirely in the form of software called by processing elements; or entirely in the form of hardware; or some modules can be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the large language model training device 110 based on trainable residual connections and two-scale convolutional transformers can be a separate processing element, or it can be integrated into a chip of the above device. In addition, it can be stored in the form of program code in the memory of the above device, and called by a processing element of the above device to execute the functions of the above large language model training device 110 based on trainable residual connections and two-scale convolutional transformers. The implementation of other modules is similar. Furthermore, all or part of these modules may be integrated together or implemented independently. The processing element described herein may be an integrated circuit having signal processing capabilities. In implementation, each step of the above method or each of the above modules may be performed by hardware integrated logic circuits in the processor element or by software instructions.

[0324] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code on a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0325] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned large language model training device 110 based on trainable residual connections and dual-scale convolutional transformers. Figure 7 As shown, Figure 7 This is a block diagram of a computer device 100 according to an embodiment of the present invention. The computer device 100 includes a training device 110 for a large language model based on trainable residual connections and a two-scale convolutional transformer, a memory 111 , a processor 112 , and a communication unit 113 .

[0326] In order to realize data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, the electrical connection between these elements can be achieved through one or more communication buses or signal lines. The training device 110 for a large language model based on trainable residual connections and two-scale convolutional transformers includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the training device 110 for a large language model based on trainable residual connections and two-scale convolutional transformers stored in the memory 111, such as the software function modules and computer programs included in the training device 110 for a large language model based on trainable residual connections and two-scale convolutional transformers.

[0327] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is running, it controls the computer device where the readable storage medium is located to execute the aforementioned training device 110 for the large language model based on trainable residual connections and dual-scale convolutional Transformer.

[0328] For illustrative purposes, the foregoing description has been made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise forms disclosed. Numerous modifications and variations are possible in light of the above teachings. These embodiments have been selected and described in order to best illustrate the principles of the present disclosure and its practical application, thereby enabling those skilled in the art to best utilize the present disclosure and to utilize various embodiments with various modifications as appropriate for the specific application contemplated.

Claims

1. A training method for a large language model based on trainable residual connections and a two-scale convolutional Transformer, characterized in that: include: Obtaining a base model based on a multi-layer Transformer architecture, wherein the base model includes multiple cascaded Transformer decoder layers, each Transformer decoder layer includes a self-attention mechanism and a feedforward neural network, each Transformer decoder layer is embedded with a two-scale convolution module, the two-scale convolution modules of different scales are used to capture different local features, the outputs of the two-scale convolution modules are fused as the output of the corresponding Transformer decoder layer, and a trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer, and the trainable weight matrix is ​​used to adjust the strength of the residual connection between each Transformer decoder layer; Obtain preprocessed sample document data to build a training set; The text data included in the training set is converted into a plurality of digital sequences through a word segmenter; the numbers included in the digital sequences are index numbers of the text data in the dictionary; Obtaining multiple high-dimensional vectors of high dimension by embedding the multiple digital sequences; Performing reasoning operations based on the multiple high-dimensional vectors, normalizing the high-dimensional vectors corresponding to each layer of the Transformer architecture, and processing the normalized original features through a linear layer; The processed original features are divided into blocks on the time axis to obtain the first features; the block operation adopts an overlap factor of 50%; A multi-scale Transformer using a trainable residual connection structure and adding a convolutional module uses an intra-frame Transformer and an inter-frame Transformer to respectively learn the short-term and long-term dependencies of the first features to obtain a second feature, wherein the intra-frame Transformer and the inter-frame Transformer have the same structure; The second feature generated by the multi-scale Transformer is then passed through a PReLU activation function and a linear layer to obtain the third feature; Performing an overlap-add operation on the third feature to obtain a fourth feature; the overlap rate of the overlap-add operation is 0.5; Processing the fourth feature through a ReLU activation function and a feedforward network layer, using nonlinear transformation to extract and learn the feature representation of each token vocabulary; When reaching the last layer of the Transformer architecture, the SoftMax function is used to obtain the sampling probability of all digital sequences, and the next digital sequence is sampled according to the preset sampling algorithm; Repeating the steps of normalizing the high-dimensional vector corresponding to each layer of the Transformer architecture and processing the normalized original features through a linear layer, to the step of processing the fourth feature through a ReLU activation function and a feedforward network layer, extracting and learning the feature representation of each token vocabulary using a nonlinear transformation, until a preset number of cycles is reached, and then generating an index sequence; Using the word segmenter in combination with the index sequence to perform text restoration to obtain the target natural language, so as to complete the pre-training of the basic model based on the multi-layer Transformer architecture to learn the statistical characteristics and contextual relationships of the language; Fine-tune the pre-trained base model based on the multi-layer Transformer architecture according to the preset downstream tasks; The basic model based on the multi-layer Transformer architecture after supervised fine-tuning is combined with the reward model through a preset strategy optimization algorithm for reinforcement learning until the preset training termination condition is reached. The trained large language model that integrates trainable residual connections and dual-scale convolutional Transformer is obtained.

2. The method according to claim 1, characterized in that The multi-scale Transformer using a trainable residual connection structure and adding a convolution module uses an intra-frame Transformer and an inter-frame Transformer to respectively learn the short-term and long-term dependencies of the first feature to obtain the second feature, including: Assigning a unique representation to each position of the first feature using absolute position encoding to obtain an intermediate first feature; Performing a linear transformation of weights and biases on the intermediate first feature through a feedforward neural network, and introducing a nonlinear transformation through a ReLU function to obtain an intermediate second feature; The intermediate second feature is processed through a multi-head attention mechanism to obtain an intermediate third feature; Processing the intermediate third feature through a dual-scale convolution module to obtain an intermediate fourth feature and an intermediate fifth feature; Processing the intermediate fourth feature and the intermediate fifth feature through the Mish activation function to obtain the intermediate sixth feature and the intermediate seventh feature; splicing the intermediate sixth feature and the intermediate seventh feature to obtain an intermediate eighth feature; The intermediate eighth feature is subjected to adaptive average pooling to obtain an intermediate ninth feature, and the intermediate ninth feature is processed by the feedforward neural network to obtain the second feature.

3. The method according to claim 2, characterized in that The step of processing the intermediate ninth feature through the feedforward neural network to obtain the second feature includes: The intermediate ninth feature in the intra-frame Transformer and the inter-frame Transformer is added to the input of each feedforward neural network layer through a trainable residual connection and then subjected to a layer normalization operation to obtain the second feature.

4. The method according to claim 3, characterized in that The trainable residual connection is represented by the formula: Among them, represents the input, represents the trainable weight matrix; represents the nonlinear mapping function from input to output, and represents the output.

5. A training device for a large language model based on trainable residual connections and a dual-scale convolutional Transformer, characterized in that: include: An acquisition module is used to obtain a basic model based on a multi-layer Transformer architecture, wherein the basic model includes multiple cascaded Transformer decoder layers, each of which includes a self-attention mechanism and a feedforward neural network, and each of which is embedded with a two-scale convolution module. The two-scale convolution modules of different scales are used to capture different local features. The outputs of the two-scale convolution modules are fused as the output of the corresponding Transformer decoder layer. A trainable weight matrix is ​​configured between the input and output of each Transformer decoder layer, and the trainable weight matrix is ​​used to adjust the strength of the residual connection between each Transformer decoder layer; obtaining preprocessed sample document data to construct a training set; A training module is used to train the basic model based on the multi-layer Transformer architecture based on the training set until a preset training termination condition is reached, thereby obtaining a trained large language model that integrates trainable residual connections and dual-scale convolutional Transformers; The training module is specifically used to: The text data included in the training set is converted into multiple digital sequences through a word segmenter; the numbers included in the digital sequence are the index numbers of the text data in the dictionary; the multiple digital sequences are obtained by embedding multiple high-dimensional vectors; according to the multiple high-dimensional vectors, the high-dimensional vectors corresponding to each layer of the Transformer architecture are normalized, and the normalized original features are processed by a linear layer; the processed original features are divided into blocks on the time axis to obtain the first feature; the block operation adopts a 50% overlap factor; a multi-scale Transformer with a trainable residual connection structure and an added convolution module adopts an intra-frame Transformer and an inter-frame Transformer to learn the short-term and long-term dependencies of the first feature respectively to obtain the second feature, and the intra-frame Transformer and the inter-frame Transformer have the same structure; the second feature generated by the multi-scale Transformer is subjected to a PReLU activation function and a linear layer to obtain a third feature; the third feature is obtained by an overlap-addition operation; the overlap rate of the overlap-addition operation is 0.5; the fourth feature is obtained by ReLU The activation function and feedforward network layer are processed, and the feature representation of each token vocabulary is extracted and learned using nonlinear transformation; when the last layer of Transformer architecture is reached, the sampling probability of all digital sequences is obtained by the SoftMax function, and the next digital sequence is sampled according to the preset sampling algorithm; the steps of normalizing the high-dimensional vector corresponding to each layer of Transformer architecture and processing the normalized original features through the linear layer are repeated, until the fourth feature is processed through the ReLU activation function and the feedforward network layer, and the feature representation of each token vocabulary is extracted and learned using nonlinear transformation, until the preset number of cycles is reached, an index sequence is generated; the word segmenter is used in combination with the index sequence to restore the text to obtain the target natural language, so as to complete the pre-training of the basic model based on the multi-layer Transformer architecture to learn the statistical characteristics and contextual relationships of the language; the pre-trained basic model based on the multi-layer Transformer architecture is supervised and fine-tuned according to the preset downstream task; the basic model based on the multi-layer Transformer architecture after supervision and fine-tuning is combined with the reward model for reinforcement learning through a preset strategy optimization algorithm.

6. A computer device, characterized in that: The computer device includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device executes the method according to any one of claims 1 to 4.

7. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to any one of claims 1 to 4.

Images