Large model training and reasoning method based on long memory semantic mode driving

Abstract

The invention discloses a large model training and reasoning method based on long memory semantic mode driving, which comprises the following steps: collecting long text data of a plurality of channels, and carrying out structured format storage and category labeling; initializing model parameters; starting to carry out multi-batch training; adding noise in data input and each hidden layer; the model carries out forward propagation on the input data; calculating loss according to an actual label and a prediction result, and performing first-order and second-order moment estimation by using a bias correction mechanism to update parameters of the model; dynamically adjusting a learning rate and a regularization coefficient; and reasoning by using the trained model. According to the method, the capturing capability of the model on the long-distance dependency relationship is improved, the performance is better in the face of complex and variable input, the training efficiency is improved, the risk of overfitting is reduced, the sparsity and the dispersion degree of the model weight can be effectively controlled, and the generalization capability of the model is further improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a large model training and reasoning method driven by a long memory semantic pattern. Background Art

[0002] With the development of information technology and the surge in data volume, traditional natural language processing (NLP) methods can no longer meet the needs of complex semantic analysis and long text processing. Especially in long text scenarios such as social media, online comments, and academic articles, the richness and complexity of information require models to have stronger understanding capabilities and better context capture capabilities. Existing deep learning models, especially Transformer-based architectures, have achieved remarkable results in many NLP tasks, but often face the limitations of self-attention mechanisms when processing long texts. These models often perform poorly in capturing long-distance dependencies, resulting in information loss and deviations in semantic understanding. In addition, the static parameter settings and lack of robustness mechanisms during the training process make the model vulnerable to noise and data changes in practical applications, further limiting its scope of application.

[0003] Specifically, the existing artificial intelligence large model technology has at least the following deficiencies:

[0004] (1) Traditional Transformer models tend to lose important contextual information when processing very long texts, resulting in decreased model performance;

[0005] (2) Many existing models fail to effectively cope with noisy data or changing inputs, resulting in insufficient model stability in practical applications;

[0006] (3) The existing technology often uses static learning rates and regularization coefficients, lacking a mechanism for dynamic adjustment based on data characteristics and model status, which affects the training efficiency and effect of the model;

[0007] (4) In complex tasks, traditional models perform well on training sets, but often overfit in test sets or actual applications, limiting their scope of application.

[0008] In view of this, it is necessary to provide a large model training and reasoning method driven by long-memory semantic patterns to overcome the above defects. Summary of the invention

[0009] The purpose of the present invention is to provide a large model training and reasoning method driven by long-memory semantic patterns, aiming to solve the problems that existing models perform poorly in capturing long-distance dependencies and are easily affected by noise and data changes in practical applications.

[0010] In order to achieve the above-mentioned object, the first aspect of the present invention provides a large model training and reasoning method based on long memory semantic pattern driving, comprising:

[0011] Step S10: Collect long text data from multiple channels, store them in a structured format and label them by categories, and generate training sets and validation sets;

[0012] Step S20: construct a Transformer model based on long memory semantic understanding and initialize the model parameters;

[0013] Step S30: inputting the training set into the Transformer model to start multi-batch training, so that the Transformer model learns various semantic patterns from simple to complex;

[0014] Step S40: adding noise to the data input and each hidden layer of the Transformer model using an adaptive noise injection mechanism; wherein the intensity and distribution of the noise are dynamically adjusted according to the statistical characteristics of the current batch of data;

[0015] Step S50: In each training batch, the Transformer model will forward propagate the input data and calculate the output based on the current model parameters to enhance the attention to the long-distance context;

[0016] Step S60: After the Transformer model is output, the loss is calculated according to the actual label and the prediction result, and the first-order and second-order moment estimation is performed using the bias correction mechanism to update the parameters of the Transformer model; wherein the loss function uses a composite regularization method to control the sparsity and dispersion of the weights;

[0017] Step S70: Utilizing a dynamic parameter adjustment mechanism, by analyzing the changing trend of the gradient in real time, the learning rate and the regularization coefficient are dynamically adjusted; wherein the learning rate is dynamically adjusted according to the gradient variance of several batches preset before, and the regularization coefficient is adjusted according to the moving average of the absolute value of the gradient;

[0018] Step S80: Repeat the above steps S30-S70 until the preset stop iteration condition is met, and then use the trained model for inference.

[0019] In a preferred embodiment, in step S20, the initialization method for the weight parameters of the Transformer model is expressed as:

[0020] Where W p is the weight parameter of the Transformer model, is the number of nodes in the input layer of the Transformer model, ∈ p is a random number that follows a standard normal distribution.

[0021] In a preferred embodiment, in step S30, the input data of the training set is represented as:

[0022] In the formula, X p Represents the matrix form of the entire training set, is the feature vector of a single sample, m is the number of samples, and T is the matrix transpose.

[0023] In a preferred embodiment, in step S40, the noise injection method is expressed as:

[0024] In the formula, represents the data after noise injection, α p is the coefficient for controlling noise intensity, N p To obey the normal distribution The noise generated, Yes X p The standard deviation of .

[0025] In a preferred embodiment, the coefficient α controlling the noise intensity p The performance feedback of the Transformer model on the validation set is dynamically adjusted, and the adjustment strategy is expressed as:

[0026] In the formula, is the noise intensity of the tth iteration, is the initial noise intensity, β p is the adjustment factor, is the change in the accuracy of the validation set of the Transformer model at the t-1th iteration.

[0027] In a preferred embodiment, in step S50, a custom attention mechanism is used to process data to enhance the model's ability to capture long-distance dependencies, which is specifically expressed as follows:

[0028]

[0029] In the formula, A P is the attention weight matrix, Q P , K P 、V P are query matrix, key matrix and value matrix respectively, is the corresponding weight matrix, d k is the dimension of the key vector.

[0030] In a preferred embodiment, for the query matrix Q P , key matrix K P Sum value matrix V P , based on the multi-head attention mechanism, each head uses a different weight matrix, so that the model can capture different subspaces of information, specifically expressed as:

[0031]

[0032] In the formula, represents the concatenation operation on the feature dimension, H is the number of heads, They are the query weight matrix, key weight matrix, and value weight matrix for each head respectively.

[0033] In a preferred embodiment, in step S60:

[0034] The calculation method of the loss function is as follows:

[0035]

[0036] Where, L p is the loss function of the Transformer model, and are the probabilities of the true label and the predicted label, respectively. is the output layer weight, C is the number of categories; λ p is the regularization coefficient;

[0037] The parameter update method of the Transformer model is as follows:

[0038]

[0039] In the formula, and are the parameters before and after updating, η p is the learning rate of the Transformer model, and are the first-order and second-order moment estimates, respectively, are the attenuation rates, g p is the gradient, ∈ p To stabilize the small amount, t represents the current iteration number.

[0040] In a preferred embodiment, in step S70:

[0041] The learning rate update method is expressed as follows:

[0042] In the formula, is the learning rate of the tth iteration, is the initial learning rate, δp is the adjustment factor, is the variance of the gradient of the first t-1 iterations;

[0043] The update method of the regularization coefficient is expressed as follows:

[0044] In the formula, is the regularization coefficient of the tth iteration, is the initial regularization coefficient, γ p is the attenuation constant, is the moving average of the absolute value of the gradient in the first t-1 iterations.

[0045] In a preferred embodiment, step S80 includes: using the embedding layer of the trained Transformer model to encode the input text to be inferred, specifically including the following sub-steps:

[0046] Step S81: Divide the text into words or clauses;

[0047] Step S82: Use word embedding technology to convert each word and clause into a vector representation;

[0048] Step S83: The vectorized data is input into the Transformer model for forward propagation, the relationship between each word and other words is calculated to focus on long-distance dependencies, and then processed by multiple layers of self-attention and feedforward networks to generate an output vector `Y`;

[0049] Step S84: According to the output vector `Y`, the linear layer and activation function of the Transformer model are used to generate the final inference result;

[0050] Step S85: Arrange and format the inference results into an output that is easy to understand.

[0051] A second aspect of the present invention provides a large model training and reasoning device driven by a long memory semantic pattern, comprising:

[0052] The data processing module is used to collect long text data from multiple channels, store it in a structured format, annotate it with categories, and generate training sets and validation sets;

[0053] Initialization module, used to build a Transformer model based on long memory semantic understanding and initialize model parameters;

[0054] A data input module, used for inputting the training set into the Transformer model to start multi-batch training, so that the Transformer model learns various semantic patterns from simple to complex;

[0055] A noise adding module, used to add noise to the data input and each hidden layer of the Transformer model using an adaptive noise injection mechanism; wherein the intensity and distribution of the noise are dynamically adjusted according to the statistical characteristics of the current batch of data;

[0056] A forward propagation module, used for enabling the Transformer model to forward propagate the input data in each training batch and calculate the output based on the current model parameters to enhance the attention to the long-distance context;

[0057] A parameter updating module, which is used to calculate the loss according to the actual label and the prediction result after the output of the Transformer model, and to update the parameters of the Transformer model by estimating the first-order and second-order moments using a bias correction mechanism; wherein the loss function uses a composite regularization method to control the sparsity and dispersion of the weights;

[0058] The dynamic adjustment module is used to dynamically adjust the learning rate and regularization coefficient by using the dynamic parameter adjustment mechanism and analyzing the gradient change trend in real time. The learning rate is dynamically adjusted according to the gradient variance of several batches preset before, and the regularization coefficient is adjusted according to the moving average of the absolute value of the gradient.

[0059] The iterative reasoning module is used to repeatedly iterate and execute the above modules until the preset stop iteration condition is met, and then use the trained model for reasoning.

[0060] A third aspect of the present invention provides a terminal, comprising a memory, a processor, and a computer program stored in the memory, wherein the computer program, when executed by the processor, implements the various steps of the large model training and reasoning method driven by long memory semantic patterns as described in any one of the above-mentioned embodiments.

[0061] A fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various steps of the large model training and reasoning method based on long memory semantic pattern driving as described in any of the above embodiments.

[0062] A fifth aspect of the present invention provides a computer program product, including a computer program or instructions, which, when processed and executed, implement the various steps of the large model training and reasoning method driven by long memory semantic patterns as described in any of the above-mentioned embodiments.

[0063] The large model training and reasoning method based on long memory semantic pattern driving provided by the present invention has the following features:

[0064] Beneficial effects:

[0065] (1) The traditional Transformer model is improved by adding an extended self-attention mechanism to process very long texts, which improves the model's ability to capture long-distance dependencies and solves the limitations of the traditional Transformer in processing long texts.

[0066] (2) Dynamically injecting noise into the data input and each hidden layer enhances the robustness and generalization ability of the model. The noise intensity can be adjusted according to the statistical characteristics of the current batch of data, making the model perform better when faced with complex and changing inputs;

[0067] (3) Real-time analysis of gradient change trends, dynamic adjustment of learning rate and regularization coefficient to adapt to the needs of the model at different training stages, improve training efficiency, and reduce the risk of overfitting;

[0068] (4) The composite regularization method can effectively control the sparsity and dispersion of model weights, thereby further improving the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0070] Figure 1 A flowchart of a large model training and reasoning method based on long memory semantic pattern drive provided by the present invention;

[0071] Figure 2 A framework diagram of the large model training and reasoning device driven by long-memory semantic patterns provided by the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solution and beneficial technical effect of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.

[0073] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0074] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0075] Embodiment 1

[0076] In an embodiment of the present invention, a large model training and reasoning method driven by a long-memory semantic pattern is provided, which is used to construct and train a model with high generalization ability and capable of processing ultra-long texts.

[0077] like Figure 1 As shown, the large model training and reasoning method driven by long memory semantic patterns includes steps S10-S80.

[0078] Step S10: Collect long text data from multiple channels, store them in a structured format, and label them by categories to generate training sets and validation sets.

[0079] Among them, the sources of training data collection for large model training mainly rely on online content platforms, professional forums, academic databases, and related news websites to ensure the comprehensiveness and diversity of information. In order to ensure the quality and practicality of the data set, data collection can use automated web crawler technology combined with manual screening methods to ensure the relevance and high quality of the data. The collected data is stored in a structured format, specifically in JSON format, which includes but is not limited to fields such as data ID, text content, source link, and collection time.

[0080] Furthermore, the data of the present invention is long text data, and the annotation categories involved include but are not limited to sentiment tendency, topic category, factual relationship recognition and semantic role annotation. Through the annotation of these categories, the fine-grained information of the long text data can be deeply mined, providing rich semantic information for subsequent model training and feature extraction. To more specifically illustrate the structure and annotation of the data, five exemplary embodiment data are provided as shown in Table 1 below:

[0081] Table 1: Structure and category annotation of training data

[0082]

[0083] Step S20: Construct a Transformer model based on long memory semantic understanding and initialize the model parameters.

[0084] It should be noted that the present invention proposes a Transformer model based on long memory semantic understanding as a large model. The basic structure of the Transformer has been widely used in a variety of natural language processing tasks, but in the long memory semantic mode, the traditional Transformer is often limited in effect when processing ultra-long texts due to the limitations of the self-attention mechanism. To this end, the present invention improves the original Transformer structure. Through the extended self-attention mechanism, this model can process longer sequences and enhance the model's ability to capture implicit information in long texts. Secondly, the parameter sparsification strategy is used to reduce the computational burden of the model while maintaining the accuracy of semantic understanding, so that the model can run efficiently even in a resource-constrained environment.

[0085] Specifically, the initialization method for the weight parameters of the Transformer model is expressed as:

[0086] Where W p is the weight parameter of the Transformer model, is the number of nodes in the input layer of the Transformer model, ∈ p is a random number that follows a standard normal distribution.

[0087] Step S30: Input the training set into the Transformer model to start multi-batch training, so that the Transformer model can learn various semantic patterns from simple to complex.

[0088] Among them, the input data of the training set is expressed as:

[0089] In the formula, X p Represents the matrix form of the entire training set, is the feature vector of a single sample, m is the number of samples, and T is the matrix transpose.

[0090] Step S40: using an adaptive noise injection mechanism to add noise to the data input and each hidden layer of the Transformer model; wherein the intensity and distribution of the noise are dynamically adjusted according to the statistical characteristics of the current batch of data.

[0091] Specifically, an adaptive noise injection mechanism is used to add noise to the data input and each hidden layer to enhance the robustness and generalization ability of the model. Specifically, in each batch of data X p X p In the case of injected noise N p N p ,The intensity and distribution of noise are dynamically adjusted according to the statistical characteristics of the current batch of data.

[0092] Among them, the noise injection method is expressed as:

[0093] In the formula, represents the data after noise injection, α p is the coefficient for controlling noise intensity, N p To obey the normal distribution The noise generated, Yes X p The standard deviation of .

[0094] Furthermore, the coefficient α that controls the noise intensity p The performance feedback of the Transformer model on the validation set is dynamically adjusted, and the adjustment strategy is expressed as:

[0095] In the formula, is the noise intensity of the tth iteration, is the initial noise intensity, β p is the adjustment factor, is the change in the accuracy of the validation set of the Transformer model at the t-1th iteration.

[0096] Step S50: In each training batch, the Transformer model will forward propagate the input data and calculate the output based on the current model parameters to enhance the attention to long-distance context.

[0097] It should be noted that, in the present invention, based on the capture of long-range dependency information, the attention to long-range context is enhanced in the forward propagation. Specifically, a custom attention mechanism is used for data processing to enhance the model's ability to capture long-range dependencies, which is specifically expressed as follows:

[0098]

[0099] In the formula, A P is the attention weight matrix, Q P , K P 、V P are query matrix, key matrix and value matrix respectively, is the corresponding weight matrix, d k is the dimension of the key vector.

[0100] Furthermore, for the query matrix Q P , key matrix K P Sum value matrix V P, based on the multi-head attention mechanism, each head uses a different weight matrix, allowing the model to learn information in different representation subspaces, enhancing the diversity of feature capture and the expressiveness of the model, so that the model can capture different subspaces of information, specifically expressed as:

[0101]

[0102] In the formula, represents the concatenation operation on the feature dimension, H is the number of heads, They are the query weight matrix, key weight matrix, and value weight matrix for each head respectively.

[0103] Step S60: After the Transformer model is output, the loss is calculated based on the actual label and the predicted result, and the bias correction mechanism is used to perform first-order and second-order moment estimation to update the parameters of the Transformer model; wherein the loss function uses a composite regularization method to control the sparsity and dispersion of the weights.

[0104] Specifically, after the Transformer model is output, the loss will be calculated based on the actual label and the predicted result. The loss function of the Transformer model takes into account the characteristics of long text data, which can more accurately evaluate the model's understanding of complex semantic structures. The loss function uses a composite regularization method to maintain the sparsity of the weights and control the dispersion of the weight values ​​to prevent overfitting. The calculation method of the loss function is expressed as follows:

[0105]

[0106] Where, L p is the loss function of the Transformer model, and are the probabilities of the true label and the predicted label, respectively. is the output layer weight, C is the number of categories; λ p is the regularization coefficient.

[0107] Among them, in order to perform sparse processing on the parameters of the Transformer model, ensure that the model will not lose too much semantic understanding ability while improving efficiency, and at the same time, use the bias correction mechanism to estimate the first-order and second-order moments, so that the model can estimate the first-order and second-order moments more stably in the early stage of training. The parameter update method of the Transformer model is expressed as follows:

[0108]

[0109] In the formula, and are the parameters before and after updating, ηp is the learning rate of the Transformer model, and are the first-order and second-order moment estimates, respectively, are the attenuation rates, g p is the gradient, ∈ p To stabilize the small amount, ensure that the denominator is non-zero; t represents the current number of iterations. and The bias correction is performed by the inverse of the exponentially decaying repeated product of , which ensures the accuracy of the estimation in the early stage of training when there are fewer samples.

[0110] Step S70: Utilize the dynamic parameter adjustment mechanism to dynamically adjust the learning rate and regularization coefficient by analyzing the changing trend of the gradient in real time to adapt to the different characteristics of the data and the complexity requirements of the model during the training process. The learning rate is dynamically adjusted according to the gradient variance of several batches preset before, and the regularization coefficient is adjusted according to the moving average of the absolute value of the gradient.

[0111] The learning rate is dynamically adjusted according to the gradient variance of the past few batches to ensure that the learning step size is reduced when the gradient changes greatly, thereby avoiding over-adjustment. Specifically, the learning rate update method is expressed as follows:

[0112] In the formula, is the learning rate of the tth iteration, is the initial learning rate, δ p is the adjustment factor, is the variance of the gradient of the first t-1 iterations.

[0113] The regularization coefficient is adjusted according to the moving average of the absolute value of the gradient to adapt to the overfitting or underfitting tendency of the model at different training stages. Specifically, the update method of the regularization coefficient is expressed as follows:

[0114] In the formula, is the regularization coefficient of the tth iteration, is the initial regularization coefficient, γ p is the attenuation constant, is the moving average of the absolute value of the gradient in the first t-1 iterations.

[0115] Step S80: Repeat the above steps S30-S70 until the preset stop iteration condition is met, and then use the trained model for inference.

[0116] Specifically, the above steps are repeated until a preset stop iteration condition is met, indicating that the Transformer model training is completed. For example, the preset stop iteration condition is reaching a preset maximum number of iterations, preferably, the preset maximum number of iterations is set to 1000 times.

[0117] After the training of the large model is completed, the trained large model is used for reasoning. Step S80 includes: using the embedding layer of the trained Transformer model to encode the input text to be reasoned, specifically including the following sub-steps S81-S85.

[0118] Step S81: Divide the text into words or clauses.

[0119] Step S82: Use word embedding technology (such as Word2Vec or BERT) to convert each word and clause into a vector representation. For example, assume that the output is a matrix `X` with a dimension of `[sequence length, embedding dimension]`.

[0120] Step S83: The vectorized data is input into the Transformer model for forward propagation (forward propagation involves the model's internal data processing rules), the relationship between each word and other words is calculated to focus on long-distance dependencies, and then processed by multiple layers of self-attention and feedforward networks to generate an output vector `Y`;

[0121] Step S84: According to the output vector `Y`, the linear layer and activation function of the Transformer model are used to generate the final inference result;

[0122] Step S85: Arrange and format the inference results into an output that is easy to understand.

[0123] The following will combine an exemplary embodiment to further assist in explaining the reasoning process of a trained large model on a long text.

[0124] Suppose there is a long text as follows:

[0125] “As global climate change intensifies, crop production patterns are also constantly adjusting. The latest research shows that agricultural yields in some areas have increased after climate adaptation measures were implemented. However, there are still some areas facing drought and water shortages, resulting in reduced yields.”

[0126] Convert this text into the model input format, structured as JSON:

[0127] ```json

[0128] {

[0129] "id":"001",

[0130] "text":"As global climate change intensifies, crop production patterns are also constantly adjusting. The latest research shows that agricultural yields in some areas have increased after climate adaptation measures were implemented. However, there are still some areas facing drought and water shortages, resulting in reduced yields.",

[0131] "timestamp":"2024-10-07"

[0132] }

[0133] ```

[0134] Furthermore, the input text is encoded using the embedding layer of the trained Transformer model. The specific steps are as follows:

[0135] (1) Word segmentation: divide the text into words or clauses, such as:

[0136] ```

[0137] [As global climate change intensifies, the production patterns of crops are also constantly adjusting.]

[0138] ```

[0139] (2) Vectorization: Use word embedding technology (such as Word2Vec or BERT) to convert each word into a vector representation. Assume that the output is a matrix `X` with the dimension `[sequence length, embedding dimension]`.

[0140] (3) Forward propagation: The vectorized data is input into the Transformer model for forward propagation. Forward propagation involves the internal data processing rules of the model, specifically:

[0141] (3.1) Self-attention mechanism: Through the improved self-attention mechanism, the model will calculate the relationship between each word and other words, focusing on long-distance dependencies.

[0142] (3.2) Processing long texts: During the processing, the model retains the key information in the long text, for example, identifying the relationship between "climate change" and "drought".

[0143] (3.3) Generate output vector: After multiple layers of self-attention and feedforward network processing, the output vector `Y` is generated, and the dimension is also `[sequence length, embedding dimension]`.

[0144] (4) Output generation: Based on the output vector `Y`, a linear layer and an activation function are used to generate the final inference result. Assuming that sentiment analysis and topic classification are to be performed, the output is as follows:

[0145] Sentiment Analysis:

[0146] - Forecast: Neutral

[0147] -Confidence: 0.85

[0148] Subject classification:

[0149] - Forecast topic: Climate change

[0150] - Confidence: 0.90

[0151] (5) Post-processing of results

[0152] Arrange and format the inference results into understandable output:

[0153] ```json

[0154] {

[0155] "id":"001",

[0156] "original_text":"As global climate change intensifies, crop production patterns are also constantly adjusting. The latest research shows that agricultural yields in some areas have increased after climate adaptation measures were implemented. However, there are still some areas facing drought and water shortages, resulting in reduced yields.",

[0157] "sentiment":{

[0158] "label":"Neutral",

[0159] "confidence":0.85

[0160] },

[0161] "topic":{

[0162] "label":"Climate Change",

[0163] "confidence":0.90

[0164] }

[0165] }

[0166] ```

[0167] In summary, the large model training and reasoning method based on long memory semantic pattern driving provided by the present invention has the following beneficial effects:

[0168] (1) The present invention improves the traditional Transformer model and adds an extended self-attention mechanism to process ultra-long texts, thereby improving the model's ability to capture long-distance dependencies and solving the limitations of the traditional Transformer in processing long texts.

[0169] (2) Dynamically injecting noise into the data input and each hidden layer enhances the robustness and generalization ability of the model. The noise intensity can be adjusted according to the statistical characteristics of the current batch of data, making the model perform better when faced with complex and changing inputs;

[0170] (3) Real-time analysis of gradient change trends, dynamic adjustment of learning rate and regularization coefficient to adapt to the needs of the model at different training stages, improve training efficiency, and reduce the risk of overfitting;

[0171] (4) The composite regularization method can effectively control the sparsity and dispersion of model weights, thereby further improving the generalization ability of the model.

[0172] The second aspect of the present invention provides a large model training and reasoning device 100 driven by a long memory semantic pattern, which is used to construct and train a model with high generalization ability and capable of processing ultra-long texts. It should be noted that the implementation principle and specific implementation method of the large model training and reasoning device 100 driven by a long memory semantic pattern can refer to the above-mentioned large model training and reasoning method driven by a long memory semantic pattern, and will not be repeated below.

[0173] like Figure 2 As shown, the large model training and reasoning device 100 driven by long memory semantic pattern includes:

[0174] The data processing module 10 is used to collect long text data from multiple channels, store them in a structured format, annotate them by category, and generate training sets and validation sets;

[0175] Initialization module 20, used to build a Transformer model based on long memory semantic understanding and initialize model parameters;

[0176] A data input module 30 is used to input the training set into the Transformer model to start multi-batch training, so that the Transformer model learns various semantic patterns from simple to complex;

[0177] A noise adding module 40, for adding noise to the data input and each hidden layer of the Transformer model using an adaptive noise injection mechanism; wherein the intensity and distribution of the noise are dynamically adjusted according to the statistical characteristics of the current batch of data;

[0178] A forward propagation module 50, configured to enable the Transformer model to forward propagate the input data in each training batch and calculate outputs based on current model parameters to enhance attention to long-distance contexts;

[0179] A parameter updating module 60 is used to calculate the loss according to the actual label and the prediction result after the output of the Transformer model, and to update the parameters of the Transformer model by estimating the first and second order moments using a bias correction mechanism; wherein the loss function uses a composite regularization method to control the sparsity and dispersion of the weights;

[0180] A dynamic adjustment module 70 is used to dynamically adjust the learning rate and the regularization coefficient by using a dynamic parameter adjustment mechanism and analyzing the gradient change trend in real time; wherein the learning rate is dynamically adjusted according to the gradient variance of a plurality of batches preset before, and the regularization coefficient is adjusted according to the moving average of the absolute value of the gradient;

[0181] The iterative reasoning module 80 is used to repeatedly iterate and execute the above modules until a preset stop iteration condition is met, and then use the trained model to perform reasoning.

[0182] Embodiment 3

[0183] The present invention provides a terminal, which includes a memory, a processor, and a computer program stored in the memory. When the computer program is executed by the processor, the computer program implements the various steps of the large model training and reasoning method driven by long memory semantic patterns as described in any of the above-mentioned implementation modes.

[0184] Embodiment 4

[0185] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various steps of the large model training and reasoning method based on long memory semantic pattern driving as described in any of the above-mentioned implementation modes.

[0186] Embodiment 5

[0187] The present invention provides a computer program product, including a computer program or instructions, which, when processed and executed, implement the various steps of the large model training and reasoning method driven by long memory semantic patterns as described in any of the above-mentioned implementation modes.

[0188] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0189] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0190] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0191] In the embodiments provided by the present invention, it should be understood that the disclosed systems or devices / terminal equipment and methods can be implemented in other ways. For example, the system or device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0192] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0193] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0194] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.

Claims

1. A large model training and reasoning method based on long memory semantic pattern driving, characterized in that: include: Step S10: Collect long text data from multiple channels, store them in a structured format and label them by categories, and generate training sets and validation sets; Step S20: construct a Transformer model based on long memory semantic understanding and initialize the model parameters; Step S30: inputting the training set into the Transformer model to start multi-batch training, so that the Transformer model learns various semantic patterns from simple to complex; Step S40: adding noise to the data input and each hidden layer of the Transformer model using an adaptive noise injection mechanism; wherein the intensity and distribution of the noise are dynamically adjusted according to the statistical characteristics of the current batch of data; Step S50: In each training batch, the Transformer model will forward propagate the input data and calculate the output based on the current model parameters to enhance the attention to the long-distance context; Step S60: After the Transformer model is output, the loss is calculated according to the actual label and the prediction result, and the first-order and second-order moment estimation is performed using the bias correction mechanism to update the parameters of the Transformer model; wherein the loss function uses a composite regularization method to control the sparsity and dispersion of the weights; Step S70: Utilizing a dynamic parameter adjustment mechanism, by analyzing the changing trend of the gradient in real time, the learning rate and the regularization coefficient are dynamically adjusted; wherein the learning rate is dynamically adjusted according to the gradient variance of several batches preset before, and the regularization coefficient is adjusted according to the moving average of the absolute value of the gradient; Step S80: Repeat the above steps S30-S70 until the preset stop iteration condition is met, and then use the trained model for inference.

2. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 1, characterized in that: In step S20, the initialization method of the weight parameters of the Transformer model is expressed as: Where W p is the weight parameter of the Transformer model, is the number of nodes in the input layer of the Transformer model, ∈ p is a random number that follows a standard normal distribution.

3. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 2, characterized in that: In step S30, the input data of the training set is represented as: Where, X p Represents the matrix form of the entire training set, is the feature vector of a single sample, m is the number of samples, and T is the matrix transpose.

4. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 3, characterized in that: In step S40, the noise injection method is expressed as: In the formula, represents the data after noise injection, α p is the coefficient for controlling noise intensity, N p To follow the normal distribution The noise generated, Yes X p The standard deviation of .

5. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 4, characterized in that: The coefficient α that controls the noise intensity p The performance feedback of the Transformer model on the validation set is dynamically adjusted, and the adjustment strategy is expressed as: In the formula, is the noise intensity of the tth iteration, is the initial noise intensity, β p is the adjustment factor, is the change in the accuracy of the validation set of the Transformer model at the t-1th iteration.

6. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 4, characterized in that: In step S50, a custom attention mechanism is used to process data to enhance the model's ability to capture long-distance dependencies, which is specifically expressed as follows: In the formula, A P is the attention weight matrix, Q P , K P 、V P are query matrix, key matrix and value matrix respectively, is the corresponding weight matrix, d k is the dimension of the key vector.

7. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 6, characterized in that: For the query matrix Q P , key matrix K P Sum value matrix V P , based on the multi-head attention mechanism, each head uses a different weight matrix, so that the model can capture different subspaces of information, specifically expressed as: In the formula, represents the concatenation operation on the feature dimension, H is the number of heads, They are the query weight matrix, key weight matrix, and value weight matrix for each head respectively.

8. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 6, characterized in that: In step S60: The calculation method of the loss function is as follows: Where, L p is the loss function of the Transformer model, and are the probabilities of the true label and the predicted label, respectively. is the output layer weight, C is the number of categories; λ p is the regularization coefficient; The parameter update method of the Transformer model is as follows: In the formula, and are the parameters before and after updating, η p is the learning rate of the Transformer model, and are the first-order and second-order moment estimates, respectively, are the attenuation rates, g p is the gradient, ∈ p To stabilize the small amount, t represents the current iteration number.

9. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 8, characterized in that: In step S70: The learning rate update method is expressed as follows: In the formula, is the learning rate of the tth iteration, is the initial learning rate, δ p is the adjustment factor, is the variance of the gradient of the first t-1 iterations; The update method of the regularization coefficient is expressed as follows: In the formula, is the regularization coefficient of the tth iteration, is the initial regularization coefficient, γ p is the attenuation constant, is the moving average of the absolute value of the gradient in the previous t-1 iterations.

10. The large model training and reasoning method based on long memory semantic pattern driving as claimed in claim 1, characterized in that: The step S80 includes: using the embedding layer of the trained Transformer model to encode the input text to be inferred, which specifically includes the following sub-steps: Step S81: Divide the text into words or clauses; Step S82: Use word embedding technology to convert each word and clause into a vector representation; Step S83: The vectorized data is input into the Transformer model for forward propagation, the relationship between each word and other words is calculated to focus on long-distance dependencies, and then processed by multiple layers of self-attention and feedforward networks to generate an output vector `Y`; Step S84: According to the output vector `Y`, the linear layer and activation function of the Transformer model are used to generate the final inference result; Step S85: Arrange and format the inference results into an output that is easy to understand.

Images