Long text cooperation generation method based on master-slave model cooperation

Abstract

The invention discloses a master-slave model collaboration-based long text collaboration generation method, which comprises the following steps of: acquiring training data through a public data set, and extracting abstracts, keywords and paragraph levels by using a natural language processing tool so as to provide structured information for the training data; layering a long text corpus in the training data, and extracting a theme layer and a paragraph layer for training a model to generate a theme framework; the layered data is converted into an input / output pair acceptable to a model, the input is a global context, and the output is a framework of the theme layer; a text generation model is constructed based on a Transform network, and then training is carried out; and performing long text collaborative generation through the trained text generation model. Structured information is provided for training data, the target and generation constraints of the model are defined, and the task definition problem that a traditional generation model is difficult to generate a hierarchical long text is solved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a long-text collaborative generation method based on master-slave model collaboration. Background Art

[0002] At present, with the development of artificial intelligence technology, long text generation has gradually become one of the important research directions in the field of natural language processing. Long text generation not only requires effective modeling of the rich information in large-scale corpora, but also requires the generated text to have a high degree of thematic coherence and logical consistency to meet the diverse needs in practical applications. However, since long texts usually have a complex hierarchical structure, including multi-level semantic relationships such as global themes, paragraph themes, and specific details, existing technologies face many challenges in processing long text generation. For example, traditional generation models usually use flat data structures, which are difficult to capture the hierarchical characteristics of long texts, resulting in the generated content being thematically scattered and loosely structured. In addition, the model's lack of understanding of global semantic information and control of the logic between paragraphs when processing long texts often leads to semantic breaks and lack of coherence in the generated results.

[0003] In order to meet the above challenges, the Transformer network based on deep learning has become the mainstream method in the field of long text generation due to its powerful feature modeling capabilities. However, the Transformer network still has certain limitations when applied to long text generation. For example, when dealing with deep neural networks, it may face problems such as gradient vanishing and local optimality, which affects the model performance and generation effect. In addition, the existing technology lacks a clear task definition for long text generation, and the input and output targets are unclear, which makes it difficult for the training process to converge efficiently, and the generation results are difficult to meet the requirements of practical applications in terms of logic and structure. Therefore, how to design a generation method that can fully capture the hierarchical structure information of long texts and improve the coherence and logical consistency of the generated content has become a problem that needs to be solved urgently.

[0004] Therefore, the prior art has at least the following deficiencies:

[0005] (1) Traditional text generation models are usually based on flat data structures and lack a clear representation of topics and paragraph levels. The generated long texts have scattered topics and chaotic structures, which makes it difficult to meet the logic and clarity requirements in applications.

[0006] (2) Models using the gradient descent method are prone to problems such as gradient vanishing or local optimality when processing deep neural networks, resulting in unstable quality of long text generation, poor generalization ability, and difficulty in adapting to complex high-dimensional feature distributions;

[0007] (3) Traditional attention mechanisms and decoding methods are difficult to effectively capture the global semantics and contextual logic in long texts. The generated texts are prone to semantic incoherence between paragraphs, which affects the readability and logical consistency of the texts.

[0008] (4) The task definition of existing generative models is not clear enough, and the correspondence between input and output is weak, resulting in unclear training objectives. The generated text is difficult to meet the application's requirements for hierarchical and logically clear text.

[0009] In view of this, it is necessary to provide a long-text collaborative generation method based on the master-slave model to overcome the above defects. Summary of the invention

[0010] The purpose of the present invention is to provide a long text collaborative generation method based on master-slave model collaboration, aiming to solve the problem of how to design a generation method that can fully capture the hierarchical structure information of long texts and improve the coherence and logical consistency of generated content.

[0011] In order to achieve the above object, the present invention provides a long-text collaborative generation method based on master-slave model collaboration, comprising:

[0012] Step S10: obtaining training data through public data sets, and using natural language processing tools to extract summaries, keywords, and paragraph levels, thereby providing structured information for the training data;

[0013] Step S20: Layer the long text corpus in the training data, extract the topic layer and the paragraph layer, and use them to train the model to generate a topic framework; wherein the topic layer describes the global topic and paragraph framework of the article, and the paragraph layer is a specific description of the paragraph topic;

[0014] Step S30: converting the stratified data into an input / output pair acceptable to the model, where the input is the global context and the output is the framework of the subject layer;

[0015] Step S40: constructing a text generation model based on a Transformer network, and then importing the input / output pairs into the text generation model for training;

[0016] Step S50: Collaboratively generate long texts using the trained text generation model; wherein the output is a generated topic framework, including article titles and paragraph topics.

[0017] In a preferred embodiment, the text generation model includes an encoder and a decoder; wherein the encoder is used to map input features into a series of context-aware vector representations, and the decoder is used to gradually decode the vector representations into target features;

[0018] The encoder includes N identical encoding layers, each encoding layer includes two sublayers, a multi-head attention mechanism sublayer and a feedforward network sublayer, and the encoder adds a residual connection and a normalization operation after each sublayer;

[0019] The decoder includes N identical decoding layers, each decoding layer includes two multi-head attention mechanism sublayers and one feedforward network sublayer, and the decoder adds residual connections and normalization operations after each sublayer;

[0020] The second multi-head attention mechanism sublayer of the decoding layer is implemented by a multi-head cross-attention mechanism; wherein the multi-head cross-attention mechanism is a mechanism for information exchange between the encoder and the decoder, which takes the output features of the encoder and the input features of the decoder as input, and uses the attention mechanism to perform cross-attention calculations on the input, thereby obtaining a set of weighted decoder output features.

[0021] In a preferred embodiment, the multi-head attention mechanism sublayer of the encoder and the first multi-head attention mechanism sublayer of the decoder are both implemented by a multi-head self-attention mechanism; wherein the multi-head self-attention mechanism comprises the following steps:

[0022] Step S61: Use the Softmax function to process the attention score to obtain the attention weight of each position in the input data sequence, and make it processable as a probability distribution, expressed as:

[0023]

[0024] In the formula, head i is the attention weight of the i-th head, Q i is the query vector of the i-th head, K i is the key vector of the i-th head, is the transpose of the key vector of the i-th head, d ks is the dimension of the input feature, h ks is the number of attention heads, V i is the value vector of the i-th head; Sof( ) is Softmax 函 number;

[0025] Step S62: Use the obtained attention weights to perform weighted summation on the value vector to obtain an output vector;

[0026] Step S63: Concatenate the output vectors of all heads together, and then generate the output of the final multi-head self-attention mechanism sub-layer through a linear transformation layer.

[0027] In a preferred embodiment, in the encoder and the decoder, the output of each sub-layer is the result of layer normalization and residual connection, which is expressed as:

[0028] y cgy =LN c (x trans ++Sublayer(x trans )),

[0029] In the formula, LN c ( ) is the layer normalization function, Sublayer( ) represents the output of the residual connected sublayer, x trans Represents the input of the sublayer; y cgy is the feature output after layer normalization.

[0030] In a preferred embodiment, in step S40, the feedforward network of the feedforward network sublayer is trained based on the photon reverberation optimization algorithm as a feature extraction model, and the specific training process includes:

[0031] Step S41: Simulate the basic characteristics of light waves, initialize weights and biases, ensure that the initial parameter values ​​have good distribution and periodicity, thereby improving the diversity of initial data feature capture and reducing the risk of the feedforward network model falling into a bad convergence path. and the bias of the jth layer Its initialization method is expressed as:

[0032]

[0033] Where, lay(i) represents the layer index of the i-th layer of the feedforward network; lay(j) represents the layer index of the j-th layer of the feedforward network; A p is the light wave amplitude, which is used to control the initial range of the parameter value; p is the wavelength of the light wave, set to 2π to cover a complete period; φ p is the initial phase; the basic characteristics of the light wave include wavelength and amplitude;

[0034] Step S42: In each iteration, the propagation behavior of light waves in the medium is simulated, and the parameter values ​​of the feedforward network are adjusted. The update amount of the weight and bias of the feedforward network is calculated as follows:

[0035]

[0036] Where η p is the learning rate of the feedforward network; is the weight update amount of the i-th layer of the feedforward network; is the bias update of the jth layer of the feedforward network; L pis the loss function of the feedforward network, used to evaluate the performance of the current model; σ p To simulate environmental noise, it is used to generate random disturbances; is the phase offset of the i-th layer of the feedforward network; is the phase offset of the jth layer of the feedforward network; the medium refers to the parameter space of the feedforward network;

[0037] Step S43: According to the characteristics of the current medium, the reflection and refraction of the light wave are calculated. The reflectivity is mainly used to review the historical features in the parameter adjustment, and the refractive index ensures that the model can quickly capture the deep features in the new input data. The calculation formulas of reflectivity and refractive index are:

[0038]

[0039] In the formula, is the reflectivity parameter of the i-th layer of the feedforward network; is the refractive index parameter of the jth layer of the feedforward network; is the incident refractive index parameter of the medium to characterize the layer input influencing factors of the feedforward network; is the refractive index parameter of the output of the medium, to characterize the factors affecting the layer output of the feedforward network;

[0040] Step S44: Update the medium characteristics according to the propagation results of the light wave, adjust the weights and biases of the feedforward network to characterize the adjustment effect of the light wave on the parameters after passing through the medium, and adjust the amplitude and direction of the weights and biases by simulating the update mechanism of the medium characteristics in combination with reflection and refraction to adapt to the complex distribution of data characteristics. The update method is expressed as:

[0041]

[0042] In the formula, is the weight of the i-th layer of the feedforward network after update; is the bias of the jth layer of the updated feedforward network; is the time delay increment of the weight of the i-th layer of the feedforward network; is the time delay increment of the bias of the jth layer of the feedforward network;

[0043] Among them, the role of the time delay increment is to enhance the utilization of historical information, and its calculation method is expressed as:

[0044]

[0045] In the formula, γ p is the delay coefficient, used to smooth the impact of historical updates; is the weight update amount of the i-1th layer of the feedforward network; is the bias update of the j-1th layer of the feedforward network;

[0046] Step S45: Using the loss function L p Evaluate the matching degree between the current network output and the target, which is used to calculate the current gradient amplitude of the feedforward network, and then adjust the light wave characteristics. The loss function of the feedforward network contains regularization terms and nonlinear penalty terms to increase the complexity and robustness of the model. Its calculation method is expressed as:

[0047]

[0048] Where npe is the number of samples input to the feedforward network in the current batch; is the true label of the kth sample; is the predicted label of the kth sample calculated by the preset Softmax function; pa is the regularization parameter of the feedforward network, which is used to control the model complexity; || || is the L2 norm; ρ p is the nonlinear adjustment parameter of the feedforward network to enhance flexibility; mc is the total number of layers of the feedforward network;

[0049] Step S46: Repeat the above steps S42-S45 until a preset number of iterations is reached or the loss function converges below a preset threshold, which indicates that the feedforward network training of the text generation model is completed.

[0050] In a preferred embodiment, in step S42, the phase offset is calculated according to the light wave interference characteristics and used for parameter update, so that the parameter update can adjust the update direction and amplitude according to the multi-dimensional characteristics of the data, and avoid the gradient information from gradually disappearing due to the complexity of the high-dimensional features. The calculation method is expressed as:

[0051]

[0052] In the formula, is the historical change trend of the i-th layer of the feedforward network; is the current gradient amplitude of the feedforward network; β p is the parameter affecting light wave interference.

[0053] In a preferred embodiment, in step S42, the historical change trend of the feedforward network represents the cumulative contribution of parameter update, which is calculated as follows:

[0054]

[0055] In the formula, α p The historical attenuation coefficient; Tmax is the number of iterations of the feedforward network; tp is the current number of iterations.

[0056] In a preferred embodiment, in step S42, the calculation method of the current gradient amplitude of the feedforward network is expressed as:

[0057]

[0058] In the formula, is the current gradient magnitude of the feedforward network.

[0059] In a preferred embodiment, in step S43, the calculation method of the refractive index parameter of the medium is expressed as:

[0060]

[0061] In the formula, pi is the permittivity, which is dynamically adjusted according to the activation function type of the feedforward network sublayer; is the magnetic permeability, which is related to the amplitude of parameter changes of the feedforward network.

[0062] In a preferred embodiment, in the step S50, the sequential relationship of the input of the text generation model is distinguished by position coding.

[0063] The long-text collaborative generation method based on master-slave model collaboration provided by the present invention has the following beneficial effects:

[0064] (1) By performing hierarchical processing on the input text data, information at the topic level and paragraph level is extracted to describe the global topic framework and specific paragraph content of the article respectively, thereby solving the problems of unclear topics and structures in the long text generation process. In addition, natural language processing tools are used to extract abstracts, keywords and paragraph levels, providing structured information for training data, so that the model can more effectively learn the generation mode of hierarchical long texts;

[0065] (2) By constructing training samples with background information as input and hierarchical topic framework as output, the model's objectives and generation constraints are clarified, solving the task definition problem that traditional generative models have difficulty in generating hierarchical long texts. In addition, in the preferred embodiment, position encoding is used to enable the model to distinguish the sequential relationship of input data, thereby improving the discourse coherence of long text generation;

[0066] (3) In some preferred embodiments, a photon reverberation optimization algorithm is used to simulate the propagation behavior of light waves in nonlinear media, and the weights and bias parameters of the feedforward network are dynamically adjusted to solve the problems of gradient vanishing and local optimality that are prone to occur in the traditional gradient descent method when training deep neural networks. The light wave interference characteristic calculation amplifies the contribution of subtle feature gradients, and the update mechanism of reflectivity and refractive index is combined to improve the model's ability to capture complex multidimensional features;

[0067] (4) In some preferred implementations, a multi-head attention mechanism is used to capture the global semantic information in long texts, and a cross-attention mechanism is used to integrate contextual relationships during the decoding process, thereby effectively improving the semantic coherence and logical consistency problems in long text generation. Layer normalization and residual connection mechanisms are used to enhance the stability and convergence performance of the network, thereby solving the instability problem of the model training process in long text generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0069] Figure 1 A flowchart of a long-text collaborative generation method based on master-slave model collaboration provided by the present invention;

[0070] Figure 2 An architectural diagram of a text generation model in an exemplary embodiment. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and beneficial technical effect of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. It should be understood that the specific implementation methods described in this specification are only for explaining the present invention, not for limiting the present invention.

[0072] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0073] It should be further understood that the term "and / or" used in the present description and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0074] In an embodiment of the present invention, a long text collaborative generation method based on master-slave model collaboration is provided, and the long text collaborative generation is realized through a large model based on the Transformer network.

[0075] like Figure 1 As shown, the long-text collaborative generation method based on the master-slave model collaboration includes steps S10-S50.

[0076] Step S10: Obtain training data through public data sets, and use natural language processing tools (such as SpaCy) to extract summaries, keywords, and paragraph levels, so as to provide structured information for the training data.

[0077] The training data source of the training model needs to contain rich hierarchical information, such as the relationship between topics, paragraphs, and details. In this embodiment, the data comes from a public dataset, including:

[0078] CNN / DailyMail dataset, which is a news corpus, including article bodies and summaries;

[0079] BooksCorpus dataset, a large-scale book corpus used to capture the discourse coherence of long texts;

[0080] The Wikipedia dataset is a long text corpus that provides multi-level structured data.

[0081] Step S20: Layer the long text corpus in the training data, extract the topic layer and paragraph layer, and use them to train the model to generate a topic framework. That is, when processing long text corpus through natural language processing tools, the input text data is processed in a hierarchical manner to solve the problems of unclear topics and unclear structures in the long text generation process.

[0082] The topic layer describes the global theme and paragraph framework of the article, for example:

[0083] {"title":"Applications of Artificial Intelligence","section_titles":["AI in the Medical Field","Autonomous Driving Technology","AI in Education"]}.

[0084] The paragraph level is a detailed description of the paragraph topic, for example:

[0085] {"section_title":"AI in the medical field","paragraphs":["The development of artificial intelligence in the medical field has greatly improved diagnostic efficiency.","AI has been widely used in image analysis, drug development and personalized treatment."]}.

[0086] Step S30: Convert the stratified data into input / output pairs acceptable to the model.

[0087] The input is the global context (such as the article background or title), such as "discussing the multi-domain application of artificial intelligence". The output is a framework at the topic level, such as a list of titles and paragraph topics. An illustrative example is as follows:

[0088] Input: Background information: Discuss the multi-domain applications of artificial intelligence.

[0089] Output: Title: Applications of Artificial Intelligence, Paragraph Topic: [AI in the medical field, autonomous driving technology, AI in education].

[0090] Step S40: construct a text generation model based on the Transformer network, and then import the input / output pairs into the text generation model for training.

[0091] Step S50: Collaboratively generate long texts using the trained text generation model; wherein the output is a generated topic framework, including article titles and paragraph topics.

[0092] For example, the input to the text generation model is: "Background information: discuss the various fields of application of artificial intelligence."

[0093] Furthermore, the order relationship of the input of the text generation model is distinguished by position encoding, for example, represented as: "background information position encoding: P1, P2, ..., PN."

[0094] The output is the generated topic framework, including the article title and paragraph topic, which is expressed as:

[0095] "Title: Application of Artificial Intelligence

[0096] Paragraph topics: [AI in healthcare, autonomous driving technology, AI in education]”

[0097] In an embodiment of the present invention, Figure 2 As shown in the figure, the text generation model is a model based on the Transformer network. The Transformer network includes an encoder and a decoder. The encoder is used to map the input features into a series of context-aware vector representations, and the decoder is used to gradually decode the vector representation into target features.

[0098] The encoder consists of N identical encoding layers, each of which includes two sublayers: a multi-head attention mechanism sublayer and a feedforward network sublayer, and the encoder adds residual connections and normalization operations after each sublayer.

[0099] The decoder includes N identical decoding layers, each of which includes two multi-head attention mechanism sublayers and a feedforward network sublayer, and the decoder adds residual connections and normalization operations after each sublayer. That is, unlike the encoding layer, each decoding layer has an additional multi-head attention mechanism sublayer. Among them, the second multi-head attention mechanism sublayer of the decoding layer is implemented by a multi-head cross-attention mechanism. It should be noted that the multi-head cross-attention mechanism is a mechanism for information exchange between the encoder and the decoder. It takes the output features of the encoder and the input features of the decoder as input, and uses the attention mechanism to perform cross-attention calculations on the input, thereby obtaining a set of weighted decoder output features. Therefore, the multi-head attention mechanism is used to capture the global semantic information in long texts, and the cross-attention mechanism is used to integrate contextual relationships in the decoding process, which effectively improves the semantic coherence and logical consistency problems in long text generation.

[0100] In this embodiment, the multi-head attention mechanism sublayer of the encoder and the first multi-head attention mechanism sublayer of the decoder are both implemented through a multi-head self-attention mechanism; wherein the multi-head self-attention mechanism includes the following steps S61-S63.

[0101] Step S61: Use the Softmax function to process the attention score to obtain the attention weight of each position in the input data sequence. The Softmax function ensures that all weights add up to 1 and can be processed as a probability distribution, which is specifically expressed as:

[0102]

[0103] In the formula, head i is the attention weight of the i-th head, Q i is the query vector of the i-th head, K i is the key vector of the i-th head, is the transpose of the key vector of the i-th head, d ks is the dimension of the input feature, h ks is the number of attention heads, V i is the value vector of the i-th head; Sof( ) is the Softmax function.

[0104] Step S62: Use the obtained attention weights to perform weighted summation on the value vector to obtain an output vector.

[0105] Step S63: Concatenate the output vectors of all heads together, and then generate the output of the final multi-head self-attention mechanism sub-layer through a linear transformation layer.

[0106] It should be noted that the multi-head attention mechanism sublayer is implemented by a multi-head self-attention mechanism, which is the core module of the encoder and is responsible for learning the importance of each position from the input features and combining the information of all positions to obtain a global context representation, thereby improving the performance and generalization ability of the model. Specifically, the multi-head self-attention mechanism is implemented in that the input features pass through three different linear transformation layers to generate "query", "key" and "value" vectors respectively, and the linear transformation is implemented by multiplying the weight matrix. Further, the generated query, key and value vectors are divided into multiple smaller parts, namely "heads", so that the model processes multiple different representation subspaces in parallel, thereby capturing the diversity information in the input data. Further, for each head, the dot product of the query and key vectors is calculated respectively to obtain an attention score, which reflects the relative importance of different positions in the sequence of input data. In order to make the numerical stability of the attention score, the attention score is divided by a scaling factor, which can be the square root of the dimension of the key vector.

[0107] Furthermore, in both the encoder and decoder, the output of each sub-layer is the result of layer normalization and residual connection, which is expressed as:

[0108] y cgy =LN c (x trans +Sublayer(x trans )),

[0109] In the formula, LN c ( ) is the layer normalization function, Sublayer( ) represents the output of the residual connected sublayer, x trans Represents the input of the sublayer; y cgy It is the feature output after layer normalization. The layer normalization and residual connection mechanism are adopted to enhance the stability and convergence performance of the network and solve the instability problem of the model training process in long text generation.

[0110] The feedforward network sublayer consists of two fully connected layers and an activation function, which is used to perform nonlinear transformation on the vector at each position. In this sublayer, the input vector first passes through the first fully connected layer for linear transformation, then passes through a ReLU activation function for nonlinear transformation, and finally passes through the second fully connected layer for linear transformation to improve the expressiveness and generalization ability of the model, so that the model can better handle complex input data.

[0111] It should be noted that the feedforward network of the Transformer network is a neural network structure composed of neurons, which plays a role in enhancing the nonlinear processing ability of the model. The neural network structure includes weights and bias parameters. In order to avoid the gradient vanishing and gradient descent phenomena caused by the traditional neural network using the gradient descent method to update parameters, and to avoid the phenomenon that the parameters fall into the local optimal solution after the training, the present invention adopts a feedforward network algorithm based on photon echo optimization as a feature extraction model during the training process of the feedforward network. During the training process of the feedforward network, the photon echo optimization simulates the propagation, reflection, refraction and time delay behavior of light waves in nonlinear media, and regards each parameter in the feedforward network as a propagation of light waves in the medium. These parameters are adjusted and optimized by simulating the propagation, reflection and refraction process of light waves, so as to solve the problems of gradient vanishing and local optimal traps that are easy to occur in the traditional gradient descent optimization mechanism, and improve the generalization ability and robustness of the algorithm in complex high-dimensional space.

[0112] Specifically, the feedforward network of the feedforward network sublayer is trained, and the specific training process includes steps S41-S46.

[0113] Step S41: Simulate the basic characteristics of light waves, initialize weights and biases, ensure that the initial parameter values ​​have good distribution and periodicity, thereby improving the diversity of initial data feature capture and reducing the risk of the feedforward network model falling into a bad convergence path. and the bias of the jth layer Its initialization method is expressed as:

[0114]

[0115] Where, lay(i) represents the layer index of the i-th layer of the feedforward network; lay(j) represents the layer index of the j-th layer of the feedforward network; A p is the light wave amplitude, which is used to control the initial range of the parameter value; p is the wavelength of the light wave, set to 2π to cover a complete period; φ p is the initial phase; the basic characteristics of the light wave include wavelength and amplitude. Preferably, A p Set to 0.1.

[0116] Step S42: In each iteration, the propagation behavior of light waves in the medium is simulated, and the parameter values ​​of the feedforward network are adjusted. The update amount of the weight and bias of the feedforward network is calculated as follows:

[0117]

[0118] Where η p is the learning rate of the feedforward network; is the weight update amount of the i-th layer of the feedforward network; is the bias update of the jth layer of the feedforward network; L p is the loss function of the feedforward network, used to evaluate the performance of the current model; σ p To simulate environmental noise, it is used to generate random disturbances; is the phase offset of the i-th layer of the feedforward network; is the phase offset of the jth layer of the feedforward network; the medium refers to the parameter space of the feedforward network. Preferably, η p Set to 0.01, σ p Set to 0.03.

[0119] Further, in step S42, the phase offset is calculated according to the light wave interference characteristics and used for parameter update, so that the parameter update can adjust the update direction and amplitude according to the multi-dimensional characteristics of the data, and avoid the gradient information from gradually disappearing due to the complexity of the high-dimensional features. The calculation method is expressed as:

[0120]

[0121] In the formula, is the historical change trend of the i-th layer of the feedforward network; is the current gradient amplitude of the feedforward network; β p is the light wave interference influencing parameter. Preferably, β p Set to 0.2.

[0122] Further, in step S42, the historical change trend of the feedforward network represents the cumulative contribution of parameter update, which is calculated as follows:

[0123]

[0124] In the formula, α p The historical attenuation coefficient; Tmax is the number of iterations of the feedforward network; tp is the current number of iterations. Preferably, α p Set to 0.95.

[0125] Further, in step S42, the calculation method of the current gradient amplitude of the feedforward network is expressed as:

[0126]

[0127] In the formula, is the current gradient magnitude of the feedforward network.

[0128] Step S43: According to the characteristics of the current medium, the reflection and refraction of the light wave are calculated. The reflectivity is mainly used to review the historical features in the parameter adjustment, and the refractive index ensures that the model can quickly capture the deep features in the new input data. The calculation formulas of reflectivity and refractive index are:

[0129]

[0130] In the formula, is the reflectivity parameter of the i-th layer of the feedforward network; is the refractive index parameter of the jth layer of the feedforward network; is the incident refractive index parameter of the medium to characterize the layer input influencing factors of the feedforward network; is the refractive index parameter of the output medium, which is used to characterize the factors affecting the layer output of the feedforward network.

[0131] Further, in step S43, the calculation method of the refractive index parameter of the medium is expressed as:

[0132]

[0133] In the formula, pi is the capacitance, which is related to the activation function of the feedforward network sublayer and is dynamically adjusted according to the activation function type of the feedforward network sublayer, such as the ReLU activation function or the Sigmoid activation function The middle value of the output range;

[0134] is the magnetic permeability, which is related to the parameter change amplitude of the feedforward network, such as the mean value of the parameter change amplitude of the feedforward network.

[0135] Step S44: Update the medium characteristics according to the propagation results of the light wave, adjust the weights and biases of the feedforward network to characterize the adjustment effect of the light wave on the parameters after passing through the medium, and adjust the amplitude and direction of the weights and biases by simulating the update mechanism of the medium characteristics in combination with reflection and refraction to adapt to the complex distribution of data characteristics. The update method is expressed as:

[0136]

[0137] In the formula, is the weight of the i-th layer of the feedforward network after update; is the bias of the jth layer of the updated feedforward network; is the time delay increment of the weight of the i-th layer of the feedforward network; is the time delay increment of the bias of the jth layer of the feedforward network.

[0138] Among them, the role of the time delay increment is to enhance the utilization of historical information, and its calculation method is expressed as:

[0139]

[0140] In the formula, γ p is the delay coefficient, used to smooth the impact of historical updates; is the weight update amount of the i-1th layer of the feedforward network; is the bias update of the j-1th layer of the feedforward network. Preferably, γ p Set to 0.9.

[0141] Step S45: Using the loss function L p Evaluate the matching degree between the current network output and the target, which is used to calculate the current gradient amplitude of the feedforward network, and then adjust the light wave characteristics. The loss function of the feedforward network contains regularization terms and nonlinear penalty terms to increase the complexity and robustness of the model. Its calculation method is expressed as:

[0142]

[0143] Where npe is the number of samples input to the feedforward network in the current batch; is the true label of the kth sample; is the predicted label of the kth sample calculated by the preset Softmax function; pa is the regularization parameter of the feedforward network, which is used to control the model complexity; || || is the L2 norm; ρ p is the nonlinear adjustment parameter of the feedforward network to enhance flexibility; mc is the total number of layers of the feedforward network. Preferably, λ pa Set to 0.2, ρ p Set to 0.1.

[0144] Step S46: Repeat the above steps S42-S45 until the termination condition is met, such as reaching a preset number of iterations or the loss function converges below a preset threshold, which indicates that the feedforward network training of the text generation model is completed.

[0145] In summary, the long-text collaborative generation method based on master-slave model collaboration provided by the present invention has the following beneficial effects:

[0146] (1) By performing hierarchical processing on the input text data, information at the topic level and paragraph level is extracted to describe the global topic framework and specific paragraph content of the article respectively, thereby solving the problems of unclear topics and structures in the long text generation process. In addition, natural language processing tools are used to extract abstracts, keywords and paragraph levels, providing structured information for training data, so that the model can more effectively learn the generation mode of hierarchical long texts;

[0147] (2) By constructing training samples with background information as input and hierarchical topic framework as output, the model's objectives and generation constraints are clarified, solving the task definition problem that traditional generative models have difficulty in generating hierarchical long texts. In addition, in the preferred embodiment, position encoding is used to enable the model to distinguish the sequential relationship of input data, thereby improving the discourse coherence of long text generation;

[0148] (3) In some preferred embodiments, a photon reverberation optimization algorithm is used to simulate the propagation behavior of light waves in nonlinear media, and the weights and bias parameters of the feedforward network are dynamically adjusted to solve the problems of gradient vanishing and local optimality that are prone to occur in the traditional gradient descent method when training deep neural networks. The light wave interference characteristic calculation amplifies the contribution of subtle feature gradients, and the update mechanism of reflectivity and refractive index is combined to improve the model's ability to capture complex multidimensional features;

[0149] (4) In some preferred implementations, a multi-head attention mechanism is used to capture the global semantic information in long texts, and a cross-attention mechanism is used to integrate contextual relationships during the decoding process, thereby effectively improving the semantic coherence and logical consistency problems in long text generation. Layer normalization and residual connection mechanisms are used to enhance the stability and convergence performance of the network, thereby solving the instability problem of the model training process in long text generation.

[0150] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0151] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0152] Those of ordinary skill in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0153] In the embodiments provided by the present invention, it should be understood that the disclosed systems or devices / terminal equipment and methods can be implemented in other ways. For example, the system or device / terminal equipment embodiments described above are only schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the system or unit can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0156] The present invention is not limited to what is described in the specification and implementation modes, and therefore additional advantages and modifications can be easily realized by those skilled in the art. Therefore, without departing from the spirit and scope of the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details, representative devices, and illustrative examples shown and described herein.

Claims

1. A long-text collaborative generation method based on master-slave model collaboration, characterized in that: include: Step S10: obtaining training data through public data sets, and using natural language processing tools to extract summaries, keywords, and paragraph levels, thereby providing structured information for the training data; Step S20: Layer the long text corpus in the training data, extract the topic layer and the paragraph layer, and use them to train the model to generate a topic framework; wherein the topic layer describes the global topic and paragraph framework of the article, and the paragraph layer is a specific description of the paragraph topic; Step S30: converting the stratified data into an input / output pair acceptable to the model, where the input is the global context and the output is the framework of the subject layer; Step S40: constructing a text generation model based on a Transformer network, and then importing the input / output pairs into the text generation model for training; Step S50: Collaboratively generate long texts using the trained text generation model; wherein the output is a generated topic framework, including article titles and paragraph topics.

2. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 1, characterized in that: The text generation model includes an encoder and a decoder; wherein the encoder is used to map input features into a series of context-aware vector representations, and the decoder is used to gradually decode the vector representations into target features; The encoder includes N identical encoding layers, each encoding layer includes two sublayers, a multi-head attention mechanism sublayer and a feedforward network sublayer, and the encoder adds a residual connection and a normalization operation after each sublayer; The decoder includes N identical decoding layers, each decoding layer includes two multi-head attention mechanism sublayers and one feedforward network sublayer, and the decoder adds residual connections and normalization operations after each sublayer; The second multi-head attention mechanism sublayer of the decoding layer is implemented by a multi-head cross-attention mechanism; wherein the multi-head cross-attention mechanism is a mechanism for information exchange between the encoder and the decoder, which takes the output features of the encoder and the input features of the decoder as input, and uses the attention mechanism to perform cross-attention calculations on the input, thereby obtaining a set of weighted decoder output features.

3. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 2, characterized in that: The multi-head attention mechanism sublayer of the encoder and the first multi-head attention mechanism sublayer of the decoder are both implemented by a multi-head self-attention mechanism; wherein the multi-head self-attention mechanism comprises the following steps: Step S61: Use the Softmax function to process the attention score to obtain the attention weight of each position in the input data sequence, and make it processable as a probability distribution, expressed as: In the formula, head i is the attention weight of the i-th head, Q i is the query vector of the i-th head, K i is the key vector of the i-th head, is the transpose of the key vector of the i-th head, d ks is the dimension of the input feature, h ks is the number of attention heads, V i is the value vector of the i-th head; Sof() is the Softmax function; Step S62: Use the obtained attention weights to perform weighted summation on the value vector to obtain an output vector; Step S63: Concatenate the output vectors of all heads together, and then generate the output of the final multi-head self-attention mechanism sub-layer through a linear transformation layer.

4. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 2, characterized in that: In the encoder and the decoder, the output of each sub-layer is the result of layer normalization and residual connection, which is expressed as: y cgy =LN c (x trans +Sublayer(x trans )), In the formula, LN c () is the layer normalization function, Sublayer() represents the output of the residual connected sublayer, X trans Represents the input of the sublayer; y cgy is the feature output after layer normalization.

5. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 2, characterized in that: In the step S40, the photon reverberation optimization algorithm is used as a feature extraction model to train the feedforward network of the feedforward network sublayer, and the specific training process includes: Step S41: Simulate the basic characteristics of light waves, initialize weights and biases, ensure that the initial parameter values ​​have good distribution and periodicity, thereby improving the diversity of initial data feature capture and reducing the risk of the feedforward network model falling into a bad convergence path. and the bias of the jth layer Its initialization method is expressed as: Where, lay(i) represents the layer index of the i-th layer of the feedforward network; lay(j) represents the layer index of the j-th layer of the feedforward network; A p is the light wave amplitude, which is used to control the initial range of the parameter value; p is the wavelength of the light wave, set to 2π to cover a complete period; φ p is the initial phase; the basic characteristics of the light wave include wavelength and amplitude; Step S42: In each iteration, the propagation behavior of light waves in the medium is simulated, and the parameter values ​​of the feedforward network are adjusted. The update amount of the weight and bias of the feedforward network is calculated as follows: Where η p is the learning rate of the feedforward network; is the weight update amount of the i-th layer of the feedforward network; is the bias update of the jth layer of the feedforward network; L p is the loss function of the feedforward network, used to evaluate the performance of the current model; σ p To simulate environmental noise, it is used to generate random disturbances; is the phase offset of the i-th layer of the feedforward network; is the phase offset of the jth layer of the feedforward network; the medium refers to the parameter space of the feedforward network; Step S43: According to the characteristics of the current medium, the reflection and refraction of the light wave are calculated. The reflectivity is mainly used to review the historical features in the parameter adjustment, and the refractive index ensures that the model can quickly capture the deep features in the new input data. The calculation formulas of reflectivity and refractive index are: In the formula, is the reflectivity parameter of the i-th layer of the feedforward network; is the refractive index parameter of the jth layer of the feedforward network; is the incident refractive index parameter of the medium to characterize the layer input influencing factors of the feedforward network; is the refractive index parameter of the output of the medium, to characterize the factors affecting the layer output of the feedforward network; Step S44: Update the medium characteristics according to the propagation results of the light wave, adjust the weights and biases of the feedforward network to characterize the adjustment effect of the light wave on the parameters after passing through the medium, and adjust the amplitude and direction of the weights and biases by simulating the update mechanism of the medium characteristics in combination with reflection and refraction to adapt to the complex distribution of data characteristics. The update method is expressed as: In the formula, is the updated weight of the i-th layer of the feedforward network; is the bias of the jth layer of the updated feedforward network; is the time delay increment of the weight of the i-th layer of the feedforward network; is the time delay increment of the bias of the jth layer of the feedforward network; Among them, the role of the time delay increment is to enhance the utilization of historical information, and its calculation method is expressed as: In the formula, γ p is the delay coefficient, used to smooth the impact of historical updates; is the weight update amount of the i-1th layer of the feedforward network; is the bias update of the j-1th layer of the feedforward network; Step S45: Using the loss function L p Evaluate the matching degree between the current network output and the target, which is used to calculate the current gradient amplitude of the feedforward network, and then adjust the light wave characteristics. The loss function of the feedforward network contains regularization terms and nonlinear penalty terms to increase the complexity and robustness of the model. Its calculation method is expressed as: Where npe is the number of samples input to the feedforward network in the current batch; is the true label of the kth sample; is the predicted label of the kth sample calculated by the preset Softmax function; pa is the regularization parameter of the feedforward network, which is used to control the complexity of the model; |||| is the L2 norm; ρ p is the nonlinear adjustment parameter of the feedforward network to enhance flexibility; mc is the total number of layers of the feedforward network; Step S46: Repeat the above steps S42-S45 until a preset number of iterations is reached or the loss function converges below a preset threshold, which indicates that the feedforward network training of the text generation model is completed.

6. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 5, characterized in that: In step S42, the phase offset is calculated according to the light wave interference characteristics and used for parameter update, so that the parameter update can adjust the update direction and amplitude according to the multi-dimensional characteristics of the data, and avoid the gradient information from gradually disappearing due to the complexity of the high-dimensional features. The calculation method is expressed as: In the formula, is the historical change trend of the i-th layer of the feedforward network; is the current gradient amplitude of the feedforward network; β p is the parameter affecting light wave interference.

7. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 6, characterized in that: In step S42, the historical change trend of the feedforward network represents the cumulative contribution of parameter update, which is calculated as follows: In the formula, α p The historical attenuation coefficient; Tmax is the number of iterations of the feedforward network; tp is the current number of iterations.

8. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 6, characterized in that: In step S42, the calculation method of the current gradient amplitude of the feedforward network is expressed as: In the formula, is the current gradient magnitude of the feedforward network.

9. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 5, characterized in that: In step S43, the calculation method of the refractive index parameter of the medium is expressed as: In the formula, p i is the permittivity, which is dynamically adjusted according to the activation function type of the feedforward network sublayer; is the magnetic permeability, which is related to the amplitude of parameter changes of the feedforward network.

10. The long-text collaborative generation method based on master-slave model collaboration as claimed in claim 1, characterized in that: In the step S50, the sequential relationship of the input of the text generation model is distinguished by position encoding.

Images