Story generation method and device, model, and model generation method and compression method

Abstract

The present application relates to the technical field of natural language processing, and discloses a story generation method and device, a model, and a generation method and compression method of the model.The story generation model comprises an embedding layer, a user preference prefix layer, a GRU layer, a low-layer Transformer decoder, and a high-layer Transformer decoder, the GRU layer uses a first hidden layer representation output by the low-layer Transformer decoder and a user preference prefix of the user preference prefix layer as input, generates a second hidden layer representation by using a gated recurrent unit, and obtains a story text based on the second hidden layer representation by the high-layer Transformer decoder; and a generation method and compression method of the model, a story generation method and device are also provided.The present application effectively improves the coherence, rationality, emotional color and individuality of the story, and the compression method uses a knowledge distillation method to realize model compression, which facilitates efficient deployment of the model on edge devices.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to story generation methods, apparatus, models, and methods for generating and compressing models. Background Technology

[0002] In recent years, pre-trained language models (PLMs) based on the Transformer architecture have performed exceptionally well in various applications, especially autoregressive story generation models such as GPT-2, which have demonstrated outstanding performance in generative tasks. However, in the open-ended task of story generation, the extremely limited input information presents some challenges when using existing story generation models to generate stories.

[0003] First, in story generation, it is extremely difficult for machine learning models to understand the semantic and logical structure of a story. Due to the lack of a global semantic and logical structure, stories generated by existing models often suffer from repetition, logical inconsistencies, and a lack of long-term coherence. These problems can lead to a decline in story quality and negatively impact the reader's experience.

[0004] Secondly, existing story generation models often rely solely on massive amounts of training data to generate stories, resulting in a lack of personalization and emotional depth. Furthermore, due to data imbalance, these models tend to generate stories with a set of generic plots, further hindering the development of personalized and emotionally resonant narratives. Consequently, these stories often fail to evoke resonance or interest from readers. Summary of the Invention

[0005] This invention provides a story generation method, apparatus, model, and a method for generating and compressing the model, which solves the technical problems of stories generated based on existing story generation models having defects such as repetition, logical conflict, lack of long-distance coherence, and lack of personalization and emotional color.

[0006] The first aspect of this invention provides a story generation model, comprising:

[0007] An embedding layer is used to encode and map the input text into a high-dimensional matrix; the input text includes background information of the story.

[0008] A low-level Transformer decoder is used to decode the high-dimensional matrix to obtain the first hidden layer representation of the corresponding decoding result;

[0009] The user preference prefix layer stores user preference prefixes corresponding to the input text, and the user preference prefixes record user preference information;

[0010] The GRU layer is used to generate a second hidden layer representation using a gated recurrent neural network, taking the first hidden layer representation and the user preference prefix as input.

[0011] A high-level Transformer decoder is used to decode the second hidden layer representation to obtain the story text.

[0012] A second aspect of the present invention provides a method for generating a story generation model, the method comprising:

[0013] Acquire training sample data; the training sample data includes multiple prefaces of the story and the corresponding story text.

[0014] The training sample data is input into the story generation model described above and trained until convergence is obtained, resulting in a well-trained story generation model.

[0015] A third aspect of the present invention provides a method for compressing a story generation model, the method comprising:

[0016] The trained story generation model is compressed using a knowledge distillation method to obtain a compressed story generation model; the trained story generation model is obtained using the story generation model generation method described above.

[0017] According to a third aspect of the present invention, the compression of the trained story generation model using the knowledge distillation method includes:

[0018] Using a trained story generation model as the teacher model, let the total number of layers of the teacher model's Transformer decoder be t, where layers 1 to k belong to the lower-level Transformer decoder of the trained story generation model, and layers k+1 to t belong to the higher-level Transformer decoder of the trained story generation model. Define a layer mapping function to transfer the token-granularity encoding representation and span-granularity encoding representation of layers 1 to k of the teacher model to layers 1 to m of the student model, and transfer the span-granularity and sample-granularity encoding representation of layers k+1 to t of the teacher model to layers m+1 to s of the student model, where s is the total number of layers of the student model, and m is a custom intermediate layer of the student model.

[0019] According to a third aspect of the present invention, the transfer of the token-granularity encoding representation and span-granularity encoding representation of layers 1 to k of the teacher model to layers 1 to m of the student model via the layer mapping function, and the transfer of the span-granularity and sample-granularity encoding representation of layers k+1 to t of the teacher model to layers m+1 to s of the student model, comprises:

[0020] For layers 1 to k of the teacher model, given the token-level and span-level encoding representations of the teacher model. And the token-level and span-level encoding representations of the student model. By minimizing the differences between them, the teacher model teaches the student model:

[0021]

[0022] In the formula, L token To address the differences in token-level encoding representations between teacher and student models at layers 1 to k of the teacher model, L span To address the differences in the span-granular encoding representation between the teacher model and the student model at layers 1 to k of the teacher model, g(·) is the layer mapping function, loss(·) is the MSE loss function, and l represents the layer in which knowledge distillation is being performed.

[0023] For layers k+1 to t of the teacher model, given the span-level and sample-level encoding representation of the teacher model. And the encoding representation of the student model at the span and sample granularities. By minimizing the differences between them, the teacher model teaches the student model:

[0024]

[0025] In the formula, L span ′ represents the difference in token-level encoding representation between the teacher model and the student model at layers k+1 to t of the teacher model. sample This refers to the difference in the encoding representation of the teacher model and the student model at the sample granularity for layers k+1 to t of the teacher model.

[0026] A fourth aspect of the present invention provides a story generation method, the method comprising:

[0027] Obtain the context information of the target story input by the target user;

[0028] The preceding information of the target story is input into the target story generation model to generate the corresponding target story text; the target story generation model is a trained story generation model obtained based on the story generation model generation method described above, or a compressed story generation model obtained based on the compression method of the story generation model described above.

[0029] A fifth aspect of the present invention provides an electronic device comprising:

[0030] A memory for storing instructions; wherein the instructions are used to implement the story generation method as described above, the compression method of the story generation model as described in any of the above methods, and / or the story generation method as described above;

[0031] A processor for executing instructions in the memory.

[0032] A sixth aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the story generation method described above, the compression method of the story generation model as described in any of the above embodiments, and / or the story generation method described above.

[0033] A seventh aspect of the present invention provides a story generation apparatus, comprising:

[0034] The first acquisition module is used to acquire the background information of the target story input by the target user.

[0035] The generation module is used to input the background information of the target story into the target story generation model to generate the corresponding target story text; the target story generation model is a trained story generation model obtained based on the story generation model generation method described above, or a compressed story generation model obtained based on the compression method of the story generation model described above.

[0036] According to one embodiment of the seventh aspect of the present invention, the first acquisition module includes:

[0037] The acquisition unit is used to acquire the voice input from the target user regarding the preceding text of the target story;

[0038] The speech-to-text unit is used to convert the speech input into textual information about the target story.

[0039] According to one embodiment of the seventh aspect of the present invention, the story generation apparatus further includes:

[0040] The second acquisition module is used to acquire the rating value given by the target user to the target story text;

[0041] An update module is used to update the user preference prefix stored in the target story generation model in the generation module according to the score value.

[0042] According to one embodiment of the seventh aspect of the present invention, the story generation apparatus further includes:

[0043] The output module is used to output the target story text.

[0044] As can be seen from the above technical solutions, the present invention has the following advantages:

[0045] The story generation model of this invention includes an embedding layer, a low-level Transformer decoder, a user preference prefix layer, a GRU layer, and a high-level Transformer decoder. The GRU layer takes the first hidden layer representation output by the low-level Transformer decoder and the user preference prefix stored in the user preference prefix layer as input, and uses a gated recurrent neural network to generate a second hidden layer representation. Finally, the high-level Transformer decoder decodes the second hidden layer representation to obtain the story text. This invention also provides a story generation method, apparatus, and compression and generation method for the model based on this model. The compression method uses multi-granularity hierarchical knowledge distillation technology, achieving model compression through layer-by-layer abstraction and information fusion. This invention employs a story generation model based on autoregression and a preference prefix matrix to generate personalized storylines, and uses the preference prefix matrix to store user preferences, enhancing the emotional color and personalization of the story, making it easier for users to resonate. By adding a GRU layer to the autoregressive model to capture long-range information of the story, the generated story becomes more coherent and reasonable. The use of multi-granularity hierarchical knowledge distillation technology for model compression facilitates efficient deployment of the story generation model on edge devices, reducing computational resource requirements. Attached Figure Description

[0046] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0047] Figure 1 The diagram shows a structural connection block diagram of a story generation model provided in an optional embodiment of the present invention;

[0048] Figure 2 The diagram shows a structural schematic of a story generation model provided in an optional embodiment of the present invention;

[0049] Figure 3 A flowchart illustrating a method for generating a story generation model, provided as an optional embodiment of the present invention;

[0050] Figure 4 A flowchart illustrating a compression method for a story generation model, provided as an optional embodiment of the present invention;

[0051] Figure 5This is a schematic diagram illustrating the granular encoding representation transfer using knowledge distillation, provided as an optional embodiment of the present invention.

[0052] Figure 6 A flowchart of a story generation method provided in an optional embodiment of the present invention;

[0053] Figure 7 A hardware interaction diagram of a story generation device provided in an optional embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of a user preference prefix update process provided in an optional embodiment of the present invention.

[0055] Figure label:

[0056] 1-Embedding layer; 2-Low-level Transformer decoder; 3-User preference prefix layer; 4-GRU layer; 5-High-level Transformer decoder; 10-First acquisition module; 20-Generation module; 30-Output module; 40-Second acquisition module; 50-Update module. Detailed Implementation

[0057] The present invention provides a story generation method, apparatus, model, and a method for generating and compressing the model, which are used to solve the technical problems of stories generated based on existing story generation models having defects such as repetition, logical conflict, lack of long-distance coherence, and lack of personalization and emotional color.

[0058] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0059] Please see Figure 1 , Figure 2 , Figure 1 The diagram shows a structural connection block diagram of a story generation method provided by an embodiment of the present invention. Figure 2 The diagram shows a structural schematic of a story generation method provided by an embodiment of the present invention.

[0060] An embodiment of the present invention provides a story generation model, comprising:

[0061] Embedding layer 1 is used to encode and map the input text into a high-dimensional matrix; the input text includes background information of the story.

[0062] The low-level Transformer decoder 2 is used to decode the high-dimensional matrix to obtain the first hidden layer representation of the corresponding decoding result;

[0063] User preference prefix layer 3 stores user preference prefixes corresponding to the input text, and the user preference prefixes record user preference information;

[0064] GRU layer 4 is used to generate a second hidden layer representation using the first hidden layer representation and the user preference prefix as inputs, and a gated recurrent neural network.

[0065] A high-level Transformer decoder 5 is used to decode the second hidden layer representation to obtain the story text.

[0066] As one implementation, the embedding layer 1 encodes and maps the input text into a high-dimensional matrix according to the following formula:

[0067] z i =encode(input text)

[0068] In the formula, `encode` maps the input text encoding to a high-dimensional matrix using a mapping table, and `z`... i This represents the encoding matrix for each character in the input text.

[0069] As one implementation, the low-level Transformer decoder 2 decodes the high-dimensional matrix according to the following formula to obtain the first hidden layer representation of the corresponding decoding result:

[0070]

[0071] In the formula, For autoregressive models (such as GPT2), k is a user-defined number of layers. This represents the first k layers (excluding k) of the autoregressive model, x i The first hidden layer representation of the decoding result of the lower-level Transformer decoder 2, x ＜i This represents the representation of the first i hidden layers.

[0072] As one implementation method, the user preference prefix is ​​with x i A matrix of the same dimensions is used to record user preferences.

[0073] As one implementation, the high-level Transformer decoder 5 decodes the second hidden layer representation to obtain the story text according to the following formula:

[0074]

[0075] In the formula, For autoregressive models (such as GPT2), k is a user-defined number of layers. Token represents the sequence from k to t (where t is the total number of layers in the model) in an autoregressive model. i Let i represent the newly generated i-th text, and softmax be the normalization exponential function.

[0076] As an example, such as Figure 2 As shown, embedding layer 1 maps the story context information and the newly generated text encoding into a high-dimensional matrix z1, z2, z3, z4, z5, z6, z7, z8, z9, which is then decoded by the lower-level Transformer decoder 2 into a first hidden layer representation x1, x2, x3, x4, x5, x6, x7, x8, x9. GRU layer 4 then uses the first hidden layer representation x1, x2, x3, x4, x5, x9 output by the lower-level Transformer decoder 2. Using x1, x2, x3, x4, h5, h6, h7, h8, and user preference prefixes p1, p2 as input, a gated recurrent neural network is used to generate a second hidden layer representation h1, h2, h3, h4, h5, h6, h7, h8, h9. Finally, a high-level Transformer decoder 5 outputs the story text y1, y2, y3, y4, y5, x6, x7, x8, x9 based on this second hidden layer representation h1, h2, h3, h4, h5, h6, h7, h8, h9.

[0077] Existing story generation models suffer from repetitive stories, logical inconsistencies, and a lack of long-range coherence. Furthermore, they fail to effectively preserve and process user preferences, resulting in stories that lack personalization and emotional depth, making it difficult to resonate with users. To improve the coherence and logic of the generated stories, this invention incorporates a GRU layer in the intermediate layer of the autoregressive model to capture the story's semantics, logical structure, and long-range dependencies, thereby enhancing the coherence and logic of the generated stories. By using the first hidden layer representation and the user preference prefix as input, a gated recurrent neural network generates a second hidden layer representation. This allows the story generation model to learn user preference information and contextual information to autoregressively generate storylines that better align with user preferences, thus increasing the emotional depth and personalization of the stories and making it easier for users to connect with them.

[0078] The present invention also provides a method for generating a story generation model.

[0079] Please see Figure 3 , Figure 3 A flowchart of a story generation model generation method provided by an embodiment of the present invention is shown.

[0080] The method for generating a story generation model provided in this embodiment of the invention includes:

[0081] Step S1: Obtain training sample data; the training sample data includes multiple story prefaces and corresponding story texts;

[0082] Step S2: Input the training sample data into the story generation model as described above for training until convergence, and obtain the trained story generation model.

[0083] The present invention also provides a method for compressing a story generation model.

[0084] Please see Figure 4 , Figure 4 A flowchart of a compression method for a story generation model provided by an embodiment of the present invention is shown.

[0085] The compression method described in this embodiment includes:

[0086] Step S10: The trained story generation model is compressed using the knowledge distillation method to obtain the compressed story generation model; the trained story generation model is obtained using the story generation model generation method described above.

[0087] In knowledge distillation, the model is divided into a teacher model and a student model. The teacher model is a powerful autoregressive story generation model with high accuracy and rich knowledge representation capabilities. Compared to the teacher model, the student model is a relatively smaller and simpler model, specifically reflected in the fewer layers of the Transformer decoder. It learns from the teacher model and replicates its performance as much as possible. The knowledge distillation method uses the larger model that needs to be compressed as the teacher model, transferring the knowledge from the teacher model to a smaller student model. This knowledge transfer is achieved by minimizing the kl-divergence between the prediction distributions of the teacher and student models.

[0088] In recent years, PLMs based on Transformer encoder or decoder architectures have performed exceptionally well in various applications, achieving remarkable success in a wide range of natural language understanding and generation tasks. Despite their powerful performance, these PLMs typically suffer from very high computational and memory costs, hindering their deployment on resource-scarce edge devices. In this embodiment of the invention, a lightweight model is obtained by compressing the trained story generation model using knowledge distillation, facilitating efficient deployment on edge devices and reducing computational resource requirements.

[0089] In one feasible approach, the compression of the trained story generation model using knowledge distillation includes:

[0090] Using a trained story generation model as the teacher model, let the total number of layers of the Transformer decoder of the teacher model be t, where layers 1 to k belong to the lower-level Transformer decoder 2 of the trained story generation model, and layers k+1 to t belong to the higher-level Transformer decoder 5 of the trained story generation model. Define a layer mapping function to transfer the token-granularity encoding representation and span-granularity encoding representation of layers 1 to k of the teacher model to layers 1 to m of the student model, and transfer the span-granularity and sample-granularity encoding representation of layers k+1 to t of the teacher model to layers m+1 to s of the student model, where s is the total number of layers of the student model, and m is the intermediate layer of the custom student model.

[0091] The corresponding schematic diagram of granular encoding representation transfer using the knowledge distillation method is shown below. Figure 5 As shown.

[0092] In this embodiment, the token-level, span-level, and sample-level encodings of the corresponding sentences are obtained respectively.

[0093] As a specific implementation method, when obtaining the multi-granularity encoded representation, the following is specifically executed:

[0094] Regarding token granularity:

[0095] Divide the input text into a continuous sequence T = [t1, t2, ..., tn] characters by character. n The story generation model described above directly yields the token-granularity encoding representation H of layer l. l ;

[0096] For span granularity:

[0097] CoreNLP's parsing tools are used to extract noun phrases (NP), verb phrases (VP), and prepositional phrases (PP) from the input text, and these phrases are then concatenated into a sequence S. span =[s1,s1,...,s n ],in, s i Let n represent the i-th phrase in the sequence, j represent the j-th element of sequence T, and n represent the n-th phrase in the sequence T. i s i The token length is determined, and then a span-level encoding representation is constructed based on the token-level encoding representation and average pooling.

[0098]

[0099] In the formula, pool is the average pooling function. Let be the matrix representation of the i-th span in the l-th layer;

[0100] By concatenating the encoded representations of each span granularity of the input text, the encoded representation of each layer of span granularity can be obtained.

[0101] For sample granularity:

[0102] For the entire input text, a sample-level encoded representation is constructed based on token-level encoded representation and average pooling:

[0103]

[0104] In the formula, pool is the average pooling function. This represents the sample granularity encoding of the l-th layer.

[0105] Existing knowledge distillation methods only employ representations of single-granularity language units (i.e., token-level or sample-level), neglecting other granularities and failing to consider the different functionalities of different Transformer layers in PLMs, adopting a uniform teaching approach for each layer. In reality, lower layers of PLMs focus more on capturing low-granularity features (such as token-level and span-level), while higher layers focus more on capturing high-granularity features (sample-level). The embodiments of this invention improve upon existing knowledge distillation methods by incorporating not only single-granularity language unit representations but also multi-granularity language unit representations, thereby better capturing the multi-level features of PLMs and improving knowledge distillation performance. These improvements reduce the computational and memory costs of story generation models while maintaining high performance, improving the deployment efficiency and application scope of story generation models.

[0106] In one feasible manner, the transfer of the token-granularity and span-granularity encoding representations of layers 1 to k of the teacher model to layers 1 to m of the student model via the layer mapping function, and the transfer of the span-granularity and sample-granularity encoding representations of layers k+1 to t of the teacher model to layers m+1 to s of the student model, includes:

[0107] For layers 1 to k of the teacher model, given the token-level and span-level encoding representations of the teacher model. And the token-level and span-level encoding representations of the student model. By minimizing the differences between them, the teacher model teaches the student model:

[0108]

[0109] In the formula, Ltoken To address the differences in token-level encoding representations between teacher and student models at layers 1 to k of the teacher model, L span To address the differences in the span-granular encoding representation between the teacher model and the student model at layers 1 to k of the teacher model, g(·) is the layer mapping function, loss(·) is the MSE loss function, and l represents the layer in which knowledge distillation is being performed.

[0110] For layers k+1 to t of the teacher model, given the span-level and sample-level encoding representation of the teacher model. And the encoding representation of the student model at the span and sample granularities. By minimizing the differences between them, the teacher model teaches the student model:

[0111]

[0112] In the formula, L span ′ represents the difference in token-level encoding representation between the teacher model and the student model at layers k+1 to t of the teacher model. sample This refers to the difference in the encoding representation of the teacher model and the student model at the sample granularity for layers k+1 to t of the teacher model.

[0113] The goal of the entire knowledge distillation training is to minimize the following loss function:

[0114] Loss=λ1L token +λ2L span +λ3L span ′+λ4L sampIe

[0115] In the formula, λ1, λ2, λ3 and λ4 are definable weight parameters.

[0116] In this embodiment of the invention, to address the issue of different divisions of labor among different Transformer layers, a teaching method for different layers is proposed, enabling student models to better learn the multi-level features of PLMs and further improving the distillation effect.

[0117] The present invention also provides a story generation method.

[0118] Please see Figure 6 , Figure 6 A flowchart of a story generation method provided by an embodiment of the present invention is shown.

[0119] The story generation method provided in this embodiment of the invention includes:

[0120] Step S100: Obtain the background information of the target story input by the target user;

[0121] Step S200: Input the target story context information into the target story generation model to generate the corresponding target story text; the target story generation model is a trained story generation model obtained based on the story generation model generation method described above, or a compressed story generation model obtained based on the compression method of the story generation model described above.

[0122] The present invention also provides an electronic device, comprising:

[0123] A memory for storing instructions; wherein the instructions are used to implement the story generation method as described above, the compression method of the story generation model as described in any of the above methods, and / or the story generation method as described above;

[0124] A processor for executing instructions in the memory.

[0125] The present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the story generation method described above, the compression method of the story generation model as described in any of the above-described ways, and / or the story generation method described above.

[0126] The present invention also provides a story generation device.

[0127] Please see Figure 7 , Figure 7 The diagram shows a structural connection block diagram of a story generation device provided in an embodiment of the present invention.

[0128] The story generation apparatus provided in this embodiment of the invention includes:

[0129] The first acquisition module 10 is used to acquire the background information of the target story input by the target user.

[0130] The generation module 20 is used to input the background information of the target story into the target story generation model to generate the corresponding target story text; the target story generation model is a trained story generation model obtained based on the story generation model generation method described above, or a compressed story generation model obtained based on the compression method of the story generation model described above.

[0131] As a specific implementation method, the Raspberry Pi can perform the functions of the generation module 20.

[0132] In one feasible implementation, the first acquisition module 10 includes:

[0133] The acquisition unit is used to acquire the voice input from the target user regarding the preceding text of the target story;

[0134] The speech-to-text unit is used to convert the speech input into textual information about the target story.

[0135] In this embodiment, users can input the preceding information of the story through simple verbal commands, thereby allowing the story generation device to generate the following story, effectively improving the user experience.

[0136] In one feasible manner, the story generation device further includes:

[0137] Output module 30 is used to output the target story text.

[0138] In one implementation, the output module 30 outputs data in the form of voice playback, specifically by converting the target story text into voice for playback.

[0139] In one feasible manner, the story generation device further includes:

[0140] The second acquisition module 40 is used to acquire the rating value given by the target user to the target story text;

[0141] The update module 50 is used to update the user preference prefix stored in the target story generation model in the generation module 20 according to the score value.

[0142] In one implementation, the second acquisition module 40 can acquire the target user's rating of the target story text via key input. After the story generation device generates a story, the user can rate the story according to their preferences and experience, and input the corresponding rating value through the keys of the second acquisition module 40 to express their emotions and feedback.

[0143] As one implementation method, the ratings given by target users to the target story text are collected and stored, and then uploaded to the update module 50 when there is an internet connection. This updates the user preference prefix, helping the story generation device to better understand the user's tastes and needs, and further improve the quality and personalization of the generated story.

[0144] Using voice input of the story's background information and voice playback of the target story text as examples, a detailed diagram illustrating the user preference prefix update process can be seen as follows: Figure 8 As shown.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed story generation method, apparatus, model generation method, and compression method can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0146] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0147] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0148] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0149] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A story generation method, characterized in that, This method is applied to obtain a story generation model, which includes: An embedding layer is used to encode and map the input text into a high-dimensional matrix; the input text includes background information of the story. A low-level Transformer decoder is used to decode the high-dimensional matrix to obtain the first hidden layer representation of the corresponding decoding result; The user preference prefix layer stores user preference prefixes corresponding to the input text, and the user preference prefixes record user preference information; The GRU layer is used to generate a second hidden layer representation using a gated recurrent neural network, taking the first hidden layer representation and the user preference prefix as input. A high-level Transformer decoder is used to decode the second hidden layer representation to obtain the story text.

2. A method for generating a story generation model, characterized in that, The method includes: Acquire training sample data; the training sample data includes multiple prefaces of the story and the corresponding story text. The training sample data is input into the story generation model in the story generation method as described in claim 1 and trained until convergence is obtained to obtain the trained story generation model.

3. A compression method for a story generation model, characterized in that, The method includes: The trained story generation model is compressed using a knowledge distillation method to obtain a compressed story generation model; the trained story generation model is obtained using the story generation model generation method described in claim 2.

4. The compression method for the story generation model according to claim 3, characterized in that, The compression of the trained story generation model using knowledge distillation includes: Using a trained story generation model as the teacher model, let the total number of layers of the teacher model's Transformer decoder be t, where layers 1 to k belong to the lower-level Transformer decoder of the trained story generation model, and layers k+1 to t belong to the higher-level Transformer decoder of the trained story generation model. Define a layer mapping function to transfer the token-granularity encoding representation and span-granularity encoding representation of layers 1 to k of the teacher model to layers 1 to m of the student model, and transfer the span-granularity and sample-granularity encoding representation of layers k+1 to t of the teacher model to layers m+1 to s of the student model, where s is the total number of layers of the student model, and m is a custom intermediate layer of the student model.

5. The compression method for the story generation model according to claim 4, characterized in that, The step of transferring the token-level and span-level encoding representations of layers 1 to k of the teacher model to layers 1 to m of the student model via the layer mapping function, and transferring the span-level and sample-level encoding representations of layers k+1 to t of the teacher model to layers m+1 to s of the student model, includes: For layers 1 to k of the teacher model, given the token-level and span-level encoding representations of the teacher model. And the token-level and span-level encoding representations of the student model. The teacher model teaches the student model by minimizing the differences between them: In the formula, To address the differences in token-level encoding representation between teacher and student models at layers 1 to k of the teacher model, To address the differences in span-level encoding representations between teacher and student models at layers 1 to k of the teacher model, For layer mapping functions, Represents the MSE loss function. This indicates the layer where knowledge distillation is currently being performed; For layers k+1 to t of the teacher model, given the span-level and sample-level encoding representation of the teacher model. And the encoding representation of the student model at the span and sample granularities. The teacher model teaches the student model by minimizing the differences between them: In the formula, To address the differences in token-level encoding representation between the teacher and student models at layers k+1 to t of the teacher model, This refers to the difference in the encoding representation of the teacher model and the student model at the sample granularity for layers k+1 to t of the teacher model.

6. A story generation method, characterized in that, The method includes: Obtain the context information of the target story input by the target user; The preceding information of the target story is input into the target story generation model to generate the corresponding target story text; the target story generation model is a trained story generation model obtained based on the generation method of the story generation model as described in claim 2, or a compressed story generation model obtained based on the compression method of the story generation model as described in any one of claims 3-5.

7. An electronic device, characterized in that, include: A memory for storing instructions; wherein the instructions are used to implement the method for generating the story generation model as described in claim 2, the method for compressing the story generation model as described in any one of claims 3-5, and / or the method for generating the story as described in claim 6; A processor for executing instructions in the memory.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method for generating a story generation model as described in claim 2, the method for compressing a story generation model as described in any one of claims 3-5, and / or the method for generating a story as described in claim 6.

9. A story generation device, characterized in that, include: The first acquisition module is used to acquire the background information of the target story input by the target user. The generation module is used to input the context information of the target story into the target story generation model to generate the corresponding target story text; The target story generation model is a trained story generation model obtained based on the story generation model generation method as described in claim 2, or a compressed story generation model obtained based on the compression method of the story generation model as described in any one of claims 3-5.

10. The story generation apparatus according to claim 9, characterized in that, The first acquisition module includes: The acquisition unit is used to acquire the voice input from the target user regarding the preceding text of the target story; The speech-to-text unit is used to convert the speech input into textual information about the target story.

11. The story generation apparatus according to claim 9, characterized in that, The story generation device also includes: The second acquisition module is used to acquire the rating value given by the target user to the target story text; An update module is used to update the user preference prefix stored in the target story generation model in the generation module according to the score value.

12. The story generation apparatus according to claim 9, characterized in that, The story generation device also includes: The output module is used to output the target story text.

Images