Sound effect generation method and system

Abstract

The invention provides a sound effect generation method and system. The method comprises the steps of obtaining target text data; inputting the target text data into a sound effect description generation fine tuning model to obtain sound effect voice data corresponding to target sound effect content description information in the target text data output by the sound effect description generation fine tuning model; wherein the sound effect description generation fine tuning model is obtained by training a target generation type large language model in sequence based on a first training data set and a second training data set; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is smaller than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set. According to the invention, the sound effect generation result is more accurate.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a sound effect generation method and system. Background Art

[0002] In recent years, the market for audio novels as a form of reading has shown a booming growth trend. In this trend, sound effects, as an indispensable element in audio novels, play a vital role, greatly enhancing the audience's immersion and emotional resonance.

[0003] In the process of audiobook production, it is often necessary for humans to read the novel text first and then perform artistic processing based on the text. This not only increases the production cost, but also places high demands on the creator's ability to associate sound effects and control the timing of adding sound effects.

[0004] The existing sound effect description prediction technology mainly uses the BERT (Bidirectional Encoder Representation from Transformers) framework to perform unsupervised pre-training on non-precisely labeled domain datasets, and then uses the pre-trained model to fine-tune downstream tasks (i.e., sound effect text prediction) on a small amount of precisely labeled datasets. Although this improves the efficiency of sound effect generation to a certain extent, it still has the problem of low accuracy in sound effect prediction. Therefore, there is an urgent need for a sound effect generation method and system to solve the above problems. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a sound effect generating method and system.

[0006] The present invention provides a sound effect generation method, comprising: Get target text data; Inputting the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; Among them, the sound effect description generation fine-tuning model is obtained by training the target generative large language model in sequence based on the first training data set and the second training data set; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is smaller than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0007] According to a sound effect generation method provided by the present invention, the sound effect description generation fine-tuning model is trained by the following steps: Acquire sample text data, wherein the sample text data includes sample text data with sound effect content and sample text data without sound effect content; Labeling the sample text data with sound effect content with the corresponding sound effect content description label to obtain a first training sample; Labeling the text data without sound effect content with a label describing the content without sound effect to obtain a second training sample; According to the number of the first training samples and the second training samples, construct the first training data set and the second training data set respectively; According to the first training data set, the target generative large language model is trained to obtain a sound effect description generation base model; The sound effect description generation base model is trained according to the second training data set to obtain the sound effect description generation fine-tuning model.

[0008] According to a sound effect generation method provided by the present invention, the first training data set and the second training data set are constructed according to the number of the first training samples and the second training samples, respectively, including: Determine the total number of the first training samples and the total number of the second training samples respectively; Constructing the first training data set according to all the first training samples and all the second training samples; The second training data set is constructed according to all the first training samples and a preset number of the second training samples; wherein the total number of the second training samples is greater than the preset number.

[0009] According to a sound effect generation method provided by the present invention, the step of obtaining sample text data comprises: Constructing a first prompt word for generating a text title according to the text genre type information and the text content element information; Based on the pre-trained generative large language model, generate a corresponding text genre title according to the first prompt word; Constructing a second prompt word for generating text content according to the text genre title; Based on the pre-trained generative large language model, generating corresponding enhanced text data according to the second prompt word; The sample text data is obtained according to the real text data and the enhanced text data.

[0010] According to a sound effect generation method provided by the present invention, the step of labeling sample text data having sound effect content with corresponding sound effect content description labels to obtain a first training sample includes: Constructing a corresponding third prompt word for marking sound effect content according to the text content of the real text data or the text content of the enhanced text data; Based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is annotated according to the third prompt word, and the first training sample is obtained when it is determined that the sound effect description content of the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect content description tag with which the target sound effect content has been annotated.

[0011] According to a sound effect generation method provided by the present invention, the method further includes: The first training sample or the second training sample is segmented according to sentences to obtain text segments corresponding to the first training sample or the second training sample, respectively, wherein the number of sentences in the text segment is an integer; The step of constructing the first training data set and the second training data set according to the number of the first training samples and the number of the second training samples comprises: The first training data set and the second training data set are constructed respectively according to the number of the text segments corresponding to the first training sample and the second training sample.

[0012] The present invention also provides a sound effect generation system, comprising: A text data acquisition module is used to obtain target text data; A sound effect generation module, used for inputting the target text data into a sound effect description generation fine-tuning model, and obtaining sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; Among them, the sound effect description generation fine-tuning model is obtained by training the target generative large language model in sequence based on the first training data set and the second training data set; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is smaller than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned sound effect generation methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the sound effect generation method as described above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the sound effect generating method described above is implemented.

[0016] The sound effect generation method and system provided by the present invention train a target generative large language model through a training sample set determined by a training sample ratio coefficient corresponding to a sound effect content description label, thereby obtaining a sound effect description generation fine-tuning model, and then inputting the acquired target text data into the sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data, thereby making the sound effect generation result more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of the prediction process of a masked language model in the prior art; Figure 2 A schematic diagram of the prediction process of the next sentence prediction technology of the prior art; Figure 3 A schematic diagram of fine-tuning training of BERT pre-trained in the prior art; Figure 4 A schematic diagram of a flow chart of the sound effect generation method provided by the present invention; Figure 5 A schematic diagram of the model training process provided by the present invention; Figure 6 A schematic diagram of the novel data enhancement process provided by the present invention; Figure 7 A schematic diagram of the process of obtaining the sound effect annotation data provided by the present invention; Figure 8 A schematic diagram of the structure of the sound effect generation system provided by the present invention; Fig. 9 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] As an increasingly popular form of reading, audiobooks have seen rapid market growth in recent years. Among them, sound effects, as an important component of audiobooks, are crucial to enhancing the audience's sense of immersion and emotional experience. Appropriate sound effects can enhance the atmosphere of the story and improve the emotional communication of the narrative, making the content of the audiobook more vivid and attractive.

[0021] Although natural language processing (NLP) technology has been widely used in the field of text analysis, most studies focus more on the mapping between sound effect description text and sound effect audio, namely Text To Audio (TTA). This type of mapping is often limited to the commonly used sound effect sets in reality, but ignores a large number of non-realistic complex sound effects that may exist in literary works, such as "roaring dragons", "spinning and dancing wooden beds" and "shock waves emitted by flying swords" in fantasy and fairy tale novels. In addition, the sound effects generated by the TTA method still have great uncertainty, and it is difficult to use them directly in the production process of some high-quality audiobooks with high requirements. Therefore, it is still necessary to manually perform audio-assisted processing based on the sound effect description.

[0022] Limited by the quality and quantity of existing sound effect description data, most studies can only perform simple predictions of a fixed number of sound effect types. The richness of the predicted sound effects is related to the size of the sound effect library, and it is impossible to predict sound effects outside the sound effect library, which greatly limits the application of sound effect prediction products. To address this problem, sound effect description prediction technology is currently used to assist in the creation of audio novels. This technical solution first performs unsupervised pre-training on non-precisely labeled domain datasets based on the BERT framework, and then uses the pre-trained model to fine-tune the sound effect text prediction task on a small number of precisely labeled datasets.

[0023] BERT is a language representation model with a bidirectional text context. Its multi-layer self-attention mechanism and feedforward neural network enable it to easily capture long-distance contextual dependencies within the text. It has powerful semantic understanding and text generation capabilities and is widely used in various NLP tasks, such as sentiment analysis and text classification.

[0024] In the BERT pre-training process on non-precise data, BERT pre-training includes two key tasks: masked language model (MLM) and next sentence prediction (NSP). Figure 1 This is a schematic diagram of the prediction process of the masked language model in the prior art, which can be referenced Figure 1As shown in Figure 2, the MLM task randomly masks some words in the input sentence and lets the model predict these masked words based on the context, thereby learning rich language representations. Figure 2 This is a schematic diagram of the prediction process of the next sentence prediction technology in the prior art, which can be referred to Figure 2 As shown in the figure, the NSP task requires the model to determine whether the two given sentences are continuous texts, so as to understand the relationship between sentences.

[0025] For the BERT fine-tuning process on the precise standard data, Figure 3 This is a diagram of fine-tuning the pre-trained BERT in the prior art. For details, please refer to Figure 3 As shown in the figure, after BERT is pre-trained on a large amount of non-precisely labeled novel data, a pre-trained BERT is obtained, and then the pre-trained BERT is fine-tuned for the novel sound effect description generation task.

[0026] Although the existing BERT-based pre-training combined with fine-tuning method is simple and fast, it still has some defects: first, the number of sound effect types predicted by the model is highly correlated with the size of the sound effect library, and it is impossible to predict sound effects outside the sound effect library; second, BERT does not take sound effect information into consideration during pre-training. Although it is trained on a large amount of novel texts, there is still a large gap between the pre-training task and the downstream novel sound effect prediction task; and, the currently required novel sound effect text precise labeled data required for downstream tasks is difficult to collect and is highly dependent on the annotator's ability to associate novel sound effects. This not only limits the size of the sound effect dataset, but may also make some common sound effects account for an extremely high proportion in the dataset, greatly reducing the richness and availability of the model's predicted sound effects. For example, if the sound effect library only contains "footsteps" and "knocks on the door", then it cannot predict "rapid footsteps", "squeaky door opening" and "puppy running"; for another example, in the novel "He walked to the door, knocked softly at first, then gradually increased his strength, and finally became furious and knocked wildly", BERT can only predict three "knocks on the door", but ideally it should give three sound effects: "gentle knocks on the door", "knocks on the door" and "angry and rapid knocks on the door". Obviously, the modifiers of the sound effect prefix and the sound effects outside the sound effect set can give audiobook creators more choices and imagination space, however, the current fixed sound effect database cannot cover all types of sound effects.

[0027] In view of the problems existing in the above-mentioned prior art, the present invention provides a sound effect generation method for adding sound effects and voice descriptions of the atmosphere to audio books during the creation of audio novels. The method uses natural language processing technology and deep learning algorithms to provide creative sound effect description suggestions that are appropriate to the novel text, thereby enhancing the overall auditory experience of the audio novel. It should be noted that the sound effect generation method of the present invention can also be applied to other sound effect generation scenarios, such as game text sound effect generation, virtual anchor sound effect generation, etc.

[0028] Figure 4 A schematic diagram of a flow chart of the sound effect generation method provided by the present invention, such as Figure 4 As shown, the present invention provides a sound effect generation method, comprising: Step 401, obtaining target text data; Step 402, inputting the target text data into a sound effect description generation fine-tuning model, and obtaining sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; Among them, the sound effect description generation fine-tuning model is obtained by training the target generative large language model in sequence based on the first training data set and the second training data set; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is smaller than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0029] In the present invention, when creating an audio novel based on a target novel, the description of the sound effect content in the target novel is first obtained to obtain target text data. These target text data contain a detailed description of the sound effect (i.e., the target sound effect content description information), such as "a car speeding on the highway, making a roar and a whizzing sound of tires rubbing against the ground", and these description information will serve as the basis for generating sound effect voice data.

[0030] Furthermore, the acquired target text data is input into the sound effect description generation fine-tuning model. The sound effect description generation fine-tuning model is a specially trained generative large language model (such as a Transformer deep learning model), which has the ability to generate corresponding sound effect voice data from the text description.

[0031] In the present invention, after receiving the target text data, the sound effect description generation fine-tuning model will parse and process the target text data, and then generate sound effect voice data corresponding to the target sound effect content description information in the target text data. These sound effect voice data can be simulated car roar, tire friction sound, etc., depending on the description in the target text data.

[0032] Specifically, the sound effect description generation fine-tuning model in the present invention is trained in sequence based on two training data sets: a first training data set and a second training data set. Both data sets contain description labels for sound effects and corresponding training samples, but they differ in the training sample ratio coefficients corresponding to the sound effect content description labels. Among them, the training sample ratio coefficient corresponding to the sound effect content description labels in the first training data set is relatively small, and it contains training samples related to the sound effect description, as well as other types of training samples, so that the model can process other types of text inputs while maintaining sensitivity to the sound effect description. The training sample ratio coefficient corresponding to the sound effect content description labels in the second training data set is relatively large, which means that it contains more training samples related to the sound effect description, so that the model pays more attention to the details of the sound effect description during the training process, thereby improving the accuracy of generating sound effect speech data.

[0033] The present invention trains the target generative large language model in sequence through the first training data set and the second training data set, and can accurately understand and generate sound effect speech data corresponding to the sound effect description in the target text data. At the same time, by adjusting the training sample ratio coefficient corresponding to the sound effect content description label in the training data set, the performance of the model can be further optimized.

[0034] The sound effect generation method provided by the present invention trains a target generative large language model through a training sample set determined by a training sample ratio coefficient corresponding to a sound effect content description label to obtain a sound effect description generation fine-tuning model, and then inputs the acquired target text data into the sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data, so that the sound effect generation result is more accurate.

[0035] Based on the above embodiment, the sound effect description generation fine-tuning model is trained by the following steps: Acquire sample text data, wherein the sample text data includes sample text data with sound effect content and sample text data without sound effect content; Labeling the sample text data with sound effect content with the corresponding sound effect content description label to obtain a first training sample; Labeling the text data without sound effect content with a label describing the content without sound effect to obtain a second training sample; According to the number of the first training samples and the second training samples, construct the first training data set and the second training data set respectively; According to the first training data set, the target generative large language model is trained to obtain a sound effect description generation base model; The sound effect description generation base model is trained according to the second training data set to obtain the sound effect description generation fine-tuning model.

[0036] In the present invention, a large amount of novel text data with and without sound effects is first obtained as samples for training the model. Sound effects can refer to the parts mentioned in the novel text that can be expressed by sound, such as "dog barking" and "thunder". Novel text data without sound effects refers to ordinary texts that do not contain such descriptions.

[0037] For text data containing sound effect content, it is necessary to manually or automatically add a label describing the sound effect. For example, if the text mentions "suddenly, there was a continuous rumbling of thunder in the distance", the corresponding sound effect content description label may be "continuous thunder". These labels will be used to guide the model to learn how to identify and generate corresponding sound effect descriptions from the text. In the present invention, the sample text data with sound effect content is annotated with the corresponding sound effect content description label. The result of this step is the first training sample, that is, a data set containing sound effect content and its corresponding label.

[0038] Text that does not contain sound effects also needs to be labeled, but this time it is labeled with a "no sound effects" label, which helps the model learn to distinguish which texts do not contain sound effect descriptions, thereby avoiding unnecessary sound effect descriptions when generating text. The result of labeling text data without sound effects with a no sound effect description label is the second training sample, which is a dataset of text without sound effects and its corresponding labels.

[0039] Further, according to the number of the first training sample and the second training sample, two independent training data sets with different training sample proportion coefficients corresponding to the sound effect content description label are constructed respectively. The two data sets will be used for two aspects of the training model respectively: one is the ability to identify and generate sound effect descriptions (using the first training data set), and the other is to improve the generalization ability of the model (using the second training data set) so that the sound effect voice data output by it is more consistent with the text description of the sound effect content. Specifically, the present invention uses the first training data set to train a target generative large language model, that is, the pre-training stage. This model will learn to identify the sound effect content from the text and try to generate the corresponding description. The result of this step is a sound effect description generation base model, which is a model that can preliminarily complete the sound effect description generation task. Then, the second training data set is used to further train or fine-tune the sound effect description generation base model. The purpose of this step is to improve the accuracy and applicability of the model. The final model is a sound effect description generation fine-tuning model, which combines the knowledge learned from the two training data sets and can more accurately generate the sound effect voice data corresponding to the sound effect content description. The present invention uses a generative large language model to generate novel sound effect descriptions, converting the classification task of fixed-class results into a non-fixed generation task, so that the sound effect descriptions output by the model are not limited to the existing data in the sound effect database, reducing the dependence on high-quality manual annotations, while improving the flexibility and richness of the generated results, and having strong adaptability and generalization for the prediction of novel sound effect descriptions in multiple genres.

[0040] On the basis of the above embodiment, the step of constructing the first training data set and the second training data set according to the number of the first training samples and the second training samples, respectively, includes: Determine the total number of the first training samples and the total number of the second training samples respectively; Constructing the first training data set according to all the first training samples and all the second training samples; The second training data set is constructed according to all the first training samples and a preset number of the second training samples; wherein the total number of the second training samples is greater than the preset number.

[0041] In the present invention, it is necessary to prepare data sets with different sound effect densities (i.e., the training sample ratio coefficients corresponding to the sound effect content description labels) for the pre-training stage and the fine-tuning stage.

[0042] When training the target generative large language model in the pre-training stage, a low-density sound effect dataset containing the full amount of data is used, that is, all the collected first training samples and second training samples are used as the first training dataset for model training; in the fine-tuning stage, all samples containing sound effect descriptions (i.e., first training samples) are first screened out, and second training samples without sound effect descriptions are added according to a certain preset number or proportion, so that the proportion of training samples corresponding to the sound effect content description labels in the formed second training dataset is relatively high, thereby constructing a high-density sound effect fine-tuning dataset.

[0043] Specifically, in the present invention, the first training samples refer to text data containing sound effect descriptions, which have been annotated with corresponding sound effect content description tags. In order to construct the first training data set, it is first necessary to count the total number of samples containing sound effect descriptions. This number will determine the size of the first training data set.

[0044] The second training samples refer to those text data that do not contain sound effect descriptions. They are marked as no sound effect content description labels. Similarly, in order to construct the second training data set, it is necessary to count the total number of these samples that do not contain sound effect descriptions. This number will serve as a reference when constructing the second training data set.

[0045] Furthermore, a first training dataset is constructed based on all first training samples (i.e., text data containing sound effect descriptions and their labels) and all second training samples (i.e., text data not containing sound effect descriptions and their labels). This dataset will be used in the pre-training phase to help the model learn how to identify sound effect content from text and generate corresponding descriptions. Since a full amount of training samples are used, this dataset can be considered a low-density sound effect dataset that contains a full amount of data, where "low density" refers to the proportion of sound effect descriptions in the overall data.

[0046] Furthermore, when constructing the second training data set, the requirements of the fine-tuning stage need to be considered. In the present invention, the fine-tuning stage requires a high-density sound effect fine-tuning data set, that is, the proportion of sound effect descriptions in the data set is high. Therefore, in the present invention, all samples containing sound effect descriptions (i.e., first training samples) are first screened out, and then second training samples without sound effect descriptions are added according to a certain proportion. In the present invention, "preset number" refers to the specific number selected from the total number of second training samples for constructing the data set when constructing the second training data set. This number can be determined according to the requirements of the fine-tuning stage and the experimental design. The purpose is to increase the proportion of sound effect descriptions in the data set while ensuring the diversity of the data set, thereby constructing a high-density sound effect fine-tuning data set. It should be noted that when constructing the second training data set, not all second training samples will be used, but according to the preset number and screening conditions (such as the proportion of sound effect descriptions, the balance of the data set, etc.), a part of the second training samples is selected from the second training samples and mixed with the sound effect description samples in the first training samples to form the second training data set together. Moreover, in the second training data set, the first training samples account for P%, and the second training samples account for Q%, and the first training samples account for P% much larger than the second training samples account for Q%. For example, the first training samples (with sound effect data) account for P% 80%, and the second training samples (without sound effect data) account for Q% 20%.

[0047] Figure 5 For a schematic diagram of the model training process provided by the present invention, please refer to Figure 5 As shown, during the pre-training process, the target generative large language model is used as the base model A, and the knowledge transfer in the field of novel sound effects is performed based on the full amount of novel sound effect description data (including real text data and enhanced text data), and the sound effect description generation base model B is obtained. The sound effect description generation base model B is fine-tuned on data with higher sound effect density (for example, a training data set in which the proportion of sound effect data P% is greater than the proportion of data without sound effects Q%). Through the above-mentioned pre-training combined with fine-tuning method, the robustness and generalization ability of the sound effect description generation fine-tuning model can be effectively improved, while increasing the proportion of sound effect descriptions in the model output. At the same time, in the actual reasoning application stage of the sound effect description generation fine-tuning model, the novel data input to the model is cut, and the maximum input length of each time is controlled to N characters, which can reduce the amount of data processed at a single time, optimize the sound effect description extraction, and make the data processing process more efficient.

[0048] The present invention utilizes a generative large speech model to learn rich novel domain knowledge and sound effect description methods on the entire novel sound effect data (with a low sound effect density), and then performs fine-tuning on a data set with a higher sound effect density, so that the model can maintain good generalization ability while adjusting the frequency and density of the generated sound effect description according to actual needs.

[0049] Based on the above embodiment, the method further includes: The first training sample or the second training sample is segmented according to sentences to obtain text segments corresponding to the first training sample or the second training sample, respectively, wherein the number of sentences in the text segment is an integer; The step of constructing the first training data set and the second training data set according to the number of the first training samples and the number of the second training samples comprises: The first training data set and the second training data set are constructed respectively according to the number of the text segments corresponding to the first training sample and the second training sample.

[0050] Since the length of novel texts varies, and the distribution of sound effect descriptions also varies, it is necessary to perform certain data processing on the collected data before model training. In the present invention, before constructing the training data set, the first training sample or the second training sample can also be subjected to text segmentation.

[0051] Specifically, the novel data is cut according to the specified word length N. During the cutting process, it is necessary to ensure that the number of sentences contained in each cut text segment (sample) is an integer. If cutting directly according to the word length N will cause the sentence to be incompletely segmented, the number of sentences in the sample will be increased until the number of text words contained in the sample exceeds N and the number of sentences is an integer. For example, N is 50, and the set of sentence lengths is [29, 10, 25, 49, 20,...], then the first sample may contain three sentences of length 29, 10, and 25 (the total length is 64, which exceeds 50 but the number of sentences is an integer). Then, according to the number of text segments obtained, the first training data set and the second training data set are constructed respectively. Since each text segment is already a complete set of sentences, the constructed data set is coherent and complete at the sentence level.

[0052] Based on the above embodiment, the step of obtaining sample text data includes: Constructing a first prompt word for generating a text title according to the text genre type information and the text content element information; Based on the pre-trained generative large language model, generate a corresponding text genre title according to the first prompt word; Constructing a second prompt word for generating text content according to the text genre title; Based on the pre-trained generative large language model, generating corresponding enhanced text data according to the second prompt word; The sample text data is obtained according to the real text data and the enhanced text data.

[0053] There are a lot of dialogues and descriptions of psychological activities in novel texts, but there is a lack of obvious sound effects. Too much of this kind of data is not conducive to model learning and training. In addition, due to the restrictions of novel copyright and quality issues, it is very challenging to obtain a large amount of high-quality novel data that meets the requirements. Therefore, it is necessary to expand the existing novel dataset. Figure 6 The schematic diagram of the novel data enhancement process provided by the present invention can be referred to Figure 6 As shown, the present invention adopts prompt word engineering technology to guide the generation of a large language model for novel creation. Among them, Prompt1 (i.e., the first prompt word) and Prompt2 (i.e., the second prompt word) are used to guide the generation of novel titles and novel contents respectively. In this way, the sound effect marking rate of the collected novel data can be effectively improved.

[0054] Specifically, in the present invention, the genre type of the novel text is firstly clarified, for example, martial arts novels, science fiction novels, historical novels, etc. Then, the key content elements of the novel are analyzed, including the theme, main characters, background setting, emotional tone, etc. For example, if the novel is about the adventure story of a young explorer on a mysterious island, then the content elements may include "exploration", "young explorer", "mysterious island", etc.

[0055] Based on the above information, a prompt word is constructed to guide the generative large language model to generate a title that matches the genre and content of the novel. For example, Prompt1 can be: "Please write a title for me about an adventure novel about a young explorer searching for treasure on a mysterious island." Further, the constructed Prompt1 is input into the pre-trained generative large language model, and the model will generate one or more novel titles based on Prompt1, and then select the title that best matches the theme and style of the novel from the generated titles as the final text genre title.

[0056] Then, based on the generated novel title, the prompt words are further refined to guide the generative large language model to generate novel content that matches the title. In the present invention, the second prompt words are constructed based on the text genre title generated in the previous embodiment. For example, Prompt2 can be "Based on the title "Mysterious Island Treasure Mystery", please create a story that includes a young explorer discovering clues, encountering challenges, and finally finding the treasure and growing up." By inputting the second prompt word into the pre-trained generative large language model, the model will generate one or several paragraphs of enhanced novel text content (i.e., obtain enhanced text data) based on this more detailed prompt, and perform a quality assessment on the generated content to ensure that it conforms to the style of the novel, is logically coherent, and contains all important content elements.

[0057] At the same time, real text data is collected, including existing novel texts, authors' manuscripts or other relevant text resources. The generated enhanced text data is combined with the real text data to improve the diversity and richness of the collected novel data to a certain extent, and to improve the sound effect labeling rate for the subsequent sound effect labeling process.

[0058] Based on the above embodiment, the step of labeling the sample text data with sound effect content with a corresponding sound effect content description label to obtain a first training sample includes: Constructing a corresponding third prompt word for marking sound effect content according to the text content of the real text data or the text content of the enhanced text data; Based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is annotated according to the third prompt word, and the first training sample is obtained when it is determined that the sound effect description content of the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect content description tag with which the target sound effect content has been annotated.

[0059] In the present invention, it is necessary to analyze the descriptive content related to sound effects in the text content of real text data or enhanced text data, which is specifically reflected in environmental sound effects (such as wind, rain, thunder, etc.), action sound effects (such as fighting sounds, footsteps, door opening sounds, etc.) and special sound effects (such as magic casting sounds, mysterious item activation sounds, etc.).

[0060] Furthermore, based on the analysis of the text content, keywords and features related to the sound effects are extracted, and these features will serve as the basis for constructing the third prompt word. Figure 7 The schematic diagram of the acquisition process of the sound effect annotation data provided by the present invention can be referred to Figure 7 As shown, the present invention combines the sound effect features and the overall style of the novel to construct a third prompt word that can guide the generative large language model to perform sound effect annotation. For example, for a text describing a battle scene, the third prompt word can be: "Please annotate and generate all the sound effects related to the battle in this text for me, including the collision of weapons, the shouting of characters, and the echo of the environment, etc." In the present invention, the constructed third prompt word and the text content to be annotated (real text data or enhanced text data) are input into the pre-trained generative large language model, and the pre-trained generative large language model automatically generates sound effect annotations based on the third prompt word and text content. These annotations may appear in the form of text, describing the specific content and characteristics of the sound effect; or they may appear in the form of tags, directly classifying and annotating the sound effect.

[0061] Furthermore, in order to ensure that the generated sound effect annotations are consistent with the target sound effect content in the text content, the annotation results need to be verified. In this process, this can be achieved through the consistency check algorithm within the model, which can compare the annotation content with the actual description of the sound effect in the text to ensure the match between them. After obtaining the sound effect annotations, those annotations that are unreasonable or inconsistent with the text content are automatically eliminated through algorithms or preset screening rules. These screening rules can be based on factors such as the rationality of the sound effect description, the match with the text content, and the clarity of the annotations.

[0062] Finally, after screening, the reasonable sound effect annotations are combined with the corresponding text content to form the first training samples, which will be used for subsequent training and learning of sound effect recognition, sound effect generation or other related tasks. In addition, in the present invention, if there is no sound effect content in the text content, the pre-trained generative large language model will annotate the text content with a label describing no sound effect content, thereby forming the second training sample.

[0063] The present invention uses prompt engineering technology to guide the generative large language model to annotate novel sound effects, and at the same time screens the sound effect annotation results, thereby generating a large amount of reliable sound effect annotation data with high sound effect richness in a short period of time, which is beneficial to the subsequent learning and understanding of the model.

[0064] The sound effect generation system provided by the present invention is described below. The sound effect generation system described below and the sound effect generation method described above can be referenced to each other.

[0065] Figure 8 A structural diagram of the sound effect generation system provided by the present invention, such as Figure 8As shown, the present invention provides a sound effect generation system, including a text data acquisition module 801 and a sound effect generation module 802, wherein the text data acquisition module 801 is used to obtain target text data; the sound effect generation module 802 is used to input the target text data into a sound effect description generation fine-tuning model, and obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is based on a first training data set and a second training data set, and is obtained by training a target generative large language model in sequence; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is less than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0066] The sound effect generation system provided by the present invention trains a target generative large language model through a training sample set determined by a training sample ratio coefficient corresponding to a sound effect content description label to obtain a sound effect description generation fine-tuning model, and then inputs the acquired target text data into the sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data, thereby making the sound effect generation result more accurate.

[0067] On the basis of the above embodiment, the sound effect generation system also includes a training module, which is specifically used to obtain sample text data, wherein the sample text data includes sample text data with sound effect content and sample text data without sound effect content; annotate the sample text data with sound effect content with the corresponding sound effect content description label to obtain a first training sample; annotate the text data without sound effect content with a no sound effect content description label to obtain a second training sample; construct the first training data set and the second training data set respectively according to the number of the first training samples and the second training samples; train the target generative large language model according to the first training data set to obtain a sound effect description generation base model; train the sound effect description generation base model according to the second training data set to obtain the sound effect description generation fine-tuning model.

[0068] Based on the above embodiment, the training module is also used to determine the total number of the first training samples and the second training samples respectively; construct the first training data set based on all the first training samples and all the second training samples; construct the second training data set based on all the first training samples and a preset number of the second training samples; wherein the total number of the second training samples is greater than the preset number.

[0069] On the basis of the above embodiment, the training module is also used to construct a first prompt word for generating a text title according to text genre type information and text content element information; based on a pre-trained generative large language model, generate a corresponding text genre title according to the first prompt word; based on the text genre title, construct a second prompt word for generating text content; based on the pre-trained generative large language model, generate corresponding enhanced text data according to the second prompt word; and obtain the sample text data according to the real text data and the enhanced text data.

[0070] On the basis of the above embodiment, the training module is also used to construct a corresponding third prompt word for marking the sound effect content according to the text content of the real text data or the text content of the enhanced text data; based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is marked according to the third prompt word, and when it is determined that the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect description content between the sound effect content description label with which the target sound effect content has been marked, the first training sample is obtained.

[0071] Based on the above embodiment, the training module is also used to segment the first training sample or the second training sample according to sentences to obtain text segments corresponding to the first training sample or the second training sample, respectively, wherein the number of sentences in the text segments is an integer; and construct the first training data set and the second training data set respectively according to the number of text segments corresponding to the first training sample and the second training sample.

[0072] The system provided in the embodiment of the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific process and detailed contents, which will not be repeated here.

[0073] Fig. 9 A schematic diagram of the structure of an electronic device provided by the present invention, such as Fig. 9As shown, the electronic device may include: a processor (Processor) 901, a communication interface (Communications Interface) 902, a memory (Memory) 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 complete mutual communication through the communication bus 904. The processor 901 may call the logic instructions in the memory 903 to execute the sound effect generation method, which includes: obtaining target text data; inputting the target text data into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is obtained by training the target generative large language model in sequence based on the first training data set and the second training data set; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is less than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0074] In addition, the logic instructions in the above-mentioned memory 903 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0075] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the sound effect generation method provided by the above-mentioned methods, and the method includes: obtaining target text data; inputting the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is based on a first training data set and a second training data set, and is obtained by training a target generative large language model in sequence; the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is less than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0076] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the sound effect generation method provided by the above-mentioned embodiments, the method comprising: obtaining target text data; inputting the target text data into a sound effect description generation fine-tuning model, and obtaining sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is based on a first training data set and a second training data set, and is obtained by training a target generative large language model in sequence; and the training sample proportion coefficient corresponding to the sound effect content description label in the first training data set is less than the training sample proportion coefficient corresponding to the sound effect content description label in the second training data set.

[0077] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0079] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating sound effects, characterized in that: include: Get target text data; Inputting the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; The sound effect description generation fine-tuning model is obtained by training the target generative large language model in sequence based on the first training data set and the second training data set; The training sample ratio coefficient corresponding to the sound effect content description label in the first training data set is smaller than the training sample ratio coefficient corresponding to the sound effect content description label in the second training data set.

2. The sound effect generation method according to claim 1, characterized in that: The sound effect description generation fine-tuning model is trained by the following steps: Acquire sample text data, wherein the sample text data includes sample text data with sound effect content and sample text data without sound effect content; Labeling the sample text data with sound effect content with the corresponding sound effect content description label to obtain a first training sample; Labeling the text data without sound effect content with a label describing the content without sound effect to obtain a second training sample; According to the number of the first training samples and the second training samples, construct the first training data set and the second training data set respectively; According to the first training data set, the target generative large language model is trained to obtain a sound effect description generation base model; The sound effect description generation base model is trained according to the second training data set to obtain the sound effect description generation fine-tuning model.

3. The sound effect generation method according to claim 2, characterized in that: The step of constructing the first training data set and the second training data set according to the number of the first training samples and the number of the second training samples comprises: Determine the total number of the first training samples and the total number of the second training samples respectively; Constructing the first training data set according to all the first training samples and all the second training samples; The second training data set is constructed according to all the first training samples and a preset number of the second training samples; wherein the total number of the second training samples is greater than the preset number.

4. The sound effect generation method according to claim 2, characterized in that: The obtaining of sample text data includes: Constructing a first prompt word for generating a text title according to the text genre type information and the text content element information; Based on the pre-trained generative large language model, generate a corresponding text genre title according to the first prompt word; Constructing a second prompt word for generating text content according to the text genre title; Based on the pre-trained generative large language model, generating corresponding enhanced text data according to the second prompt word; The sample text data is obtained according to the real text data and the enhanced text data.

5. The sound effect generation method according to claim 4, characterized in that: The step of labeling the sample text data having the sound effect content with the corresponding sound effect content description label to obtain the first training sample includes: Constructing a corresponding third prompt word for marking sound effect content according to the text content of the real text data or the text content of the enhanced text data; Based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is annotated according to the third prompt word, and the first training sample is obtained when it is determined that the sound effect description content of the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect content description tag with which the target sound effect content has been annotated.

6. The sound effect generation method according to any one of claims 2 to 5, characterized in that: The method further comprises: The first training sample or the second training sample is segmented according to sentences to obtain text segments corresponding to the first training sample or the second training sample, respectively, wherein the number of sentences in the text segment is an integer; The step of constructing the first training data set and the second training data set according to the number of the first training samples and the number of the second training samples comprises: The first training data set and the second training data set are constructed respectively according to the number of the text segments corresponding to the first training sample and the second training sample.

7. A sound effect generation system, characterized in that: include: A text data acquisition module is used to obtain target text data; A sound effect generation module, used for inputting the target text data into a sound effect description generation fine-tuning model, and obtaining sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; The sound effect description generation fine-tuning model is obtained by training the target generative large language model in sequence based on the first training data set and the second training data set; The training sample ratio coefficient corresponding to the sound effect content description label in the first training data set is smaller than the training sample ratio coefficient corresponding to the sound effect content description label in the second training data set.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the sound effect generating method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the sound effect generating method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the sound effect generating method according to any one of claims 1 to 6 is implemented.

Images