Sound effect generation method and system

Abstract

The application provides an audio effect generation method and system, which comprises the following steps: obtaining target text data; inputting the target text data into an audio effect description generation fine-tuning model to obtain audio effect voice data corresponding to target audio effect content description information in the target text data output by the audio effect description generation fine-tuning model; wherein the audio effect description generation fine-tuning model is obtained by sequentially training a target generative large language model based on a first training data set and a second training data set; and a training sample proportion coefficient of an audio effect content description label in the first training data set is smaller than a training sample proportion coefficient of the audio effect content description label in the second training data set. The application makes the audio effect generation result more accurate.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a method and system for generating sound effects. Background Technology

[0002] In recent years, audiobooks, as a reading method, have experienced booming market growth. Within this trend, sound effects, as an indispensable element of audiobooks, play a crucial role, greatly enhancing the listener's immersion and emotional resonance.

[0003] In the process of producing audiobooks, it is often necessary to read the novel text manually first, and then make artistic processing based on the text. This not only increases the production cost, but also places high demands on the creator's ability to associate sound effects and control the timing of adding sound effects.

[0004] Existing sound effect description prediction techniques primarily employ the BERT (Bidirectional Encoder Representation from Transformers) framework for unsupervised pre-training on non-precision-calibrated domain datasets. The pre-trained model is then fine-tuned on a small number of precision-calibrated datasets for downstream tasks (i.e., sound effect text prediction). While this improves sound effect generation efficiency to some extent, it suffers from low accuracy in sound effect prediction. Therefore, a sound effect generation method and system are urgently needed to address these issues. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention provides a sound effect generation method and system.

[0006] This invention provides a sound effect generation method, comprising:

[0007] Obtain the target text data;

[0008] The target text data is input into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model;

[0009] The sound effect description generation fine-tuning model is obtained by training the target generative large language model sequentially on the first training dataset and the second training dataset; the proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0010] According to a sound effect generation method provided by the present invention, the sound effect description generation fine-tuning model is obtained through the following steps:

[0011] Obtain sample text data, wherein the sample text data includes sample text data with sound effects and sample text data without sound effects;

[0012] The first training sample is obtained by labeling the sample text data containing sound effects with the corresponding sound effect content description tags.

[0013] Text data without sound effects is labeled with descriptive tags to obtain the second training sample;

[0014] Based on the number of the first training samples and the number of the second training samples, the first training dataset and the second training dataset are constructed respectively;

[0015] Based on the first training dataset, the target generative large language model is trained to obtain a sound effect description generation base model;

[0016] Based on the second training dataset, the sound effect description generation base model is trained to obtain the sound effect description generation fine-tuning model.

[0017] According to a sound effect generation method provided by the present invention, the step of constructing a first training dataset and a second training dataset based on the number of the first training samples and the number of the second training samples respectively includes:

[0018] Determine the total number of the first training sample and the second training sample respectively;

[0019] The first training dataset is constructed based on all the first training samples and all the second training samples;

[0020] The second training dataset is constructed based on all the first training samples and a preset number of second training samples; wherein the total number of the second training samples is greater than the preset number.

[0021] According to a sound effect generation method provided by the present invention, the step of obtaining sample text data includes:

[0022] Based on the text genre type information and text content element information, construct the first prompt word for generating the text title;

[0023] Based on a pre-trained generative large language model, a corresponding text genre title is generated according to the first prompt word;

[0024] Based on the text genre title, construct a second prompt word for generating text content;

[0025] Based on the pre-trained generative large language model, corresponding enhanced text data is generated according to the second prompt word;

[0026] The sample text data is obtained based on the real text data and the enhanced text data.

[0027] According to a sound effect generation method provided by the present invention, the step of labeling sample text data with sound effect content with corresponding sound effect content description tags to obtain a first training sample includes:

[0028] Based on the text content of the real text data or the text content of the enhanced text data, construct a corresponding third prompt word for annotating the sound effect content;

[0029] Based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is labeled according to the third prompt word. If it is determined that the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect description content of the target sound effect content that has been labeled with the sound effect content description tag, the first training sample is obtained.

[0030] According to a sound effect generation method provided by the present invention, the method further includes:

[0031] The first training sample or the second training sample is segmented according to sentences to obtain text segments corresponding to the first training sample or the second training sample respectively, wherein the number of sentences in the text segment is an integer;

[0032] The step of constructing the first training dataset and the second training dataset according to the number of the first training samples and the second training samples respectively includes:

[0033] The first training dataset and the second training dataset are constructed based on the number of text segments corresponding to the first training sample and the second training sample, respectively.

[0034] The present invention also provides a sound effect generation system, comprising:

[0035] The text data acquisition module is used to acquire target text data;

[0036] The sound effect generation module is used to input the target text data into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model.

[0037] The sound effect description generation fine-tuning model is obtained by training the target generative large language model sequentially on the first training dataset and the second training dataset; the proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the sound effect generation method as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the sound effect generation method as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the sound effect generation method as described above.

[0041] The sound effect generation method and system provided by this invention trains a target generative large language model with a training sample set determined by the proportion coefficient of training samples corresponding to the sound effect content description tags to obtain a sound effect description generation fine-tuning model. Then, the obtained target text data is input into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data, making the sound effect generation result more accurate. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a schematic diagram of the prediction process of a masked language model in the prior art;

[0044] Figure 2 A schematic diagram illustrating the prediction process of existing next-sentence prediction techniques.

[0045] Figure 3 This is a schematic diagram of fine-tuning training of pre-trained BERT in existing technology;

[0046] Figure 4 A flowchart illustrating the sound effect generation method provided by this invention;

[0047] Figure 5 A schematic diagram illustrating the model training process provided by this invention;

[0048] Figure 6 A schematic diagram illustrating the novel data enhancement process provided by this invention;

[0049] Figure 7 This is a schematic diagram illustrating the process of acquiring sound effect annotation data provided by the present invention;

[0050] Figure 8 This is a schematic diagram of the sound effect generation system provided by the present invention;

[0051] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0053] Audiobooks, as an increasingly popular reading format, have experienced rapid market growth in recent years. Among the various elements of audiobooks, sound effects are crucial for enhancing listeners' immersion and emotional experience. Appropriate sound effects can strengthen the story's atmosphere, improve the emotional delivery of the narrative, and thus make the audiobook content more vivid and engaging.

[0054] Although Natural Language Processing (NLP) technology has been widely applied in text analysis, most research focuses on the mapping between audio descriptions and audio effects, known as Text-to-Audio (TTA). This mapping is often limited to commonly used sound effects in reality, neglecting the numerous complex and unrealistic sound effects that may exist in literary works. Examples include descriptions of "dragon roars," "a wooden bed spinning and dancing," and "shockwaves emitted while flying on a sword" in fantasy and fairy tales. Furthermore, the sound effects generated by TTA methods still have significant uncertainties, making them difficult to use directly in the production of high-quality audiobooks. Therefore, manual audio processing based on sound effect descriptions is still necessary.

[0055] Limited by the quality and quantity of existing sound effect description data, most studies can only make simple predictions for a fixed number of sound effect types. The richness of the predicted sound effects is related to the size of the sound effect library, and it cannot predict sound effects outside the sound effect library, which greatly limits the application of sound effect prediction products. To address this problem, existing technologies employ sound effect description prediction to assist in the creation of audiobooks. This approach first performs unsupervised pre-training on non-precision-calibrated domain datasets based on the BERT framework, and then fine-tunes the pre-trained model on a small number of precision-calibrated datasets for sound effect text prediction tasks.

[0056] BERT is a language representation model with bidirectional textual context. Its multi-layered self-attention mechanism and feedforward neural network enable it to easily capture long-distance contextual dependencies within text. It has powerful semantic understanding and text generation capabilities and is widely used in various NLP tasks, such as sentiment analysis and text classification.

[0057] During the BERT pre-training process on non-standard data, BERT pre-training includes two key tasks: masked language model (MLM) and next sentence prediction (NSP). Figure 1 This is a schematic diagram illustrating the prediction process of a masked language model in the prior art. (For reference only.) Figure 1 As shown, the MLM task learns rich linguistic representations by randomly masking parts of the words in the input sentence and having the model predict these masked words based on the context. Figure 2 This is a schematic diagram illustrating the prediction process of existing next-sentence prediction techniques. (For reference only.) Figure 2 As shown, the NSP task asks the model to determine whether two given sentences are consecutive texts in order to understand the relationship between the sentences.

[0058] For the BERT fine-tuning process on the precise labeled data Figure 3 This is a schematic diagram illustrating the fine-tuning training of pre-trained BERT in existing technologies. For details, please refer to [link / reference]. Figure 3 As shown, after BERT is pre-trained on a large amount of non-standard novel data, a pre-trained BERT is obtained, and then the pre-trained BERT is fine-tuned for the novel sound effect description generation task.

[0059] While existing methods combining BERT pre-training and fine-tuning are simple and quick, they still have some drawbacks: First, the number of sound effect types predicted by the model is highly correlated with the size of the sound effect library, and it cannot predict sound effects outside the library; second, BERT does not consider sound effect information during pre-training, and although it is trained on a large amount of novel text, there is still a significant gap between the pre-training task and the downstream novel sound effect prediction task; furthermore, the precise labeled data of novel sound effect texts required for downstream tasks is currently difficult to collect, and it relies heavily on the annotator's ability to associate novel sound effects. This not only limits the size of the sound effect dataset, but may also result in a very high proportion of some common sound effects in the dataset, greatly reducing the richness and usability of the sound effects predicted by the model. For example, if the sound effects library only contains "footsteps" and "knocking," then it cannot predict "rapid footsteps," "creaking door opening," and "dog running." Similarly, in a passage from a novel where the text reads, "He walked to the door, first knocking softly, then gradually increasing the force, and finally banging angrily," BERT can only predict three "knocking" sounds, but ideally it should provide three sound effects: "soft knocking," "knocking," and "angry and rapid knocking." Clearly, modifiers for sound effects and sound effects outside the sound effects library can give audiobook creators more choices and room for imagination; however, the current fixed sound effects database cannot cover all sound effect types.

[0060] To address the problems existing in the prior art, this invention provides a sound effect generation method for adding sound effects and atmospheric voice descriptions to audiobooks during the audiobook creation process. This method utilizes natural language processing technology and deep learning algorithms to provide fitting and creative sound effect description suggestions, thereby enhancing the overall auditory experience of the audiobook. It should be noted that the sound effect generation method of this invention can also be applied to other sound effect generation scenarios, such as game text sound effect generation and virtual narrator sound effect generation.

[0061] Figure 4 This is a flowchart illustrating the sound effect generation method provided by the present invention, as shown below. Figure 4 As shown, the present invention provides a sound effect generation method, including:

[0062] Step 401: Obtain the target text data;

[0063] Step 402: Input the target text data into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model;

[0064] The sound effect description generation fine-tuning model is obtained by training the target generative large language model sequentially on the first training dataset and the second training dataset; the proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0065] In this invention, when creating an audiobook from a target novel, the description of sound effects in the target novel is first obtained, thus obtaining target text data. This target text data contains a detailed description of the sound effects (i.e., target sound effect content description information), such as "a car speeds by on the highway, making a roaring sound and the whooshing sound of tires rubbing against the ground." This description information will serve as the basis for generating sound effect speech data.

[0066] Furthermore, the acquired target text data is input into the sound effect description generation fine-tuning model, which is a specially trained generative large language model (such as the Transformer deep learning model) that has the ability to generate corresponding sound effect speech data from text descriptions.

[0067] In this invention, the sound effect description generation fine-tuning model, upon receiving target text data, parses and processes the target text data, and then generates sound effect speech data corresponding to the target sound effect content description information in the target text data. This sound effect speech data can be simulated car engine noises, tire friction sounds, etc., depending specifically on the description in the target text data.

[0068] Specifically, the sound effect description generation fine-tuning model in this invention is trained sequentially on two training datasets: a first training dataset and a second training dataset. Both datasets contain description labels for sound effects and corresponding training samples, but they differ in the proportion of training samples corresponding to the sound effect description labels. The first training dataset has a smaller proportion of training samples corresponding to the sound effect description labels, containing training samples related to sound effect descriptions as well as other types of training samples. This allows the model to maintain sensitivity to sound effect descriptions while also handling other types of text input. The second training dataset has a larger proportion of training samples corresponding to the sound effect description labels, meaning it contains more training samples related to sound effect descriptions. This allows the model to pay more attention to the details of sound effect descriptions during training, thereby improving the accuracy of the generated sound effect speech data.

[0069] This invention trains a target generative large language model sequentially using a first training dataset and a second training dataset, enabling it to accurately understand and generate audio data corresponding to the audio descriptions in the target text data. Furthermore, by adjusting the proportion of training samples corresponding to the audio content description labels in the training dataset, the model's performance can be further optimized.

[0070] The sound effect generation method provided by this invention trains a target generative large language model with a training sample set determined by the proportion coefficient of training samples corresponding to the sound effect content description tags to obtain a sound effect description generation fine-tuning model. Then, the obtained target text data is input into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data, making the sound effect generation result more accurate.

[0071] Based on the above embodiments, the sound effect description generation fine-tuning model is obtained through the following steps:

[0072] Obtain sample text data, wherein the sample text data includes sample text data with sound effects and sample text data without sound effects;

[0073] The first training sample is obtained by labeling the sample text data containing sound effects with the corresponding sound effect content description tags.

[0074] Text data without sound effects is labeled with descriptive tags to obtain the second training sample;

[0075] Based on the number of the first training samples and the number of the second training samples, the first training dataset and the second training dataset are constructed respectively;

[0076] Based on the first training dataset, the target generative large language model is trained to obtain a sound effect description generation base model;

[0077] Based on the second training dataset, the sound effect description generation base model is trained to obtain the sound effect description generation fine-tuning model.

[0078] In this invention, a large amount of novel text data, including and excluding sound effects, is first acquired as samples for training the model. Sound effects refer to parts of the novel text that can be expressed by sound, such as "dog barking" or "thunder." Novel text data without sound effects refers to ordinary text that does not contain such descriptions.

[0079] For text data containing sound effects, it is necessary to manually or automatically add labels describing the sound effects. For example, if the text mentions "suddenly, a continuous rumble of thunder came from afar," the corresponding sound effect description label might be "continuous thunder." These labels will be used to guide the model in learning how to identify and generate corresponding sound effect descriptions from the text. In this invention, the sample text data containing sound effects is labeled with the corresponding sound effect description labels. The result of this step is the first training sample, i.e., a dataset containing sound effect content and its corresponding labels.

[0080] Text that does not contain sound effects also needs to be labeled, but this time the label is "no sound effects content". This helps the model learn to distinguish which texts do not contain sound effect descriptions, thus avoiding the generation of unnecessary sound effect descriptions when generating text. Labeling text data without sound effects with the "no sound effects content" label results in the second training sample, which is the dataset of texts without sound effects content and their corresponding labels.

[0081] Furthermore, based on the number of the first and second training samples, two independent training datasets with different proportions of training samples corresponding to the sound effect content description labels are constructed. These two datasets will be used to train two aspects of the model: first, the ability to recognize and generate sound effect descriptions (using the first training dataset); and second, to improve the model's generalization ability (using the second training dataset), making its output sound effect speech data more consistent with the sound effect content text description. Specifically, this invention uses the first training dataset to train a target generative large language model, i.e., the pre-training stage. This model will learn to recognize sound effect content from text and attempt to generate corresponding descriptions. The result of this step is a basic sound effect description generation model, which is a model capable of initially completing the sound effect description generation task. Then, the second training dataset is used to further train or fine-tune the basic sound effect description generation model. The purpose of this step is to improve the model's accuracy and applicability. The final model is a fine-tuned sound effect description generation model, which combines the knowledge learned from the two training datasets and can more accurately generate the sound effect speech data corresponding to the sound effect content description. This invention uses a generative large language model to generate sound effect descriptions for novels, transforming the classification task of fixed-class results into a non-fixed generation task. This allows the sound effect descriptions output by the model to be not limited to existing data in the sound effect database, reducing the reliance on high-quality manual annotation. At the same time, it improves the flexibility and richness of the generated results, and has strong adaptability and generalization for predicting sound effect descriptions for novels of multiple genres.

[0082] Based on the above embodiments, the step of constructing the first training dataset and the second training dataset according to the number of the first training samples and the second training samples respectively includes:

[0083] Determine the total number of the first training sample and the second training sample respectively;

[0084] The first training dataset is constructed based on all the first training samples and all the second training samples;

[0085] The second training dataset is constructed based on all the first training samples and a preset number of second training samples; wherein the total number of the second training samples is greater than the preset number.

[0086] In this invention, datasets with different sound effect densities (i.e., the proportion coefficient of training samples corresponding to sound effect content description tags) need to be prepared for the pre-training and fine-tuning stages.

[0087] During the pre-training phase, when training the target generative large language model, a low-density sound effect dataset containing all the data is used. That is, all the collected first and second training samples are used as the first training dataset for model training. In the fine-tuning phase, all samples containing sound effect descriptions (i.e., the first training samples) are first selected, and second training samples without sound effect descriptions are added according to a certain preset number or proportion. This makes the proportion coefficient of training samples corresponding to sound effect content description labels in the second training dataset relatively high, thereby constructing a high-density sound effect fine-tuning dataset.

[0088] Specifically, in this invention, the first training samples refer to text data containing sound effect descriptions, which have been labeled with corresponding sound effect content description tags. To construct the first training dataset, it is first necessary to count the total number of these samples containing sound effect descriptions. This number will determine the size of the first training dataset.

[0089] The second training samples refer to the text data that does not contain sound effect descriptions; these are labeled as "no sound effect content description". Similarly, to construct the second training dataset, it is necessary to count the total number of these samples without sound effect descriptions. This number will serve as a reference when constructing the second training dataset.

[0090] Furthermore, a first training dataset is constructed based on all the first training samples (i.e., text data containing sound effect descriptions and their labels) and all the second training samples (i.e., text data without sound effect descriptions and their labels). This dataset will be used in the pre-training phase to help the model learn how to recognize sound effect content from text and generate corresponding descriptions. Since the full set of training samples is used, this dataset can be considered a low-density sound effect dataset containing all the data, where "low-density" refers to the proportion of sound effect descriptions in the overall data.

[0091] Furthermore, when constructing the second training dataset, the needs of the fine-tuning stage need to be considered. In this invention, the fine-tuning stage requires a high-density sound effect fine-tuning dataset, meaning that sound effect descriptions constitute a high proportion of the dataset. Therefore, in this invention, all samples containing sound effect descriptions (i.e., the first training samples) are first selected, and then second training samples without sound effect descriptions are added in a certain proportion. In this invention, the "preset quantity" refers to the specific number selected from the total number of second training samples for constructing the second training dataset. This quantity can be determined according to the needs of the fine-tuning stage and the experimental design. The aim is to increase the proportion of sound effect descriptions in the dataset while ensuring dataset diversity, thereby constructing a high-density sound effect fine-tuning dataset. It should be noted that not all second training samples are used when constructing the second training dataset. Instead, a portion of the second training samples is selected and mixed with the sound effect description samples in the first training samples according to the preset quantity and selection criteria (such as the proportion of sound effect descriptions, dataset balance, etc.) to jointly constitute the second training dataset. Furthermore, in the second training dataset, the proportion of the first training sample is P%, and the proportion of the second training sample is Q%. The proportion of the first training sample P% is much greater than the proportion of the second training sample Q%. For example, the proportion of the first training sample (with sound effect data) P% is 80%, and the proportion of the second training sample (without sound effect data) Q% is 20%.

[0092] Figure 5 This is a schematic diagram of the model training process provided by the present invention, which can be referred to. Figure 5 As shown, during pre-training, the target generative large language model is used as the base model A. Knowledge transfer from the novel's sound effect domain is performed based on the full set of novel sound effect description data (including real text data and enhanced text data) to obtain a base model B for sound effect description generation. This base model B is then fine-tuned on data with higher sound effect density (e.g., a training dataset where the percentage of data with sound effects is P% greater than the percentage of data without sound effects is Q%). This pre-training combined with fine-tuning method effectively improves the robustness and generalization ability of the fine-tuned sound effect description generation model, while also increasing the proportion of sound effect descriptions in the model output. Furthermore, in the actual inference application stage of the fine-tuned sound effect description generation model, the novel data input to the model is segmented, and the maximum input length for each iteration is controlled to N characters. This reduces the amount of data processed per iteration, optimizes sound effect description extraction, and makes the data processing more efficient.

[0093] This invention utilizes a generative large speech model to learn rich novel domain knowledge and sound effect description methods on a full set of novel sound effect data (with low sound effect density), and then fine-tunes it on a dataset with high sound effect density, so that the model can maintain good generalization ability while adjusting the frequency and density of the generated sound effect descriptions according to real-world needs.

[0094] Based on the above embodiments, the method further includes:

[0095] The first training sample or the second training sample is segmented according to sentences to obtain text segments corresponding to the first training sample or the second training sample respectively, wherein the number of sentences in the text segment is an integer;

[0096] The step of constructing the first training dataset and the second training dataset according to the number of the first training samples and the second training samples respectively includes:

[0097] The first training dataset and the second training dataset are constructed based on the number of text segments corresponding to the first training sample and the second training sample, respectively.

[0098] Because the lengths of novel texts vary and the distribution of sound effects also differs, the collected data needs to be processed before model training. In this invention, the first or second training sample can be segmented before constructing the training dataset.

[0099] Specifically, the novel data is segmented according to a specified word length N. During the segmentation process, it is necessary to ensure that each segmented text fragment (sample) contains an integer number of sentences. If directly segmenting according to the word length N results in incomplete sentence division, the number of sentences in the sample is increased until the number of words in the sample exceeds N and the number of sentences is an integer. For example, if N is 50 and the sentence length set is [29, 10, 25, 49, 20,...], then the first sample might contain three sentences of lengths 29, 10, and 25 (total length 64, exceeding 50 but the number of sentences is an integer). Then, based on the number of obtained text fragments, a first training dataset and a second training dataset are constructed respectively. Since each text fragment is already a complete set of sentences, the constructed datasets are coherent and complete at the sentence level.

[0100] Based on the above embodiments, the step of obtaining sample text data includes:

[0101] Based on the text genre type information and text content element information, construct the first prompt word for generating the text title;

[0102] Based on a pre-trained generative large language model, a corresponding text genre title is generated according to the first prompt word;

[0103] Based on the text genre title, construct a second prompt word for generating text content;

[0104] Based on the pre-trained generative large language model, corresponding enhanced text data is generated according to the second prompt word;

[0105] The sample text data is obtained based on the real text data and the enhanced text data.

[0106] Given the prevalence of dialogue and psychological descriptions in novel texts, coupled with a lack of explicit sound effects, excessive amounts of this type of data are detrimental to model learning and training. Furthermore, due to copyright and quality restrictions, obtaining a large amount of high-quality, compliant novel data presents a significant challenge. Therefore, it is necessary to expand the existing novel dataset. Figure 6 A schematic diagram of the novel data augmentation process provided by the present invention can be referred to. Figure 6 As shown, this invention employs prompt word engineering technology to guide the generation of a large-scale language model for novel creation. Specifically, Prompt1 (the first prompt word) and Prompt2 (the second prompt word) are used to guide the generation of the novel's title and content, respectively. This approach effectively improves the sound effect labeling rate of the collected novel data.

[0107] Specifically, in this invention, the genre of the novel text is first clearly defined, such as martial arts novels, science fiction novels, and historical novels. Next, the key content elements of the novel are analyzed, including themes, main characters, setting, and emotional tone. For example, if the novel is about a young explorer's adventure on a mysterious island, then the content elements might include "adventure," "young explorer," and "mysterious island."

[0108] Based on the above information, a prompt word is constructed to guide the generative large language model to generate a title that matches the novel's genre and content. For example, Prompt1 could be: "Please create a title for an adventure novel about a young explorer searching for treasure on a mysterious island." Furthermore, the constructed Prompt1 is input into the pre-trained generative large language model, which generates one or more novel titles based on Prompt1, and then selects the title that best matches the novel's theme and style as the final text title.

[0109] Then, based on the generated novel title, the prompt words are further refined to guide the generative large language model in generating novel content that matches the title. In this invention, based on the text genre title generated in the previous embodiment, a second prompt word is constructed. For example, Prompt2 could be "Based on the title 'The Mystery of the Treasure on the Mysterious Island,' please create a story that includes a young explorer discovering clues, encountering challenges, and ultimately finding the treasure and growing as a person."

[0110] By inputting the second cue word into a pre-trained generative large language model, the model generates one or more enhanced novel texts based on this more detailed cue (i.e., obtains enhanced text data), and evaluates the quality of the generated content to ensure that it conforms to the style of the novel, is logically coherent, and contains all the important content elements.

[0111] Simultaneously, authentic text data is collected, including existing novel texts, author manuscripts, or other relevant text resources. Combining the generated enhanced text data with authentic text data improves the diversity and richness of the collected novel data to a certain extent, thereby increasing the sound effect labeling rate in the subsequent sound effect tagging process.

[0112] Based on the above embodiments, the step of labeling sample text data with sound effects content with corresponding sound effect content description tags to obtain the first training sample includes:

[0113] Based on the text content of the real text data or the text content of the enhanced text data, construct a corresponding third prompt word for annotating the sound effect content;

[0114] Based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is labeled according to the third prompt word. If it is determined that the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect description content of the target sound effect content that has been labeled with the sound effect content description tag, the first training sample is obtained.

[0115] In this invention, it is necessary to analyze the descriptions related to sound effects in the text content of real text data or enhanced text data, specifically in environmental sound effects (such as wind, rain, thunder, etc.), action sound effects (such as fighting sounds, footsteps, door opening sounds, etc.), and special sound effects (such as magic casting sounds, mysterious item activation sounds, etc.).

[0116] Furthermore, based on the analysis of the text content, keywords and features related to sound effects are extracted, and these features will serve as the basis for constructing third-party prompts. Figure 7 This is a schematic diagram illustrating the process of acquiring sound effect annotation data provided by the present invention, which can be referred to. Figure 7 As shown, this invention combines sound effect features with the overall style of a novel to construct a third cue word that can guide a generative large language model to annotate sound effects. For example, for a text describing a battle scene, the third cue word could be: "Please annotate and generate all battle-related sound effects in this text, including the sounds of weapons clashing, characters shouting, and environmental echoes."

[0117] In this invention, the constructed third cue word and the text content to be labeled (real text data or augmented text data) are input into a pre-trained generative large language model. The pre-trained generative large language model automatically generates sound effect annotations based on the third cue word and the text content. These annotations may appear in text form, describing the specific content and features of the sound effect; or they may appear in the form of tags, directly classifying and labeling the sound effect.

[0118] Furthermore, to ensure that the generated sound effect annotations are consistent with the target sound effect content in the text, the annotation results need to be verified. This process can be achieved through a consistency check algorithm within the model. This algorithm compares the annotated content with the actual sound effect described in the text to ensure their matching degree. After obtaining the sound effect annotations, algorithms or preset filtering rules automatically remove those annotations that are unreasonable or do not conform to the text content. These filtering rules can be based on factors such as the reasonableness of the sound effect description, the degree of matching with the text content, and the clarity of the annotation.

[0119] Finally, after screening, reasonable sound effect annotations are combined with corresponding text content to form the first training samples. These training samples will be used for training and learning in subsequent sound effect recognition, sound effect generation, or other related tasks. Additionally, in this invention, for cases where no sound effect content appears in the text content, the pre-trained generative large language model will annotate the text content with a "no sound effect content" descriptive label, thus forming the second training sample.

[0120] This invention uses prompt engineering technology to guide generative large language models to annotate sound effects in novels, and at the same time filters the sound effect annotation results, thereby generating a large amount of reliable sound effect annotation data with high sound effect richness in a short period of time, which is beneficial to the model's subsequent learning and understanding.

[0121] The sound effect generation system provided by the present invention is described below. The sound effect generation system described below can be referred to in correspondence with the sound effect generation method described above.

[0122] Figure 8 This is a schematic diagram of the sound effect generation system provided by the present invention, such as... Figure 8As shown, this invention provides a sound effect generation system, including a text data acquisition module 801 and a sound effect generation module 802. The text data acquisition module 801 is used to acquire target text data; the sound effect generation module 802 is used to input the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data, output by the sound effect description generation fine-tuning model. The sound effect description generation fine-tuning model is obtained by training a target generative large language model sequentially on a first training dataset and a second training dataset. The proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0123] The sound effect generation system provided by this invention trains a target generative large language model with a training sample set determined by the proportion coefficient of training samples corresponding to the sound effect content description tags to obtain a sound effect description generation fine-tuning model. Then, the obtained target text data is input into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data, making the sound effect generation result more accurate.

[0124] Based on the above embodiments, the sound effect generation system further includes a training module. Specifically, the training module is used to: acquire sample text data, wherein the sample text data includes sample text data with sound effect content and sample text data without sound effect content; label the sample text data with sound effect content with corresponding sound effect content description tags to obtain a first training sample; label the text data without sound effect content with no sound effect content description tags to obtain a second training sample; construct a first training dataset and a second training dataset based on the number of the first training samples and the second training samples, respectively; train the target generative large language model based on the first training dataset to obtain a base model for sound effect description generation; and train the base model for sound effect description generation based on the second training dataset to obtain a fine-tuned model for sound effect description generation.

[0125] Based on the above embodiments, the training module is further configured to: determine the total number of the first training samples and the second training samples respectively; construct the first training dataset based on all the first training samples and all the second training samples; construct the second training dataset based on all the first training samples and a preset number of second training samples; wherein the total number of the second training samples is greater than the preset number.

[0126] Based on the above embodiments, the training module is further configured to: construct a first prompt word for generating a text title based on text genre type information and text content element information; generate a corresponding text genre title based on the first prompt word using a pre-trained generative large language model; construct a second prompt word for generating text content based on the text genre title; generate corresponding enhanced text data based on the second prompt word using the pre-trained generative large language model; and obtain the sample text data based on the real text data and the enhanced text data.

[0127] Based on the above embodiments, the training module is further configured to: construct a corresponding third prompt word for labeling sound effect content based on the text content of the real text data or the text content of the enhanced text data; based on the pre-trained generative large language model, label the sound effect content in the text content of the real text data or the text content of the enhanced text data according to the third prompt word; and obtain the first training sample when it is determined that the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect description content between the target sound effect content and the sound effect content description tag that has been labeled.

[0128] Based on the above embodiments, the training module is further configured to: segment the first training sample or the second training sample according to sentences to obtain text segments corresponding to the first training sample or the second training sample respectively, wherein the number of sentences in the text segments is an integer; and construct the first training dataset and the second training dataset respectively according to the number of text segments corresponding to the first training sample and the second training sample respectively.

[0129] The system provided in this embodiment of the invention is used to execute the above-described method embodiments. For specific processes and details, please refer to the above embodiments, which will not be repeated here.

[0130] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9As shown, the electronic device may include: a processor 901, a communications interface 902, a memory 903, and a communication bus 904, wherein the processor 901, the communications interface 902, and the memory 903 communicate with each other via the communication bus 904. The processor 901 can call logical instructions in the memory 903 to execute a sound effect generation method, which includes: acquiring target text data; inputting the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data, output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is obtained by training a target generative large language model sequentially based on a first training dataset and a second training dataset; the proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0131] Furthermore, the logical instructions in the aforementioned memory 903 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0132] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, the computer program comprising program instructions, wherein when the program instructions are executed by a computer, the computer is able to execute the sound effect generation method provided by the above methods, the method comprising: acquiring target text data; inputting the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is obtained by training a target generative large language model sequentially based on a first training dataset and a second training dataset; the proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0133] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the sound effect generation method provided in the above embodiments. The method includes: acquiring target text data; inputting the target text data into a sound effect description generation fine-tuning model to obtain sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; wherein the sound effect description generation fine-tuning model is obtained by sequentially training a target generative large language model based on a first training dataset and a second training dataset; the proportion coefficient of training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of training samples corresponding to the sound effect content description labels in the second training dataset.

[0134] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0135] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0136] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating sound effects, characterized in that, include: Obtain the target text data; The target text data is input into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model; The sound effect description generation fine-tuning model is obtained by training the target generative large language model sequentially on the first training dataset and the second training dataset; the proportion coefficient of the training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of the training samples corresponding to the sound effect content description labels in the second training dataset. The methods obtained by sequentially training the target generative large language model include: Based on the first training dataset, the target generative large language model is trained to obtain a sound effect description generation base model; Based on the second training dataset, the sound effect description generation base model is trained to obtain the sound effect description generation fine-tuning model.

2. The sound effect generation method according to claim 1, characterized in that, The method further includes: Obtain sample text data, wherein the sample text data includes sample text data with sound effects and sample text data without sound effects; The first training sample is obtained by labeling the sample text data containing sound effects with the corresponding sound effect content description tags. Text data without sound effects is labeled with descriptive tags to obtain the second training sample; Based on the number of the first training samples and the number of the second training samples, the first training dataset and the second training dataset are constructed respectively.

3. The sound effect generation method according to claim 2, characterized in that, The step of constructing the first training dataset and the second training dataset according to the number of the first training samples and the number of the second training samples respectively includes: Determine the total number of the first training sample and the second training sample respectively; The first training dataset is constructed based on all the first training samples and all the second training samples; The second training dataset is constructed based on all the first training samples and a preset number of second training samples; wherein the total number of the second training samples is greater than the preset number.

4. The sound effect generation method according to claim 2, characterized in that, The acquisition of sample text data includes: Based on the text genre and text content element information, construct the first prompt word for generating the text title; Based on a pre-trained generative large language model, a corresponding text genre title is generated according to the first prompt word; Based on the text genre title, construct a second prompt word for generating text content; Based on the pre-trained generative large language model, corresponding enhanced text data is generated according to the second prompt word; The sample text data is obtained based on the real text data and the enhanced text data.

5. The sound effect generation method according to claim 4, characterized in that, The first training sample is obtained by labeling the sample text data containing sound effects with corresponding sound effect content description tags, including: Based on the text content of the real text data or the text content of the enhanced text data, construct a corresponding third prompt word for annotating the sound effect content; Based on the pre-trained generative large language model, the sound effect content in the text content of the real text data or the text content of the enhanced text data is labeled according to the third prompt word. If it is determined that the target sound effect content in the text content of the real text data or the text content of the enhanced text data is consistent with the sound effect description content of the target sound effect content that has been labeled with the sound effect content description tag, the first training sample is obtained.

6. The sound effect generation method according to any one of claims 2 to 5, characterized in that, The method further includes: The first training sample or the second training sample is segmented according to sentences to obtain text segments corresponding to the first training sample or the second training sample respectively, wherein the number of sentences in the text segment is an integer; The step of constructing the first training dataset and the second training dataset according to the number of the first training samples and the number of the second training samples respectively includes: The first training dataset and the second training dataset are constructed based on the number of text segments corresponding to the first training sample and the second training sample, respectively.

7. A sound effect generation system, characterized in that, include: The text data acquisition module is used to acquire target text data; The sound effect generation module is used to input the target text data into the sound effect description generation fine-tuning model to obtain the sound effect speech data corresponding to the target sound effect content description information in the target text data output by the sound effect description generation fine-tuning model. The sound effect description generation fine-tuning model is obtained by training the target generative large language model sequentially on the first training dataset and the second training dataset; the proportion coefficient of the training samples corresponding to the sound effect content description labels in the first training dataset is less than the proportion coefficient of the training samples corresponding to the sound effect content description labels in the second training dataset. The system is also used for: Based on the first training dataset, the target generative large language model is trained to obtain a sound effect description generation base model; Based on the second training dataset, the sound effect description generation base model is trained to obtain the sound effect description generation fine-tuning model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the sound effect generation method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the sound effect generation method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the sound effect generation method as described in any one of claims 1 to 6.

Images