Audio generation method and device, electronic equipment, storage medium and program product
By querying matching audio elements in the audio database and combining the audio to generate audio, the problems of low audio production efficiency and lack of personalization in the prior art are solved, and highly personalized and accurate audio generation is achieved to meet the unique needs of users.
Patent Information
- Application Number
- CN202411014841.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-26
- Publication Date
- 2024-12-03
AI Technical Summary
In the prior art, audio production methods rely on designer experience and manual operation, resulting in inefficient and lack of personalization and innovation in output audio, making it difficult to meet the customized needs of unique emotional expressions or specific application scenarios.
By receiving audio generation requests, including audio attribute information and audio type tags, we query matching target audio elements in the audio database, and generate target audio based on these information combinations. We use convolutional neural networks, recursive neural networks and other technologies to ensure the generated audio Meet the specific needs and expectations of users.
It realizes highly personalized audio generation, improves the user experience, ensures that the generated audio is exactly matched with user preferences and application scenarios, and can achieve more accurate and effective information communication in various applications.
Smart Images

Figure CN119068850A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to an audio generation method, device, electronic device, storage medium and program product. Background Art
[0002] Well-designed audio can not only enhance the audio-visual experience of multimedia content such as movies and games, but also inject unique emotional colors and atmosphere into the content, making it more expressive and appealing.
[0003] In the prior art, audio production methods mainly rely on the designer's experience and manual operation. Although this method can produce high-quality audio, when faced with a large number of audio production tasks, designers often face the problem of running out of inspiration or falling into clichés in design ideas. Another common method is to select spare audio segments from the material library based on the trigger information input by the user, and then piece these audio segments together to generate the final audio file, but it usually results in the output audio lacking personalization and innovation, and it is difficult to meet the customization requirements for unique emotional expression or specific application scenarios.
[0004] Therefore, there is an urgent need for an automated audio generation method that can meet personalized needs. Summary of the invention
[0005] In view of this, an embodiment of the present invention provides an audio generation method. One or more embodiments of the present invention also relate to an audio generation device, a computing device, a computer-readable storage medium and a computer program product to solve the technical defects existing in the prior art.
[0006] According to a first aspect of an embodiment of the present invention, there is provided an audio generation method, comprising:
[0007] receiving an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, wherein the audio type tag is used to characterize auxiliary information used by the audio;
[0008] Querying an audio database for at least one target audio element corresponding to an audio type tag, wherein the audio database stores audio elements and corresponding reference type tags;
[0009] At least one target audio element is combined based on the audio attribute information to generate a target audio.
[0010] According to a second aspect of an embodiment of the present invention, there is provided an audio generating device, comprising:
[0011] A receiving module is configured to receive an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, and the audio type tag is used to characterize auxiliary information used by the audio;
[0012] A query module is configured to query an audio database for at least one target audio element corresponding to an audio type tag, wherein the audio database stores audio elements and corresponding reference type tags;
[0013] The generating module is configured to generate target audio by combining at least one target audio element based on the audio attribute information.
[0014] According to a third aspect of an embodiment of the present invention, there is provided a computing device, including:
[0015] Memory and processor;
[0016] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer programs / instructions are executed by the processor, the steps of the above-mentioned audio generation method are implemented.
[0017] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores a computer program / instruction, and the computer program / instruction implements the steps of the above-mentioned audio generation method when executed by a processor.
[0018] According to a fifth aspect of an embodiment of the present invention, there is provided a computer program product, comprising a computer program / instruction, which implements the steps of the above-mentioned audio generation method when executed by a processor.
[0019] An embodiment of the present invention implements receiving an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, and the audio type tag is used to characterize auxiliary information used by the audio; querying at least one target audio element corresponding to the audio type tag in an audio database, wherein the audio element and the corresponding reference type tag are correspondingly stored in the audio database; and generating target audio by combining at least one target audio element based on the audio attribute information. By receiving an audio generation request and combining the audio attribute information with the audio type tag, highly personalized audio generation is achieved. By querying the audio database for target audio elements that match the tag, the system can accurately select and combine audio content to ensure that the generated audio meets the user's specific needs and expectations. This personalized generation not only improves the user experience, enabling it to obtain audio that fully matches its preferences and application scenarios, but also enables more accurate and effective information communication in various applications, thereby better meeting the user's actual needs and expected goals. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 is a schematic diagram of an interactive structure of an audio generation system provided by an embodiment of the present invention;
[0021] Figure 2 is a flow chart of an audio generation method provided by an embodiment of the present invention;
[0022] Figure 3 is a processing flow chart of an audio generation method provided by an embodiment of the present invention;
[0023] Figure 4 is a structural schematic diagram of an audio generating device provided by an embodiment of the present invention;
[0024] Figure 5 It is a structural block diagram of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0025] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.
[0026] The terms used in one or more embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present invention. The singular forms of "a", "said" and "the" used in one or more embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.
[0027] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present invention, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0028] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of the present invention are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0029] First, the terms involved in one or more embodiments of the present invention are explained.
[0030] Convolutional Neural Network (CNN): It is an artificial neural network specially designed for processing data with a grid-like structure. CNN has achieved great success in image and video recognition, speech recognition and other fields. Its structure includes input layer, hidden layer and output layer, among which the hidden layer usually includes convolution layer, pooling layer and fully connected layer. The convolution layer extracts the features of the input data through convolution operation, the pooling layer reduces the dimension of the feature map through pooling operation, and the fully connected layer is used to output the final result. Through convolution operation and weight sharing, CNN can effectively identify patterns and features in images, thereby realizing efficient processing and learning of complex data.
[0031] Recurrent Neural Network (RNN): is a neural network model for sequential data. Sequential data includes text, speech signals, etc. The special thing about RNN is that the connections between its neurons are cyclic, which means that the output of the network is affected not only by the current input, but also by the previous state. Due to its internal loop structure, RNN is particularly good at processing and generating sequential data.
[0032] Long Short-Term Memory (LSTM): is a special type of recurrent neural network (RNN) designed to solve the vanishing gradient problem in traditional RNNs. LSTM preserves long-term dependencies by introducing a structure called a "memory cell", which enables it to more effectively learn and memorize long-term patterns in sequence data.
[0033] Genetic algorithm: A heuristic optimization algorithm that searches for the optimal solution or a solution close to the optimal solution in the solution space by simulating processes such as natural selection and genetic mechanisms in biological evolution.
[0034] Digital signal processing technology: refers to a technology and method that uses digital computing means to sample, quantize and process continuous-time signals. Its main purpose is to analyze, synthesize, improve or compress signals to meet specific needs.
[0035] Decision Tree and Random Forest: Decision tree is a supervised learning algorithm for classification and regression. It divides data into different categories or prediction values through a tree structure. Each node represents a feature, the branch represents the possible value of the feature, and the leaf node represents the final classification or regression result. Random forest is an ensemble learning method composed of multiple decision trees. It improves the accuracy and stability of the model by combining the prediction results of multiple decision trees.
[0036] In the process of audio design, designers often have to face a large number of audio production tasks, and are prone to running out of inspiration or falling into clichés. Traditional audio production methods mainly rely on the designer's experience and manual operations, which are inefficient and lack novelty. In the existing technology, there is a solution that selects spare audio segments from the material library according to the trigger information input by the user, pieces together the spare audio segments, and generates audio files. However, this method usually results in the output audio lacking personalization and innovation.
[0037] In order to solve the above problems, in the present invention, an audio generation method is provided. The present invention also relates to a screen generation device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.
[0038] See also Figure 1 , Figure 1 It is a schematic diagram of the interactive structure of an audio generation system provided by an embodiment of the present invention, wherein the audio generation system specifically includes a user end, a server end and an audio database, wherein the user end is used to send an audio generation request to the server end; the server end is used to receive the audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, the audio type tag is used to characterize auxiliary information used for the audio, and at least one target audio element corresponding to the audio type tag is queried in the audio database, wherein the audio elements and the corresponding reference type tags are correspondingly stored in the audio database, and the target audio is generated by combining at least one target audio element based on the audio attribute information, and the target audio is sent to the user end; the audio database is used to query the target audio element based on the audio type tag and return it to the server end.
[0039] Applied to the audio generation system, by querying the audio database for target audio elements that match the tags, the system can accurately select and combine audio content to ensure that the generated audio meets the user's specific needs and expectations. This personalized generation not only improves the user experience, enabling them to obtain audio that fully matches their preferences and application scenarios, but also enables more accurate and effective information communication in various applications, thereby better meeting the user's actual needs and expected goals.
[0040] See also Figure 2 , Figure 2 It is a flowchart of an audio generation method provided by an embodiment of the present invention, which specifically includes the following steps.
[0041] Step 202: Receive an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, and the audio type tag is used to characterize auxiliary information used by the audio.
[0042] The audio attribute information refers to the attributes that describe the audio characteristics, such as pitch, loudness, frequency, waveform envelope, etc. These attributes directly describe the physical characteristics or acoustic characteristics of the audio.
[0043] Audio type labels are auxiliary information used to describe the non-physical characteristics of audio, usually based on usage scenarios, emotional characteristics or other semantic information. These labels help the system understand and classify audio, allowing the system to accurately select and generate audio content that meets specific situations or preferences based on user needs.
[0044] For example, suppose a user wants to generate a cheerful background music. The audio attribute information may include acoustic features such as high pitch, moderate loudness, and relatively continuous frequency; and the audio type label may include labels such as "cheerful" and "background music" that describe emotions and usage scenarios. The system can then combine this information to accurately select appropriate audio elements from the audio database and generate audio content that meets the user's expectations.
[0045] Therefore, the audio attribute information mainly describes the acoustic characteristics of the audio, while the audio type label provides a description of non-attribute features.
[0046] In actual applications, after receiving an audio generation request, the system first parses the audio generation request to obtain the audio attribute information and at least one dimension of the audio type tag. The audio attribute information describes the acoustic characteristics of the audio, such as pitch, loudness, and frequency. This information helps the system understand the physical characteristics of the audio. At the same time, the audio type tag provides auxiliary information about the audio usage scenario, emotional characteristics, or other non-physical characteristics. These tags can include descriptions such as "background music", "cheerful", and "relaxed" to help the system determine the specific characteristics of the audio that the user expects to generate.
[0047] In a specific embodiment of the present invention, for example, a user wants to generate an audio suitable for use as a morning alarm. At this time, the system will receive the following information as an audio generation request: audio attribute information includes: high pitch, so as to clearly stimulate attention; moderate loudness, loud enough to wake up the user, but not too harsh; stable frequency, without obvious drastic changes, to ensure that it is not too annoying; audio type tags include: "morning alarm", indicating that this is an audio used to wake up the user; "urgent": emphasizes that the audio emotional characteristics are more tense.
[0048] Step 204: querying an audio database for at least one target audio element corresponding to an audio type tag, wherein the audio database stores audio elements and corresponding reference type tags.
[0049] The audio database is a special storage device for storing audio content and its corresponding type tags. The reference type tag is a type tag corresponding to an audio element stored in the audio database. One audio element may correspond to multiple reference type tags. It is understandable that the audio type tag in step 202 is a type tag included in the audio generation request, which is not the same as the reference type tag here.
[0050] The target audio element is an audio element in the audio database with at least one reference type label having the same audio type label.
[0051] In practical applications, each audio element in the audio database is accompanied by one or more reference type tags that describe its characteristics, such as usage scenarios, emotions, or audio styles. Based on the audio type tags in the request, the system searches the audio database for audio elements that have at least one reference type tag that matches the audio type tag in the request. Since an audio element can have multiple reference type tags, the system considers the combination of these tags during the matching process. The target audio element that the system is looking for should contain at least one reference type tag that corresponds to the audio type tag in the request to ensure that the selected element is as close to the user's needs as possible. The matching process ensures that the system can accurately select audio elements from the database that meet the user's expectations. These elements not only meet the requirements in terms of acoustic properties, but also meet non-physical requirements such as usage scenarios or emotional characteristics mentioned in the request.
[0052] In a specific embodiment of the present invention, take the example that a game development team is designing a forest scene for an adventure game. This scene is an important area for players to explore, and a background music that can create a sense of mystery and adventure is needed. The audio generation request includes: physical audio features suitable for adventure scenes, such as specific values or value ranges of parameters such as pitch, loudness, and frequency; audio type tags: including "background music", "adventure", and "mystery". The audio generation system searches for audio elements that meet the requirements in the audio database based on the above audio type tags. Each audio element has multiple reference type tags, such as "background music", "forest environment", and "adventure exploration". The system matches several audio elements that meet the conditions in the audio database, such as: a piece of background instrumental music with a medium rhythm and a sense of mystery, an audio clip of natural environment sounds (such as birds singing, rustling leaves), and low-key melodies and harmony sound effects to enhance the depth and sense of exploration of the scene.
[0053] Furthermore, querying at least one target audio element corresponding to an audio type tag includes: for any audio type tag, querying at least one candidate audio element in an audio database based on any audio type tag, wherein at least one candidate audio element corresponds to at least one candidate audio type tag that is the same as any audio type tag; for any candidate audio element, calculating the matching similarity based on the number of candidate audio type tags and the number of audio type tags; and determining the candidate audio element whose matching similarity is greater than a first preset threshold as the target audio element corresponding to the audio type tag.
[0054] The candidate audio element refers to an audio element in the audio database that has at least one reference type label that matches the audio type label. The candidate audio type label is the reference type label that matches the queried audio type label corresponding to the candidate audio element. The matching similarity is an indicator that measures the degree of overlap between the candidate audio type label of the candidate audio element and the queried audio type label, and can be defined as the ratio of the number of candidate audio type labels to the number of audio type labels.
[0055] In actual applications, in an audio database, all audio elements containing this tag are queried as candidate audio elements based on the audio type tag. For each candidate audio element, the number of tags of all its candidate audio type tags is counted, and then divided by the total number of audio type tags to obtain the matching similarity. A preset threshold is set, such as 0.5 (indicating that at least 50% of the audio type tags appear in the reference type tags of the candidate audio elements). Candidate audio elements with matching similarity greater than or equal to this threshold are determined as target audio elements.
[0056] In a specific embodiment of the present invention, taking the audio generation request containing the tag: {A, B, C, D} as an example, there is a candidate audio element X in the audio database, and its reference type tag set is {C, D, E, F}. Search all audio elements containing at least one query tag (A, B, C, D) in the database. Assume that the candidate audio element X is matched, which contains tags C and D. Among the reference type tags of element X, C and D match the query tag, that is, the candidate audio type tag set is {C, D}. Therefore, the number of matched tags is 2. Divide the number of matched candidate audio type tags (2) by the number of audio type tags (4), and the matching similarity is 0.5. The preset threshold is 0.5, and the calculated similarity is also 0.5, which meets or exceeds this threshold. Since the similarity reaches or exceeds the preset threshold, the candidate audio element X is confirmed as the target audio element.
[0057] Based on this, the system can quickly and effectively retrieve target audio elements that match the audio type tags requested by the user from the audio database, providing the necessary material basis for subsequent audio generation or other applications.
[0058] Step 206: Generate target audio by combining at least one target audio element based on the audio attribute information.
[0059] Among them, the audio attribute information exists as a constraint condition in the process of generating the target audio.
[0060] In actual applications, for each target audio element, the system determines whether it matches the audio attribute information in the request based on its acoustic characteristics. If a target audio element is not sufficient to meet all constraints, the system adjusts its attribute information to meet the target audio generation requirements. The selected target audio elements are combined and adjusted to ensure that they are consistent with the audio attribute information in the request in terms of both acoustic and non-physical characteristics. The system may perform audio processing on each audio element, such as mixing, pitch adjustment, or adding effects, to ensure that the final generated audio content meets expectations. Finally, the system will generate and output target audio content that meets the requirements based on the user's request. In this process, audio attribute information plays the role of constraints, helping the system to accurately select and combine audio elements during the audio generation process, thereby achieving personalized and accurate audio generation.
[0061] In a specific embodiment of the present invention, for example, a game developer wants to create an exciting, tense, and epic background music for an ancient battlefield scene to enhance the player's immersion and gaming experience. The audio generation request includes: pitch: a high-pitched and exciting range to enhance the battle atmosphere, and rhythm: a fast and strong range to convey tension. The audio type tag includes emotion: an epic sense of magnificence to inspire the player's heroic spirit. The matched target audio elements include: a war drum sound clip with a strong and fast rhythm; an epic orchestral melody, such as a grand trumpet and deep string music.
[0062] The system combines these selected audio elements to ensure that they meet the audio attribute information constraints in the request: Use war drum beats as the basic rhythm of the background music to create a tense and continuous battle atmosphere. On this basis, overlay epic orchestral melodies to enhance the magnificence and excitement of the music. As needed, fine-tune the volume, pitch, and rhythm of each audio element to ensure that they blend harmoniously and jointly construct background music that meets the needs of the scene. After combination and adjustment, the system generates a passionate, tense, and epic background music. This music not only meets the audio attribute information constraints provided by the game developer, but also effectively enhances the gaming experience of the ancient battlefield scene. Finally, the system outputs the generated background music to the game developer for integration into the game.
[0063] Furthermore, based on the audio attribute information, at least one target audio element is combined to generate target audio, including: inputting the audio attribute information and at least one target audio element into a pre-trained audio generation model to obtain the target audio, wherein the audio attributes are used to constrain the attribute characteristics of the target audio.
[0064] In practical applications, the target audio element and audio attribute information are used as inputs to a pre-trained audio generation model. The pre-trained audio generation model can be a generative adversarial network based on deep learning, an autoencoder, or a variational autoencoder, etc., which is used to generate target audio that meets the requirements. During the model training process, the audio attribute information is used as a constraint to ensure that the generated audio matches the input requirements in terms of acoustic and non-physical properties. The audio generation model generates target audio that meets the requirements based on the input audio attribute information and the target audio element. The model may adjust the interaction between elements or combine different audio features to ensure that the generated audio content can accurately reflect the required acoustic and non-physical properties. Finally, the pre-trained audio generation model generates target audio that meets the input requirements. In this process, the audio attribute information acts as an important constraint on the generation process, ensuring that the generated audio can meet the needs and expectations of users.
[0065] Based on this, the system uses pre-trained audio generation models and input audio attribute information to effectively generate target audio that meets specific requirements, thereby supporting audio customization needs in various application scenarios.
[0066] Furthermore, after generating the target audio by combining at least one audio element based on the audio attribute information, it also includes: responding to the correction instruction of the front-end user, correcting the target audio to obtain the corrected audio; and tuning and training the audio generation model based on the corrected audio and the target audio.
[0067] The modification instructions are specific modification opinions or requirements put forward by the user on the generated target audio. These instructions may include but are not limited to:
[0068] Pitch Adjustment: Users may wish to raise or lower the pitch of the audio to better match a desired feel.
[0069] Tempo change: Users may request to speed up or slow down the tempo of the music to suit specific scene requirements.
[0070] Adding or removing elements: You may want to add the sound of a certain instrument to the audio, or remove certain elements.
[0071] Emotional expression: Users may want the audio to convey stronger or different emotions, such as making the music more exciting or softer.
[0072] Fine-tuning training is the process of further optimizing the audio generation model to improve the model's performance in specific audio generation tasks.
[0073] In actual applications, users audition the generated target audio. If they find something unsatisfactory, they can issue correction instructions through the front-end interface, such as adjusting the volume, changing the timbre of a certain instrument, adding or deleting certain audio elements, etc. The system modifies the corresponding part of the target audio according to the user's correction instructions to generate the corrected audio. This process may involve resynthesis, editing, or applying audio effects. Collect user feedback on the target audio, including the audio attributes requested by the user, the user's specific correction instructions, and the audio files before and after the correction. Model training: Use the comparison of the audio before and after the correction as a supervisory signal to train the audio generation model so that it can learn how to more accurately generate audio that satisfies the user based on the given audio attributes. Evaluate the performance of the model on new data, check whether it is overfitting or underfitting, adjust the model structure or optimize the algorithm as needed, and repeat the training until the model performance reaches the expected standard. Continuously collect user feedback, continuously correct and optimize the audio generation model, and improve the quality of the generated audio and user satisfaction.
[0074] Based on this, the system can continuously improve the audio generation model so that the audio it generates better meets the needs and expectations of users.
[0075] Furthermore, before receiving the audio generation request, it also includes: receiving a database update request, wherein the database update request includes the correspondence between at least one reference type tag and at least one audio element; and updating the audio database based on the correspondence between at least one reference type tag and at least one audio element.
[0076] In actual applications, the system listens to a specific communication interface and waits to receive database update requests from administrators, third-party developers, or automated scripts. Each update request usually contains at least one reference type tag (e.g., timbre, genre, emotion, style, etc.) and a corresponding audio element (e.g., an audio clip, sample, or complete musical work). This information should be clearly defined in the request data packet. The system parses the received update request and extracts the correspondence between the reference type tag and the audio element. This may involve decoding and verifying the request data packet to ensure the integrity and correctness of the data. For each audio element, the system also needs to extract or generate necessary metadata, such as the audio format, duration, sampling rate, etc., as well as other information related to the reference type tag. Before updating the database, the system checks the availability and quality of the audio elements and ensures that there are no duplicate entries. Based on the parsed information, the system adds the new audio element and its related metadata to the database, or updates the information of the existing entry to reflect the new reference type tag and the corresponding relationship. In order to maintain the efficient query capability of the database, the system may need to update the index to ensure that subsequent audio generation requests can quickly locate the relevant audio elements. After the update is complete, the system performs a consistency check on the database to ensure that all updates have been applied correctly without omissions or errors. The system sends a confirmation message to the party that requested the update, indicating that the update has been completed, along with any problems or warnings encountered during the update process.
[0077] Based on this, the database of the audio generation system can be continuously enriched and optimized, thereby improving the quality and diversity of audio generation.
[0078] Furthermore, before receiving a database update request, it also includes: obtaining any audio element in the audio database; performing audio feature analysis on any audio element, determining the classification results of the audio element in each preset classification dimension, and generating a reference type label based on the classification results; generating a database update request based on the reference type label and any audio element.
[0079] In actual applications, an audio file is selected from an external source or internal storage as the object of analysis. This can be audio uploaded by the user, audio resources captured from the web, or new audio clips generated by the system. Audio signal processing and machine learning techniques are used to analyze the characteristics of the audio. These characteristics may include but are not limited to frequency distribution, rhythm, melody, harmonic structure, timbre, intensity changes, rhythm patterns, etc. More complex analysis may involve deep learning models to identify more detailed features such as speakers, instrument types, emotional colors, language types, etc. in the audio. The preset classification dimensions can be music genres, emotional states, languages, instrument types, etc., which have been defined in the audio database design phase. Based on the above extracted audio features, the system classifies the audio elements into corresponding classification dimensions. For example, a song may be classified as "rock", "fast tempo", "English lyrics", etc. Based on the classification results, a series of reference type labels are generated for the audio elements. These labels help to quickly retrieve and classify audio elements in the database, and also provide a basis for subsequent audio generation.
[0080] Package the audio elements and their corresponding reference type tags into a database update request. This request contains detailed information about the audio element to be updated or added, and how it should be classified and labeled in the database. Send the constructed database update request to the database management system to trigger the database update process. After receiving the update request, the database management system performs the corresponding update operation to add the audio element and its reference type tag to the database, or update the existing record. Ensure that the updated database content is accurate and there is no data loss or conflict. Verify whether the audio elements can be correctly retrieved and used, and ensure that the database update does not affect the normal operation of the system.
[0081] Based on this, the audio database can be dynamically updated to ensure that the audio elements contained are rich in diversity and accurately classified, thereby improving the efficiency and quality of the audio generation system.
[0082] Furthermore, the audio type labels include emotion type labels and scene type labels; audio feature analysis is performed on any audio element to determine the classification results of the audio element in each preset classification dimension, including: waveform analysis is performed on any audio element to obtain waveform change characteristics; the waveform change characteristics are input into a pre-trained audio classification model to obtain the classification results of any audio element in each preset classification dimension, wherein the audio classification model is trained based on sample audio elements of each preset classification dimension.
[0083] In practical applications, first, the system reads the audio file and converts it into a digital waveform signal. Signal processing techniques such as Fourier transform and Mel frequency cepstral coefficients are applied to extract key features from the waveform signal. These features can include: changes in energy and intensity, frequency components, rhythm and beats, duration and pitch, clarity and reverberation of the sound, intonation and emotional color of the voice.
[0084] Use a deep learning model that has been trained on a large number of sample audio elements. These models are usually convolutional neural networks, recurrent neural networks, or long short-term memory networks, which can capture the time series dependencies of audio features. The features extracted from the audio waveform are passed as input to the model. The model predicts the category of the audio based on the correlation and pattern between the features. The model outputs a probability distribution under each preset classification dimension, such as the probability of different emotions or scenes. Usually, the classification with the highest probability will be selected as the final classification result. The preset classification dimensions can be varied, for example:
[0085] Emotion type: happy, sad, angry, calm, etc.
[0086] Scene types: indoor, outdoor, urban, natural environment, etc.
[0087] Model training requires a large number of labeled samples, and each sample audio element carries a known emotion type label and scene type label. The model parameters are adjusted through the back propagation algorithm to minimize the error between the predicted results and the actual labels, so that the model can accurately predict the type of unknown audio.
[0088] Based on the classification results of the model, corresponding emotion type labels and scene type labels are generated for the audio elements. These labels are stored in the database together with the audio elements for easy search, classification, and later use by audio synthesis or recommendation systems. As more audio elements are analyzed and classified, the model can be further optimized through online learning or periodic retraining to adapt to new audio types and classification standards.
[0089] Based on this, the audio database can effectively manage and organize various types of audio, making it easier for users to find audio materials that meet specific emotional or scene requirements.
[0090] Furthermore, before receiving the audio generation request, it also includes: receiving an initial type tag and audio attribute information input by a front-end user, and obtaining at least one reference type tag in an audio database; calculating the semantic similarity between the initial type tag and each reference type tag; screening out reference type tags whose semantic similarity is greater than a second preset threshold, and determining them as audio type tags; and generating an audio generation request based on the audio attribute information and the audio type tag.
[0091] There may be an incomplete match between the content input by the front-end user and the reference type tags in the audio database. This is because the user's description may contain non-standardized terms, colloquial expressions, or very specific personalized requirements, while the tags in the database are usually standardized, universal categories. In this case, the content input by the front-end user may not always match the reference type tags in the audio database, so a certain degree of fuzzy matching of the user's needs is required before generating an audio generation request.
[0092] In actual applications, users enter the description of the audio they want on the front-end interface, which may include information such as emotional state, scene setting, music style, and rhythm speed. These descriptions will be converted into initial type tags and specific audio attribute information. The system searches and obtains a series of reference type tags in the audio database. These tags are classification information attached to the audio elements that already exist in the database, which can help the system understand the characteristics of different types of audio. For each pair of initial type tags and reference type tags, the system uses natural language processing technology to calculate the semantic similarity between them. This may involve word embedding models, semantic similarity algorithms (such as cosine similarity), or other deep learning methods to quantify the semantic relationship between two tags. A second preset threshold is set as a screening condition, which is a predefined value used to determine whether two tags are similar enough. Only those reference type tags with a semantic similarity higher than this threshold will be retained. Finally, the system constructs an audio generation request based on the determined audio type tags and the audio attribute information provided by the user. This request contains all the necessary parameters and conditions, such as audio length, volume, timbre, rhythm, emotional color, etc., to guide the subsequent audio synthesis or generation process.
[0093] Based on this, fuzzy matching enables the system to understand the user's intention and find the closest audio type even if the user's description is not exactly consistent with the database label. The system does not have to be limited to strict keyword matching, but can handle freer and more natural language input.
[0094] Furthermore, after generating the target audio by combining at least one audio element based on the audio attribute information, it also includes: analyzing the target audio to determine the combination method of at least one target audio element; and visually displaying the audio attribute information and the combination method of the target audio on the front-end interface.
[0095] In practical applications, the generated target audio is analyzed to determine the relationship and combination of various audio elements, for example:
[0096] Timing: Determine the position and duration of individual audio elements on the timeline.
[0097] Volume & Balance: Set the volume of different elements to ensure overall harmony.
[0098] Transitions and Effects: Add effects like fade in and fade out, echo, reverb, etc. to smooth out transitions between audio.
[0099] On the front-end interface, the system presents the properties and combination of the target audio to the user in a graphical manner, for example:
[0100] Waveform: Shows the overall shape of the audio and the volume changes of each part.
[0101] Timeline Editor: Displays the time position of each audio element, and users can adjust it by dragging.
[0102] Mixing console: Allows the user to adjust the volume of each audio element, equalizer settings, etc.
[0103] Effects Panel: Provides options to preview and apply various audio effects.
[0104] Users can interact with audio directly on the visual interface, for example:
[0105] Drag and drop: Move the position of audio clips, change their order or how they overlap.
[0106] Volume and effect adjustment: Change volume, balance, and effect parameters via sliders or menus.
[0107] Real-time preview: As adjustments are made, the system updates the audio output in real time, allowing users to hear the effects of the changes instantly.
[0108] When the user has completed all necessary adjustments, the system will generate correction instructions based on the user's changes, such as updating the position on the timeline, adjusting the volume and effect parameters, and editing or deleting audio clips. Finally, the target audio is regenerated or adjusted according to the correction instructions until the user is satisfied. This process can be repeated until the ideal audio result is achieved.
[0109] Based on this, it combines audio analysis, user interface design and real-time audio processing technology to provide an intuitive and powerful audio creation platform that enables users to easily create music works that meet their personal needs and preferences.
[0110] Applied to the audio generation method, receiving an audio generation request is realized, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, and the audio type tag is used to characterize the auxiliary information used by the audio; querying at least one target audio element corresponding to the audio type tag in an audio database, wherein the audio element and the corresponding reference type tag are correspondingly stored in the audio database; and generating target audio by combining at least one target audio element based on the audio attribute information. By receiving an audio generation request and combining the audio attribute information with the audio type tag, highly personalized audio generation is realized. By querying the audio database for target audio elements that match the tag, the system can accurately select and combine audio content to ensure that the generated audio meets the user's specific needs and expectations. This personalized generation not only improves the user experience, enabling it to obtain audio that fully matches its preferences and application scenarios, but also enables more accurate and effective information communication in various applications, thereby better meeting the user's actual needs and expected goals.
[0111] The following combination Figure 3 , the audio generation method is further described. Among them, Figure 3 It is a processing flow chart of an audio generation method provided by an embodiment of the present invention, which specifically includes the following steps.
[0112] Step 302: Analyze the audio files in the audio database and extract audio features.
[0113] Specifically, deep learning algorithms can be used to conduct comprehensive analysis of audio files, which not only analyzes the basic physical properties of the audio, but also introduces advanced features such as emotional tone and usage scenarios, providing a basis for subsequent more detailed and dynamic audio generation.
[0114] The network that analyzes the audio file can be a convolutional neural network that extracts complex frequency and time features from the audio file. CNN can effectively identify local patterns when processing audio data, such as intensity changes at specific frequencies and waveform structures over time, which is very useful for label generation of audio types and emotional tones.
[0115] The network for analyzing audio files can also be a recursive neural network or a long short-term memory network, which are used to process time series information in audio data, which helps to understand the dynamic changes and long-term dependencies of the audio, and is especially important for the generation of labels for the emotional tone and usage scenarios of the audio.
[0116] In addition, decision trees and random forests can be used to learn and identify the rules of audio generation from large amounts of audio data, as well as to automatically classify audio tags. These algorithms are able to handle high-dimensional data and provide intuitive rule sets to assist in the generation of tags and subsequent audio matching.
[0117] Step 304: Generate an audio type tag according to the audio features.
[0118] Specifically, labels can be dynamically generated to describe the type, emotional tone, and usage scenarios of the audio, such as marking low-frequency and vibrating waveforms as "impact" and high-frequency and sharp tones as "whoosh", making the application of each audio file more flexible and diverse during the generation process.
[0119] Step 306: Update the audio type tag associated with the audio file to the audio.
[0120] Step 308: The user inputs audio parameters such as audio type, emotional tone, number of layers, volume, and whether the tone changes randomly in the front-end interface.
[0121] Step 310: Match the audio file in the audio database.
[0122] Specifically, a matching algorithm can be used to handle incomplete correspondence between user input and audio tags. For example, tags such as "impact, intense, natural" can be intelligently identified as similar to "impact, intense, fighting", ensuring that the final generated audio is as close to user expectations as possible.
[0123] Step 312: Align the transient positions of the matched audio files to generate multi-layer audio.
[0124] Specifically, the algorithm for generating audio can be a genetic algorithm for optimizing multiple audio parameters, such as random changes in the number of layers, volume, and pitch, to find the best audio combination. Digital signal processing technology can also be used when generating audio for actual processing and synthesis of audio, including but not limited to adding effects such as filtering, echo, reverberation, and synthesis of multiple audio tracks.
[0125] Step 314: Display the audio on the front-end interface.
[0126] Step 316: The user fine-tunes each layer of audio in the front-end interface.
[0127] For example, the user sets the audio type to "whoosh", the emotional tone to "intense", the number of layers to 3, and turns on the random change function of volume and pitch through the interface. According to the set parameters, the system matches the appropriate audio files from the tag database, aligns the transient positions, and generates multi-layered whoosh audio. Users can see the audio files used for each layer in the front-end interface, and can further adjust the independent parameters of each layer. Through innovative tag generation and intelligent matching methods, personalized audio can be quickly generated according to the specific needs of the audio, greatly improving the diversity and innovation of audio design. In addition, the system can also help designers break through traditional ideas and gain new design inspiration.
[0128] Corresponding to the above method embodiment, the present invention also provides an audio generation device embodiment, Figure 4 FIG. 1 is a schematic diagram of the structure of an audio generating device provided by an embodiment of the present invention. Figure 4 As shown, the device comprises:
[0129] The receiving module 402 is configured to receive an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, and the audio type tag is used to characterize auxiliary information used by the audio;
[0130] A query module 404 is configured to query an audio database for at least one target audio element corresponding to an audio type tag, wherein the audio database stores audio elements and corresponding reference type tags;
[0131] The generating module 406 is configured to generate target audio by combining at least one target audio element based on the audio attribute information.
[0132] Optionally, the query module 404 is further configured to query at least one candidate audio element in the audio database based on any audio type tag, wherein at least one candidate audio element corresponds to at least one candidate audio type tag that is the same as any audio type tag; for any candidate audio element, calculate the matching similarity based on the number of candidate audio type tags and the number of audio type tags; and determine the candidate audio element whose matching similarity is greater than a first preset threshold as the target audio element corresponding to the audio type tag.
[0133] Optionally, the audio generating device also includes an update module, configured to receive a database update request, wherein the database update request includes a correspondence between at least one reference type tag and at least one audio element; and update the audio database based on the correspondence between at least one reference type tag and at least one audio element.
[0134] Optionally, the audio generating device also includes an analysis module, which is configured to obtain any audio element in the audio database; perform audio feature analysis on any audio element, determine the classification results of the audio element in each preset classification dimension, and generate a reference type label based on the classification results; and generate a database update request based on the reference type label and any audio element.
[0135] Optionally, the audio type label includes an emotion type label and a scene type label; the analysis module is further configured to perform waveform analysis on any audio element to obtain waveform change characteristics; the waveform change characteristics are input into a pre-trained audio classification model to obtain a classification result of any audio element in each preset classification dimension, wherein the audio classification model is trained based on sample audio elements of each preset classification dimension.
[0136] Optionally, the generation module 406 is further configured to input the audio attribute information and at least one target audio element into a pre-trained audio generation model to obtain the target audio, wherein the audio attribute is used to constrain the attribute characteristics of the target audio.
[0137] Optionally, the generation module 406 is further configured to modify the target audio in response to a modification instruction from a front-end user to obtain a modified audio; and perform tuning training on the audio generation model based on the modified audio and the target audio.
[0138] Optionally, the audio generation device also includes a determination module, which is configured to receive an initial type tag and audio attribute information input by a front-end user, and obtain at least one reference type tag from an audio database; calculate the semantic similarity between the initial type tag and each reference type tag; screen out reference type tags whose semantic similarity is greater than a second preset threshold, and determine them as audio type tags; and generate an audio generation request based on the audio attribute information and the audio type tag.
[0139] Optionally, the audio generating device further includes a display module configured to analyze the target audio and determine a combination method of at least one target audio element; and visually display the audio attribute information and combination method of the target audio on a front-end interface.
[0140] Applied to the audio generation device, by receiving an audio generation request, which includes audio attribute information and an audio type tag of at least one dimension. Then, the query module queries at least one target audio element that matches these audio type tags in the audio database, and these elements are stored in the database along with their reference type tags. Finally, the generation module combines at least one target audio element based on the received audio attribute information to generate a target audio that meets the user's specific needs and expectations. In this way, the device can accurately select and combine audio content to ensure that the generated audio fully matches the user's preferences and application scenarios. This personalized generation not only improves the user experience, enabling it to obtain the audio that best meets its needs, but also enables more accurate and effective information communication in various applications, thereby better meeting the user's actual needs and expected goals.
[0141] The above is a schematic scheme of an audio generation device of this embodiment. It should be noted that the technical scheme of the audio generation device and the technical scheme of the above audio generation method belong to the same concept, and the details not described in detail in the technical scheme of the audio generation device can be referred to the description of the technical scheme of the above audio generation method.
[0142] Figure 5 The block diagram of a computing device 500 provided according to an embodiment of the present invention is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.
[0143] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface, wired or wireless (e.g., a network interface card (NIC)), such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, and a Near Field Communication (NFC).
[0144] In one embodiment of the present invention, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of the present invention. Those skilled in the art may add or replace other components as needed.
[0145] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 500 may also be a mobile or stationary server.
[0146] The processor 520 is used to execute the following computer program / instructions, which implement the steps of the above-mentioned audio generation method when executed by the processor.
[0147] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computing device embodiment, since it is basically similar to the audio generation method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the audio generation method embodiment.
[0148] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned audio generation method when executed by a processor.
[0149] The various embodiments of the present invention are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the computer-readable storage medium embodiment, since it is basically similar to the audio generation method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the audio generation method embodiment.
[0150] An embodiment of the present invention further provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned audio generation method when executed by a processor.
[0151] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the above-mentioned audio generation method belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be referred to the description of the technical scheme of the above-mentioned audio generation method.
[0152] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
[0153] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0154] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the embodiments of the present invention are not limited by the described action sequence, because according to the embodiments of the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of the present invention.
[0155] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0156] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of the present invention. The present invention selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.
Claims
1. An audio generation method, characterized in that: include: Receive an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, wherein the audio type tag is used to characterize auxiliary information used by the audio; Querying an audio database for at least one target audio element corresponding to the audio type tag, wherein the audio database stores audio elements and corresponding reference type tags; The at least one target audio element is combined based on the audio attribute information to generate target audio.
2. The method according to claim 1, characterized in that The querying of at least one target audio element corresponding to the audio type tag includes: For any audio type tag, query at least one candidate audio element in an audio database based on the any audio type tag, wherein the at least one candidate audio element corresponds to at least one candidate audio type tag that is the same as the any audio type tag; For any candidate audio element, calculating a matching similarity based on the number of the candidate audio type tags and the number of the audio type tags; The candidate audio elements whose matching similarity is greater than a first preset threshold are determined as the target audio elements corresponding to the audio type label.
3. The method according to claim 1, characterized in that Before receiving the audio generation request, the method further includes: receiving a database update request, wherein the database update request includes a correspondence between at least one reference type tag and at least one audio element; The audio database is updated based on the correspondence between the at least one reference type tag and the at least one audio element.
4. The method according to claim 3, characterized in that Before receiving the database update request, the method further includes: Obtain any audio element in the audio database; Performing audio feature analysis on any of the audio elements, determining classification results of the audio elements in each preset classification dimension, and generating a reference type label based on the classification results; The database update request is generated based on the reference type tag and any one of the audio elements.
5. The method according to claim 4, characterized in that The audio type label includes an emotion type label and a scene type label; and the performing audio feature analysis on any of the audio elements to determine the classification results of the audio element in each preset classification dimension includes: Performing waveform analysis on any of the audio elements to obtain waveform change characteristics; The waveform change feature is input into a pre-trained audio classification model to obtain a classification result of any audio element in each preset classification dimension, wherein the audio classification model is trained based on sample audio elements of each preset classification dimension.
6. The method according to claim 1, characterized in that The step of combining the at least one target audio element to generate the target audio based on the audio attribute information includes: The audio attribute information and the at least one target audio element are input into a pre-trained audio generation model to obtain a target audio, wherein the audio attribute is used to constrain the attribute characteristics of the target audio.
7. The method according to claim 6, characterized in that After generating the target audio by combining the at least one audio element based on the audio attribute information, the method further includes: In response to a correction instruction from a front-end user, the target audio is corrected to obtain a corrected audio; The audio generation model is fine-tuned and trained based on the modified audio and the target audio.
8. The method according to claim 1, characterized in that Before receiving the audio generation request, the method further includes: Receive an initial type tag and audio attribute information input by a front-end user, and obtain at least one reference type tag in an audio database; Calculating the semantic similarity between the initial type label and each of the reference type labels; Screening out reference type tags whose semantic similarity is greater than a second preset threshold and determining them as audio type tags; An audio generation request is generated based on the audio attribute information and the audio type tag.
9. The method according to claim 1, characterized in that: After the at least one audio element is combined based on the audio attribute information to generate the target audio, the method further includes: Analyze the target audio to determine a combination mode of the at least one target audio element; The audio attribute information of the target audio and the combination method are visually displayed on the front-end interface.
10. An audio generating device, characterized in that: include: A receiving module is configured to receive an audio generation request, wherein the audio generation request includes audio attribute information and an audio type tag of at least one dimension, wherein the audio type tag is used to characterize auxiliary information used by the audio; A query module, configured to query an audio database for at least one target audio element corresponding to the audio type tag, wherein the audio database stores audio elements and corresponding reference type tags; A generating module is configured to generate a target audio by combining the at least one target audio element based on the audio attribute information.
11. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the audio generation method according to any one of claims 1 to 9 are implemented.
12. A computer-readable storage medium, characterized in that: It stores a computer program / instruction, which, when executed by a processor, implements the steps of the audio method according to any one of claims 1 to 9.
13. A computer program product, characterized in that The invention comprises a computer program / instruction, which, when executed by a processor, implements the steps of the audio generation method according to any one of claims 1 to 9.
