Intelligent sound simulation system based on AI

Abstract

The invention belongs to the technical field of sound simulation manufacturing, and discloses an intelligent sound simulation system based on AI. A user sound data set and playing sound data are collected and preprocessed, and the user sound data set comprises user recording data, user daily sound data and user speech sound data; performing feature extraction on the user sound data set to obtain user sound features; establishing a sound synthesis model according to the user sound features; according to the voice synthesis model, the playing voice data are input, the voice output data are obtained, the voice synthesis module is matched, the playing voice data are output, and the audio file with the user voice features as the reference is obtained, so that the acceptability of the user can be improved in the learning process, and the user experience can be improved by replacing the voice of a specific character in the film and television work. In conclusion, the method has the remarkable advantages of being high in learning assisting capacity, wide in user application range and high in entertainment interestingness.

Description

Technical Field

[0001] The present invention relates to the technical field of sound simulation production, and more specifically, to an AI-based intelligent sound simulation system. Background Art

[0002] Artificial intelligence, abbreviated as AI in English, is an important driving force for a new round of scientific and technological revolution and industrial transformation. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Artificial intelligence is an important part of intelligent disciplines. It attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is a very broad science, including robots, speech recognition, image recognition, natural language processing, expert systems, machine learning, computer vision, etc. With the rapid development of my country's science and technology industry and the increasing maturity of artificial intelligence, AI, a front-end technology, has gradually entered thousands of households and has been widely used in all walks of life in society. In daily life, learning is a rather depressing thing for most people. How to make the learning process less boring has also become a major problem in today's society.

[0003] The patent application publication number CN117354444A discloses a film and television production system based on AI intelligence and audio-visual simulation restoration technology. Through the acquisition module it establishes, online or offline data collection is carried out. The electronic file type data collected online is scanned offline through the physical image, and then input into the import module for preliminary production. The quality of the film and television works synthesized by the import module is reviewed and controlled through the export module and feedback is given. Finally, the AI ​​module is learned and updated through the management module to achieve the purpose of intelligent video editing.

[0004] Although the above-mentioned film and television production system based on AI intelligence and audio and video simulation restoration technology can achieve the purpose of reducing the cost of manual editing through intelligent editing of videos, it is difficult to complete the multi-faceted conversion of audio. In the current society where the requirements for learning are constantly increasing, how to effectively improve the comfort, acceptability and fun of learning has become a major problem for many related practitioners.

[0005] In view of this, the present invention proposes an AI-based intelligent sound simulation system to solve the above problems. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solutions:

[0007] The sound collection module is used to collect user sound data sets and play sound data, and perform preprocessing. The user sound data sets include user recording data, user daily sound data, and user speech sound data;

[0008] Furthermore, the method of collecting the user voice data set and playing the voice data includes:

[0009] Through a mobile phone, computer or other sound receiving device, the sound signal of the user in the recording state is collected to obtain the user's recording data;

[0010] Through the recording pen and video recording audio, the sound signals of the user in daily life and speech state are collected to obtain the user's daily sound data and the user's speech sound data;

[0011] The audio data of the specified video is collected based on Speechnotes speech recognition software and converted into text format to obtain the playback sound data;

[0012] The methods of preprocessing include data cleaning, data denoising, filtering and noise removal;

[0013] A feature extraction module is used to extract features from a user voice data set to obtain user voice features;

[0014] Further, the step of extracting comprises:

[0015] Q1: Normalize the sound signals in the user sound dataset;

[0016] The sound signal is passed through a filter to emphasize the energy of the high-frequency part. The calculation formula of the filter is: Where Yn is the output sound signal, Xn is the input sound signal, is the pre-emphasis coefficient;

[0017] Q2: Divide the output sound signal into multiple short time frames;

[0018] Select a window function, assign a fixed length to the window, and obtain the windowed sound signal by substituting the calculation formula: Am = Bm × Cm, where Bm is the output sound signal, Cm is the window function, and the window function is one of a rectangular window, a Hamming window, or a Hanning window;

[0019] Q3: Based on Fourier transform, the time domain signal in the windowed sound signal is converted into a frequency domain signal to obtain a transformation result;

[0020] Q4: Based on the Mel filter bank, multiply the transformation result by the filter weight in the Mel filter to obtain the filtering result;

[0021] Q5: Logarithmically compress the filtering result and then perform discrete cosine transform to output the user's voice characteristics;

[0022] Furthermore, the normalization processing method includes scaling the sound signal to a preset amplitude range, traversing the sound signal, obtaining the maximum and minimum values ​​in the sound signal, and substituting into the normalization calculation formula: Obtain a standardized value, where R is the input sound signal, Rmax is the maximum value in the sound signal, and Rmin is the minimum value in the sound signal. Replace the input sound signal with the standardized value and input it into the normalization calculation formula to obtain a normalized sound signal;

[0023] Model training module, used to establish a sound synthesis model based on the user's voice characteristics;

[0024] Furthermore, the method of establishing the sound synthesis model includes:

[0025] Collect L groups of historical user voice features as a sample set, and build an initial voice synthesis model based on a generative adversarial network. The voice synthesis model consists of two parts: a generator and a discriminator.

[0026] Preset number of iterations;

[0027] Select a loss function, input the sample set and the synthesized audio generated by the generator into the discriminator, obtain the output result, calculate the generator loss function and the discriminator loss function respectively, and minimize the generator loss function and the discriminator loss function. Repeat this process until the preset number of iterations is reached to obtain the sound synthesis model.

[0028] The sound synthesis module is used to input the playback sound data according to the sound synthesis model to obtain the speech output data;

[0029] Furthermore, the method of obtaining the voice output data includes:

[0030] Inputting the played sound data into the sound synthesis model;

[0031] Recognize words, punctuation marks, abbreviations, numbers and special characters in played sound data;

[0032] Segment the text in the played sound data into independent words or phrases, and normalize the abbreviations in the words or phrases;

[0033] Based on the sound synthesis model, the timbre and rhythm of each word or phrase are established, and the rhythm includes unstressed tone, stressed tone and intonation;

[0034] Outputting the voice output data;

[0035] Further, S1: collecting user voice data sets and playback voice data, and preprocessing, the user voice data sets include user recording data, user daily voice data and user speech voice data;

[0036] S2: Extract features from the user voice data set to obtain user voice features;

[0037] S3: Establish a sound synthesis model based on the user's voice characteristics;

[0038] S4: According to the sound synthesis model, input the playback sound data to obtain the voice output data.

[0039] The technical effects and advantages of the AI-based intelligent sound simulation system of the present invention are as follows:

[0040] The present invention obtains user voice data sets and playback sound data through the sound collection module, and can complete the user's voice collection very conveniently, which is convenient for the use of subsequent systems. Through the feature extraction module, the user's voice features can be extracted in a targeted manner, which is convenient for the subsequent system to generate the user's voice. Through the sound synthesis model established according to the user's voice features, AI can output sounds that simulate the user's voice features. By adjusting the loss function, the user can obtain his own voice that is different from that heard by others. In conjunction with the sound synthesis module, the playback sound data is output to obtain an audio file based on the user's voice features. Not only can the user's acceptance be improved and the sense of rejection be reduced during the learning process, but the immersion and interest of watching movies can also be increased by replacing the voices of specific characters in film and television works. In short, the present invention has the significant advantages of strong auxiliary learning ability, a wide range of user applications and high entertainment interest. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram of an AI-based intelligent sound simulation system of the present invention;

[0042] Figure 2 Schematic diagram of an AI-based intelligent sound simulation system of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] Example 1

[0045] See also Figure 1As shown, the AI-based intelligent sound simulation system described in this embodiment includes:

[0046] The sound collection module is used to collect user sound data sets and play sound data, and perform preprocessing. The user sound data sets include user recording data, user daily sound data, and user speech sound data;

[0047] Furthermore, the method of collecting the user voice data set and playing the voice data includes:

[0048] Through a mobile phone, computer or other sound receiving device, the sound signal of the user in the recording state is collected to obtain the user's recording data;

[0049] Through the recording pen and video recording audio, the sound signals of the user in daily life and speech state are collected to obtain the user's daily sound data and the user's speech sound data;

[0050] The audio data of the specified video is collected based on Speechnotes speech recognition software and converted into text format to obtain the playback sound data;

[0051] The methods of preprocessing include data cleaning, data denoising, filtering and noise removal;

[0052] A feature extraction module is used to extract features from a user voice data set to obtain user voice features;

[0053] Further, the steps of extracting include:

[0054] Q1: Normalize the sound signals in the user sound dataset;

[0055] The sound signal is passed through a filter to emphasize the energy of the high-frequency part. The calculation formula of the filter is: Where Yn is the output sound signal, Xn is the input sound signal, is the pre-emphasis coefficient;

[0056] Q2: Divide the output sound signal into multiple short time frames;

[0057] It should be explained that a frame refers to a continuous sound signal divided into multiple short-term sound signals, and the sound signal after the division is completed is a frame;

[0058] Select a window function, assign a fixed length to the window, and obtain the windowed sound signal by substituting the calculation formula: Am = Bm × Cm, where Bm is the output sound signal, Cm is the window function, and the window function is one of a rectangular window, a Hamming window, or a Hanning window;

[0059] Q3: Based on Fourier transform, the time domain signal in the windowed sound signal is converted into a frequency domain signal to obtain a transformation result;

[0060] Q4: Based on the Mel filter bank, multiply the transformation result by the filter weight in the Mel filter to obtain the filtering result;

[0061] It needs to be explained that the filter weight is realized by calculating the overlapping area between each filter and the transformation result;

[0062] Q5: Logarithmically compress the filtering result and then perform discrete cosine transform to output the user's voice characteristics;

[0063] Furthermore, the normalization processing method includes scaling the sound signal to a preset amplitude range, traversing the sound signal, obtaining the maximum and minimum values ​​in the sound signal, and substituting them into the normalization calculation formula: Obtain a standardized value, where R is the input sound signal, Rmax is the maximum value in the sound signal, and Rmin is the minimum value in the sound signal. Replace the input sound signal with the standardized value and input it into the normalization calculation formula to obtain a normalized sound signal;

[0064] It should be explained that the preset amplitude range refers to the amplitude of the set sound signal, and the preset amplitude range can be [-1, 1];

[0065] Model training module, used to establish a sound synthesis model based on the user's voice characteristics;

[0066] Further, the method of establishing the sound synthesis model includes:

[0067] Collect L groups of historical user voice features as a sample set, and build an initial voice synthesis model based on a generative adversarial network. The voice synthesis model consists of two parts: a generator and a discriminator.

[0068] Preset number of iterations;

[0069] Select a loss function, input the sample set and the synthesized audio generated by the generator into the discriminator, obtain the output result, calculate the generator loss function and the discriminator loss function respectively, and minimize the generator loss function and the discriminator loss function. Repeat this process until the preset number of iterations is reached to obtain the sound synthesis model.

[0070] The sound synthesis module is used to input the playback sound data according to the sound synthesis model to obtain the speech output data;

[0071] Further, the method of obtaining the voice output data includes:

[0072] Inputting the played sound data into the sound synthesis model;

[0073] Recognize words, punctuation marks, abbreviations, numbers and special characters in played sound data;

[0074] Segment the text in the played sound data into independent words or phrases, and normalize the abbreviations in the words or phrases;

[0075] It should be explained that normalization refers to expanding the abbreviations in words or phrases, such as converting 1st to first;

[0076] Based on the sound synthesis model, the timbre and rhythm of each word or phrase are established, and the rhythm includes unstressed tone, stressed tone and intonation;

[0077] Outputting the voice output data;

[0078] The beneficial effects of this embodiment are that the user voice data set and the playback sound data are obtained by the sound collection module, and the user's voice collection can be completed very conveniently, which is convenient for the use of the subsequent system. The user's voice features can be extracted in a targeted manner through the feature extraction module, so that the subsequent system can generate the user's voice. Through the sound synthesis model established according to the user's voice features, AI can output a sound that simulates the user's voice features. By adjusting the loss function, the user can obtain his own voice that is different from that heard by others. In conjunction with the sound synthesis module, the playback sound data is output to obtain an audio file based on the user's voice features. Not only can the user's acceptance be improved and the sense of rejection be reduced during the learning process, but the immersion and fun of watching movies can be increased by replacing the voices of specific characters in film and television works. In short, the present invention has the significant advantages of strong auxiliary learning ability, a wide range of user applicability and high entertainment fun.

[0079] Example 2

[0080] See also Figure 2 As shown, the part not described in detail in this embodiment is described in Example 1, and an AI-based intelligent sound simulation method is provided, including: S1: collecting a user sound data set and playing sound data, and performing preprocessing, the user sound data set includes user recording data, user daily sound data and user speech sound data;

[0081] S2: Extract features from the user voice data set to obtain user voice features;

[0082] S3: Establish a sound synthesis model based on the user's voice characteristics;

[0083] S4: According to the sound synthesis model, input the playback sound data to obtain the voice output data.

[0084] Example 3

[0085] This embodiment discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the operation mode of the AI-based intelligent sound simulation method provided above is implemented.

[0086] Since the electronic device introduced in this embodiment is an electronic device used to implement an AI-based intelligent sound simulation method in the embodiment of the present application, based on the AI-based intelligent sound simulation method introduced in the embodiment of the present application, the technical personnel of the field can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present application is not described in detail here. As long as the technical personnel of the field implement the electronic device used in the AI-based intelligent sound simulation method in the embodiment of the present application, it belongs to the scope of protection of this application.

[0087] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and thresholds in the formula are set by technicians in this field according to actual conditions.

[0088] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technical users in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An AI-based intelligent sound simulation system, characterized in that: include: The sound collection module is used to collect user sound data sets and play sound data, and perform preprocessing. The user sound data sets include user recording data, user daily sound data, and user speech sound data; A feature extraction module is used to extract features from a user voice data set to obtain user voice features; Model training module, used to establish a sound synthesis model based on the user's voice characteristics; The sound synthesis module is used to input the playback sound data according to the sound synthesis model to obtain the voice output data.

2. The AI-based intelligent sound simulation system according to claim 1, characterized in that: Methods for collecting user voice data sets and playing voice data include: Through a mobile phone, computer or other sound receiving device, the sound signal of the user in the recording state is collected to obtain the user's recording data; Through the recording pen and video recording audio, the sound signals of the user in daily life and speech state are collected to obtain the user's daily sound data and the user's speech sound data; The audio data of the specified video is collected based on Speechnotes speech recognition software and converted into text format to obtain the specified sound data; The preprocessing methods include data cleaning, data denoising, filtering and noise removal.

3. The AI-based intelligent sound simulation system according to claim 2, characterized in that: The steps to perform the extraction include: Q1: Normalize the sound signals in the user sound dataset; The sound signal is passed through a filter to emphasize the energy of the high-frequency part. The calculation formula of the filter is: Where Yn is the output sound signal, Xn is the input sound signal, is the pre-emphasis coefficient; Q2: Divide the output sound signal into multiple short time frames; Select a window function, assign a fixed length to the window, and obtain the windowed sound signal by substituting the calculation formula: Am = Bm × Cm, where Bm is the output sound signal, Cm is the window function, and the window function is one of a rectangular window, a Hamming window, or a Hanning window; Q3: Based on Fourier transform, the time domain signal in the windowed sound signal is converted into a frequency domain signal to obtain a transformation result; Q4: Based on the Mel filter bank, multiply the transformation result by the filter weight in the Mel filter to obtain the filtering result; Q5: Logarithmically compress the filtering result and then perform discrete cosine transform to output the user's voice characteristics.

4. The AI-based intelligent sound simulation system according to claim 3, characterized in that: The normalization process includes scaling the sound signal to a preset amplitude range, traversing the sound signal, obtaining the maximum and minimum values ​​in the sound signal, and substituting the normalization calculation formula into: A standardized value is obtained, where R is the input sound signal, Rmax is the maximum value in the sound signal, and Rmin is the minimum value in the sound signal. The standardized value replaces the input sound signal and is input into the normalization calculation formula to obtain a normalized sound signal.

5. The AI-based intelligent sound simulation system according to claim 4, characterized in that: Ways to build a sound synthesis model include: Collect L groups of historical user voice features as a sample set, and build an initial voice synthesis model based on a generative adversarial network. The voice synthesis model consists of two parts: a generator and a discriminator. Preset number of iterations; Select a loss function, input the sample set and the synthesized audio generated by the generator into the discriminator, obtain the output result, calculate the generator loss function and the discriminator loss function respectively, and minimize the generator loss function and the discriminator loss function. Repeat this process until the preset number of iterations is reached to obtain the sound synthesis model.

6. The AI-based intelligent sound simulation system according to claim 5, characterized in that: Ways to obtain voice output data include: Inputting the played sound data into the sound synthesis model; Recognize words, punctuation marks, abbreviations, numbers and special characters in played sound data; Segment the text in the played sound data into independent words or phrases, and normalize the abbreviations in the words or phrases; Based on the sound synthesis model, the timbre and rhythm of each word or phrase are established, and the rhythm includes unstressed tone, stressed tone and intonation; The output is the voice output data.

7. An AI-based intelligent sound simulation method, implemented according to an AI-based intelligent sound simulation system according to claims 1-6, characterized in that: S1: Collect user voice data sets and playback voice data, and perform preprocessing. The user voice data sets include user recording data, user daily voice data, and user speech voice data; S2: Extract features from the user voice data set to obtain user voice features; S3: Establish a sound synthesis model based on the user's voice characteristics; S4: According to the sound synthesis model, input the playback sound data to obtain the speech output data.

Images