Method, system and device for realizing personalized voice interaction and medium

Abstract

The invention relates to a method, system and device for achieving personalized voice interaction and a medium, and belongs to the technical field of intelligent voice, and the method for achieving personalized voice interaction comprises the following steps: obtaining sound sampling data; preprocessing the sound sampling data to obtain recording sample data; performing voice clone reasoning on the recording sample data to obtain a personalized voice packet; and synchronizing the personalized voice packet to a vehicle-mounted audio system to realize personalized voice interaction. According to the method, the voice library package similar to the user in timbre and speaking mode is generated by collecting the short-time recording data of the user, and the voice library package is broadcasted through the image TTS of the virtual character on the vehicle machine, so that the driving experience is improved, and the personalized customization of the vehicle is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent voice technology, and in particular to a method, system, device and medium for realizing personalized voice interaction. Background Art

[0002] With the increasing demand for automobile intelligence and personalization, the advantages of the voice replication function in cars are significant. In voice social scenarios, car users can increase the convenience and fun of voice interaction through voice replication, which is the most convenient and mature personal sound library customization product that users can experience.

[0003] Therefore, it is necessary to provide a new method, system, device and medium for realizing personalized voice interaction to solve the above technical problems. Summary of the invention

[0004] The purpose of the present disclosure is to provide a method, system, device and medium for realizing personalized voice interaction in order to solve the above problems.

[0005] The present invention achieves the above-mentioned purpose through the following technical solutions:

[0006] A method for implementing personalized voice interaction comprises the following steps:

[0007] Get sound sampling data;

[0008] Preprocessing the sound sampling data to obtain recording sample data;

[0009] Performing voice cloning inference on the recording sample data to obtain a personalized voice package;

[0010] By synchronizing the personalized voice package to the vehicle audio system, personalized voice interaction can be achieved.

[0011] As a further optimization solution of the present disclosure, the sound sampling data is preprocessed to obtain the recording sample data including:

[0012] The sound sampling data is subjected to accompaniment vocal separation, reverberation removal, echo removal and audio noise reduction.

[0013] As a further optimization solution of the present disclosure, voice cloning inference is performed on the recording sample data to obtain a personalized voice package, including:

[0014] Extracting features from the recorded sample data to obtain speech features;

[0015] Build a speech synthesis model;

[0016] Training the speech synthesis model based on the speech features to obtain a final model;

[0017] The text and customized parameters are input into the final model as required to generate a personalized voice package.

[0018] As a further optimization solution of the present disclosure, feature extraction is performed on the recording sample data to obtain speech features including:

[0019] The recording sample data is subjected to feature extraction using deep learning technology, wherein the speech features include Mel-frequency cepstral coefficients, frequency domain features, and prosodic features.

[0020] As a further optimization solution of the present disclosure, the speech synthesis model is a three-layer architecture integrating the Tacotron model, the FastSpeech model and the WaveNet model.

[0021] As a further optimization scheme of the present disclosure, the speech synthesis model is trained based on the speech features to obtain a final model including:

[0022] Dividing the speech features into a training set and a test set according to a preset ratio;

[0023] Inputting the training set into the speech synthesis model for training, and using the test set for verification;

[0024] Use data augmentation techniques to increase the diversity of training data;

[0025] Select the corresponding loss function;

[0026] Using an optimizer to update the weights of the speech synthesis model;

[0027] Performing a grid search on key hyperparameters of the speech synthesis model until an optimal configuration is found;

[0028] An emotion control mechanism is added to the speech synthesis model so that the generated speech can express specific emotions, thereby generating a final model.

[0029] As a further optimization scheme of the present disclosure, the key hyperparameters include learning rate, batch size and number of layers.

[0030] A system for realizing personalized voice interaction, comprising:

[0031] A data acquisition module, used for acquiring sound sampling data;

[0032] A preprocessing module, used to preprocess the sound sampling data to obtain recording sample data;

[0033] A cloning inference module, used to perform voice cloning inference on the recording sample data to obtain a personalized voice package;

[0034] The voice interaction module is used to synchronize the personalized voice package to the vehicle audio system to realize personalized voice interaction.

[0035] An electronic device comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0036] Memory, for storing computer programs;

[0037] The processor is used to execute the program stored in the memory to implement a method for personalized voice interaction.

[0038] A computer-readable storage medium stores a computer program, which implements a method for personalized voice interaction when executed by a processor.

[0039] The beneficial effects of the present disclosure are:

[0040] Improved driving experience: The voice replication function allows users to record their own voices or those of their relatives and friends to form a personalized voice package, making the interaction with the vehicle system more intimate and personalized. By using the voice of a close person as the navigation or system prompt tone, it can increase the emotional pleasure when driving and reduce the loneliness when driving alone. Using the voice of a close person as an interactive medium can establish a more emotional connection between the driver and the vehicle;

[0041] Enhance vehicle customization: Car owners can choose voices that match their personalities, so that the vehicle's personalization is not only reflected in appearance and configuration, but also in the intelligent interactive experience;

[0042] Improve operational convenience: The sound replication system is usually used in conjunction with intelligent voice recognition technology, which can achieve fast and accurate command recognition and feedback, thereby improving operational efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is a method flow chart in an embodiment of the present disclosure;

[0044] Figure 2 is the working principle in the embodiments of the present disclosure;

[0045] Figure 3 is a system structure block diagram in an embodiment of the present disclosure;

[0046] Figure 4 It is a block diagram of the device structure in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] The present application is further described in detail below in conjunction with the accompanying drawings. It is necessary to point out here that the following specific implementation methods are only used to further illustrate the present application and cannot be understood as limiting the scope of protection of the present application. Technical personnel in this field can make some non-essential improvements and adjustments to the present application based on the above application content.

[0048] like Figure 1 As shown, a method for realizing personalized voice interaction includes the following steps:

[0049] Step 1: Obtain sound sampling data, including:

[0050] First, the car system needs to be equipped with APK software applications, and users can enter through the APPlist interface. The application guide interface contains text guidance and an entry QR code. Users can scan the QR code on the right through the voice copy APP or mini program on the mobile phone.

[0051] For sound sampling, when the mobile phone and the car computer have a smooth network connection, you can scan the car computer QR code to enter the mini program. Follow the instructions and record in a quiet environment, such as a noise-free room, recording studio, or in the car. When recording, use Mandarin, pronounce words clearly, maintain a normal volume, and read aloud at a normal speed. At the same time, keep a distance of 20 cm from the mobile phone microphone.

[0052] Step 2: preprocessing the sound sampling data to obtain recording sample data, specifically including:

[0053] After recording is completed and the sound sampling data is obtained, it is necessary to use software to edit and clean the sound sampling data to remove noise, clip incoherent parts, and ensure the quality of the recording file.

[0054] Batch processing of vocal accompaniment separation uses the UVR5 model; the model is divided into three categories:

[0055] 1. Preserve vocals: Choose this for audio without harmony, it preserves the main vocals better than HP5. There are two models built-in, HP2 and HP3. HP3 may slightly miss the accompaniment but preserves the main vocals better than HP2.

[0056] 2. Keep only the main voice: Select this for audio with harmony, which may weaken the main voice. A model of HP5 is built in;

[0057] 3. De-reverberation and de-delay model (by FoxJoy):

[0058] (1) MDX-Net (onnx_dereverb): It is the best choice for dual-channel reverberation and cannot remove single-channel reverberation;

[0059] (2) DeEcho: Removes delay effects. Aggressive removes delay effects more thoroughly than Normal. DeReverb removes reverberation in addition. It can remove mono reverberation, but it cannot remove high-frequency plate reverberation completely.

[0060] To remove reverb / delay, use MDX-Net first and then DeEcho-Agggressive. Enter the interface to upload the audio and select "HP2_all_vocals" as the model. For the file path of the output audio, use the default settings and start the "conversion process". Finally, one main sound and the other background sound are generated.

[0061] Audio noise reduction: Use the obb voice noise reduction tool, enter the folder path of the audio you just cut, and turn on voice noise reduction.

[0062] Step 3: Perform voice cloning inference on the recorded sample data to obtain a personalized voice package, specifically including:

[0063] 1. Use deep learning technology to extract features from the recorded sample data, where the speech features include Mel frequency cepstral coefficients, frequency domain features, and prosodic features:

[0064] MFCC (Mel-Frequency Cepstral Coefficients): Extracts the Mel-Frequency Cepstral Coefficients of the audio signal, which is one of the commonly used features in speech processing.

[0065] Spectrogram: Generates a spectrogram of an audio signal, which can capture the frequency domain characteristics of the audio.

[0066] Prosody: Extracts prosodic features such as pitch, duration, and intensity, which are crucial to the naturalness of speech.

[0067] 2. Build a speech synthesis model:

[0068] The speech synthesis model is a three-layer architecture integrating the Tacotron model, the FastSpeech model and the WaveNet model.

[0069] Tacotron: A Sequence-to-Sequence (Seq2Seq) based Text-to-Speech (TTS) model that can generate natural and fluent speech.

[0070] FastSpeech: FastSpeech is faster than Tacotron while maintaining high-quality speech generation capabilities.

[0071] WaveNet: A generative adversarial network (GAN) or autoregressive model for generating high-quality raw audio waveforms.

[0072] 3. Model training:

[0073] The speech features are divided into a training set and a test set in a ratio of 7:3;

[0074] Inputting the training set into the speech synthesis model for training, and using the test set for verification;

[0075] Data augmentation: Use data augmentation techniques (e.g. speed variation, pitch change) to increase the diversity of training data.

[0076] Loss Function: Choose an appropriate loss function, such as L1 loss, L2 loss, or adversarial loss, depending on the type of model.

[0077] Optimizer: Use optimizers such as Adam and RMSprop to update the weights of the model.

[0078] Grid Search: Grid search is performed on key hyperparameters of the model (such as learning rate, batch size, number of layers) to find the best configuration.

[0079] Early stopping: Monitor performance on the test set and stop training when performance stops improving to avoid overfitting.

[0080] Tone adjustment: Change the generated tone by adjusting certain parameters in the model to make it more in line with the user's personalized needs.

[0081] Emotion control: Add an emotion control mechanism to the model so that the generated speech can express specific emotions (such as happiness, sadness).

[0082] Inference speed: The model is optimized for faster inference, enabling it to generate speech based on user input in real time.

[0083] API interface: Develop an API interface so that other applications can easily call the speech synthesis service.

[0084] Step 4: Synchronize the personalized voice package to the car audio system to achieve personalized voice interaction.

[0085] The account in the car audio system is bound and synchronized with the mini program account, and the voice is displayed on the mobile phone. After the production is completed, click the button in the car to refresh the current list and update the voice package. Find the system settings in the control panel of the car, enter the sound settings and switch to the customized voice, and you can start to experience the unique personalized voice service.

[0086] like Figure 2 As shown, in this embodiment, the user's operation process is as follows:

[0087] 1. The user downloads and installs the developed APP or applet on the mobile phone, and the car computer also pre-installs the sound replica APP developed by the car company. The user logs in to the account on the mobile phone, which is consistent with the account on the car computer.

[0088] 2. The user chooses a quiet indoor environment to record the sound, avoiding the interference of echo or noise. Through the guidance prompts on the mobile phone, enter the uniquely identified audio information and upload it to the voice cloud platform. After uploading, the cloud will process the audio, such as editing, removing noise, adjusting the sound quality, etc., to ensure the quality of the final output voice package. Then, the platform's powerful voice synthesis capabilities use a small sample to train the user's complete sound library and transmit it back to the mobile phone.

[0089] 3. The user logs in to the account on the car computer. After entering the APP for the first time in the car audio system, click the refresh control, and the audio library package customized by the voice cloud platform will be synchronized to the car computer.

[0090] 4. Enter the system settings and find the system settings in the control panel of the car computer. Enter the sound settings and select the personalized sound library to start experiencing the unique personalized voice service.

[0091] The advantages of equipping cars with related technologies are as follows:

[0092] 1. Personalized experience

[0093] Customized service: The voice replication function allows users to record their own voices or those of their relatives and friends, making the interaction with the car system more personalized.

[0094] Emotional connection: By using the voice of a close person as the navigation or assistant voice, the emotional connection between the user and the vehicle is increased.

[0095] User Preferences: Users can select or change sounds according to their preferences, providing more freedom of choice.

[0096] 2. Technical convenience

[0097] Simple recording process: The recording process of sound replication is simple and quick, and users only need to follow the instructions to complete the recording.

[0098] Fast training: The system takes a short time to analyze and train the sound, and it takes about 20 minutes to complete the sound customization.

[0099] Easy to manage: Users can easily manage and switch between different sounds in the device's sound list.

[0100] 3. Wide range of application scenarios

[0101] Diverse uses: In addition to car systems, sound replication technology can also be applied to smart homes, smart devices and other fields.

[0102] Social interaction: Using customized voices on social media or messaging apps can add interest and intimacy to communications.

[0103] Content creation: Content creators can use voice replication technology to dub characters and increase the appeal of their works.

[0104] 4. Improve safety

[0105] Reduce distractions: Using a familiar voice as the car's voice may reduce distractions while driving.

[0106] Emergency situations: In emergency situations, familiar voice prompts may be more likely to attract the driver's attention.

[0107] 5. Enhance interactivity

[0108] Simulate interpersonal communication: Sound replication technology makes the communication between the car system and people closer to real interpersonal communication.

[0109] Improved entertainment: On long drives or in traffic jams, interesting customized voices can make driving more entertaining.

[0110] 6. Improved sense of technology

[0111] Demonstrating advanced technology: The sound reproduction function demonstrates the advanced technology of vehicle intelligence and enhances the user's technological experience.

[0112] Brand influence: Car brands that provide sound reproduction functions may improve their competitiveness in the market.

[0113] 7. Enhanced user stickiness

[0114] Dependence on personalized settings: Once users use customized sounds, they may be more willing to use the vehicle out of habit.

[0115] Long-term use: Personalized sound settings may be one of the reasons why users use a certain model for a long time.

[0116] 8. Economic Benefits

[0117] Value-added service: Sound reproduction can be used as a value-added service for vehicles to increase the added value of the product.

[0118] Maintenance costs: Compared with traditional recording methods, digital sound reproduction technology may reduce long-term maintenance and update costs.

[0119] like Figure 3As shown, an embodiment of the present disclosure provides a system for implementing personalized voice interaction, including:

[0120] A data acquisition module 11 is used to obtain sound sampling data;

[0121] A preprocessing module 12 is used to preprocess the sound sampling data to obtain recording sample data;

[0122] The cloning inference module 13 is used to perform voice cloning inference on the recording sample data to obtain a personalized voice package;

[0123] The voice interaction module 14 is used to synchronize the personalized voice package to the vehicle audio system to realize personalized voice interaction.

[0124] The implementation process of the functions and effects of each module in the above system is specifically described in the implementation process of the corresponding steps in the above method, which will not be repeated here.

[0125] For the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The system embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the disclosed solution. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0126] In the above-mentioned embodiment, any multiple of all modules can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least some functions of one or more of these modules can be combined with at least some functions of other modules and implemented in one module. At least one of all modules can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of all modules can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0127] See also Figure 3, an electronic device provided by an embodiment of the present disclosure includes a processor 1110, a communication interface 1120, a memory 1130 and a communication bus 1140, wherein the processor 1110, the communication interface 1120, and the memory 1130 communicate with each other through the communication bus 1140;

[0128] Memory 1130, for storing computer programs;

[0129] The processor 1110 is used to implement the method for realizing personalized voice interaction as shown below when executing the program stored in the memory 1130.

[0130] The communication bus 1140 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0131] The communication interface 1120 is used for communication between the above electronic device and other devices.

[0132] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory 1130 may also be at least one storage device located away from the processor 1110.

[0133] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0134] The embodiment of the present disclosure further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method for realizing personalized voice interaction as described above is implemented.

[0135] The computer-readable storage medium may be included in the device / apparatus described in the above embodiment; or it may exist independently without being assembled into the device / apparatus. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method for realizing personalized voice interaction according to the embodiment of the present disclosure is realized.

[0136] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0137] The above-mentioned embodiments only express several implementation methods of the present disclosure, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present disclosure. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present disclosure, and these all belong to the protection scope of the present disclosure.

Claims

1. A method for realizing personalized voice interaction, characterized in that: The following steps are involved: Get sound sampling data; Preprocessing the sound sampling data to obtain recording sample data; Performing voice cloning inference on the recording sample data to obtain a personalized voice package; By synchronizing the personalized voice package to the vehicle audio system, personalized voice interaction can be achieved.

2. A method for realizing personalized voice interaction according to claim 1, characterized in that: Preprocessing the sound sampling data to obtain recording sample data includes: The sound sampling data is subjected to accompaniment vocal separation, reverberation removal, echo removal and audio noise reduction.

3. The method for realizing personalized voice interaction according to claim 1, characterized in that: Perform voice cloning inference on the recorded sample data to obtain a personalized voice package, including: Extracting features from the recorded sample data to obtain speech features; Build a speech synthesis model; Training the speech synthesis model based on the speech features to obtain a final model; The text and customized parameters are input into the final model as required to generate a personalized voice package.

4. A method for realizing personalized voice interaction according to claim 3, characterized in that: Feature extraction is performed on the recording sample data to obtain speech features including: The recording sample data is subjected to feature extraction using deep learning technology, wherein the speech features include Mel-frequency cepstral coefficients, frequency domain features, and prosodic features.

5. The method for realizing personalized voice interaction according to claim 3, characterized in that: The speech synthesis model is a three-layer architecture integrating the Tacotron model, the FastSpeech model and the WaveNet model.

6. The method for realizing personalized voice interaction according to claim 3, characterized in that: The speech synthesis model is trained based on the speech features to obtain a final model including: Dividing the speech features into a training set and a test set according to a preset ratio; Inputting the training set into the speech synthesis model for training, and using the test set for verification; Use data augmentation techniques to increase the diversity of training data; Select the corresponding loss function; Using an optimizer to update the weights of the speech synthesis model; Performing a grid search on key hyperparameters of the speech synthesis model until an optimal configuration is found; An emotion control mechanism is added to the speech synthesis model so that the generated speech can express specific emotions, thereby generating a final model.

7. A method for realizing personalized voice interaction according to claim 6, characterized in that: The key hyperparameters include learning rate, batch size, and number of layers.

8. A system for realizing personalized voice interaction, characterized in that: include: A data acquisition module, used for acquiring sound sampling data; A preprocessing module, used to preprocess the sound sampling data to obtain recording sample data; A cloning inference module, used to perform voice cloning inference on the recording sample data to obtain a personalized voice package; The voice interaction module is used to synchronize the personalized voice package to the vehicle audio system to realize personalized voice interaction.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, for storing computer programs; A processor is used to execute the program stored in the memory to implement the method for realizing personalized voice interaction described in any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for realizing personalized voice interaction described in any one of claims 1 to 7 is implemented.

Images