Live broadcasting room voice dialogue voice changing method and device, equipment and medium

Abstract

The invention relates to the field of network live broadcast, and discloses a live broadcast room voice dialogue voice changing method and device, equipment and a medium, and the method comprises the steps: responding to a voice speaking event of a speaking user in a live broadcast room, and detecting and determining a voice segment in target audio data; according to a preset target pitch value, performing tone modification processing on the segment pitch feature of the voice segment to obtain a tone tuning pitch feature; based on the tuning pitch feature and the target timbre feature, performing voice changing processing on the voice segment to obtain a voice changing segment; and replacing the corresponding voice segment in the target audio data with the variable voice segment to obtain variable voice audio data, and sending the variable voice audio data to a called user in the live broadcast room. According to the method and the device, the performance of a voice changing technology is remarkably improved, and the problems in the aspects of real-time performance, personalized service, pitch adjustment naturalness, tone true restoration and the like in the traditional technology are solved.

Description

Technical Field

[0001] The present application relates to network live broadcast technology, and in particular to a method for changing the voice of a live broadcast room voice conversation and its device, equipment, and medium. Background Art

[0002] In the field of modern communication and entertainment, especially in the live broadcast industry, voice communication is one of the core interactive methods. Users often want to increase the fun of interaction during live broadcasts, or need to change their voices for privacy protection. However, traditional voice changing technology often has some limitations, which are particularly evident in terms of real-time and personalized services.

[0003] First, traditional voice-changing technologies often have difficulty balancing efficiency and effectiveness when processing real-time voices. In situations where real-time requirements are high, such as live broadcasts, these technologies often affect user experience due to processing delays. In addition, traditional technologies also have shortcomings in timbre matching. They usually lack flexible timbre adjustment capabilities and are difficult to perform precise timbre matching and voice-changing processing according to the user's personalized needs.

[0004] Secondly, many existing technologies fail to fully consider the natural adjustment of pitch and the true restoration of timbre during the voice changing process. This results in significant differences between the voice after voice changing and the target timbre or the user's expected effect, and cannot meet the user's requirements for naturalness and similarity. Especially in highly interactive scenes such as live broadcasts, this unnatural feeling will reduce the audience's participation and satisfaction.

[0005] Furthermore, when processing audio data, traditional technologies often ignore the user's personalized needs for sound quality. Users may want to adjust the pitch according to their preferences to achieve a voice-changing effect that is more in line with their personal characteristics. However, existing technologies often cannot provide such personalized services, limiting the improvement of user experience.

[0006] Finally, traditional voice-changing technology fails to effectively utilize advanced audio processing technology when integrating timbre and pitch features, resulting in a large difference in sound quality between the voice-changed voice segment and the original voice, and cannot achieve the ideal voice-changing effect. This difference is particularly prominent in applications that require high sound quality, such as live broadcasts, which affects the practicality and popularity of voice-changing technology.

[0007] It can be seen that although the existing voice changing technology has met the market demand to a certain extent, it still has obvious deficiencies in many aspects such as real-time performance, personalized services, natural pitch adjustment, and true restoration of timbre. These deficiencies limit the application potential of voice changing technology in real-time interactive scenarios such as live broadcasts. A new technical solution is urgently needed to overcome these challenges and provide more natural, more realistic, and more personalized voice changing services. Summary of the invention

[0008] The purpose of this application is to solve the above-mentioned problems and to provide a method for changing the voice of a live broadcast conversation and its corresponding device, equipment, and non-volatile readable storage medium.

[0009] According to one aspect of the present application, a method for changing voice in a live broadcast room voice dialogue is provided, comprising the following steps:

[0010] Responding to a voice speech event triggered by a user speaking in the live broadcast room, detecting and determining a voice segment in the target audio data corresponding to the event;

[0011] According to a preset target pitch value, the segment pitch feature of the speech segment is subjected to pitch shift processing to obtain an optimized pitch feature;

[0012] Based on the tuned pitch feature and the target timbre feature, the voice segment is subjected to voice change processing to obtain a voice change segment, wherein the target timbre feature is a template timbre feature of a target object obtained by matching the timbre feature of the speaking user from a sound information template library;

[0013] The voice-changed segment replaces the corresponding voice segment in the target audio data to obtain voice-changed audio data, and the voice-changed audio data is sent to the recipient user in the live broadcast room.

[0014] According to another aspect of the present application, a voice changing device for live broadcast room voice dialogue is provided, comprising:

[0015] A segment analysis module is configured to respond to a speech event triggered by a user speaking in the live broadcast room, and detect and determine a speech segment in the target audio data corresponding to the event;

[0016] A pitch change processing module is configured to perform pitch change processing on the segment pitch feature of the speech segment according to a preset target pitch value to obtain an optimized pitch feature;

[0017] A voice change processing module is configured to perform voice change processing on the voice segment based on the tuned pitch feature and the target timbre feature to obtain a voice change segment, wherein the target timbre feature is a template timbre feature of a target object obtained by matching the timbre feature of the speaking user from a sound information template library;

[0018] The segment updating module is configured to replace the corresponding voice segment in the target audio data with the voice-changed segment to obtain voice-changed audio data, and send the voice-changed audio data to the recipient user in the live broadcast room.

[0019] According to another aspect of the present application, a live broadcast room voice changing device is provided, comprising a central processing unit and a memory, wherein the central processing unit is used to call and run a computer program stored in the memory to execute the steps of the live broadcast room voice changing method described in the present application.

[0020] According to another aspect of the present application, a non-volatile readable storage medium is provided, which stores a computer program implemented according to the live broadcast room voice conversation voice changing method in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the method are executed.

[0021] This application effectively solves a series of technical problems existing in traditional technologies through its technical solutions, and achieves remarkable technical effects, showing its technical advantages, including but not limited to:

[0022] First of all, in response to the problem of insufficient real-time performance in traditional technologies, this application proposes a method for changing voice in live broadcast rooms. The method can respond to voice speech events triggered by users speaking in the live broadcast room, and detect and determine voice segments in the target audio data in real time, ensuring the real-time performance of the voice change processing and meeting the needs of real-time interactive scenarios such as live broadcasts.

[0023] Secondly, this application introduces a preset target pitch value to accurately change the pitch characteristics of the speaking user, and obtains the tuned pitch characteristics for subsequent voice change processing. This improvement not only improves the naturalness and authenticity of the voice after voice change, but also makes the voice change effect more in line with the user's personalized needs and enhances the user's interactive experience.

[0024] Furthermore, the present application performs voice change processing on the speech fragment by combining the tuning pitch feature and the target timbre feature, wherein the target timbre feature is the template timbre feature of the target object matched from the sound information template library based on the timbre feature of the speaking user. This not only improves the accuracy of timbre matching, but also makes the voice after voice change closer to the target timbre and expected pitch, enhancing the naturalness and similarity of the voice change effect.

[0025] In addition, after completing the voice change processing, the application can replace the corresponding human voice segment in the original audio data with the changed voice segment, and send the changed voice audio data to the user in the live broadcast room in real time. This not only improves the efficiency and accuracy of the voice change processing, but also ensures the continuity and fluency of the voice communication in the live broadcast room, providing a richer and more interesting experience for voice interaction in the live broadcast scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is an exemplary network architecture suitable for applying the live broadcast room voice conversation voice changing method of this application;

[0027] Figure 2 A flowchart of an embodiment of a method for changing voice in a live broadcast room voice conversation of the present application;

[0028] Figure 3 An exemplary deep learning-based network architecture proposed in this application to implement an end-to-end voice changing process;

[0029] Figure 4 This is an exemplary voice change configuration interface of the present application, which can be used to select a character object with a target timbre characteristic and set a target pitch value;

[0030] Figure 5 This is a principle block diagram of the voice changing device for live broadcast conversation in this application;

[0031] Figure 6 This is a structural diagram of a voice changing device for live broadcast room voice conversation adopted in this application. DETAILED DESCRIPTION

[0032] To facilitate understanding of the various embodiments of the live broadcast room voice conversation voice changing method of the present application, an exemplary network architecture is first introduced. Figure 1 As shown, the exemplary network architecture includes multiple computer devices, including multiple servers and multiple terminal devices. Through the interaction between these devices, the network live broadcast service can be ensured, and information transmission can be achieved by transmitting live streams, specifically including image streams and audio streams. For example, a live server 83 and a media server 85 are deployed in the network architecture, and at the same time, the terminal device 90 of the anchor user and the terminal device 92 of the audience user are allowed to access the network architecture to communicate with each server.

[0033] The media server 85 is responsible for processing the media data of the live broadcast room of the network live broadcast, mainly processing the live broadcast stream including its image stream and / or audio stream. Adapting to the situation that the present application aims to realize audio voice changing processing, in one embodiment, the media server 85 receives the audio stream generated by the speaking user, decodes the audio stream, and in the decoding space, according to the live broadcast room voice conversation voice changing method of the present application, performs voice changing processing on the corresponding target audio data in the audio stream to obtain voice-changed audio data, and then the media server 85 encodes the voice-changed audio data corresponding to each terminal device 90, 92 in the live broadcast room and transmits it to each audience user as the receiving user for playback.

[0034] In another embodiment, the terminal devices 90 and 92 of the host user or the audience user who is the speaking user can perform voice change processing on the target audio data in the audio stream generated in the terminal device according to the voice change method of the live broadcast room voice dialogue of the present application to obtain the corresponding voice-changed audio data. The terminal devices 90 and 92 can obtain the target audio data in real time from their audio acquisition units, detect the voice segments therein, perform voice change processing, and then encode them into an audio stream and send them to the live broadcast room through the media server, so as to be delivered to the called user, such as other audience users or host users in the live broadcast room, through instant messaging, through the live broadcast room public screen chat area, or through comments.

[0035] The live broadcast server 83 is a bridge that maintains the operation of the live broadcast room to connect the anchor user and the audience user. To this end, the live broadcast server 83 is not only responsible for providing the users in the live broadcast room with the address for pulling the live broadcast stream from the media server 85, but also handles interactive functions related to the live broadcast, such as public screen chat, barrage, gifts and comments, etc., which enhances the interactivity of the live broadcast and the audience's sense of participation.

[0036] In such a network architecture, the computer program product that implements the live broadcast room voice dialogue voice changing method of the present application can be flexibly deployed in different network nodes. For anchor users and audience users, the program product can be integrated into the network live broadcast program and deployed in their terminal devices 90, 92, such as smart phones, tablet computers, personal computers or game consoles. On the terminal devices 90, 92, the program can obtain the audio data generated by the speaking user in real time as the target audio data to be processed by voice change, which can reflect the real-time advantage.

[0037] In addition, the computer program product can also be directly deployed in the media server 85 for centralized processing of audio streams in live streaming. On the media server 85, the program can simultaneously process a large number of audio streams from different speaking users, perform voice change processing on the audio stream of each speaking user, and send it to the terminal device 92 of each audience user in the corresponding live broadcast room, so that the corresponding receiving user can play the corresponding voice-changed audio data.

[0038] From the above description of the network architecture, it can be seen that the method of the present application can be implemented on the terminal devices 90, 92 or the media server 85, providing a high degree of flexibility and a wide range of application potentials.

[0039] See also Figure 2 According to a method for changing the voice of a live broadcast room provided by the present application, the method can be implemented as a computer program product, which is installed and run in a computer device such as a terminal device or a server, and the target audio data generated by the speaking user is changed and then sent to the live broadcast room for other receiving users to play. In some embodiments thereof, the method includes the following steps:

[0040] Step S3100, responding to a voice speech event triggered by a user speaking in the live broadcast room, detecting and determining a voice segment in the target audio data corresponding to the event;

[0041] In the live broadcast room, when an audience user or anchor user can act as a speaking user and send a voice conversation to other audience users and / or anchor users as the receiving users through the public screen chat area, comment area, real-time chat controls, etc., a specific voice speech event will be triggered. For example, when the speaking user wants to speak in the live broadcast, they can click the speech control on the live broadcast room interface, and this operation triggers the voice speech event. Then, the audio acquisition unit of the computer device of the speaking user begins to collect audio data, which can be used as target audio data. According to the inherent business logic of the live broadcast room, after the speech is finished, these target audio data are sent to the live broadcast room for other users to listen to.

[0042] In this process, the computer device can obtain the target audio data in real time, or obtain the data when the user finishes speaking, so as to perform subsequent processing of this application. In order to determine the human voice speaking segment contained in the target audio data, a variety of technical means can be used for detection. These means include but are not limited to audio signal processing technology, such as Fourier transform, to identify the speech part in the audio, or use a voice activity detection algorithm to determine the segment containing the human voice in the audio.

[0043] In some embodiments, before performing speech segment detection, in order to improve the accuracy of detection, the target audio data may be pre-processed for noise reduction. Noise reduction can be achieved through a variety of algorithms, such as Wiener filter, spectral subtraction, or deep learning driven noise reduction technology. These noise reduction technologies can reduce the impact of background noise on speech detection, thereby improving the accuracy and efficiency of subsequent speech processing steps.

[0044] In one embodiment, specifically for noise reduction technology, a Wiener filter is selected, which is a linear filter based on statistical methods, and it reduces noise by minimizing the mean square error between the original signal and the estimated signal. In some embodiments, spectral subtraction can also be used for implementation. Spectral subtraction is a simpler method, which achieves noise reduction by estimating the power spectrum of the noise and subtracting it from the power spectrum of the signal. Other embodiments may use noise reduction technology based on deep learning, which identifies and removes noise by training neural networks. This method can adapt to various complex noise environments and can provide better noise reduction effects.

[0045] Through these technical means, the computer device can effectively detect and extract the voice clips of human voices from the target audio data, laying the foundation for the next step of voice change processing. Such a processing flow allows the voice of the speaking user to be presented in a clearer and more natural way in the live broadcast, enhancing the interactivity of the live broadcast and the audience's sense of participation. By avoiding non-voice clips in the target audio data, it can also save the system overhead of the computer device, making it more suitable for deployment in terminal devices.

[0046] Step S3200: performing pitch shift processing on the segment pitch feature of the speech segment according to a preset target pitch value to obtain an optimized pitch feature;

[0047] By performing pitch-changing processing on the voice clips of the speaking user and adjusting the pitch characteristics of the speaking user to match the preset target pitch value, a more natural voice-changing effect that better meets user needs can be achieved.

[0048] The preset target pitch value can be achieved in a variety of ways. In one embodiment, the speaking user is allowed to directly input the pitch value they expect through the graphical user interface of the live broadcast room. This pitch value can be an absolute frequency value, such as 440Hz representing the A note in music, or a relative value, such as a semitone higher or lower than the original sound. In another embodiment, the user can select a preset pitch value provided in the sound information template library corresponding to the target object selected by the user, so as to reflect the tonal characteristics of the voice of a specific person that the user wants to imitate.

[0049] Extracting the corresponding segment pitch features from the speech segment is the basis for completing the pitch change processing. The segment pitch feature extraction can be achieved through a variety of audio signal processing technologies. One embodiment is to use Fourier transform to convert the audio signal from the time domain to the frequency domain, and then determine the fundamental frequency, that is, the pitch, as the segment pitch feature by finding the energy concentration area in the spectrum. Another embodiment may be to use the autocorrelation method to estimate the periodicity by measuring the correlation between the signal and its delayed version, thereby obtaining the segment pitch feature. In addition, in other embodiments, deep learning methods can also be used for pitch feature extraction, by training a neural network to make it suitable for identifying pitch features in audio, and then using it to extract the corresponding segment audio features for the speech segment.

[0050] After the pitch feature extraction is completed, the segment pitch feature can be transposed to match the segment pitch feature of the speech segment with the preset target pitch value. The key to this process is to adjust the frequency components of the audio signal to achieve precise pitch adjustment while keeping other characteristics of the speech intact.

[0051] In one embodiment, the PSOLA algorithm can be used to complete the pitch conversion of the segment pitch features. The algorithm can adjust the pitch without significantly changing the duration and length of the speech. The PSOLA algorithm achieves the upward or downward shift of the pitch by synchronously cutting, moving and superimposing the original audio signal, thereby obtaining an audio signal that matches the target pitch value.

[0052] Another embodiment is a method based on phase addition, which adjusts the pitch characteristics of a speech segment by directly operating the phase information of the speech segment. Phase addition technology can maintain the naturalness and clarity of the speech signal while adjusting the pitch, avoiding distortion or sound quality degradation caused by pitch changes.

[0053] In another embodiment, a deep learning-based pitch-shifting technique can also be used, in particular, a WaveNet model. WaveNet is a technique that uses a deep convolutional neural network to generate audio waveforms, which can generate high-quality pitch-shifted speech according to a target pitch value and obtain corresponding tuned pitch features. The WaveNet model can generate pitch-shifted speech with a high degree of naturalness and clarity by learning a large amount of audio data, while accurately retaining the intonation and emotional features of the original speech.

[0054] Through the above various embodiments, the pitch change processing of the segment pitch feature of the voice segment can be completed, and the tuned pitch feature, that is, the adjusted pitch feature, can be obtained, which matches the target pitch value, thereby not only improving the authenticity and naturalness of the voice change, but also enhancing the interactivity of the live broadcast and the audience's sense of participation. By accurately controlling the pitch, the computer device can generate a voice change effect that better meets the user's expectations.

[0055] Step S3300: Based on the tuned pitch feature and the target timbre feature, the voice segment is subjected to voice change processing to obtain a voice change segment, wherein the target timbre feature is a template timbre feature of a target object obtained by matching the timbre feature of the speaker from a sound information template library;

[0056] After completing the pitch adjustment, the voice clip can be voice-changed. To this end, the adjusted pitch features are combined with the target timbre features to generate voice-changed audio that is different from the original sound of the voice clip but natural and realistic.

[0057] The pitch tuning feature needs to be combined with the target timbre feature during voice change processing. The target timbre feature can be a template timbre feature that matches the timbre of the speaking user extracted from the sound information template library. The template that is closest to the timbre of the speaking user can be determined from the sound information template library based on a variety of algorithms, such as cosine similarity or Euclidean distance.

[0058] After obtaining the target timbre features, the computer device uses these features to perform voice change processing on the voice segment. This can be implemented specifically through a variety of technologies. One embodiment is to use traditional signal processing technology, such as a spectrum transformation method, to adjust the spectrum envelope of the voice segment to match the spectrum characteristics of the target timbre to obtain a voice-changed segment. Another embodiment is to use a speech synthesis technology based on an HMM (hidden Markov model) to generate a voice-changed segment that matches the target timbre by modeling the statistical characteristics of timbre features and pitch features. In another embodiment, deep learning technology, especially a neural network-based method, is used to provide a more advanced solution for voice change processing. For example, the WaveNet model can directly learn the distribution of the original audio waveform and generate high-quality voice-changed segments. In addition, technology based on GAN (generative adversarial network) can be used to generate realistic voice-changed segments, in which the generator network is responsible for generating the voice-changed segments, and the discriminator network ensures that the generated voice-changed segments are indistinguishable from the real audio.

[0059] When implementing voice change processing, feature fusion technology can also be used to jointly encode the tuning pitch features, target timbre features, and other speech features such as the content semantic segments of speech segments to form voice change audio features, so as to decode and generate corresponding voice change segments based on the voice change audio features. This encoding process can be achieved through a deep neural network, and the network structure can include convolutional layers, recurrent layers, or fully connected layers to extract and fuse multiple speech features.

[0060] After the encoding is completed, the computer device will decode the voice-changing audio features to generate the final voice-changing segment. This decoding process can also be achieved through a neural network, specifically using a sequence-to-sequence (Seq2Seq) model to convert the encoded voice-changing audio features back into an audio waveform.

[0061] Based on this, the computer device can accurately perform voice change processing on the speech clips of the speaking user, generating natural and realistic voice-changed audio that not only meets the target timbre characteristics but also has the characteristics of adjusted pitch. This voice change processing not only improves the fun and interactivity of the voice, but also enhances the entertainment of the live broadcast and the participation of the audience.

[0062] Step S3400: Replace the corresponding voice segment in the target audio data with the voice-changed segment to obtain voice-changed audio data, and send the voice-changed audio data to the recipient user in the live broadcast room.

[0063] The generation of the voice-changing segment is to simulate a specific timbre and pitch to meet the needs of the speaking user or to provide entertainment effects. The generated voice-changing segment needs to be aligned in time with the speech segment in the target audio data to ensure the coherence and synchronization during the replacement. To this end, the computer device can use audio editing techniques, such as timestamp matching and audio alignment algorithms, to ensure that the voice-changing segment can accurately replace the corresponding speech segment.

[0064] The replacement operation can be implemented by programming, wherein the computer device uses a custom script to process the target audio data. Specifically, the computer device first locates the voice segment in the target audio data and replaces it with the voice-changed segment.

[0065] In order to ensure that the replaced audio data sounds natural and smooth, the computer device can also perform some post-processing, such as smoothing the connection points of the audio clips to reduce possible audio distortion and noise. This can be achieved by using a cross-fade technique, which gradually increases the volume of one clip while gradually reducing the volume of the other clip at the connection point of the two audio clips, thereby achieving a smooth transition.

[0066] In addition, after the computer device replaces the corresponding voice segment with the voice-changed segment, it obtains voice-changed audio data and sends the voice-changed audio data to the recipient user in the live broadcast room.

[0067] After receiving the voice-changed audio data, the recipient can play and hear the audio in the live broadcast room. The voice-changed audio data is transmitted through the audio stream of the live broadcast room, ensuring that all audience users can receive the same audio content at the same time. When the recipient hears the voice-changed audio, they will experience richer and more interesting interactions because the voice-changed effect adds a new dimension to the live broadcast. For example, the voice of the speaking user may sound like a completely different person, adding mystery and entertainment to the live broadcast.

[0068] In addition, the user being called can respond to the voice-changing effect through interactive functions in the live broadcast room, such as bullet screen, comments or instant messages, and provide feedback information through reactions. This feedback information can be used to evaluate the voice-changing effect so that the user being called can further participate in the live broadcast interaction. This instant feedback mechanism not only enhances the user's sense of participation, but also provides the speaking user with a basis for adjusting the voice-changing parameters, so that the voice-changing effect can be optimized according to the audience's preferences. In this way, the audio communication in the live broadcast room becomes more dynamic and personalized, improving the overall experience of the live broadcast.

[0069] It can be seen from the above embodiments that the present application effectively solves a series of technical problems existing in the traditional technology and achieves significant technical effects, showing its technical advantages, including but not limited to:

[0070] First of all, in response to the problem of insufficient real-time performance in traditional technologies, this application proposes a method for changing voice in live broadcast rooms. The method can respond to voice speech events triggered by users speaking in the live broadcast room, and detect and determine voice segments in the target audio data in real time, ensuring the real-time performance of the voice change processing and meeting the needs of real-time interactive scenarios such as live broadcasts.

[0071] Secondly, this application introduces a preset target pitch value to accurately change the pitch characteristics of the speaking user, and obtains the tuned pitch characteristics for subsequent voice change processing. This improvement not only improves the naturalness and authenticity of the voice after voice change, but also makes the voice change effect more in line with the user's personalized needs and enhances the user's interactive experience.

[0072] Furthermore, the present application performs voice change processing on the speech fragment by combining the tuning pitch feature and the target timbre feature, wherein the target timbre feature is the template timbre feature of the target object matched from the sound information template library based on the timbre feature of the speaking user. This not only improves the accuracy of timbre matching, but also makes the voice after voice change closer to the target timbre and expected pitch, enhancing the naturalness and similarity of the voice change effect.

[0073] In addition, after completing the voice change processing, the application can replace the corresponding human voice segment in the original audio data with the changed voice segment, and send the changed voice audio data to the user in the live broadcast room in real time. This not only improves the efficiency and accuracy of the voice change processing, but also ensures the continuity and fluency of the voice communication in the live broadcast room, providing a richer and more interesting experience for voice interaction in the live broadcast scene.

[0074] Based on any one of the embodiments of the present application, the pitch shifting and voice changing processing can be completed end-to-end with the help of the deep learning model network architecture built in advance. Figure 3 As shown in FIG. 1 , the network architecture includes a content encoder, a pitch extraction model, a joint encoder, and a generator. The content encoder is used to extract the deep semantic features corresponding to the speech content from the speech segment to obtain the content semantic features, the pitch extraction model is used to extract the pitch features of the speech segment and perform pitch change according to the target pitch value to obtain the tuned pitch features, the content semantic features, the tuned pitch features, and the target timbre features are jointly encoded by the joint encoder and sent to the generator for decoding to generate the corresponding audio data as the voice-changed segment.

[0075] Both the content encoder and the pitch extraction model can be implemented through convolutional layers, which can capture the corresponding local features in the audio signal and extract deep feature representations by stacking. Therefore, these convolutional layers can constitute a content encoder that encodes the content of the speech segment and a pitch extraction model that extracts the pitch features of the speech segment. Among them, the content encoder can also be implemented using existing models such as Whisper, Hubert, and ContentVec.

[0076] For the pitch adjustment process of step S3200, the pitch features of the speech segment can be extracted by using the pitch extraction model in the network architecture, and then these pitch features can be adjusted according to the preset target pitch value. This pitch extraction model can be regarded as a pitch conversion network, which can adjust the fundamental frequency of the speech segment according to the target pitch value, obtain the tuned pitch features, realize the upward or downward shift of the pitch, and thus obtain the segment pitch features that match the target pitch value.

[0077] For the voice change processing of step S3300, the network architecture combines the content semantic features, the adjusted tuning pitch features and the target timbre features, and then passes through a joint encoder, which encodes them into an embedding vector, and then inputs this embedding vector into the generator of the network architecture. The generator generates a voice-changed segment based on this embedding vector. By learning a large amount of audio data, the generator is able to generate voice-changed audio with high naturalness and clarity, while accurately retaining the intonation and emotional characteristics of the original speech.

[0078] The network architecture can be trained in advance using a generative adversarial approach, where the generator generates mutated voice clips and the discriminator evaluates the difference between the generated audio and the real audio. In this way, the generator can learn to generate high-quality mutated voice clips that are difficult to distinguish from real audio.

[0079] The voice-changing segments output by the network architecture obtained according to the present embodiment can be used to replace the corresponding voice segments in the target audio data. The advantage of applying this network architecture is that it can generate high-quality voice-changing segments end-to-end while maintaining the naturalness and clarity of the audio, providing users with a seamless and natural voice-changing experience. In this way, the network architecture not only improves the authenticity and naturalness of the voice change, but also enhances the interactivity of the live broadcast and the audience's sense of participation. When the network architecture is used to serve the voice segment voice change processing of a large number of speaking users in the live broadcast platform, its efficiency advantage is more significant.

[0080] Based on any embodiment of the method of the present application, responding to a voice sending event triggered by a user speaking in a live broadcast room, detecting and determining a voice segment in target audio data corresponding to the event includes:

[0081] Step S3110, responding to a voice speech event triggered by a user in the live broadcast room, and obtaining target audio data corresponding to the real-time speech of the user;

[0082] In the live broadcast room, when the user starts speaking, a specific speech event is triggered. The computer device responds to this event and obtains the target audio data corresponding to the real-time speech of the user. In the computer device as a terminal device, the real-time collection of audio data can be achieved through the microphone equipped on the user device. The sound signal captured by the microphone is then converted into digital form so that the computer device can further process it.

[0083] The collection of audio data can be done in many ways, for example, it can be started by the speech button on the live studio interface. Once the user activates this button, the audio collection unit of the computer device, that is, the microphone, starts working and captures the user's voice in real time. The collected raw audio data contains the user's voice and possible environmental noise, which is the basis for subsequent processing steps.

[0084] During the audio acquisition process, computer equipment records information such as the amplitude and frequency of the audio signal, which is then used to determine the vocal part of the audio. The acquisition of audio data not only needs to capture clear sound, but also needs to ensure the integrity and accuracy of the data so that the vocal fragments can be accurately identified and extracted in the subsequent processing steps.

[0085] The collected audio data usually exists in the form of audio waveforms, which contain rich information, such as the pitch, volume, and timbre of the voice. In order to ensure that this data can be accurately analyzed and processed, computer equipment will use specific algorithms to process these waveforms in order to extract useful information from them. These algorithms may include signal processing techniques such as Fourier transform, which can convert audio signals from the time domain to the frequency domain, making it easier to identify and analyze features in the audio.

[0086] Through the above process, the computer device can accurately obtain the real-time audio data of the speaking user as the target audio data, which provides a basis for subsequent audio noise reduction and human voice activity detection. This implementation method enables the speaking user's voice to be presented in a clearer and more natural way in the live broadcast, and also lays a solid foundation for achieving high-quality voice change processing.

[0087] Step S3120: performing audio noise reduction processing on the target audio data to obtain low-noise audio data with suppressed noise;

[0088] After obtaining the target audio data corresponding to the real-time speech of the speaking user, audio noise reduction processing can be performed to remove or reduce the background noise in the target audio data and more accurately identify and extract the voice segments with human voice activity. Audio noise reduction processing is achieved by applying specific algorithms to the target audio data, which can identify and suppress unwanted noise components while retaining the core characteristics of the speech signal.

[0089] As disclosed above, there are many implementation methods for audio noise reduction technology, including but not limited to Wiener filters, spectral subtraction, and deep learning-driven noise reduction technology. When implementing audio noise reduction, a computer device analyzes the spectrum of the target audio data, identifies the characteristics of noise and speech, and adjusts the signal accordingly. For example, a deep learning model can be trained to identify specific types of noise, such as fan noise, traffic noise, or background conversation, and remove them from the audio data. Such processing not only improves the accuracy of speech detection, but also provides clearer audio input for subsequent voice change processing.

[0090] Through audio noise reduction processing, computer equipment can obtain low-noise audio data with noise suppressed, providing high-quality audio signals for subsequent human voice activity detection and voice change processing.

[0091] Step S3130: Perform human voice activity detection based on the low-noise audio data to determine a speech segment containing human voice.

[0092] After the audio noise reduction process is completed, the computer equipment will perform voice activity detection to accurately identify the passages containing human voices from the low-noise audio data. Voice activity detection is a key step in audio processing. It uses the characteristics of the sound to distinguish between human voice and non-human voice parts, ensuring that the voice change process only acts on the actual speech content.

[0093] Human voice activity detection can be achieved through a variety of algorithms. For example, a common method is to use features based on short-time Fourier transform (STFT), which can convert time series signals into time-frequency representation, making it easier to identify the human voice frequency range. Based on STFT, energy threshold detection can be applied, that is, an energy level threshold is set. When the energy of the audio signal in a specific frequency range exceeds this threshold, it is considered to be human voice activity.

[0094] Another embodiment is to use deep learning based algorithms, such as recurrent neural networks (RNN) or long short-term memory networks (LSTM), which are able to learn complex patterns of human voices and take into account the temporal continuity of speech signals during the detection process. Deep learning methods usually require a large amount of labeled data to train the model to ensure its accuracy and robustness.

[0095] Another embodiment is to use Mel Frequency Cepstral Coefficient (MFCC) as a feature, which is a feature extraction method widely used in speech recognition, which can capture the key spectral features of human voice signals. By analyzing the changes of MFCC features over time, the computer device can distinguish between human voice and background noise, thereby accurately locating the human voice segment.

[0096] In practical applications, computer devices can combine multiple technologies to improve the accuracy of human voice activity detection. For example, energy threshold detection can be used first to quickly locate possible human voice areas, and then a deep learning-based model can be used for more detailed recognition to ensure the accuracy of the detection results.

[0097] By executing all the steps of the above embodiments, more efficient and accurate voice processing can be achieved compared to other embodiments, thereby significantly improving the performance of voice changing technology. This embodiment not only enhances the naturalness and clarity of the voice through the combination of its various steps, but also ensures that the audio data after voice change can truly reflect the original voice characteristics of the speaking user, while avoiding unnecessary noise interference, providing users with a more natural, more realistic, and more personalized voice changing service. In addition, this comprehensive technical solution can also adapt to various complex live broadcast environments, whether in a noisy background or in a scene that requires a high degree of privacy protection, it can provide a stable and reliable voice changing effect, greatly enhancing the interactivity of the live broadcast and the audience's sense of participation.

[0098] Based on any embodiment of the method of the present application, according to a preset target pitch value, the segment pitch feature of the speech segment is subjected to pitch shifting processing to obtain an optimized pitch feature, including:

[0099] Step S3210: determining whether the speaking user has preset a pitch value, and if so, using the pitch value as the target pitch value;

[0100] The speaking user can enter the voice change configuration interface through the voice change configuration control in the live broadcast room, and set the desired pitch value in it for use as the target pitch value. Whether the speaking user has preset this pitch value can be marked by a status mark. When the user sets the pitch value, the status mark directly stores the pitch value preset by the user. When the user does not set the pitch value or marks it as the default state, the target pitch value expected by the user is the preset pitch value corresponding to the target object corresponding to the target timbre feature determined by the user. This pitch value is usually mapped with the target timbre feature and stored in the sound information template library.

[0101] Accordingly, the computer device can determine whether the user has preset a pitch value by reading the value of the status identifier. When it is recognized that the user has manually preset a pitch value, the pitch value is used as the target pitch value. Otherwise, step S3220 is continued to determine the target pitch value.

[0102] Step S3220: When the speaking user has not preset a pitch value, the preset pitch value corresponding to the target object determined by the speaking user from the character objects in the sound information template library is used as the target pitch value;

[0103] When the speaking user does not preset the pitch value, the computer device will automatically retrieve and adopt the preset pitch value of the target object from the sound information template library. The sound information template library is a database that stores a variety of timbre features and corresponding pitch values. These data can be based on the statistical results of a large number of samples, or customized for specific person objects. In this library, each timbre feature has its corresponding pitch value, which reflects the tonal characteristics of different timbres.

[0104] The computer device has previously determined the target timbre characteristics corresponding to the target object that the speaking user wishes to imitate. Specifically, the speaking user has selected the target object through a pre-provided timbre sample recommendation list. The target object may be a preset person, character, or other voice characteristics that the user wishes to imitate. Therefore, according to the identity of the target object, the preset pitch value associated with the target object can be retrieved from the voice information template library.

[0105] Step S3230: Perform pitch shifting processing on the segment pitch feature of the speech segment according to the target pitch value, so that the segment pitch feature reaches the target pitch value.

[0106] By performing pitch shifting on the speech segment of the speaking user, the pitch characteristics of the speech segment can be matched with the target pitch value. As disclosed above, pitch shifting can be implemented by a variety of deep learning models, such as the pitch extraction model based on convolutional neural network disclosed above. The model can analyze the spectral characteristics of the speech segment and identify the fundamental frequency components in the speech signal, that is, the characteristics that determine the pitch of the speech. By adjusting these fundamental frequency components, the model can change the pitch of the original speech, moving it up or down to the target pitch value.

[0107] As disclosed above, in the pitch shifting process, the computer device may also apply a series of other algorithms to finely adjust the pitch of the speech segment. These algorithms include traditional signal processing techniques, such as the PSOLA algorithm, which can stretch or compress the speech signal in the time domain without changing the duration and length of the speech, thereby adjusting the pitch. In addition, phase estimation-based methods may also be used, which change the pitch by modifying the phase spectrum of the speech signal while maintaining the naturalness of the speech.

[0108] Through this embodiment, intelligent adjustment of the pitch characteristics of the speaking user's voice segment is achieved, which significantly improves the performance of the voice changing technology, especially in meeting the user's personalized needs and providing a natural voice changing experience.

[0109] First, this embodiment allows the user to directly set the desired pitch value through the voice change configuration interface. This direct user input enables the voice change process to accurately reflect the user's personalized needs and enhances the user's sense of participation and control. When the user sets the pitch value, it can immediately respond and use the value as the target pitch value, thereby ensuring that the voice change result is consistent with the user's expectations.

[0110] For users who do not manually set the pitch value, the preset pitch value of the target object can be automatically retrieved from the sound information template library and adopted. This intelligent processing provides convenience for users, especially when users want to imitate a specific person or timbre. In this way, it not only simplifies the user's operation process, but also ensures the naturalness and accuracy of the voice change processing, so that even if the user does not explicitly set the pitch value, a high-quality voice change effect can be achieved.

[0111] It can be seen that this embodiment realizes a flexible and accurate pitch adjustment scheme by intelligently considering the user's personalized needs for pitch values ​​and providing preset pitch values ​​for the target object when the user does not manually set the pitch value, thereby significantly improving the practicality and user experience of speech changing technology.

[0112] On the basis of any embodiment of the method of the present application, before the voice segment is subjected to voice change processing based on the tuned pitch feature and the target timbre feature, the method includes:

[0113] Step S1100: acquiring speech voice data of the speaking user, and extracting the timbre features of the speaking user based on the speech voice data, wherein the speech voice data is the target audio data generated in real time, or the historical audio data pre-recorded by the speaking user;

[0114] Acquiring the speaking user's voice data can be adapted to two different situations: one is to process the audio data generated in real time, that is, the target audio data in the audio stream generated when the speaking user speaks in real time during the live broadcast. This target audio data is more convenient for extracting the speaking user's own timbre characteristics in real time; the other is the historical audio data composed of the audio data that the speaking user has recorded in advance. This historical audio data is usually generated by requiring the user to read a preset text, so it can usually provide richer semantics to more accurately and efficiently extract the speaking user's own timbre characteristics. Whether it is real-time audio or historical audio, the goal is to extract the unique timbre characteristics of the speaking user and obtain his own timbre characteristics. These characteristics are parameters in the speech signal that represent the characteristics of the individual voice.

[0115] The extraction of timbre features can be achieved through audio signal processing technology. Computer equipment uses specific algorithms to analyze these signals based on the speaking voice data. The audio data is first transformed from the time domain to the frequency domain by Fourier transform and then its Mel-frequency cepstral coefficients (MFCC) are determined, thereby revealing the frequency components of the audio signal. Then, the key spectral features in the audio signal are identified, and these features can be used to characterize the timbre of the speaking user and constitute timbre features. In practice, a timbre feature extraction model that has been pre-trained to convergence can be used to implement the extraction of the timbre of audio data.

[0116] Timbre features play an important role in speech processing. They can be used not only to identify and verify the identity of the speaker, but also as adjustment targets in voice modification to generate audio output that matches a specific timbre. By accurately matching or adjusting these features, computer devices can generate voice-modified audio that sounds natural and similar to the target timbre, which is crucial to improving the authenticity and naturalness of speech.

[0117] Step S1200, calculating the timbre similarity between the timbre feature of the user and the timbre features of the templates corresponding to the various character objects preset in the sound information template library;

[0118] The present application prepares a sound information template library in advance. The sound information template library is a pre-built database that contains the timbre features and pitch values ​​of multiple character objects. In addition, it also associates the speech audio clips of each character object. These data are associated with the identity of the character object and stored in the library. To build such a database, it is necessary to collect speech audio clips of different characters, and then use audio processing technology such as the timbre feature extraction model disclosed above to extract the timbre features of each person.

[0119] In the sound information template library, the timbre characteristics and pitch values ​​of each character object are encoded as a feature vector, which is stored in the database and associated with the unique identifier of the character object, so that each character object has its own corresponding template timbre characteristics. In this way, when timbre matching is required, the computer device can quickly retrieve and compare these template timbre characteristics.

[0120] Based on the sound information template library, the timbre similarity between the timbre features of the speaking user and each template timbre feature can be determined. To this end, the data distance between the timbre features of the speaking user and the template timbre features of each character object in the sound information template library is calculated to determine the similarity between them. The calculation of the data distance can be implemented by selecting one of a variety of algorithms, such as cosine similarity, Euclidean distance or neural network.

[0121] For example, cosine similarity measures the similarity of two vectors by calculating the cosine of the angle between the vectors corresponding to the two timbre features. The closer the cosine value is to 1, the more similar the two vectors are. Euclidean distance measures the distance between two vectors in multidimensional space. The smaller the distance, the higher the similarity. In addition, deep learning methods such as convolutional neural networks (CNN) or recurrent neural networks (RNN) can also be used. These models can learn complex patterns of timbre features and calculate high-precision similarity scores.

[0122] Step S1300: Select some character objects with relatively high timbre similarity, construct the character objects and their corresponding speech audio clips into a timbre sample recommendation list, and push it to the speaking user;

[0123] After calculating the timbre similarity between the timbre characteristics of the speaking user and the timbre characteristics of the templates of each character object in the sound information template library, the character objects with relatively high similarity to the timbre characteristics of the speaking user are screened out according to these similarity scores. In one embodiment, a threshold is pre-set, and the timbre similarity exceeding the threshold is defined as relatively high timbre similarity.

[0124] After determining the person objects with high similarity to the voice characteristics of the speaking user, these person objects and their corresponding speaking audio clips are constructed into a voice sample recommendation list. This recommendation list thus contains multiple candidate voice samples, i.e., speaking audio clips, each of which has a certain similarity to the voice of the speaking user, thereby providing the user with a series of possible voice choices. The purpose of the recommendation list is to allow users to select the voice they think is most suitable for subsequent voice changing processing.

[0125] The generation of the recommendation list can also be achieved in the following way: a sorting algorithm can be used to sort all character objects according to their similarity with the timbre characteristics of the speaking user, and then the top N (N is a preset integer value) character objects with the highest similarity are selected.

[0126] The determined list of recommended timbre samples is finally pushed to the speaking user, who can listen to the sample audio of each candidate timbre and select a timbre that best meets the expectations based on personal preferences. This step not only improves the user experience because it allows users to select according to their preferences, but also enhances the personalization and naturalness of the voice changing process because the timbre finally selected is more likely to be a timbre that the user is satisfied with and comfortable with.

[0127] Step S1400: taking the character object determined by the speaking user based on the speaking audio segment in the timbre sample recommendation list as the target object, and taking the template timbre feature of the target object as the target timbre feature.

[0128] After the speaking user selects a timbre that best meets his or her expectations from the timbre sample recommendation list, the computer device will determine the target timbre features based on the user's selection. Specifically, the computer device displays the timbre sample recommendation list to the speaking user for selection. When the user selects a specific timbre sample from the timbre sample recommendation list, the computer device will identify the person object corresponding to the sample, use it as the target object, and extract the template timbre features of the target object from the sound information template library as the target timbre features. After determining the target timbre features, the computer device will use these features to guide the voice changing process to ensure that the final generated voice changing audio matches the timbre selected by the user.

[0129] This embodiment achieves highly personalized and accurate timbre matching by accurately extracting the timbre features of the speaking user and comparing them with the timbre features of the templates in the sound information template library. This process first effectively extracts the timbre features of the user based on the speaking voice data of the speaking user, and then, by calculating the timbre similarity between the timbre features of the speaking user and the timbre features of the templates of the character objects in the template library, the character object closest to the user's timbre can be identified, which not only provides the user with a list of timbre sample recommendations, but also ensures that these recommendations are based on accurate matching results, greatly improving the relevance and accuracy of timbre selection.

[0130] When the speaking user selects a character object as the target object from the recommended list of timbre samples and determines its corresponding template timbre features as the target timbre features, it can be used for subsequent voice changing processing. This not only improves the naturalness and authenticity of the voice changing process, but also enhances the user's sense of participation and satisfaction, because users can choose a timbre that they think best suits their personal preferences. In addition, this intelligent timbre selection and matching process makes voice changing technology more flexible and adaptable, and can adapt to the personalized needs of different users, whether in live broadcasts, entertainment or other application scenarios that require voice changing.

[0131] It can be seen that the technical advantage of this embodiment is that it can provide an intelligent timbre matching and selection mechanism based on the user's timbre characteristics, which not only improves the personalization and naturalness of the voice change processing, but also improves the overall user experience. In this way, this embodiment realizes a timbre selection and voice change processing method that is both efficient and user-friendly, providing users with a new and highly interactive live broadcast experience.

[0132] On the basis of any embodiment of the method of the present application, based on the tuned pitch feature and the target timbre feature, the voice segment is subjected to voice change processing to obtain the voice change segment, including:

[0133] Step S3310, extracting features from the speech segment to obtain its content semantic features;

[0134] This embodiment can be implemented by applying the pre-built network architecture disclosed in the previous text of this application. In this network architecture, the content encoder extracts features from the speech fragments, and obtains its content semantic features from the speech fragments. These features can represent the content information of the speech, that is, the meaning of the specific words and sentences spoken by the speaker. The content encoder plays a core role, which is responsible for extracting deep semantic features from the input speech fragments. These features include not only the physical properties of the speech such as pitch, volume and timbre, but also the semantic information in the speech, such as phonemes, words and phrases.

[0135] Content encoders are usually composed of multiple layers of convolutional neural networks that can abstract and extract features of speech signals layer by layer. For example, the first layer may focus on extracting basic spectral features of the audio signal, while subsequent layers can recognize more complex patterns such as phonemes and words. In this way, the content encoder can capture the semantic content in the speech and encode this information into a set of high-dimensional feature vectors, namely content semantic features.

[0136] These content semantic features are then used in other steps of the voice change process. For example, in the joint encoding step, the content semantic features are combined with the tuned pitch features and the target timbre features to form the voice change audio features.

[0137] Step S3320, jointly encode the content semantic feature, the tuned pitch feature, and the target timbre feature to obtain a voice-changed audio feature;

[0138] Referring to the network architecture shown in the previous text of this application, the extracted content semantic features, tuned pitch features and target timbre features are jointly encoded by a joint encoder to generate voice-changed audio features. By fusing multiple features into a unified representation, the representation can fully capture the characteristics of the original speech segment and guide the generator to generate the corresponding voice-changed audio.

[0139] The content semantic feature contains the content information of the speech, the tuned pitch feature represents the adjusted pitch information, and the target timbre feature describes the desired timbre properties. The task of the joint encoder is to fuse these features into an embedding vector that can be understood and used by the generator to produce the voice-changed audio as a voice-changed segment.

[0140] The joint encoder can be implemented using a variety of techniques. One implementation is to use a deep neural network, such as a recurrent neural network (RNN) or a long short-term memory network (LSTM), which is able to process sequential data and take into account temporal dependencies during the encoding process. In this network, the input feature sequence is passed through a series of layers, each of which can contain non-linear activation functions to enhance the expressive power of the model. In this way, the joint encoder is able to learn how to map different features into a common embedding space.

[0141] Another embodiment is to use an attention mechanism to enhance the effect of the joint encoder. The attention mechanism allows different weights to be assigned to different parts of the input features during encoding, so that more attention can be paid to features that have a greater impact on the voice change effect. For example, if a specific pitch feature is very important for timbre recognition, the attention mechanism can ensure that this feature is given enough attention during the encoding process.

[0142] Another embodiment is to use an autoencoder structure for joint encoding. The autoencoder compresses the input features into a low-dimensional representation through an encoding stage, and then restores this representation to the original data through a decoding stage. In this process, the encoder learns how to capture the essential information of the input features, and the decoder learns how to reconstruct the audio signal from this compressed representation.

[0143] Through the application of the joint encoder, multiple features can be effectively fused and an embedded vector containing rich information can be generated as the voice-changed audio feature. This vector is then used in the generator to produce the voice-changed audio clip. This joint encoding process not only improves the accuracy and naturalness of the voice-changed processing, but also enhances the sound quality and expressiveness of the final audio output.

[0144] Step S3330: Decode the voice-changed audio features to generate corresponding audio data as a voice-changed segment.

[0145] According to the network architecture given in the previous text of this application, the decoding generation of the voice-changing audio features can be completed by the generator in the network architecture, which is responsible for converting the encoded voice-changing audio features back into audio data to generate voice-changing clips. The generator is a key component of the deep learning model network architecture, which is usually composed of multiple layers and can reconstruct high-quality audio signals from the embedded vector.

[0146] The design of the generator can be implemented in many ways. A common implementation is to use a deep convolutional neural network (DCNN), which can gradually upsample the encoded audio features to restore the spectrum of the original audio signal. In this process, each convolutional layer of the generator increases the dimensionality of the audio signal until it reaches the sampling rate and bit depth of the original audio. For example, the WaveNet model is a special DCNN that uses stacked convolutional layers and the concept of dilation gating to generate audio waveforms.

[0147] Another embodiment is to use a recurrent neural network (RNN) or a long short-term memory network (LSTM), which are able to process sequence data and take into account temporal dependencies in the generation process. These networks generate continuous audio waveforms by recursively processing the encoded features and gradually building each time step of the audio signal.

[0148] Another embodiment is to use an autoencoder structure, where the encoder compresses the input features into a low-dimensional representation, and the generator (decoder) reconstructs the audio signal from this representation. In this process, the generator learns how to recover the details and structure of the original audio signal from the compressed features.

[0149] Generator training usually requires a large amount of audio data, which is used to adjust the parameters of the generator so that it can generate voice-changed audio that is indistinguishable from real audio. During training, the output of the generator is compared with the target audio data, and the difference is calculated through a loss function. Common loss functions include mean square error (MSE) and adversarial loss. These loss functions help the generator learn how to better reconstruct audio signals.

[0150] Through this embodiment, an efficient and accurate voice change processing flow is realized, and its technical advantages are significant. First, the process can accurately extract deep semantic features from voice fragments, which not only contain the physical properties of the voice, but also cover semantic information, providing rich input information for voice change processing. Secondly, through the application of the joint encoder, the content semantic features, the tuning pitch features and the target timbre features are effectively integrated to generate a comprehensive voice change audio feature vector, which not only improves the accuracy and naturalness of the voice change processing, but also enhances the sound quality and expressiveness of the final audio output. In addition, the design of the generator is flexible and diverse, and a high-quality audio signal can be reconstructed from the encoded voice change audio features, whether using a deep convolutional neural network, a recurrent neural network or an autoencoder structure, the naturalness and clarity of the voice change fragment can be ensured. Finally, this process provides users with a seamless and natural voice change experience, and at the same time, when processing the voice fragments of a large number of speaking users in the live broadcast platform, it shows its efficiency advantage, significantly improves the performance of voice change technology, and solves the problems of real-time, personalized services, natural pitch adjustment and true restoration of timbre in traditional technologies.

[0151] On the basis of any embodiment of the method of the present application, after sending the voice-changed audio data to the called user in the live broadcast room, the method includes:

[0152] Step S5100: Acquire feedback information replied by the called user after playing the voice-changed audio data, and determine the acceptance degree of the called user to the voice-changed audio data according to the feedback information;

[0153] In a live broadcast environment, the feedback of the receiver on the voice-changing audio data can be used to measure the voice-changing effect. To this end, the corresponding feedback information can be collected and analyzed to determine the corresponding acceptance degree of the voice-changing audio data by the receiver. Feedback information can be obtained through a variety of channels, including any channel suitable for realizing a dialogue between the speaking user and the receiver, such as chat messages, bullet screens, comments in the comment area, instant chat dialogue windows, etc. in the public screen chat area in the live broadcast room. The receiver can provide their feedback information in the form of voice or text through these channels, and the feedback information may include positive evaluations, critical suggestions, or other forms of comments.

[0154] In order to process these feedback information, a preset deep learning scoring model can be used to infer these feedback information. This model can classify and map the collected feedback information to determine the recipient's acceptance of the voice-changing audio data. Specifically, the scoring model analyzes the emotional tendency in the text feedback through the feedback information, or identifies key words and intonation in the voice feedback, and then maps these analysis results to a predefined category to obtain the corresponding acceptance probability value, and determines this acceptance probability value as a representation of the degree of acceptance. Among them, positive feedback will correspond to a higher acceptance probability value, while negative feedback will correspond to a lower value.

[0155] The training of the scoring model requires a large amount of labeled data, which includes historical feedback information and its corresponding acceptance labels. Through training, the model can learn how to accurately extract emotions and intentions from feedback information and convert them into quantitative acceptance levels. In addition, the model can also optimize its performance by continuously learning from user feedback to more accurately reflect the user's true feelings.

[0156] In actual operation, the feedback information flow in the live broadcast room is monitored in real time and analyzed using the scoring model. Once enough feedback information is collected, the model will calculate a comprehensive acceptance probability value, which reflects the overall acceptance of the corresponding user to the current voice-changing audio data.

[0157] Step S5200: Detect whether the acceptance level is lower than a preset threshold. If it is lower than the preset threshold, reselect other character objects from the sound information template library, construct the other character objects and their speech audio segments into a re-voice change request and send it to the speaking user;

[0158] When the computer device detects that the acceptance level of the called user is lower than the preset threshold, it means that the voice changing effect fails to meet the expectations of the called user and needs to be adjusted. At this time, new character objects are selected from the sound information template library, which have timbre characteristics different from the previous target timbre characteristics, in order to provide a voice changing effect that better meets the preferences of the called user.

[0159] There are many specific implementation methods for selecting a new character object. For example, the computer device can automatically select one or more new character objects based on the historical feedback, preference settings, or specific content of the current feedback of the called user. In addition, by analyzing the feedback pattern of the called user on different timbres, a machine learning algorithm can be used to predict and select the timbre that is most likely to improve user satisfaction. A simpler method can also be to randomly determine a character object as a new character object.

[0160] After the new character object is determined, its template timbre features are used to construct a re-voice change request and sent to the speaking user. This request contains the speaking audio clip of the new character object, and the speaking user can preview the effect of the new timbre and decide whether to accept the voice change request.

[0161] Step S5300: When the speaking user confirms the re-voice changing request, the other character object is used as a new target object, and for the subsequently generated voice segment, the step of performing pitch changing processing on the segment pitch feature of the voice segment according to a preset target pitch value is restarted to perform voice changing processing on the subsequently generated voice segment.

[0162] After the speaking user confirms the request for re-voice change, the newly selected character object can be used as the target object, and its template timbre characteristics can be used as the target timbre characteristics to perform voice change processing on the subsequently generated voice segments. Specifically, the process from step S3200 to step S3400 can be performed on the subsequently generated voice segments, so that the subsequent voice segments can complete the voice change processing based on the target timbre characteristics of the new character object, in order to improve the satisfaction of the receiving user with the voice-changed audio data.

[0163] This embodiment significantly improves the interactivity and adaptability of voice changing processing technology by introducing a mechanism for collecting and analyzing feedback information from the receiving user, and solves the problem that the feelings of the receiving user are often ignored in traditional technologies. In traditional voice changing processing, usually only the needs of the speaking user are considered, and there is a lack of consideration of the acceptance level of the receiving user, which may lead to a deviation between the voice changing effect and the expectations of the receiving user. This embodiment can dynamically adjust the voice changing parameters by monitoring the feedback of the receiving user in real time, ensuring that the voice changing effect is more in line with the preferences of the receiving user, thereby improving the overall user experience.

[0164] By performing sentiment analysis and intent recognition on the feedback of the called user through a deep learning scoring model, this embodiment can quantify the acceptance of the called user and adjust the voice changing strategy based on this feedback. This dynamic adjustment mechanism based on feedback makes the voice changing process no longer a one-way process, but a two-way interactive process, in which the feedback of the called user is considered and responded to in real time. When the acceptance level of the called user is lower than the preset threshold, a new character object will be automatically selected from the sound information template library, and the option of re-changing the voice will be provided. This intelligent processing method improves the flexibility and personalization level of the voice changing process.

[0165] In addition, this embodiment also allows the speaking user to re-change his voice according to the feedback of the user being called. This user-involved voice change process increases the speaking user's sense of control and makes the voice change process more accurately meet the needs of both parties. In this way, not only the satisfaction of the user being called is improved, but also the participation of the speaking user is enhanced, thereby creating a more active and positive interactive atmosphere on the live broadcast platform.

[0166] It can be seen that the technical advantage of this embodiment is that it can realize voice changing processing driven by user feedback. This user-centric approach improves the actual application effect and user satisfaction of voice changing technology, and provides a richer and more personalized voice communication experience for the live broadcast platform.

[0167] On the basis of any embodiment of the method of the present application, after sending the voice-changed audio data to the called user in the live broadcast room, the method includes:

[0168] Step S7100: In response to a user configuration event, a voice change configuration interface is popped up in the live broadcast room, and voice change control parameters of the speaking user are received through the voice change configuration interface, wherein the voice change control parameters include target timbre characteristics of a target object selected by the speaking user from character objects in the sound information template library and a target pitch value set by the speaking user;

[0169] This embodiment allows the speaking user to trigger a user configuration event through the controls provided in the live broadcast room, and in response to the event, a voice change configuration interface pops up in the live broadcast room, such as Figure 4 As shown, the speaking user can further customize and optimize the voice changing effect.

[0170] The voice changer configuration interface provides an intuitive operating environment. The speaking user can play the speaking audio clips of each character object through the recommended list of timbre samples displayed on the interface to preview the timbre effect, and select the character object they want to imitate according to their personal preferences to determine its timbre information, that is, the target timbre characteristics. The speaking user can select a character object from the recommended list of timbre samples and determine it as the target object, thereby determining its corresponding target timbre characteristics.

[0171] In addition to selecting the target timbre characteristics, the user can also set a target pitch value in the voice changer configuration interface. This parameter allows the user to adjust the pitch of the changed voice to suit their personal preferences or the needs of a specific scenario. The target pitch value can be absolute, such as a specific number of hertz, or relative, such as a semitone higher or lower than the original voice.

[0172] After the user makes a selection and sets the parameters, the computer device will collect these voice change control parameters, specifically using the target object's identity and the preset target pitch value as voice change control parameters for subsequent voice processing. These parameters will guide the computer device on how to perform pitch change processing on the new voice segment to ensure that the voice change effect meets the user's expectations. For example, if the user selects a deep male voice as the target timbre and sets a lower target pitch value, the computer device will adjust the pitch of the subsequent voice segment accordingly to match this setting.

[0173] Step S7200, based on the target timbre features corresponding to the target object selected by the speaking user and the target pitch value set by the speaking user, for the subsequently generated voice segment, start again from the step of performing tone shifting processing on the segment pitch features of the voice segment according to the preset target pitch value, and perform voice shifting processing on the subsequently generated voice segment.

[0174] After the user selects the target timbre characteristics and sets the target pitch value through the voice change configuration interface, these parameters can be used to perform subsequent voice change processing. Specifically, the computer device can execute the process of steps S3200 to S3400 of this application for the subsequently generated voice fragments according to the target timbre characteristics selected by the user and the set target pitch value to generate corresponding voice-changed audio data, realize voice-changed processing for the user's subsequently generated voice fragments, and then send the generated voice-changed audio data back to the live broadcast room for the called user to play.

[0175] This embodiment significantly improves user experience and satisfaction by providing a user-friendly voice change configuration interface. The speaking user can select a specific character object according to his or her preferences and needs to determine the target timbre characteristics. This personalized selection not only increases the user's control over the voice change effect, but also makes the voice change result more in line with the personal style and expectations of the speaking user, improving the authenticity and naturalness of the voice change.

[0176] In addition, the speaking user can also set a target pitch value to further fine-tune the voice changing effect to make it closer to the user's voice characteristics or the needs of a specific scene. This flexibility allows users to adjust their voices in different live broadcast environments, whether for increased entertainment, privacy protection, or to better fit the live broadcast content. In this way, users are not only participants in the voice changing process, but also creators, able to adjust and optimize their voice changing effects in real time.

[0177] This mechanism of user customization and participation makes voice changing technology more flexible and responsive to user needs. It not only improves user satisfaction with voice changing technology, but also enhances user loyalty and activity to the live broadcast platform. Speaking users can express themselves more confidently while enjoying the unlimited possibilities brought by voice changing technology. In general, this embodiment achieves a more personalized and accurate voice changing effect by giving speaking users more control and choice, providing users with a new and interactive live broadcast experience.

[0178] See also Figure 5 According to one aspect of the present application, a voice changing device for a live broadcast room voice dialogue is provided, comprising a segment analysis module 3100, a pitch change processing module 3200, a voice change processing module 3300, and a segment update module 3400, wherein the segment analysis module 3100 is configured to respond to a voice speech event triggered by a speaking user in the live broadcast room, and detect and determine a speech segment in the target audio data corresponding to the event; the pitch change processing module 3200 is configured to perform pitch change processing on the segment pitch feature of the speech segment according to a preset target pitch value to obtain a tuned pitch feature; the voice change processing module 3300 is configured to perform voice change processing on the speech segment based on the tuned pitch feature and a target timbre feature to obtain a voice-changed segment, wherein the target timbre feature is a template timbre feature of a target object obtained by matching the timbre feature of the speaking user from a sound information template library; the segment update module 3400 is configured to replace the corresponding speech segment in the target audio data with the voice-changed segment to obtain voice-changed audio data, and send the voice-changed audio data to the receiving user in the live broadcast room.

[0179] Based on any embodiment of the device of the present application, the segment analysis module 3100 includes: an audio determination module, which is configured to respond to a voice speech event triggered by a speaking user in the live broadcast room, and obtain target audio data corresponding to the real-time speech of the speaking user; a noise reduction processing module, which is configured to perform audio noise reduction processing on the target audio data to obtain low-noise audio data with suppressed noise; and a detection and interception module, which is configured to perform human voice activity detection based on the low-noise audio data to determine a voice segment in which a human voice is speaking.

[0180] Based on any embodiment of the device of the present application, the pitch change processing module 3200 includes: a manual application module, configured to determine whether the speaking user has preset a pitch value, and when the pitch value is preset, use the pitch value as the target pitch value; a template application module, configured to use the preset pitch value corresponding to the target object determined by the speaking user from the character object in the sound information template library as the target pitch value when the speaking user has not preset a pitch value; and a pitch change execution module, configured to perform pitch change processing on the segment pitch feature of the speech segment according to the target pitch value, so that the segment pitch feature reaches the target pitch value.

[0181] On the basis of any embodiment of the device of the present application, prior to the voice changing processing module 3300, the device includes: a timbre extraction module, configured to obtain the speaking voice data of the speaking user, and extract the timbre features of the speaking user based on the speaking voice data, wherein the speaking voice data is the target audio data generated in real time, or the historical audio data pre-recorded by the speaking user; a semantic operation module, configured to calculate the timbre similarity between the timbre features of the speaking user and the template timbre features corresponding to each character object preset in the sound information template library; an object recommendation module, configured to select some character objects with relatively high timbre similarity, construct the some character objects and their corresponding speaking audio segments into a timbre sample recommendation list, and push it to the speaking user; a user selection module, configured to take the character object determined by the speaking user based on the speaking audio segment in the timbre sample recommendation list as the target object, and take the template timbre features of the target object as the target timbre features.

[0182] Based on any embodiment of the device of the present application, the voice changing processing module 3300 includes: a content extraction module, configured to extract features from the speech segment to obtain its content semantic features; a joint encoding module, configured to jointly encode the content semantic features, the tuned pitch features and the target timbre features to obtain voice changing audio features; a decoding generation module, configured to decode the voice changing audio features to generate corresponding audio data as a voice changing segment.

[0183] On the basis of any embodiment of the device of the present application, after the segment updating module 3400, the device includes: a receiver evaluation module, configured to obtain feedback information replied by the receiver after playing the voice-changed audio data, and determine the receiver's acceptance degree of the voice-changed audio data according to the feedback information; a reset recommendation module, configured to detect whether the acceptance degree is lower than a preset threshold, and when it is lower than the preset threshold, reselect other character objects from the sound information template library, and construct the other character objects and their speaking audio segments into a re-voice-changing request and send it to the speaking user; a tone change reset module, configured to use the other character objects as new target objects after the speaking user confirms the re-voice-changing request, and restart execution from the tone change processing module 3200 for the subsequently generated voice segments, and perform voice change processing for the subsequently generated voice segments.

[0184] On the basis of any embodiment of the device of the present application, after the segment update module 3400, the device includes: a user configuration module, configured to respond to a user configuration event, pop up a voice change configuration interface in the live broadcast room, and receive the voice change control parameters of the speaking user through the voice change configuration interface, wherein the voice change control parameters include the target timbre characteristics of the target object selected by the speaking user from the character objects in the sound information template library and the target pitch value set by the speaking user; a configuration application module, configured to restart execution from the tone change processing module 3200 for subsequently generated voice segments based on the target timbre characteristics corresponding to the target object selected by the speaking user and the target pitch value set by the speaking user, and perform voice change processing for the subsequently generated voice segments.

[0185] Another embodiment of the present application also provides a voice changing device for live broadcast room voice dialogue. Figure 6 As shown, a schematic diagram of the internal structure of a voice changing device for a live broadcast room voice conversation. The voice changing device for a live broadcast room voice conversation includes a processor, a computer-readable storage medium, a memory, and a network interface connected through a system bus. Among them, the computer-readable non-volatile readable storage medium of the voice changing device for a live broadcast room voice conversation stores an operating system, a database, and computer-readable instructions. The database may store an information sequence. When the computer-readable instructions are executed by the processor, the processor may implement a method for changing the voice of a live broadcast room voice conversation.

[0186] The processor of the live broadcast room voice conversation voice changing device is used to provide computing and control capabilities to support the operation of the entire live broadcast room voice conversation voice changing device. The memory of the live broadcast room voice conversation voice changing device may store computer-readable instructions, and when the computer-readable instructions are executed by the processor, the processor may execute the live broadcast room voice conversation voice changing method of the present application. The network interface of the live broadcast room voice conversation voice changing device is used to connect and communicate with a terminal.

[0187] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a partial structure related to the present application scheme, and does not constitute a limitation on the live broadcast room voice conversation voice changing device to which the present application scheme is applied. The specific live broadcast room voice conversation voice changing device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0188] In this embodiment, the processor is used to execute Figure 5 The memory stores the program code and various data required to execute the above modules or submodules. The network interface is used to realize data transmission between user terminals or servers. The non-volatile readable storage medium in this embodiment stores the program code and data required to execute all modules in the live broadcast room voice conversation voice changing device of this application, and the server can call the program code and data of the server to execute the functions of all modules.

[0189] The present application also provides a non-volatile readable storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the live broadcast room voice conversation voice changing method of any embodiment of the present application.

[0190] The present application also provides a computer program product, including a computer program / instruction, which implements the steps of the method described in any embodiment of the present application when executed by one or more processors.

[0191] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments of the present application can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a computer-readable storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0192] To sum up, this application significantly improves the performance of voice changing technology through innovations such as real-time response, precise pitch changing, precise timbre matching and efficient audio processing, solves the problems of real-time, personalized services, natural pitch adjustment and true timbre restoration in traditional technologies, and provides users with a more natural, more realistic and more personalized voice changing service.

Claims

1. A method for changing voice in live broadcast room voice dialogue, characterized in that: include: Responding to a voice speech event triggered by a user speaking in the live broadcast room, detecting and determining a voice segment in the target audio data corresponding to the event; According to a preset target pitch value, the segment pitch feature of the speech segment is subjected to pitch shift processing to obtain an optimized pitch feature; Based on the tuned pitch feature and the target timbre feature, the voice segment is subjected to voice change processing to obtain a voice change segment, wherein the target timbre feature is a template timbre feature of a target object obtained by matching the timbre feature of the speaking user from a sound information template library; The voice-changed segment replaces the corresponding voice segment in the target audio data to obtain voice-changed audio data, and the voice-changed audio data is sent to the recipient user in the live broadcast room.

2. The method for changing voice of live broadcast room voice dialogue according to claim 1, characterized in that: Responding to a voice sending event triggered by a user speaking in the live broadcast room, detecting and determining a voice segment in the target audio data corresponding to the event, including: Respond to the voice speech event triggered by the user in the live broadcast room, and obtain the target audio data corresponding to the real-time speech of the user; Performing audio noise reduction processing on the target audio data to obtain low-noise audio data with suppressed noise; Human voice activity detection is performed based on the low-noise audio data to determine a speech segment in which a human voice is speaking.

3. The method for changing voice of live broadcast room voice dialogue according to claim 1, characterized in that: According to a preset target pitch value, the segment pitch feature of the speech segment is subjected to pitch shifting processing to obtain an optimized pitch feature, including: Determining whether the speaking user has preset a pitch value, and if the user has preset a pitch value, using the pitch value as a target pitch value; When the speaking user does not preset a pitch value, taking the preset pitch value corresponding to the target object determined by the speaking user from the character objects in the sound information template library as the target pitch value; The segment pitch feature of the speech segment is subjected to pitch shifting processing according to the target pitch value, so that the segment pitch feature reaches the target pitch value.

4. The method for changing voice of live broadcast room voice dialogue according to claim 1, characterized in that: Based on the tuned pitch feature and the target timbre feature, before the voice segment is subjected to voice change processing, the method includes: Acquire speech voice data of the speaking user, and extract the timbre characteristics of the speaking user based on the speech voice data, wherein the speech voice data is the target audio data generated in real time, or the historical audio data pre-recorded by the speaking user; Calculating the timbre similarity between the timbre feature of the party and the timbre features of the templates corresponding to the various character objects preset in the sound information template library; Selecting some character objects with relatively high timbre similarity, constructing the some character objects and their corresponding speaking audio clips into a timbre sample recommendation list, and pushing it to the speaking user; The character object determined by the speaking user based on the speaking audio segment in the timbre sample recommendation list is used as the target object, and the template timbre feature of the target object is used as the target timbre feature.

5. The method for changing voice of live broadcast room voice dialogue according to claim 1, characterized in that: Based on the tuned pitch feature and the target timbre feature, the voice segment is subjected to voice change processing to obtain the voice change segment, including: Extracting features from the speech segment to obtain semantic features of its content; The content semantic feature, the tuned pitch feature and the target timbre feature are jointly encoded to obtain a voice-changed audio feature; Decoding is performed according to the voice-changing audio features to generate corresponding audio data as a voice-changing segment.

6. The method for changing voice of live broadcast room voice dialogue according to any one of claims 1 to 5, characterized in that: After sending the voice-changed audio data to the called user in the live broadcast room, the method includes: Acquire feedback information replied by the called user after playing the voice-changed audio data, and determine the acceptance degree of the called user to the voice-changed audio data according to the feedback information; Detecting whether the acceptance level is lower than a preset threshold, and when it is lower than the preset threshold, reselecting other character objects from the sound information template library, constructing the other character objects and their speaking audio segments into a re-voice change request and sending it to the speaking user; When the speaking user confirms the re-voice changing request, the other character object is taken as the new target object, and for the subsequently generated voice segment, the step of performing pitch changing processing on the segment pitch feature of the voice segment according to the preset target pitch value is restarted to perform voice changing processing on the subsequently generated voice segment.

7. The method for changing voice of live broadcast room voice dialogue according to any one of claims 1 to 5, characterized in that: After sending the voice-changed audio data to the called user in the live broadcast room, the method includes: In response to the user configuration event, a voice change configuration interface is popped up in the live broadcast room, and the voice change control parameters of the speaking user are received through the voice change configuration interface, wherein the voice change control parameters include the target timbre characteristics of the target object selected by the speaking user from the character objects in the sound information template library and the target pitch value set by the speaking user; According to the target timbre characteristics corresponding to the target object selected by the speaking user and the target pitch value set by the speaking user, for the subsequently generated speech fragments, the step of performing tone shifting processing on the segment pitch characteristics of the speech fragment according to the preset target pitch value is restarted to perform voice shifting processing on the subsequently generated speech fragments.

8. A voice changing device for live broadcast room voice dialogue, characterized in that: include: A segment analysis module is configured to respond to a speech event triggered by a user speaking in the live broadcast room, and detect and determine a speech segment in the target audio data corresponding to the event; A pitch change processing module is configured to perform pitch change processing on the segment pitch feature of the speech segment according to a preset target pitch value to obtain an optimized pitch feature; A voice change processing module is configured to perform voice change processing on the speech segment based on the tuned pitch feature and the target timbre feature to obtain a voice change segment, wherein the target timbre feature is a template timbre feature of a target object obtained by matching the timbre feature of the speaking user from a sound information template library; The segment updating module is configured to replace the corresponding voice segment in the target audio data with the voice-changed segment to obtain voice-changed audio data, and send the voice-changed audio data to the recipient user in the live broadcast room.

9. A voice changing device for live broadcast room voice dialogue, comprising a central processing unit and a memory, characterized in that: The central processing unit is used to call and run the computer program stored in the memory to execute the steps of the method according to any one of claims 1 to 7.

10. A non-volatile readable storage medium, characterized in that: It stores a computer program implemented according to the method described in any one of claims 1 to 7 in the form of computer-readable instructions, and when the computer program is called and executed by a computer, the steps included in the corresponding method are executed.

Images