Voice data processing method, device and electronic device in instant messaging application

Abstract

The present application provides a method, device and electronic device for processing voice data in an instant messaging application, which belongs to the field of voice processing technology. In an embodiment of the present application, the instant messaging application on the voice receiving side displays the received voice message; in response to the operation of playing the voice message, the playback speed of the voice message is adjusted according to the first noise intensity of the playback environment and then played, wherein the playback speed is negatively correlated with the first noise intensity; the instant messaging application on the voice sending side responds to the recording operation triggered by the target object, obtains the voice data collected by the microphone; and processes the voice data according to the second noise intensity of the recording environment and then sends it, wherein the playback speed is negatively correlated with the second noise intensity. Since the embodiment of the present application adjusts the speech speed according to the detected noise level, the greater the noise, the slower the playback speed of the adjusted voice data, and the user is no longer required to manually adjust the playback volume, thereby improving the flexibility of the playback control method of the voice message.

Description

Background Art

[0002] With the increasing development of information technology, the voice functions of various social apps have become more and more perfect. Now, no matter where you are, our daily online communication no longer requires typing. Information can be sent in the form of voice messages through instant messaging apps, making communication more and more convenient.

[0003] At present, when an instant messaging APP receives a voice message, the user clicks to play the received voice message, and reads the voice message from the storage unit and decodes it. After decoding, playable PCM audio data is obtained, and finally the player plays the PCM audio data, so that the user can hear the received voice message. During the process of playing a voice message by the instant messaging APP, if the current playback environment is noisy (for example, the playback environment is an outdoor environment with a lot of noise), the user may not be able to hear the played voice message clearly. At this time, the user needs to manually adjust the playback volume, and the playback control method during the voice message playback process is not flexible enough. Summary of the invention

[0004] The embodiment of the present application provides a method for processing voice data in an instant messaging application, so as to improve the flexibility of the playback control method during the playback process.

[0005] In a first aspect, an embodiment of the present application provides a method for processing voice data in an instant messaging application, comprising:

[0006] Display received voice messages;

[0007] In response to the operation of playing the voice message, the voice message is played after adjusting the playing speed according to the first noise intensity of the playing environment, and the playing speed is negatively correlated with the first noise intensity.

[0008] Optionally, the first noise intensity in the noisy speech data is detected according to the following method:

[0009] For each noisy speech data frame in the noisy speech data, performing noise detection on the noisy speech data frame to obtain a noise estimation parameter corresponding to the noisy speech data frame;

[0010] The first noise intensity is determined according to a noise estimation parameter corresponding to a noisy speech data frame in the noisy speech data.

[0011] Optionally, the noise estimation parameter corresponding to the noisy speech data frame is determined according to the following method:

[0012] Determine a target subband within a preset frequency spectrum corresponding to the noisy speech data frame;

[0013] The noise estimation parameter corresponding to the noisy speech data frame is determined according to the noise estimation value corresponding to each target subband, the number of frequency points contained in each target subband, and the noise estimation parameter corresponding to the last noisy speech data frame collected.

[0014] Optionally, the noise estimation value corresponding to each target subband is determined according to the following method:

[0015] Determine the power spectrum of the target subband according to the frequency domain complex value corresponding to each frequency point in the target subband and the number of frequency points contained in the target subband; wherein the frequency domain complex value corresponding to the frequency point is obtained by Fourier transforming the collected noisy speech data frame;

[0016] Determine the speech existence probability value corresponding to the target sub-band according to a smoothed power spectrum obtained by performing time-frequency domain smoothing processing on the power spectrum of the target sub-band and a minimum power spectrum in the power spectrum of the sub-band corresponding to the noisy speech data frame;

[0017] The noise estimation value corresponding to the target subband is determined according to the power spectrum corresponding to the target subband, the speech existence probability value corresponding to the target subband, and the noise estimation value corresponding to the target subband in the previous noisy speech data frame.

[0018] Optionally, before adjusting the playback speed of the voice message according to the first noise intensity of the playback environment, the method further includes:

[0019] Perform voice enhancement processing on the voice message.

[0020] In a second aspect, an embodiment of the present application provides a method for processing voice data in an instant messaging application, the method comprising:

[0021] Respond to the recording operation triggered by the target object and obtain the voice data collected by the microphone;

[0022] The voice data is processed according to a second noise intensity of a recording environment and then sent, so that a playback speed of the processed voice data is negatively correlated with the second noise intensity.

[0023] Optionally, after obtaining the voice data collected by the microphone and before processing the voice data according to the second noise intensity of the recording environment, the method further includes:

[0024] Determining whether the collected voice data meets a second preset condition;

[0025] The second preset condition includes at least one of the following conditions:

[0026] The account receiving the voice data is a preset type account;

[0027] The emotional state determined according to the voice features in the voice data is a preset emotional state, and the emotional state is obtained by performing emotional analysis on the voice features in the voice data;

[0028] The voice data contains preset key information, and the preset key information in the voice data is obtained by performing semantic analysis on the voice data.

[0029] Optionally, the second noise intensity in the noise data is detected according to the following method:

[0030] For each noise data frame in the noise data, performing noise detection on the noise data frame to obtain a noise estimation parameter corresponding to the noise data frame;

[0031] The second noise intensity is determined according to a noise estimation parameter corresponding to a noise data frame in the noise data.

[0032] Optionally, the noise estimation parameter corresponding to the noise data frame is determined according to the following method:

[0033] Determine a target subband within a preset frequency spectrum corresponding to the noise data frame;

[0034] The noise estimation parameter corresponding to the noise data frame is determined according to the noise estimation value corresponding to each target subband, the number of frequency points contained in each target subband, and the noise estimation parameter corresponding to the last noise data frame collected.

[0035] Optionally, the noise estimation value corresponding to each target subband is determined according to the following method:

[0036] Determine the power spectrum of the target subband according to the frequency domain complex value corresponding to each frequency point in the target subband and the number of frequency points contained in the target subband; wherein the frequency domain complex value corresponding to the frequency point is obtained by performing Fourier transform processing on the collected noise data frame;

[0037] Determine the speech existence probability value corresponding to the target sub-band according to a smoothed power spectrum obtained by performing time-frequency domain smoothing processing on the power spectrum of the target sub-band and a minimum power spectrum in the power spectrum of the sub-band corresponding to the noise data frame;

[0038] The noise estimation value corresponding to the target subband is determined according to the power spectrum corresponding to the target subband, the speech existence probability value corresponding to the target subband, and the noise estimation value corresponding to the target subband in the previous noisy data frame.

[0039] In a third aspect, an embodiment of the present application provides a voice data processing device in an instant messaging application, comprising:

[0040] A display unit, used for displaying received voice messages;

[0041] The first processing unit is used to respond to the operation of playing the voice message, and adjust the playing speed of the voice message according to the first noise intensity of the playing environment before playing, and the playing speed is negatively correlated with the first noise intensity.

[0042] In a fourth aspect, an embodiment of the present application provides a voice data processing device in an instant messaging application, including:

[0043] A collection unit, used to respond to the recording operation triggered by the target object and obtain the voice data collected by the microphone;

[0044] The second processing unit is used to process the voice data according to a second noise intensity of the recording environment and then send it, so that the playback speed of the processed voice data is negatively correlated with the second noise intensity.

[0045] In a fifth aspect, an embodiment of the present application provides an electronic device, including:

[0046] at least one processor; and

[0047] a memory communicatively connected to the at least one processor; wherein,

[0048] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the voice data processing method provided in the present application.

[0049] In a sixth aspect, an embodiment of the present application provides a computer-readable medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the voice data processing method provided in the present application.

[0050] Beneficial effects of this application:

[0051] Since the instant messaging application on the voice receiving terminal in the embodiment of the present application displays the received voice message; in response to the operation of playing the voice message triggered by the target object, the instant messaging application on the voice receiving terminal adjusts the playback speed of the voice message according to the first noise intensity of the playback environment and then plays the voice receiving end; this adjustment of the playback speed of the voice message according to the first noise intensity of the playback environment makes its playback control method more flexible; in addition, the greater the noise in the user's environment, the slower the adjusted playback speed, making it easier for the user to hear the voice content clearly. At the same time, in the embodiment of the present application, in response to the recording operation triggered by the target object, the voice data collected by the microphone is obtained; the voice data is processed and sent according to the second noise intensity of the recording environment. Since in the embodiment of the present application, the noise intensity is negatively correlated with the playback speed, at the voice sending end, the greater the noise in the user's environment, the slower the adjusted recording speed, and the user on the voice receiving side can directly click to play the voice message, and automatically adjust the playback speed of the voice message according to the environmental noise, making instant messaging more convenient. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A schematic diagram of an optional application scenario illustrative of an embodiment of the present application;

[0053] Figure 2 A flow chart of voice data playback in an instant messaging application provided in an embodiment of the present application;

[0054] Figure 3 A flow chart of a method for processing voice data in an instant messaging application provided by an embodiment of the present application;

[0055] Figure 4 A display interface for receiving voice message reminders for instant messaging provided in an embodiment of the present application;

[0056] Figure 5 A chat display interface showing a received voice message in an instant messaging APP provided in an embodiment of the present application;

[0057] Figure 6 A diagram showing a display interface for prompting information for voice speed adjustment provided in an embodiment of the present application;

[0058] Figure 7 A display interface diagram for specifying the adjustment of speech speed provided in an embodiment of the present application;

[0059] Figure 8 A schematic diagram of the duration adjustment of the WSOLA algorithm provided in an embodiment of the present application;

[0060] Fig. 9 A flow chart of a method for processing voice data in an instant messaging application provided in an embodiment of the present application;

[0061] Fig.10 A flow chart of a method for processing voice data in an instant messaging application provided by an embodiment of the present application;

[0062] Fig.11 A chat display interface of an instant messaging application provided in an embodiment of the present application;

[0063] Fig.12 The display interface of the instant messaging application during the recording process provided by the embodiment of the present application;

[0064] Fig.13 A flow chart of a method for processing voice data in an instant messaging application provided in an embodiment of the present application;

[0065] Fig.14 A schematic diagram of the structure of a voice data processing device in an instant messaging application provided by an embodiment of the present application;

[0066] Fig.15 A schematic diagram of the structure of another voice data processing device in an instant messaging application provided by an embodiment of the present application;

[0067] Fig.16 It is a schematic diagram of the structure of an electronic device in an embodiment of the present application;

[0068] Fig.17 Schematic diagram of the structure of a computing device in an embodiment of the present application. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and beneficial effects of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0070] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0071] The following are explanations of some of the terms that appear in this article:

[0072] 1. Voice messages: Voice messages are the most basic form of communication and are widely used in various social apps. People who walk, the elderly who type slowly, and children who cannot type can all easily send voice messages and communicate with others through voice messages, which greatly facilitates people's online communication.

[0073] 2. Power spectrum: Power spectrum is the abbreviation of power spectrum density function, which is defined as the signal power within a unit frequency band. It shows the change of signal power with frequency, that is, the distribution of signal power in the frequency domain.

[0074] 3. Noise estimation: Noise estimation has become a critical link in speech enhancement technology. In many single-channel speech enhancement algorithms, especially when the noise source is unknown, real-time noise power spectrum estimation is very important. The accuracy of noise estimation will directly affect the final effect: if the noise estimation is too high, weak speech will be removed and the enhanced speech will be distorted; if the estimation is too low, there will be more background residual noise.

[0075] 4. Terminal: Also known as user equipment (UE), mobile station (MS), mobile terminal (MT), etc., it is a device that provides voice and / or data connectivity to users, for example, handheld devices with wireless connection functions, vehicle-mounted devices, etc. At present, some examples of terminals are: mobile phones, tablet computers, laptops, PDAs, mobile internet devices (MID).

[0076] 5. Client can refer to both software APP (Application) and terminal equipment. It has a visual display interface and can interact with users; it corresponds to the server and provides local services to customers. For software applications, except for some applications that only run locally, they are generally installed on ordinary client terminals and need to cooperate with the server to run. After the development of the Internet, more commonly used applications include email clients for sending and receiving emails, and instant messaging clients. For this type of application, there needs to be corresponding servers and service programs in the network to provide corresponding services, such as database services, configuration parameter services, etc. In this way, specific communication connections need to be established between the client terminal and the server to ensure the normal operation of the application.

[0077] The following is a brief introduction to the design concept of the embodiment of the present application:

[0078] The key technologies of speech technology (ST) in the computer field include automatic speech recognition technology (ASR) and speech synthesis technology (TTS), as well as voiceprint recognition and other technologies. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become the most promising human-computer interaction method in the future, and speech has more advantages than other interaction methods.

[0079] With the continuous development of voice technology, instant messaging (IM) operations are being performed in more and more scenarios. It refers to services that can instantly send and receive Internet messages. With the continuous development of instant messaging technology, its functions have become increasingly complete, and it has developed into a comprehensive information platform that integrates communication, information, entertainment, search, e-commerce, office collaboration and corporate customer service. Some major instant messaging providers provide services for accessing the Internet through mobile phones. Users can send and receive messages through their mobile phones and other mobile phones or computers that have installed the corresponding client software.

[0080] With the communication between people, text messages can no longer meet the needs of users. The form of messages has changed from text to voice. The existence of voice messages makes it very convenient for users when typing is inconvenient. However, when the user is in a noisy external environment, the voice sent or received by the user is mixed with noisy external environmental sounds, which will greatly affect the user's auditory comprehension ability. The user needs to pay more attention to hear the content of the voice message clearly. At present, when the noise in the playback environment is large, the user can only manually adjust the playback volume, and if the noise in the environment is too loud, even if the maximum volume is used for playback, the user may not be able to hear the content of the voice message clearly. Therefore, the current playback control method of manually adjusting the playback volume by the user is not flexible enough.

[0081] In view of this, the embodiment of the present application provides a method, device and electronic device for processing voice data in an instant messaging application. The voice data processing method provided by the embodiment of the present application, the instant messaging application on the voice receiving end displays the received voice message; responds to the operation of playing the voice message; collects the noisy voice data in the playback environment through the microphone, and performs noise detection on the noisy voice data to obtain the first noise intensity of the playback environment; and plays the voice message after adjusting the playback speed according to the first noise intensity of the playback environment. Because in the embodiment of the present application, after receiving the voice message, when playing the received voice message, the playback speed of the voice message can be adjusted according to the first noise intensity in the playback environment, and the first noise intensity is negatively correlated with the playback speed, that is, at the voice receiving end, the greater the first noise intensity in the playback environment, the slower the playback speed of the adjusted voice message; so that when the noise in the playback environment is large, the playback speed of the voice message is slowed down, the user can more easily hear the content of the voice message, better understand the content of the voice message, and no longer need the user to manually adjust the playback volume, thereby improving the flexibility of the playback control method of the voice message.

[0082] After introducing the design ideas of the embodiments of the present application, the following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0083] like Figure 1 As shown, it is a schematic diagram of an optional application scenario illustrative of an embodiment of the present application, including a first object 10 on the voice sending side, a voice sending terminal 11, a cloud server 12, a voice receiving terminal 13, and a second object 14 on the voice receiving side.

[0084] The voice sending terminal 11 is installed with an instant messaging application, and the first object 10 needs to send a voice message to the second object 14 through the instant messaging application on the voice sending terminal 11; the instant messaging application on the voice sending terminal 11 responds to the recording operation triggered by the first object 10, and collects voice data through the microphone on the voice sending terminal 11; the first object 10 determines the instant messaging application on the voice receiving terminal 13 through the triggered recording operation; the voice sending terminal 11 establishes a connection with the cloud server 12 through the access point, and the instant messaging application on the voice sending terminal 11 communicates with the cloud server 12; the instant messaging application on the voice sending terminal 11 uploads the voice data collected by the microphone on the voice sending terminal 11 to the cloud server 12.

[0085] The cloud server of the embodiment of the present application may be an independent server, or may be a server cluster composed of multiple servers; the cloud server 12 may include a server corresponding to an instant messaging application, and may also include a message push server;

[0086] The voice receiving terminal 13 establishes a connection with the cloud server 12 through the access point, and the instant messaging application on the voice receiving terminal 13 communicates with the cloud server 12; the cloud server 12 pushes the voice data uploaded by the instant messaging application on the voice sending terminal 11 to the instant messaging application on the voice receiving terminal 13; the instant messaging application on the voice receiving terminal 13 displays the received voice message; in response to the operation of playing the voice message triggered by the second object 14, the instant messaging application on the voice receiving terminal 13 adjusts the playback speed of the voice message according to the first noise intensity of the playback environment and then plays it.

[0087] In combination with the application scenarios described above, refer to Figure 2-Figure 11 To describe a method for processing voice data in an instant messaging application provided by an exemplary embodiment of the present application. It should be noted that the above application scenario is only shown to facilitate understanding of the spirit and principle of the present application, and the implementation of the present application is not limited in this respect. On the contrary, the implementation of the present application can be applied to any applicable scenario.

[0088] like Figure 2 As shown, it is a flow chart of voice data playback in an instant messaging application provided by an embodiment of the present application. In this embodiment, the user receives a voice message and clicks on the voice message to be listened to. The voice message is read from the storage and decoded. After decoding, linear PCM audio data is obtained. Then, based on the background environmental noise collection and detection analysis on the playback side, the playback speed is configured according to the detection noise size. When the detection noise becomes larger, the playback signal speed is gradually reduced to make it easier for the user to listen to and understand the played audio content. As the environmental noise becomes smaller, the speed will gradually return to the original playback speed.

[0089] It should be noted that the adjusted voice message can be played through a loudspeaker, or can also be played through an external audio playback device (headphones, etc.).

[0090] like Figure 3 As shown, it is a flow chart of a method for processing voice data in an instant messaging application provided by an embodiment of the present application. The method may include the following steps:

[0091] Step S301, displaying the received voice message;

[0092] For example, a voice message may be a voice message received in an instant messaging app. When a user opens the instant messaging app, he or she may see voice message reminders sent by friends on the home page. For example, a voice message may be one or more messages, such as Figure 4 As shown in FIG. 1 , it is a display interface for receiving voice message reminders for instant messaging provided by an embodiment of the present application. After the user clicks on the voice message to enter the chat interface, the received voice message is displayed in the chat interface of the instant messaging APP, so that the user can see the detailed content of the received voice message. Assuming that the voice message is 16 seconds, as shown in FIG. Figure 5 As shown, it is a chat display interface showing received voice messages in the instant messaging APP provided in an embodiment of the present application.

[0093] Step S302: in response to the operation of playing the voice message, the voice message is played after adjusting the playing speed according to the first noise intensity of the playing environment, and the playing speed is negatively correlated with the first noise intensity.

[0094] Among them, after the instant messaging application receives and displays the voice message, the user can click on the displayed voice message to trigger the voice playback operation.

[0095] After receiving a voice message, an optional implementation of the embodiment of the present application is to perform voice enhancement processing on the received voice message;

[0096] Since voice messages are sent by the sending user through a microphone to collect voice data, the collected voice data is sent to the receiving user. However, if there is noise in the recording environment where the sending user is located, the voice data collected by the microphone contains background noise. After the voice data containing background noise is sent to the receiving user, if the background noise is large, the receiving user may not be able to hear the voice of the sending user clearly.

[0097] Based on this problem, after receiving the voice message, the embodiment of the present application performs voice enhancement processing (or noise reduction processing) on ​​the received voice message; during implementation, a pure voice signal related to the sending user is extracted from the voice message containing background noise, thereby effectively suppressing or reducing the interference of background noise.

[0098] In the embodiment of the present application, in response to the operation of playing a voice message, the noisy voice data in the playing environment is collected by the microphone, and the instant messaging application determines the first noise intensity of the playing environment according to the noisy voice data collected by the microphone;

[0099] Alternatively, another optional implementation is that after determining that the voice message meets the first preset condition, the embodiment of the present application collects noisy voice data in the playback environment through a microphone, and the instant messaging application determines the first noise intensity of the playback environment based on the noisy voice data collected by the microphone;

[0100] The first preset condition includes at least one of the following conditions:

[0101] Condition 1: The account sending the voice message is a preset type account.

[0102] It should be noted that the instant messaging application of the embodiment of the present application can pre-set the account type, and after receiving the voice message, determine the account that sent the voice message, and judge whether the account that sent the voice message is a preset type account. If so, trigger the microphone to collect noisy voice data in the playback environment;

[0103] Among them, the preset type account can be an account that has a preset social relationship with the account receiving the voice message; for example, the preset type account can be set as an account of special attention for the account receiving the voice message.

[0104] Alternatively, the preset type account may also be an account with a preset label set for the account receiving the voice message; for example, the preset type account may be an account with a label such as family member, customer, etc. set for the account receiving the voice message.

[0105] When the account sending the voice message is a preset type account, for example, the preset type account is set as an account of special concern for the account receiving the voice message, or the preset type account is an account with labels such as family members and customers set for the account receiving the voice message, it means that the relationship between the account sending the voice message and the account receiving the voice message is relatively close, or that the account sending the voice message is relatively important to the account receiving the voice message. At this time, it is necessary to collect noisy voice data in the playback environment and adjust the playback speed of the voice message according to the first noise intensity in the playback environment, so that the playback speed of the voice message is more reasonable, so that the user who receives the voice message can better understand the content of the voice message when playing the voice message, thereby improving the intelligibility of the voice message.

[0106] Condition 2: The emotional state determined according to the voice features in the voice message is the preset emotional state, and the emotional state is obtained by performing emotional analysis on the voice features in the voice message.

[0107] It should be noted that the emotional state determined based on the voice features in the voice message is the emotional state of the user who sends the voice message.

[0108] After receiving a voice message, the embodiment of the present application can perform emotion analysis on the voice features in the voice message, and determine the current emotional state of the user who sent the voice message according to the voice features in the voice message;

[0109] An optional implementation is to perform sentiment analysis on voice features in voice messages to obtain the emotional state through machine learning.

[0110] For example, the preset emotional state may be an abnormal emotional state such as crying, sadness, excitement, etc.;

[0111] When the emotional state of the user sending the voice message is a preset emotional state, such as crying, sadness, excitement and other abnormal emotional states, the sent voice message may be intermittent, making it difficult for the receiving user to hear the voice message clearly; and when the sending user of the voice message is in an abnormal emotional state, the receiving user of the voice message needs to hear the content of the voice message as clearly as possible to understand the current state of the sending user of the voice message. Therefore, at this time, the noisy voice data in the playback environment is collected, and the playback speed of the voice message is adjusted according to the first noise intensity in the playback environment, so that the playback speed of the voice message is more reasonable, which is convenient for the receiving user of the voice message to better understand the content of the voice message and improve the intelligibility of the voice message.

[0112] Condition 3: The voice message contains preset key information, and the preset key information in the voice message is obtained by performing semantic analysis on the voice message.

[0113] It should be noted that when the preset key information is included in the voice message, it indicates that the voice message is an important voice message, or it may also indicate that the voice message is a relatively special voice message;

[0114] For example, the preset key information may be key information of the address type such as "Dad" or "Mom". When the key information of the address type such as "Dad" or "Mom" is included in the voice message, it indicates that the voice message is a voice message sent by the child to the parents, and the voice message is a relatively important voice message.

[0115] Alternatively, the preset key information may be key information related to money, such as "money", "transfer", "remittance", etc. When the preset key information is included in the voice message, it indicates that the voice message is a relatively important voice message;

[0116] Alternatively, the preset key information may be key information with special meanings such as "birthday" or "holiday". When the preset key information is included in the voice message, it indicates that the voice message is a special voice message.

[0117] Since the preset key information is contained in the voice message, it means that the voice message is a relatively important or special voice message, and the recipient of the voice message needs to hear the voice message clearly. Therefore, at this time, the noisy voice data in the playback environment is collected, and the playback speed of the voice message is adjusted according to the first noise intensity in the playback environment, so that the playback speed of the voice message is more reasonable, which is convenient for the recipient of the voice message to better understand the content of the voice message and improve the intelligibility of the voice message.

[0118] After obtaining the noisy speech data collected by the microphone, the embodiment of the present application obtains the first noise intensity of the playback environment by performing noise detection on the collected noisy speech data.

[0119] Among them, the embodiment of the present application can detect the first noise intensity in the environment in real time, or, when receiving an operation to play a voice message, detect the noisy voice data collected in the current playback environment to obtain the first noise intensity.

[0120] In implementation, the embodiment of the present application can detect noisy speech data to determine the first noise intensity of the playback environment in the following manner:

[0121] For each noisy speech data frame in the noisy speech data, perform noise detection on the noisy speech data frame to obtain a noise estimation parameter corresponding to the noisy speech data frame; determine a first noise intensity according to the noise estimation parameter corresponding to the noisy speech data frame in the noisy speech data;

[0122] In the embodiment of the present application, when detecting the noisy speech data to determine the first noise intensity of the playback environment, noise estimation parameters are obtained by performing noise detection on each noisy data frame in the noisy speech data;

[0123] In implementation, the embodiment of the present application can determine the noise estimation parameter corresponding to the noisy speech data frame according to the following method:

[0124] Determine a target subband within a preset frequency spectrum corresponding to the noisy speech data frame; determine a noise estimation parameter corresponding to the noisy speech data frame according to a noise estimation value corresponding to each target subband, the number of frequency points contained in each target subband, and a noise estimation parameter corresponding to a previous noisy speech data frame collected.

[0125] It should be noted that the preset frequency spectrum range may be a main frequency spectrum range of human voice.

[0126] After determining the target subband within the preset spectrum range corresponding to the noisy speech data frame, since it is necessary to determine the noise estimation parameter corresponding to the noisy speech data frame, the embodiment of the present application needs to determine the noise estimation value corresponding to each target subband of the noisy speech data frame.

[0127] An optional implementation manner is that the embodiment of the present application determines the noise estimation value corresponding to each target subband according to the following method;

[0128] Determine the power spectrum of the target subband according to the frequency domain complex value corresponding to each frequency point in the target subband and the number of frequency points contained in the target subband;

[0129] Specifically, the collected noisy speech data frame is Fourier transformed to obtain the frequency domain complex value corresponding to each frequency point in the frequency domain, the absolute value of the obtained frequency domain complex value is taken, the absolute value of the frequency domain complex value corresponding to all frequency points in the target sub-band is squared and summed, then the number of frequency points in the target sub-band is calculated, and finally the average value is calculated to obtain the sub-band power spectrum;

[0130] For example, the embodiment of the present application may calculate the sub-band power spectrum according to the following formula:

[0131]

[0132] Where k = 1, 2, 3, ..., N, i corresponds to the frame number, z corresponds to the frequency index value, and k corresponds to the subband number. X(i, z) is the frequency domain complex value of the zth frequency point in the i-th frame after Fourier transformation; freq1(k) is the starting frequency index value of the kth subband, and freq2(k) is the ending frequency index value of the kth subband.

[0133] In implementation, the embodiment of the present application may use the MCRA algorithm to determine the noise estimation parameters of the noisy speech data frame;

[0134] Specifically, after determining the power spectrum of the target subband, the subband power spectrum is first smoothed. The smoothing here includes frequency domain smoothing of adjacent subbands and time domain smoothing of historical frames. Specifically, the frequency domain smoothing can be performed according to the following formula:

[0135]

[0136] Wherein, x(j+w) is a frequency domain smoothing weighting factor group, for example, x[5]=[0.1, 0.2, 0.4, 0.2, 0.1];

[0137] Time domain smoothing is performed according to the following formula:

[0138]

[0139] Where C0 is the time domain smoothing factor, for example, C0 = 0.9;

[0140] According to the smoothed power spectrum data, a minimum value tracking method is used to obtain the minimum power spectrum in the power spectrum of the sub-band corresponding to the noisy speech data frame, and a preliminary estimation is made on the noisy speech data frame;

[0141] Specifically, the embodiment of the present application can obtain the minimum power spectrum in the power spectrum of the sub-band corresponding to the noisy speech data frame according to the following judgment process:

[0142]

[0143]

[0144] Wherein, T is a constant, representing the update period of the noise estimation value.

[0145] It should be noted that during speech activity, the noisy speech power of a single frequency band will usually decay to the power level of the noise, which is the starting point of the minimum tracking noise estimation algorithm. By tracking the minimum value of the noisy speech power in each frequency band, a rough estimate of the noise level in that frequency band can be obtained.

[0146] In implementation, the speech existence probability value corresponding to the target sub-band is determined according to the minimum power spectrum in the power spectrum of the sub-band corresponding to the noisy speech data frame and the power spectrum after time domain smoothing;

[0147] Specifically, the embodiment of the present application may adopt the following judgment process to determine the speech existence probability value corresponding to the target sub-band:

[0148]

[0149]

[0150]

[0151] In implementation, the noise estimation value corresponding to the target subband is determined according to the power spectrum corresponding to the target subband obtained above, the speech existence probability value corresponding to the target subband, and the noise estimation value corresponding to the target subband in the previous noisy speech data frame;

[0152] Specifically, the embodiment of the present application can determine the noise estimation value corresponding to the target subband according to the following formula:

[0153]

[0154] According to the above formula, it can be determined that the noise estimation value can be obtained by weighting the noise power spectrum under the condition of no voice and the noise power spectrum under the condition of voice presence respectively by the conditional probability that no voice exists and the conditional probability that voice exists in the noisy speech data frame in subband k, and then summing them up;

[0155] When speech exists in subband k, the mean value of the noise power spectrum under the condition of speech existence can be approximated as the noise estimation value of the previous frame.

[0156] According to the noise estimation value corresponding to each target subband obtained above, the noise estimation parameter of the i-th frame signal is determined by recursive averaging:

[0157]

[0158] Where β is the noise estimation smoothing coefficient;

[0159] According to the noise estimation value corresponding to each target sub-band and the number of frequency points contained in each target sub-band, all sub-bands within the preset spectrum range are summed. In addition, the noise estimation parameters corresponding to the previous noisy speech data frame need to be referred to; both parts of the data need to be multiplied by the smoothing coefficient for noise smoothing processing to make the resulting speech signal more continuous.

[0160] In implementation, after determining the noise estimation parameter corresponding to the noisy speech data frame, the embodiment of the present application can determine the first noise intensity in the noisy speech data according to the noise estimation parameter;

[0161] It should be noted that the first noise intensity may be the noise estimation parameter of the last noisy speech data frame in the noisy speech data; or the first noise intensity may be the average value of the noise estimation parameters of all noisy speech data frames in the noisy speech data.

[0162] During implementation, an optional implementation manner is that after determining the first noise intensity of the playback environment, the embodiment of the present application compares the first noise intensity with a preset noise threshold; when the first noise intensity is less than or equal to the noise threshold, the voice message is played directly; when the first noise intensity is greater than the preset noise threshold, the playback speed of the voice message is adjusted.

[0163] When adjusting the playback speed of a voice message in an embodiment of the present application, one optional method is to automatically adjust the playback speed according to the first noise intensity, and another optional method is to adjust the playback speed of the voice message according to the playback speed selected by the user.

[0164] The following describes these two speech speed adjustment methods respectively.

[0165] Method 1: Adjust the playback speed of the voice message according to the playback speed selected by the user.

[0166] In the embodiment of the present application, after determining that the first noise intensity is greater than a preset noise threshold, the user can manually select the playback speed of the voice message.

[0167] During implementation, when the first noise intensity is greater than a preset noise threshold, a prompt message for adjusting the playback speed is displayed in the display interface of the instant messaging application; the user can trigger a confirmation operation for adjusting the playback speed based on the prompt message; after determining that the user has triggered a confirmation operation for adjusting the playback speed, the instant messaging application adjusts the playback speed of the voice message.

[0168] For example, after receiving a voice message, the user can click on the received voice message in the chat interface of the instant messaging application to trigger the operation of playing the voice message; after the first noise intensity is greater than a preset threshold, such as Figure 6 As shown, a prompt box is displayed in the chat interface to prompt the user whether to adjust the playback speed of the voice message;

[0169] Assume that the user selects "yes" to confirm the adjustment of the playback speed of the voice message, then a slider for the user to select the playback speed is displayed in the chat interface, such as Figure 7 As shown; suppose that the user selects a playback speed of 0.8x (the normal speed is generally 1x), that is, the adjusted playback speed is 0.8 times the normal playback speed, and the speed becomes slower.

[0170] The embodiment of the present application determines a first target playback speed specified by a confirmation operation based on a confirmation operation of the user, and determines a first target compression and expansion coefficient corresponding to the voice message based on the first target playback speed; and expands the playback speed of the voice message based on the first target compression and expansion coefficient to adjust the playback speed of the voice message.

[0171] Among them, the embodiment of the present application can determine the first target compression and expansion coefficient corresponding to the first target playback speech rate according to the mapping relationship between the preset target playback speech rate and the compression and expansion coefficient.

[0172] Method 2: Automatically adjust the playback speed according to the first noise intensity.

[0173] In implementation, the embodiment of the present application may determine the first target compression expansion coefficient corresponding to the voice message according to the following method;

[0174] The voice message is divided into frames according to a preset frame spacing to obtain multiple voice data frames; for each voice data frame, a first reference compression expansion coefficient corresponding to the voice data frame is determined according to the difference between the first noise intensity and the noise threshold; the first reference compression expansion coefficient and the first target compression expansion coefficient corresponding to the previous voice data frame are weightedly summed to obtain the first target compression expansion coefficient corresponding to the voice data frame.

[0175] It should be noted that, when adjusting the speech rate of a voice message in the embodiment of the present application, the voice message needs to be divided into frames, and each frame needs to be adjusted separately. The speech rate can be controlled by adjusting the target compression expansion coefficient in the embodiment of the present application. However, if the coefficient is adjusted arbitrarily, the sound will be unnatural and uncomfortable to listen to. Therefore, in order to avoid sudden changes in the coefficient, the target compression expansion coefficient of the previous frame needs to be used to smooth the target compression expansion coefficient of the current frame.

[0176] Specifically, the embodiment of the present application can adjust the first target compression expansion coefficient according to the following α adjustment method:

[0177] 1. Determine the first reference compression expansion coefficient α target ;

[0178]

[0179] Where, f(*) is a monotonically increasing function, C is the preset noise threshold, is the first noise intensity of the playback environment;

[0180] The first noise intensity is compared with a preset noise threshold. If the first noise intensity is greater than the preset noise threshold, the first reference compression expansion coefficient α is adjusted. target Become smaller.

[0181] 2. According to the first reference compression expansion coefficient α target , determine a first target compression expansion coefficient;

[0182] For example, the first target compression expansion coefficient α(i) of the i-th voice data frame can be calculated according to the following formula:

[0183] α(i)=b*α(i-1)+(1-b)*α target (i)

[0184] Where b is a coefficient less than 1;

[0185] The first target compression expansion coefficient α determined according to the above formula is smoothed according to the first target compression expansion coefficient of the previous voice data frame to avoid sudden changes in the voice signal, so that the playback sound can be made more natural while controlling the speaking speed.

[0186] In implementation, after determining the first target compression expansion coefficient, the playback speed of the voice message can be expanded according to the following method:

[0187] For each voice data frame, the time domain of the voice data frame is expanded according to the target compression expansion coefficient corresponding to the voice data frame, and a reference voice data frame is determined from the search space corresponding to the voice data frame; the expanded voice data frame is superimposed with the reference voice data frame to obtain a target voice data frame for playing;

[0188] It should be noted that the time domain length of the reference voice data frame is the same as the time domain length of the voice data frame after the expansion process.

[0189] Specifically, the embodiment of the present application can adjust the playback speed of the voice data to be played according to the WSOLA algorithm, such as Figure 8 The figure shows a schematic diagram of the duration adjustment of the WSOLA algorithm provided in an embodiment of the present application:

[0190] 1) Processing the voice data to be played by dividing the frames according to a preset frame spacing, for example, the frame spacing may be L;

[0191] It should be noted that, in order to ensure the continuity of the synthesized signal, there will be overlapping parts between adjacent data frames;

[0192] 2) Select any voice data frame to be played after the frame processing, and perform windowing processing by Hanning window;

[0193] For example, any frame of voice data to be played here can select the kth frame of voice data to be played;

[0194] 3) The frame shift between the kth frame of the voice data to be played and the k+1th frame of the adjacent next frame of the voice data is S a ;

[0195] 4) Adjust the frame shift distance between two adjacent frames by using the target compression expansion coefficient α, S s =S a *α, that is, the kth frame of the voice data to be played is expanded in time domain, and the expanded voice data frame is obtained, and the expanded voice data frame is windowed by Hanning window;

[0196] The voice data frames after expansion processing are the K'th frame, the (K+1)'th frame, etc., corresponding to Figure 8 The time domain length of the expanded speech data frame is consistent with the kth frame of speech data to be played; when α>1, the speech is compressed and the speech speed becomes faster; when α<1, the speech is expanded and the speech speed becomes slower;

[0197] 5) Determine the voice data with the highest waveform similarity to the (K+1)'th voice data frame after expansion from the search space corresponding to the kth voice data frame to be played as the reference voice data frame, corresponding to Figure 8 The data frame D in is processed by Hanning window.

[0198] The time domain length of the reference speech data frame is consistent with the speech data frame after expansion processing; the search space is within the maximum tolerance interval ([-Δmax, Δmax]) at the KL of the data frame to be processed, corresponding to Figure 8 The space C in

[0199] 6) The reference speech data frame is superimposed with the k-th speech data frame of the speech data to be played, and a Hanning window is performed to obtain a target language data frame.

[0200] It should be noted that the adjusted playback speed of the voice message is negatively correlated with the first noise intensity.

[0201] like Fig. 9 As shown, it is a flow chart of a voice data processing method in an instant messaging application provided by an embodiment of the present application. In the following description, the instant messaging application on the voice sending terminal is referred to as a first instant messaging application, and the instant messaging application on the voice receiving terminal is referred to as a second instant messaging application, including the following steps:

[0202] Step S901: In response to a recording operation triggered by a first object, the first instant messaging application collects voice data through a microphone;

[0203] Step S902: The first instant messaging application sends the collected voice data;

[0204] Step S903: The second instant messaging application displays the received voice message;

[0205] Step S904: In response to the operation of playing the voice data, the second instant messaging application determines a first noise intensity in the playing environment;

[0206] Step S905: The second instant messaging application determines that the first noise intensity is greater than a preset noise threshold;

[0207] Step S906: The second instant messaging application divides the voice message into frames according to a preset frame spacing to obtain a plurality of voice data frames;

[0208] Step S907: The second instant messaging application determines a first reference compression expansion coefficient corresponding to the voice data frame according to the difference between the first noise intensity and the noise threshold;

[0209] Step S908: The second instant messaging application performs weighted sum processing on the first reference compression and expansion coefficient and the first target compression and expansion coefficient corresponding to the previous voice data frame to obtain the first target compression and expansion coefficient corresponding to the voice data frame;

[0210] Step S909: the second instant messaging application performs expansion processing on the playback speed of the voice message according to the first target compression expansion coefficient to obtain a target voice data frame;

[0211] Step S910: The second instant messaging application plays the expanded voice message.

[0212] The voice data processing method in the above embodiment is that when the noise in the playback environment is too loud, the voice receiving side adjusts the playback speed. When the first noise intensity in the playback environment is greater than the preset noise threshold, the playback speed of the voice message is automatically adjusted. The user no longer needs to manually adjust the playback volume. The user can clearly hear the voice message and quickly understand the content of the voice message. Similarly, in an instant messaging application, when the user at the voice sending end is in a noisy environment, after the recording operation is triggered, the voice data obtained by the recording will contain the noise in the recording environment. After the voice data containing the noise is sent to the receiving side instant messaging application, the receiving side user may not be able to hear the voice data of the sending side user when playing the voice message.

[0213] In view of this, the voice data processing method provided in the embodiment of the present application responds to the recording operation triggered by the target object, obtains the voice data collected by the microphone; the voice data is processed and sent according to the second noise intensity of the recording environment, so that the playback speed of the processed voice data is negatively correlated with the second noise intensity. Because in the embodiment of the present application, after the sending side instant messaging application collects the voice data through the microphone, the playback speed corresponding to the voice data obtained by the recording can be adjusted according to the second noise intensity in the recording environment, and the second noise intensity is negatively correlated with the playback speed, that is, at the voice sending end, the greater the environmental noise of the user, the slower the playback speed of the adjusted voice data, so that when the noise in the recording environment is large, the playback speed of the voice data is slowed down, and after the adjusted voice data is sent to the receiving side instant messaging application, the user can more easily hear the content of the voice message during the playback process, better understand the content of the voice message, and no longer need the user to manually adjust the playback volume, thereby improving the flexibility of the playback control method of the voice message.

[0214] like Fig.10 As shown, it is a flow chart of a method for processing voice data in an instant messaging application provided by an embodiment of the present application. The method may include the following steps:

[0215] Step S1001, responding to the recording operation triggered by the target object, obtaining voice data collected by the microphone;

[0216] For example, Fig.11 In the chat display interface of the instant messaging application shown in FIG. 1 , the user can trigger the recording operation by clicking the “Press and hold to speak” option in the chat display interface; after the user triggers the recording operation, the display interface of the instant messaging application can be as follows during the recording process Fig.12 shown.

[0217] Step S1002: The voice data is processed according to the second noise intensity of the recording environment and then sent, so that the playback speed of the processed voice data is negatively correlated with the second noise intensity.

[0218] It should be noted that, in the embodiment of the present application, in response to the recording operation triggered by the target object, the noise data in the voice data is extracted by analyzing the voice data collected by the microphone; the instant messaging application determines the second noise intensity of the recording environment according to the noise data collected by the microphone;

[0219] Alternatively, an optional implementation manner is that the embodiment of the present application collects voice data of the recording environment through the microphone, and after determining that the collected voice data meets the second preset condition, the instant messaging application determines the second noise intensity in the recording environment according to the voice data collected by the microphone;

[0220] The second preset condition includes at least one of the following conditions:

[0221] Condition 1: The account receiving the voice data is a preset type account.

[0222] It should be noted that the instant messaging application of the embodiment of the present application can pre-set the account type, and before sending the voice data, determine the account receiving the voice data, and judge whether the account receiving the voice data is a preset type account. If so, trigger the microphone to collect the voice data in the recording environment;

[0223] Among them, the preset type account can be an account that has a preset social relationship with the account that sends the voice data; for example, the preset type account can be set as an account of special attention for the account that sends the voice data.

[0224] Alternatively, the preset type account may also be an account with a preset label set for the account sending the voice data; for example, the preset type account may be an account with a label such as family member, customer, etc. set for the account sending the voice data.

[0225] When the account receiving the voice data is a preset type account, for example, the preset type account is set as an account of special concern for the account sending the voice data, or the preset type account is set as an account with labels such as family members and customers for the account sending the voice message, it means that the relationship between the account sending the voice data and the account receiving the voice data is relatively close, or that the account receiving the voice data is relatively important to the account sending the voice data. At this time, it is necessary to collect noisy voice data in the recording environment, and adjust the playback speed of the voice data according to the second noise intensity in the recording environment, so that the playback speed of the voice data is more reasonable, so that the user who receives the voice data can better understand the content of the voice data when playing the voice data, thereby improving the intelligibility of the voice data.

[0226] Condition 2: the emotional state determined according to the voice features in the voice data is the preset emotional state, and the emotional state is obtained by performing emotional analysis on the voice features in the voice data;

[0227] It should be noted that the emotional state determined based on the voice features in the voice data is the emotional state of the user who sends the voice data.

[0228] Before sending the voice data, the embodiment of the present application can perform sentiment analysis on the voice features in the voice message, and determine the current sentiment state of the user sending the voice data according to the voice features in the voice data;

[0229] An optional implementation is to perform sentiment analysis on speech features in speech data to obtain the emotional state through machine learning.

[0230] For example, the preset emotional state may be an abnormal emotional state such as crying, sadness, excitement, etc.;

[0231] When the emotional state of the user sending the voice data is a preset emotional state, such as crying, sadness, excitement and other abnormal emotional states, the sent voice data may be intermittent, making it difficult for the receiving user of the voice data to hear the voice data clearly; and when the sending user of the voice data is in an abnormal emotional state, the receiving user of the voice data needs to hear the content of the voice message as clearly as possible to understand the current state of the sending user of the voice data. Therefore, at this time, the noisy voice data in the recording environment is collected, and the playback speed of the voice data is adjusted according to the second noise intensity in the recording environment, so that the playback speed of the voice data is more reasonable, which is convenient for the receiving user of the voice data to better understand the content of the voice data and improve the intelligibility of the voice data.

[0232] Condition 3: The voice data contains preset key information, and the preset key information in the voice data is obtained by performing semantic analysis on the voice data.

[0233] It should be noted that when the preset key information is included in the voice data, it indicates that the voice data is important voice data, or it may also indicate that the voice data is relatively special voice data;

[0234] For example, the preset key information may be key information of the address type such as "Dad" or "Mom"; when the voice data contains key information of the address type such as "Dad" or "Mom", it means that the voice data is voice data sent by the child to the parents, and the voice data is relatively important voice data;

[0235] Alternatively, the preset key information may be key information related to money, such as "money", "transfer", "remittance", etc. When the preset key information is included in the voice data, it indicates that the voice data is relatively important voice data;

[0236] Alternatively, the preset key information may be key information with special meanings such as "birthday" or "holiday". When the preset key information is included in the voice data, it indicates that the voice data is special voice data.

[0237] Since the preset key information is contained in the voice data, it means that the voice data is relatively important or special voice data, and the receiving user of the voice data needs to hear the voice data clearly. Therefore, at this time, the noisy voice data in the recording environment is collected, and the playback speed of the voice data is adjusted according to the second noise intensity in the recording environment, so that the playback speed of the voice data is more reasonable, which is convenient for the receiving user of the voice data to better understand the content of the voice data and improve the intelligibility of the voice data.

[0238] After obtaining the voice data collected by the microphone, the embodiment of the present application obtains the second noise intensity of the recording environment by analyzing the voice data collected by the microphone.

[0239] Among them, the embodiments of the present application can collect noisy speech data in a recording environment in real time, detect the noisy speech data to obtain a second noise intensity of the recording environment; or, after triggering a recording operation, collect speech data through a microphone, analyze the collected speech data, extract noise data from the speech data, and detect the noise data to obtain a second noise intensity.

[0240] In implementation, the embodiment of the present application can detect the noise data to determine the second noise intensity of the recording environment in the following manner:

[0241] For each noise data frame in the noise data, perform noise detection on the noise data frame to obtain a noise estimation parameter corresponding to the noise data frame; determine a second noise intensity according to the noise estimation parameter corresponding to the noise data frame in the noise data;

[0242] In the embodiment of the present application, when detecting the noise data to determine the second noise intensity of the recording environment, noise estimation parameters are obtained by performing noise detection on each noise data frame in the noise data;

[0243] It should be noted that the method for performing noise detection on the noise data frame to determine the noise estimation parameters in the embodiment of the present application is the same as the method for performing noise detection on the noisy speech data frame to determine the noise estimation parameters in the above text. The specific implementation method can be found in the description above and will not be described in detail here.

[0244] In implementation, after determining the noise estimation parameter corresponding to the noise data frame, the embodiment of the present application may determine the second noise intensity in the noise data according to the noise estimation parameter;

[0245] It should be noted that the second noise intensity may be a noise estimation parameter of the last noise data frame in the noise data; or the second noise intensity may be an average value of noise estimation parameters of all noise data frames in the noise data.

[0246] During implementation, an optional implementation is that after determining the second noise intensity of the recording environment, the embodiment of the present application compares the second noise intensity with a preset noise threshold; when the second noise intensity is less than or equal to the noise threshold, the voice data is directly sent; when the second noise intensity is greater than the preset noise threshold, the playback speed of the voice data is expanded to reduce the playback speed of the voice data.

[0247] When adjusting the playback speed of voice data in the embodiment of the present application, it is necessary to adjust the playback speed of voice data according to the second target compression expansion coefficient.

[0248] In implementation, the embodiment of the present application can determine the second target compression expansion coefficient corresponding to the speech data according to the following method;

[0249] The voice data is divided into frames according to a preset frame spacing to obtain multiple voice data frames; for each voice data frame, a second reference compression expansion coefficient corresponding to the voice data frame is determined according to the difference between the second noise intensity and the noise threshold; the second reference compression expansion coefficient and the second target compression expansion coefficient corresponding to the previous voice data frame are weightedly summed to obtain the second target compression expansion coefficient corresponding to the voice data frame.

[0250] It should be noted that when adjusting the speech rate of the speech data in the embodiment of the present application, the speech data needs to be divided into frames and adjusted for each frame. The speech rate can be controlled by adjusting the target compression and expansion coefficient in the embodiment of the present application. However, if the coefficient is adjusted arbitrarily, the sound will be unnatural and uncomfortable to listen to. Therefore, in order to avoid sudden changes in the coefficient, the target compression and expansion coefficient of the previous frame needs to be used to smooth the target compression and expansion coefficient of the current frame.

[0251] Specifically, the method for adjusting the speech speed of the voice data obtained by recording in the embodiment of the present application is the same as the method for adjusting the speech speed of the received voice message mentioned above. The WSOLA algorithm can be used to adjust the speech speed of the voice data. The specific method can be found in the description above and will not be repeated here.

[0252] like Fig.13 As shown, it is a flow chart of a voice data processing method in an instant messaging application provided by an embodiment of the present application. In the following description, the instant messaging application on the voice sending terminal is referred to as a first instant messaging application, and the instant messaging application on the voice receiving terminal is referred to as a second instant messaging application, including the following steps:

[0253] Step S1301: In response to a recording operation triggered by a first object, the first instant messaging application collects voice data through a microphone;

[0254] Step S1302: The first instant messaging application analyzes the collected voice data and extracts noise data from the voice data;

[0255] Step S1303: The first instant messaging application determines a second noise intensity in the playback environment;

[0256] Step S1304: the first instant messaging application determines that the second noise intensity is greater than a preset noise threshold;

[0257] Step S1305: the first instant messaging application performs frame processing on the voice data according to a preset frame spacing to obtain a plurality of voice data frames;

[0258] Step S1306: The first instant messaging application determines a second reference compression expansion coefficient corresponding to the voice data frame according to the difference between the second noise intensity and the noise threshold;

[0259] Step S1307: the first instant messaging application performs weighted sum processing on the second reference compression and expansion coefficient and the second target compression and expansion coefficient corresponding to the previous voice data frame to obtain the second target compression and expansion coefficient corresponding to the voice data frame;

[0260] Step S1308: the first instant messaging application performs expansion processing on the recorded speech rate of the voice data according to the second target compression expansion coefficient to obtain a target voice data frame;

[0261] Step S1309: The first instant messaging application sends the processed voice message to the second instant messaging application.

[0262] like Fig.14 FIG. 1 is a schematic diagram of a structure of a voice data processing device 1400 in an instant messaging application in an embodiment of the present application, including:

[0263] Display unit 1401, used to display the received voice message;

[0264] The first processing unit 1402 is used to respond to the operation of playing the voice message, and adjust the playing speed of the voice message according to the first noise intensity of the playing environment before playing, and the playing speed is negatively correlated with the first noise intensity.

[0265] Optionally, the first processing unit 1402 is specifically configured to:

[0266] The playback speed of the voice message is adjusted according to the following method to obtain noisy voice data in the playback environment collected by the microphone;

[0267] Detecting a first noise intensity in noisy speech data;

[0268] When the first noise intensity is greater than a preset noise threshold, adjusting the playback speed of the voice message; and

[0269] When the first noise intensity is less than or equal to the noise threshold, the voice message is played.

[0270] Optionally, the first processing unit 1402 is specifically configured to:

[0271] When the first noise intensity is greater than a preset noise threshold, a prompt message for adjusting the playback speed is displayed;

[0272] In response to the confirmation operation of adjusting the playing speed, the playing speed of the voice message is adjusted.

[0273] Optionally, the first processing unit 1402 is specifically configured to:

[0274] In response to a confirmation operation for adjusting the playback speed, obtaining a first target playback speed specified by the confirmation operation;

[0275] Determining a first target compression / expansion coefficient corresponding to the voice message according to the first target playback speech speed;

[0276] The playing speed of the voice message is expanded according to the first target compression expansion coefficient to adjust the playing speed of the voice message.

[0277] Optionally, the first processing unit 1402 is specifically configured to:

[0278] Determining a first target compression expansion coefficient corresponding to the voice message according to the first noise intensity;

[0279] The playing speed of the voice message is expanded according to the first target compression expansion coefficient to adjust the playing speed of the voice message.

[0280] Optionally, the first processing unit 1402 is specifically configured to:

[0281] In response to the operation of playing the voice message, or after determining that the received voice message meets the first preset condition, collecting noisy voice data in the playing environment through the microphone;

[0282] The first preset condition includes at least one of the following conditions:

[0283] The account that sends the voice message is a preset type account;

[0284] The emotional state determined according to the voice features in the voice message is the preset emotional state, and the emotional state is obtained by performing emotional analysis on the voice features in the voice message;

[0285] The text in the voice message contains preset key information, and the preset key information in the voice message is obtained by performing semantic analysis on the voice message.

[0286] Optionally, the first processing unit 1402 is specifically configured to:

[0287] Determine a first target compression expansion coefficient corresponding to the voice message according to the first noise intensity, and divide the voice message into frames according to a preset frame spacing to obtain a plurality of voice data frames;

[0288] For each speech data frame, a first reference compression expansion coefficient corresponding to the speech data frame is determined according to the difference between the first noise intensity and the noise threshold; the first reference compression expansion coefficient and the first target compression expansion coefficient corresponding to the previous speech data frame are weightedly summed to obtain the first target compression expansion coefficient corresponding to the speech data frame.

[0289] Optionally, the first processing unit 1402 is specifically configured to:

[0290] For each voice data frame, the time domain of the voice data frame is expanded according to the target compression expansion coefficient corresponding to the voice data frame, and a reference voice data frame is determined from the search space corresponding to the voice data frame; the expanded voice data frame and the reference voice data frame are superimposed to obtain a target voice data frame for playback; wherein the time domain length of the reference voice data frame is the same as the time domain length of the expanded voice data frame.

[0291] Optionally, the first processing unit 1402 is further configured to: perform voice enhancement processing on the voice message before adjusting the playback speed of the voice message according to the first noise intensity of the playback environment.

[0292] like Fig.15 FIG. 1 is a schematic diagram of a structure of a voice data processing device 1500 in an instant messaging application in an embodiment of the present application, including:

[0293] The collection unit 1501 is used to respond to the recording operation triggered by the target object and obtain the voice data collected by the microphone;

[0294] The second processing unit 1502 is used to process the voice data according to the second noise intensity of the recording environment and then send it, so that the playback speed of the processed voice data is negatively correlated with the second noise intensity.

[0295] Optionally, the second processing unit 1502 is further configured to:

[0296] After obtaining the voice data collected by the microphone, and before processing the voice data according to a second noise intensity of the recording environment, determining that the collected voice data meets a second preset condition;

[0297] The second preset condition includes at least one of the following conditions:

[0298] The account receiving the voice data is a preset type account;

[0299] The emotional state determined according to the voice features in the voice data is the preset emotional state, and the emotional state is obtained by performing emotional analysis on the voice features in the voice data;

[0300] The voice data contains preset key information, and the preset key information in the voice data is obtained by performing semantic analysis on the voice data.

[0301] Optionally, the second processing unit 1502 is specifically configured to:

[0302] The voice data is processed according to the following method, the voice data collected by the microphone is analyzed, and the noise data in the voice data is extracted;

[0303] detecting a second noise intensity in the noise data;

[0304] When the second noise intensity is greater than a preset noise threshold, processing the voice data; and

[0305] When the second noise intensity is less than or equal to the noise threshold, the voice data is sent.

[0306] Optionally, the second processing unit 1502 is specifically configured to:

[0307] Determining a second target compression expansion coefficient corresponding to the voice data according to the second noise intensity;

[0308] The playback speed of the voice data is expanded according to the second target compression and expansion coefficient to adjust the playback speed of the voice data.

[0309] Optionally, the second processing unit 1502 is specifically configured to:

[0310] The second noise intensity in the noise data is detected according to the following method:

[0311] For each noise data frame in the noise data, noise detection is performed on the noise data frame to obtain noise estimation parameters corresponding to the noise data frame;

[0312] The second noise intensity is determined according to the noise estimation parameter corresponding to the noise data frame in the noise data.

[0313] Optionally, the second processing unit 1502 is specifically configured to:

[0314] The noise estimation parameters corresponding to the noise data frame are determined in the following manner:

[0315] Determine a target subband within a preset frequency spectrum corresponding to the noise data frame;

[0316] The noise estimation parameter corresponding to the noise data frame is determined according to the noise estimation value corresponding to each target subband, the number of frequency points contained in each target subband, and the noise estimation parameter corresponding to the last noise data frame collected.

[0317] Optionally, the second processing unit 1502 is specifically configured to:

[0318] The noise estimation value corresponding to each target subband is determined as follows:

[0319] Determine the power spectrum of the target subband according to the frequency domain complex value corresponding to each frequency point in the target subband and the number of frequency points contained in the target subband; wherein the frequency domain complex value corresponding to the frequency point is obtained by Fourier transforming the collected noise data frame;

[0320] Determine the speech existence probability value corresponding to the target sub-band according to the smoothed power spectrum obtained by time-frequency domain smoothing of the power spectrum of the target sub-band and the minimum power spectrum in the power spectrum of the sub-band corresponding to the noise data frame;

[0321] The noise estimation value corresponding to the target subband is determined according to the power spectrum corresponding to the target subband, the speech existence probability value corresponding to the target subband, and the noise estimation value corresponding to the target subband in the previous noisy data frame.

[0322] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0323] Those skilled in the art will appreciate that each aspect of the present application may be implemented as a system, method, or program product. Therefore, each aspect of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or an implementation combining hardware and software aspects, which may be collectively referred to as a "circuit", "module", or "system" herein.

[0324] In some possible implementations, the present application also provides an electronic device, referring to Fig.16 As shown, the electronic device 1600 may include at least one processor 1601 and at least one memory 1602. The memory 1602 stores program codes. When the program codes are executed by the processor 1601, the processor 1601 executes the steps of the voice data processing method according to various exemplary embodiments of the present application described above in this specification. For example, the processor 1601 may execute the following steps: Figure 3 or the steps shown in 10.

[0325] In some possible implementations, the present application also provides a computing device, which may include at least one processing unit and at least one storage unit. The storage unit stores program code, and when the program code is executed by the processing unit, the processing unit executes the steps of the voice data processing method according to various exemplary implementations of the present application described above in this specification. For example, the processor 1601 may execute the following steps: Figure 3 or the steps shown in 10.

[0326] Refer to the following Fig.17 1700 according to this embodiment of the present application is described. Fig.17 The computing device 1700 is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0327] like Fig.17 The computing device 1700 is in the form of a general computing device. The components of the computing device 1700 may include, but are not limited to: at least one processing unit 1701, at least one storage unit 1702, and a bus 1703 connecting different system components (including the storage unit 1702 and the processing unit 1701).

[0328] Bus 1703 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a processor, or a local bus using any of a variety of bus architectures.

[0329] The storage unit 1702 may include a readable medium in the form of a volatile memory, such as a random access memory (RAM) 1721 or a cache memory unit 1722 , and may further include a read-only memory (ROM) 1723 .

[0330] The storage unit 1702 may also include a program / utility 1725 having a set (at least one) of program modules 1724, such program modules 1724 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0331] The computing device 1700 may also communicate with one or more external devices 1704 (e.g., keyboards, pointing devices, etc.), one or more devices that enable a user to interact with the computing device 1700, or any device that enables the computing device 1700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1705. In addition, the computing device 1700 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), or a public network, such as the Internet) via a network adapter 1706. As shown, the network adapter 1706 communicates with other modules for the computing device 1700 via a bus 1703. It should be understood that, although not shown in the figure, other hardware or software modules may be used in conjunction with the computing device 1700, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0332] In some possible implementations, each aspect of the voice data processing method in an instant messaging application provided by the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is used to enable the computer device to execute the steps of voice data processing according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the following steps: Figure 3 or the steps shown in 10.

[0333] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0334] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0335] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for processing voice data in an instant messaging application, characterized in that: The method includes: Displaying a received voice message, the voice message being determined by the instant messaging application based on received voice data, the voice data being collected through a microphone; In response to the operation of playing the voice message, the voice message is played after adjusting the playback speed according to a first noise intensity of the playback environment, wherein the playback speed is negatively correlated with the first noise intensity; the first noise intensity is determined according to a noise estimation parameter corresponding to a noisy data frame in the playback environment, and the noise estimation parameter corresponding to the noisy data frame is determined according to a noise estimation value of each target subband within a preset spectrum range corresponding to the noisy data frame, the number of frequency points contained in each target subband, and a noise estimation parameter corresponding to a previous noisy voice data frame collected; When the first noise intensity is greater than a preset noise threshold, the playback speed of the voice message may be adjusted according to the following speed adjustment method: The voice message is divided into frames according to a preset frame spacing to obtain multiple voice data frames; for each voice data frame, a first reference compression expansion coefficient corresponding to the voice data frame is determined according to the difference between the first noise intensity and the noise threshold; the first reference compression expansion coefficient and the first target compression expansion coefficient corresponding to the previous voice data frame are weighted summed to obtain the first target compression expansion coefficient corresponding to the voice data frame; the time domain of the voice data frame is expanded according to the first target compression expansion coefficient corresponding to the voice data frame, and a reference voice data frame is determined from the search space corresponding to the voice data frame; the expanded voice data frame is superimposed with the reference voice data frame to obtain a target voice data frame for playback, so as to adjust the playback speed of the voice message; wherein the time domain length of the reference voice data frame is the same as the time domain length of the expanded voice data frame.

2. The method according to claim 1, characterized in that The playing speed of the voice message is adjusted according to the following method: Obtain noisy speech data in the playback environment collected by the microphone; Detecting a first noise intensity in the noisy speech data; When the first noise intensity is greater than a preset noise threshold, adjusting the playback speed of the voice message; and When the first noise intensity is less than or equal to the noise threshold, the voice message is played.

3. The method according to claim 2, characterized in that The obtaining of noisy speech data collected by a microphone specifically includes: In response to the operation of playing the voice message, or after determining that the received voice message meets the first preset condition, collecting noisy voice data in the playing environment through the microphone; The first preset condition includes at least one of the following conditions: The account sending the voice message is a preset type account; The emotional state determined according to the voice features in the voice message is a preset emotional state, and the emotional state is obtained by performing emotional analysis on the voice features in the voice message; The voice message contains preset key information, and the preset key information in the voice message is obtained by performing semantic analysis on the voice message.

4. A method for processing voice data in an instant messaging application, characterized in that: The method includes: Respond to the recording operation triggered by the target object and obtain the voice data collected by the microphone; The voice data is processed according to a second noise intensity of the recording environment and then sent, so that the playback speed of the processed voice data is negatively correlated with the second noise intensity; the second noise intensity is determined according to a noise estimation parameter corresponding to a noise data frame in the recording environment, and the noise estimation parameter corresponding to the noise data frame is determined according to a noise estimation value of each target subband within a preset frequency spectrum range corresponding to the noise data frame, the number of frequency points contained in each target subband, and a noise estimation parameter corresponding to a previous noise data frame collected; Wherein, when the second noise intensity is greater than a preset noise threshold, the voice data is processed in the following manner: The voice data is divided into frames according to a preset frame spacing to obtain a plurality of voice data frames; for each voice data frame, a second reference compression expansion coefficient corresponding to the voice data frame is determined according to a difference between the second noise intensity and the noise threshold; the second reference compression expansion coefficient and a second target compression expansion coefficient corresponding to a previous voice data frame are weighted and summed to obtain a second target compression expansion coefficient corresponding to the voice data frame; The time domain of the voice data frame is expanded according to the second target compression expansion coefficient corresponding to the voice data frame, and a reference voice data frame is determined from the search space corresponding to the voice data frame; the expanded voice data frame is superimposed with the reference voice data frame to obtain a target voice data frame for playback, so as to adjust the playback speed of the voice data; wherein the time domain length of the reference voice data frame is the same as the time domain length of the expanded voice data frame.

5. The method according to claim 4, characterized in that The voice data is processed in the following manner: Analyzing the voice data collected by the microphone to extract noise data from the voice data; detecting a second noise intensity in the noise data; When the second noise intensity is greater than a preset noise threshold, processing the voice data; and When the second noise intensity is less than or equal to the noise threshold, the voice data is sent.

6. A voice data processing device in an instant messaging application, characterized in that: include: A display unit, configured to display a received voice message, wherein the voice message is determined by the instant messaging application based on received voice data, wherein the voice data is collected through a microphone; A first processing unit, configured to respond to an operation of playing the voice message, and adjust a playing speed of the voice message according to a first noise intensity of a playing environment before playing the voice message, wherein the playing speed is negatively correlated with the first noise intensity; The first noise intensity is determined according to a noise estimation parameter corresponding to a noisy data frame in the playback environment, and the noise estimation parameter corresponding to the noisy data frame is determined according to a noise estimation value of each target subband within a preset spectrum range corresponding to the noisy data frame, the number of frequency points contained in each target subband, and a noise estimation parameter corresponding to a previous noisy speech data frame collected; When the first noise intensity is greater than a preset noise threshold, the playback speed of the voice message may be adjusted according to the following speed adjustment method: The voice message is divided into frames according to a preset frame spacing to obtain multiple voice data frames; for each voice data frame, a first reference compression expansion coefficient corresponding to the voice data frame is determined according to the difference between the first noise intensity and the noise threshold; the first reference compression expansion coefficient and the first target compression expansion coefficient corresponding to the previous voice data frame are weighted summed to obtain the first target compression expansion coefficient corresponding to the voice data frame; the time domain of the voice data frame is expanded according to the first target compression expansion coefficient corresponding to the voice data frame, and a reference voice data frame is determined from the search space corresponding to the voice data frame; the expanded voice data frame is superimposed with the reference voice data frame to obtain a target voice data frame for playback, so as to adjust the playback speed of the voice message; wherein the time domain length of the reference voice data frame is the same as the time domain length of the expanded voice data frame.

7. A voice data processing device in an instant messaging application, characterized in that: include: A collection unit, used to respond to the recording operation triggered by the target object and obtain the voice data collected by the microphone; A second processing unit, configured to process the voice data according to a second noise intensity of a recording environment and then send the processed voice data so that a playback speed of the processed voice data is negatively correlated with the second noise intensity; The second noise intensity is determined according to a noise estimation parameter corresponding to a noise data frame in the recording environment, wherein the noise estimation parameter corresponding to the noise data frame is determined according to a noise estimation value of each target subband within a preset frequency spectrum corresponding to the noise data frame, the number of frequency points contained in each target subband, and a noise estimation parameter corresponding to a previous noise data frame collected; Wherein, when the second noise intensity is greater than a preset noise threshold, the voice data is processed in the following manner: The voice data is divided into frames according to a preset frame spacing to obtain a plurality of voice data frames; for each voice data frame, a second reference compression expansion coefficient corresponding to the voice data frame is determined according to a difference between the second noise intensity and the noise threshold; the second reference compression expansion coefficient and a second target compression expansion coefficient corresponding to a previous voice data frame are weighted and summed to obtain a second target compression expansion coefficient corresponding to the voice data frame; The time domain of the voice data frame is expanded according to the second target compression expansion coefficient corresponding to the voice data frame, and a reference voice data frame is determined from the search space corresponding to the voice data frame; the expanded voice data frame is superimposed with the reference voice data frame to obtain a target voice data frame for playback, so as to adjust the playback speed of the voice data; wherein the time domain length of the reference voice data frame is the same as the time domain length of the expanded voice data frame.

8. An electronic device, characterized in that: It includes a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of any method described in claims 1 to 3, or the processor executes the steps of any method described in claims 4 to 5.

9. A computer-readable storage medium, characterized in that: It includes program code, and when the program code is run on an electronic device, the program code is used to enable the electronic device to execute the steps of any method described in claims 1 to 3, or to enable the electronic device to execute the steps of any method described in claims 4 to 5.

Images