Audio processing method and apparatus, computer device, readable storage medium, and program product

Abstract

An audio processing method, executed by a computer device. The method comprises: collecting an audio signal in real time in the process of performing audio interaction with an audio interaction terminal by using an identity of an object identifier (502); calculating a real-time voiceprint vector of the audio signal collected in real time, and comparing the calculated real-time voiceprint vector with a reference voiceprint vector registered for the object identifier and belonging to a registered object to obtain a comparison result (504); when it is determined, according to the comparison result, that current speaking voice of the audio signal is from the registered object, sending the audio signal to the audio interaction terminal (506); and when it is determined, according to the comparison result, that the current speaking voice of the audio signal is not from the registered object, performing voice alteration processing on the audio signal to obtain a voice-altered audio signal, and sending the voice-altered audio signal to the audio interaction terminal (508).

Description

Audio processing method and device, computer device, readable storage medium and program product

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202410725210.0, filed on June 4, 2024, and entitled "Audio processing method and device, computer device, readable storage medium and program product", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of computer, in particular to an audio processing method and device, computer device, computer readable storage medium and computer program product. BACKGROUND

[0004] With the development of computer technology and Internet technology, audio-based interaction has become a ubiquitous application scenario in daily life, such as online meetings, interactive live broadcasts, online classrooms, etc. Real-time audio and video interaction can greatly facilitate information transmission and improve communication efficiency.

[0005] However, in some audio interaction scenarios, the remote user not only hears the speaking voice of the user initiating the audio interaction, but also needs to hear the speaking voice of other personnel, or inadvertently hears the speaking voice of other personnel, which may lead to exposure of the voices of other personnel in addition to the user initiating the audio interaction, resulting in information leakage and personal privacy security being unprotected. SUMMARY

[0006] The present application provides an audio processing method and device, computer device, computer readable storage medium and computer program product.

[0007] In a first aspect, the present application provides an audio processing method, executed by a computer device, comprising:

[0008] In the process of audio interaction with the audio interaction terminal in the identity of the object identifier, the audio signal is collected in real time;

[0009] The real-time voiceprint vector of the real-time collected audio signal is calculated, and the calculated real-time voiceprint vector is compared with the reference voiceprint vector of the registered object registered for the object identifier to obtain a comparison result;

[0010] In the case where it is determined according to the comparison result that the speaking voice of the audio signal currently comes from the registered object, the audio signal is sent to the audio interaction terminal; and

[0011] In a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, the audio signal is processed to change the voice to obtain a changed audio signal, and the changed audio signal is sent to the audio interaction terminal.

[0012] In a second aspect, the present application further provides an audio processing device, comprising:

[0013] An audio interaction initiation module is configured to collect an audio signal in real time in a process of audio interaction with an audio interaction terminal in an identity of an object identifier;

[0014] A comparison module is configured to calculate a real-time voiceprint vector of the audio signal collected in real time, compare the calculated real-time voiceprint vector with a reference voiceprint vector of a registered object registered for the object identifier, and obtain a comparison result;

[0015] A sending module is configured to send the audio signal to the audio interaction terminal in a case where it is determined according to the comparison result that the current speaking voice of the audio signal comes from the registered object; and

[0016] The sending module is configured to process the audio signal to change the voice to obtain a changed audio signal in a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, and send the changed audio signal to the audio interaction terminal.

[0017] In a third aspect, the present application further provides a computer device, comprising a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0018] An audio signal is collected in real time in a process of audio interaction with an audio interaction terminal in an identity of an object identifier;

[0019] A real-time voiceprint vector of the audio signal collected in real time is calculated, the calculated real-time voiceprint vector is compared with a reference voiceprint vector of a registered object registered for the object identifier, and a comparison result is obtained;

[0020] The audio signal is sent to the audio interaction terminal in a case where it is determined according to the comparison result that the current speaking voice of the audio signal comes from the registered object; and

[0021] The audio signal is processed to change the voice to obtain a changed audio signal in a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, and the changed audio signal is sent to the audio interaction terminal.

[0022] In a fourth aspect, the present application provides a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the following steps:

[0023] In the process of audio interaction with the audio interaction terminal in the identity of the object identifier, real-time audio signals are collected;

[0024] The real-time voiceprint vector of the real-time audio signal is calculated, and the calculated real-time voiceprint vector is compared with the reference voiceprint vector of the registered object registered for the object identifier to obtain a comparison result;

[0025] In the case where it is determined according to the comparison result that the current speaking voice of the audio signal is derived from the registered object, the audio signal is sent to the audio interaction terminal; and

[0026] In the case where it is determined according to the comparison result that the current speaking voice of the audio signal is not derived from the registered object, the audio signal is processed to obtain a changed audio signal, and the changed audio signal is sent to the audio interaction terminal.

[0027] In a fifth aspect, the present application provides a computer program product, comprising a computer program, which, when executed by a processor, implements the following steps:

[0028] In the process of audio interaction with the audio interaction terminal in the identity of the object identifier, real-time audio signals are collected;

[0029] The real-time voiceprint vector of the real-time audio signal is calculated, and the calculated real-time voiceprint vector is compared with the reference voiceprint vector of the registered object registered for the object identifier to obtain a comparison result;

[0030] In the case where it is determined according to the comparison result that the current speaking voice of the audio signal is derived from the registered object, the audio signal is sent to the audio interaction terminal; and

[0031] In the case where it is determined according to the comparison result that the current speaking voice of the audio signal is not derived from the registered object, the audio signal is processed to obtain a changed audio signal, and the changed audio signal is sent to the audio interaction terminal.

[0032] The details of one or more embodiments of the present application are presented in the following drawings and description. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the technical solutions in the prior art, the accompanying drawings needed to be used in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description only represent some of the embodiments of the present application, and all other drawings obtained by those of ordinary skill in the art based on the drawings without any creative effort should be within the protection scope of the present application.

[0034] FIG. 1 is a diagram of an application environment of an audio processing method in an embodiment;

[0035] FIG. 2 is a structural block diagram of an audio processing system according to an example embodiment of the present application;

[0036] FIG. 3 is a diagram of a technical architecture of audio interaction in the related art;

[0037] FIG. 4 is a diagram of a technical architecture of audio interaction according to an example embodiment of the present application;

[0038] FIG. 5 is a flow diagram of an audio processing method in an embodiment;

[0039] FIG. 6 is a flow diagram of voice activity detection in an embodiment;

[0040] FIG. 7 is a diagram of steps of sound protection in an embodiment;

[0041] FIG. 8 is a diagram of detailed steps of audio processing in a detailed embodiment;

[0042] FIG. 9 is a structural block diagram of an audio processing apparatus in an embodiment;

[0043] FIG. 10 is an internal structure diagram of a computer device in an embodiment;

[0044] FIG. 11 is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION

[0045] The technical solutions in the embodiments of the present application will be described clearly and completely in combination with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only represent some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without any creative effort should be within the protection scope of the present application.

[0046] The audio processing method provided by the embodiments of the present application can be applied to an application environment as shown in FIG. 1. Wherein, the terminal 102 communicates with the server 104 through a network. The data storage system can store data required to be processed by the server 104. The data storage system can be integrated on the server 104, or can be placed on a cloud or other network server. In an embodiment, the terminal 102 can collect an audio signal in real time in the process of audio interaction with the audio interaction terminal 106 in the identity of an object identifier; calculate a real-time voiceprint vector of the real-time collected audio signal, compare the calculated real-time voiceprint vector with a reference voiceprint vector registered for the object identifier to obtain a comparison result; in the case that it is determined according to the comparison result that the current speaking voice of the audio signal is derived from the registered object, the audio signal is sent to the audio interaction terminal 106; in the case that it is determined according to the comparison result that the current speaking voice of the audio signal is not derived from the registered object, the audio signal is processed to obtain a changed voice audio signal, and the changed voice audio signal is sent to the audio interaction terminal 106 through the server 104. The audio interaction terminal 106 is the opposite terminal of the terminal 102.

[0047] The server 104 can be a stand-alone physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal 102 can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, a vehicle-mounted terminal, a smart television, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0048] FIG. 2 shows a structural block diagram of an audio processing system 100 provided by an exemplary embodiment of the present application. The audio processing system 100 comprises a first terminal 102, a server 104 and a second terminal 106.

[0049] The first terminal 102 is connected to the server cluster 102 through a wireless network or a wired network. The first terminal 102 is installed and runs a client (or an application) supporting audio and video functions, which can be any one of a conference client supporting an online conference function, a user content generation client supporting a live broadcast function, and a learning client supporting an online class function. The first terminal 102 is a terminal used by a registered object, and the client in the first terminal 102 is logged in with an object identifier, representing that the registered object uses the functions of the client through the first terminal 102. The first terminal 102 is connected to the server 104 through a wireless network or a wired network. Optionally, the clients installed on the first terminal 102 and the second terminal 106 are the same, or the clients installed on the two terminals are the same type of clients of different operating system platforms. Different operating systems include an Android operating system, an IOS operating system, a Windows operating system, and an operating system dedicated to a game console. The first terminal 102 can be referred to as one of a plurality of terminals, and the second terminal 106 can be referred to as one of a plurality of terminals. The terminals mentioned in this embodiment can be the first terminal 102 or the second terminal 106 for example. The second terminal 104 is an audio interaction terminal initiated by the first terminal 102.

[0050] In one application scenario, an online conference client is run on a terminal, a conference host logs in the online conference client through an account of the conference host, after the online conference starts, the terminal starts audio interaction of the online conference, in addition to the conference host, some other personnel share the account to participate in the multi-person conference, so that the audio interaction terminal user can hear the voices of the conference host and the other participants. The terminal uses the audio processing method provided in this embodiment, which can protect the voice information of the other participants in the discussion process of the online conference without affecting the voice of the conference host, protect the personal information of the user, and make the other personnel participating in the conference of the audio interaction terminal unable to identify the specific identity of the user around the conference host through the voice.

[0051] In one application scenario, a live broadcast client is run on a terminal, an anchor logs in the live broadcast client through an account of the anchor, after the live broadcast, in addition to the anchor, some other personnel (for example, some backstage assistant personnel and staff) share the live broadcast account to participate in the live broadcast, so that the live broadcast audience user can hear the voices of the anchor and the other personnel. The terminal uses the audio processing method provided in this embodiment, which can protect the voice information of the other personnel in the live broadcast process without affecting the voice of the anchor, protect the personal information of the user, and make the audience unable to identify the specific identity of the user around the anchor through the voice.

[0052] In one application scenario, an online class client is run on a terminal, a teacher logs in the online class client through his own account, after starting online teaching, the terminal starts audio interaction of the online class, in addition to the main teacher, some other personnel (for example, students in the same classroom as the main teacher) share the same account to participate in the online class, so that the audio interaction terminal user can hear the voices of the main teacher and the students (such as the classroom question and answer interaction process between the students and the main teacher). The account used by the teacher to log in is the object identifier, the voices of the students and other personnel are compared with the reference voiceprint vector of the teacher (registered object) registered for the object identifier, if the comparison result determines that the speaking voice does not come from the teacher, the audio signal is processed to change the voice, so as to protect the voice information of the students and other personnel.

[0053] The terminal uses the audio processing method provided by the embodiments of the present application, which can protect the voice information of the students in the question and answer interaction process of the online class without affecting the speaking voice of the main teacher, so that the personnel participating in the online class through the audio interaction terminal cannot identify the specific identity of the users around the main teacher through the voice, especially in the case of students being minors, the personal privacy of the minors can be protected.

[0054] It should be noted that the audio processing method provided by the embodiments of the present application can also be executed by a server, for example, in the process that the terminal performs audio interaction with the audio interaction terminal in the identity of the object identifier, the audio signal is collected in real time, the audio signal collected in real time is directly sent to the server, the server receives the audio signal uploaded by the terminal, calculates the real-time voiceprint vector of the audio signal, the server compares the real-time voiceprint vector calculated with the reference voiceprint vector registered for the object identifier to obtain the comparison result, in the case that the comparison result determines that the current speaking voice of the audio signal comes from the registered object, the server sends the audio signal to the audio interaction terminal, in the case that the comparison result determines that the current speaking voice of the audio signal does not come from the registered object, the server processes the audio signal to change the voice to obtain the changed voice signal, and sends the changed voice signal to the audio interaction terminal.

[0055] As shown in FIG. 3, it is a technical architecture diagram of audio interaction in the related art. Referring to FIG. 3, generally, after the near-end terminal initiates the audio interaction and starts the audio signal collection, the real-time collected audio signals are sequentially subjected to the preprocessing, encoding, and then the encoded code stream is obtained, and then the encoded code stream is uplinked to the server, and then the server is downlinked to the audio interaction terminal, and then the audio interaction terminal is sequentially subjected to the decoding and the sound mixing processing, and then the audio signals are played. In this way, the audio interaction is realized. As can be seen from the above process, when the near-end user side includes the registered object and also includes multiple other objects, the sound information of all the objects will be collected and uplinked to the server, and at this time, the far-end user at the audio interaction terminal side can hear the sound of all the objects, and at the same time, the corresponding near-end user can also be identified through the tone. In individual scenarios, this will cause the near-end user identity to be leaked, resulting in the information security risk.

[0056] As shown in FIG. 4, it is a technical architecture diagram of audio interaction in an embodiment of the present application. Referring to FIG. 4, after the near-end terminal initiates the audio interaction, the audio signal collection is started, and then the real-time collected audio signals are sequentially subjected to the preprocessing, the sound protection processing, and the encoding, and then the encoded code stream is obtained, and then the encoded code stream is uplinked to the server, and then the server is downlinked to the audio interaction terminal, and then the audio interaction terminal is sequentially subjected to the decoding and the sound mixing processing, and then the audio signals are played. In this way, by adding the sound protection processing, the sound information of other personnel can be protected in the audio interaction process, and the personal information of the user is protected.

[0057] In an exemplary embodiment, as shown in FIG. 5, an audio processing method is provided, and the method is applied to the computer device (terminal 102 or server 104) in FIG. 1 as an example for description, and the method includes the following steps 502 to 508. Wherein:

[0058] Step 502, in the process of audio interaction with the audio interaction terminal in the identity of the object identifier, real-time audio signals are collected.

[0059] The object identifier refers to the identifier information for uniquely identifying the registered object, which can be a mobile phone number, an email address, or other third-party application account used by the registered object. The computer device collects the real-time audio signals in the process of audio interaction with the audio interaction terminal in the identity of the object identifier. For example, in the application scenario of the multi-person online conference, after the computer device initiates the online conference, the audio signal collection is started to transmit the sound signals emitted by the conference personnel participating in the multi-person online conference to the audio interaction terminal.

[0060] Step 504, the real-time voiceprint vector of the real-time collected audio signals is calculated, the calculated real-time voiceprint vector is compared with the reference voiceprint vector registered for the object identifier, and a comparison result is obtained.

[0061] The voiceprint is a biological feature carrying language information and has stability, measurability, uniqueness, and the like. The voiceprint vector is a vectorized expression of the voiceprint and is used to uniquely represent the voice feature of the same speaker, thereby verifying the identity of the speaker. The reference voiceprint vector registered for the object identity is used to uniquely represent the voiceprint information of the object identity. The computer device can obtain the reference voiceprint vector registered for the object identity in advance and store the reference voiceprint vector locally. When voiceprint comparison is needed, the reference voiceprint vector registered for the object identity can be obtained from the local storage. Of course, the computer device can also obtain the reference voiceprint vector registered for the object identity from another device, and the embodiments of the present application do not limit this.

[0062] After the computer device collects the audio signal, the computer device can calculate the real-time voiceprint vector of the real-time collected audio signal. Alternatively, after the computer device collects the audio signal, the computer device can first perform preprocessing on the audio signal and then calculate the corresponding real-time voiceprint vector. The preprocessing is, for example, 3A processing, which is a combination of three audio algorithms, namely, acoustic echo cancellation (AEC), background noise suppression (ANS), and automatic gain control (AGC).

[0063] After the real-time voiceprint vector of the real-time collected audio signal is calculated, the computer device can obtain the reference voiceprint vector registered for the object identity, compare the two, and obtain a comparison result. In a case where the similarity between the real-time voiceprint vector of the real-time collected audio signal and the reference voiceprint vector registered for the object identity is less than a set threshold, the comparison result is that the speaking voice of the real-time collected audio signal does not come from the registered object. In a case where the similarity between the real-time voiceprint vector of the real-time collected audio signal and the reference voiceprint vector registered for the object identity is greater than or equal to the set threshold, the comparison result is that the speaking voice of the real-time collected audio signal comes from the registered object. The computer device can calculate the Euclidean distance between the real-time voiceprint vector of the real-time collected audio signal and the reference voiceprint vector registered for the object identity as the similarity.

[0064] In a case where it is determined according to the comparison result that the speaking voice of the audio signal currently comes from the registered object, the audio signal is sent to the audio interaction terminal.

[0065] In a case where it is determined according to the comparison result that the speaking voice of the audio signal currently comes from the registered object, it is indicated that the current speaker is the registered object, and the voice of the registered object does not need to be protected. Therefore, the computer device can directly send the audio signal to the audio interaction terminal.

[0066] In a case where it is determined according to the comparison result that the speaking voice of the audio signal currently does not come from the registered object, the audio signal is processed to obtain a transformed audio signal, and the transformed audio signal is sent to the audio interaction terminal.

[0067] The voice transformation processing on the audio signal can be pitch transformation processing on the audio signal, can be speed transformation processing on the audio signal, or can be both pitch transformation processing and speed transformation processing on the audio signal, so as to ensure that the voice is more natural and harmonious while the pitch is transformed. In a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, it is indicated that the current speaker is an object other than the registered object, and the voice of the other object needs to be protected. Therefore, the computer device needs to perform voice transformation processing on the audio signal to obtain a transformed audio signal, and then send the transformed audio signal to the audio interaction terminal.

[0068] The computer device can add a sound source identifier indicating that the speaking voice does not come from the registered object in the transformed audio signal, and then send the transformed audio signal with the added sound source identifier to the audio interaction terminal. The sound source identifier is identifier information added in the transformed audio signal to indicate that the speaking voice does not come from the registered object, and can prompt the audio interaction terminal about the source of the current speaking voice.

[0069] The pitch transformation processing refers to changing the pitch of the audio signal by upsampling or downsampling the frequency of the audio signal. For example, when performing linear interpolation on the audio signal, a suitable interpolation factor is selected according to actual needs to adjust the frequency, so as to change the pitch of the voice. The speed transformation processing refers to changing the playing speed of the audio signal by compressing or expanding the audio signal in the time domain. For example, the audio signal is divided into audio frames with a preset time length (such as 20 ms), each audio frame is stretched or compressed in the time dimension, and then synthesized according to the time sequence to restore or adjust the audio time length.

[0070] The above audio processing method, in the process of audio interaction between the object identifier and the audio interaction terminal, acquires an audio signal in real time, calculates a real-time voiceprint vector of the real-time acquired audio signal after starting the audio signal acquisition, compares the calculated real-time voiceprint vector with a reference voiceprint vector registered for the object identifier to obtain a comparison result, sends the audio signal to the audio interaction terminal in a case where it is determined according to the comparison result that the current speaking voice of the audio signal comes from the registered object, and performs voice transformation processing on the audio signal to obtain a transformed audio signal and sends the transformed audio signal to the audio interaction terminal in a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object. In this way, in a scenario where multiple people share audio interaction initiated by an object identifier, the voice information of members other than the registered object can be protected from being directly exposed to the audio interaction terminal in the process of audio interaction, and the audio interaction initiated by the object identifier can also not be affected, so that the voice information of other personnel can be protected and the personal information of the user can be protected.

[0071] In one embodiment, before calculating the real-time voiceprint vector of the real-time collected audio signal, the method further comprises the step of performing liveliness detection on the real-time collected audio signal to obtain a detection result.

[0072] That is, the audio processing method comprises: in the process of audio interaction with the audio interaction terminal in the identity identified by the object, real-time collection of audio signals; performing liveliness detection on the real-time collected audio signal to obtain a detection result, in the case that the detection result indicates that the audio signal contains speech sound, then calculating the real-time voiceprint vector of the real-time collected audio signal, comparing the calculated real-time voiceprint vector with the reference voiceprint vector registered for the object identification to obtain a comparison result, in the case that it is determined according to the comparison result that the current speech sound of the audio signal is derived from the registered object, then sending the audio signal to the audio interaction terminal, in the case that it is determined according to the comparison result that the current speech sound of the audio signal is not derived from the registered object, then performing voice transformation on the audio signal to obtain a transformed audio signal, and sending the transformed audio signal to the audio interaction terminal. In the case that the detection result indicates that the audio signal does not contain speech sound, the current real-time collected audio signal can be directly transmitted to the audio interaction terminal, and the steps of performing audio signal collection, performing liveliness detection on the real-time collected audio signal to obtain a detection result are continued.

[0073] Among them, the voice liveliness detection can be used to detect whether the current real-time collected audio signal includes the speaker sound, that is, whether it contains human voice. If it contains human voice, the subsequent steps for implementing sound protection are executed, if it does not contain human voice, the subsequent steps for implementing sound protection are not needed to be executed, but the current real-time collected audio signal is directly transmitted to the audio interaction terminal, and the audio signal collection and the liveliness detection on the real-time collected audio signal are continued.

[0074] In this embodiment, by performing voice liveliness detection on the real-time collected audio signal, the subsequent processing steps for implementing sound protection on the audio signal not containing human voice can be avoided, resource waste can be avoided, and audio transmission efficiency can be improved.

[0075] In one embodiment, the liveliness detection on the real-time collected audio signal to obtain a detection result comprises: dividing the real-time collected audio signal into a plurality of audio frames; converting each audio frame into a frequency domain signal; calculating the speech sound energy corresponding to the speech spectrum in the audio frame according to the frequency domain signal; calculating the total energy of the audio frame according to the frequency domain signal; when the proportion of the speech sound energy to the total energy is greater than or equal to a set threshold, obtaining a detection result that the audio signal contains speech sound; when the proportion of the speech sound energy to the total energy is less than the set threshold, obtaining a detection result that the audio signal does not contain speech sound.

[0076] wherein the audio frame refers to a data frame with a certain time length divided from the real-time collected audio signal, and generally has a time length of 20 ms or 10 ms, which can be used for subsequent audio processing and analysis. The frequency domain signal refers to the signal form presented after the audio signal is converted to the frequency domain, and the characteristic information of the audio signal at different frequencies can be obtained through analysis thereof. The speaking voice energy refers to the energy corresponding to the speaking spectrum in the audio frame, which can be obtained through analysis and calculation of the frequency domain signal, and can be used to determine whether the audio signal contains speaking voice. The total energy refers to the total energy of the audio frame, which can be obtained through analysis and calculation of the frequency domain signal, and can be used in combination with the speaking voice energy to determine whether the audio signal contains speaking voice.

[0077] Specifically, the input of the voice activity detection is an audio frame, which is generally a data frame of 20 ms or 10 ms. The audio frame is analyzed, for example, the audio frame can be converted into a frequency domain signal, the proportion of the energy of the part below 2 kHz (the speaking spectrum corresponding to the speaking voice) to the total energy of the audio frame is calculated, and when it is greater than or equal to a certain set threshold, it is considered that the audio frame contains speaking voice, otherwise it is considered that the audio frame does not contain speaking voice. Optionally, when the voice activity detection audio frame does not contain human voice for more than a certain time length threshold, it is determined that the human voice disappears. As shown in FIG. 6, it is a flowchart of voice activity detection in an embodiment. Referring to FIG. 6, the input is a piece of audio frame, and the energy of the audio frame is calculated to obtain the detection result of whether it contains human voice.

[0078] In this embodiment, by dividing the audio signal into an audio frame and converting it into a frequency domain signal, whether the current real-time collected audio signal contains a speaker voice can be effectively, quickly and accurately detected.

[0079] In one embodiment, a real-time voiceprint vector of the real-time collected audio signal is calculated, the calculated real-time voiceprint vector is compared with a reference voiceprint vector registered for an object identifier, and a comparison result is obtained, including: obtaining a general background model trained based on speech samples from different speakers, adjusting model parameters of the general background model by using the real-time collected audio signal through an adaptive algorithm to obtain a Gaussian mixture model corresponding to the real-time collected audio signal, the Gaussian mixture model being obtained by linearly combining a plurality of Gaussian distribution functions according to model parameters corresponding to each Gaussian distribution function; obtaining a real-time voiceprint vector of the real-time collected audio signal according to the model parameters corresponding to each Gaussian distribution function; obtaining a reference voiceprint vector registered for the object identifier, comparing the real-time voiceprint vector of the real-time collected audio signal with the reference voiceprint vector registered for the object identifier, and obtaining a comparison result.

[0080] Due to the differences in oral cavities and other aspects of different people, the same sound will also have differences when heard, that is, different timbres, which is reflected in the frequency domain as different frequency distributions corresponding to the same sound by different people, and the frequency distribution can be regarded as a continuous probability distribution function. A Gaussian mixture model is obtained by modeling the audio of the registered object using a universal background model, which is a continuous probability distribution function described by multiple Gaussian functions, and the parameters of the probability distribution functions corresponding to different objects are also different. Based on the universal background model, the model parameters (multiple parameters used to combine multiple Gaussian functions) of the Gaussian mixture model corresponding to the registered object can be represented as a voiceprint vector.

[0081] The universal background model (UBM) is a large speech feature distribution model used to represent speaker-independent feature distribution, and its training data comes from background data collected from speech samples of different speakers. In this embodiment, a universal background model is trained by first collecting a large number of speech samples unrelated to the registered object, and then adjusting the model parameters of the universal background model using a small amount of speech of the registered object through an adaptive algorithm to obtain a Gaussian mixture model (GMM), i.e., a speaker model of the registered object. The multiple Gaussian distribution functions are linearly combined according to the model parameters corresponding to each Gaussian distribution function, and the parameters corresponding to different objects are different. The parameters of all probability distribution functions are represented as a vector to obtain a real-time voiceprint vector of an object. After obtaining the real-time voiceprint vector of the real-time collected audio signal, the reference voiceprint vector registered for the object identification is obtained, and the real-time voiceprint vector of the real-time collected audio signal is compared with the reference voiceprint vector registered for the object identification to obtain a comparison result. According to the comparison result, it can be accurately determined whether the real-time collected audio signal is from the registered object. The Gaussian mixture model is a model obtained by linearly combining multiple Gaussian distribution functions according to the model parameters corresponding to each Gaussian distribution function, and the parameters corresponding to different objects are different, which can be used to describe the distribution of speech features.

[0082] In one embodiment, the reference voiceprint vector registered for the object identification includes:

[0083] The model parameters of the universal background model are adjusted using the sound source signal of the registered object through an adaptive algorithm to obtain a Gaussian mixture model corresponding to the sound source signal of the registered object, and the Gaussian mixture model is obtained by linearly combining multiple Gaussian distribution functions according to the model parameters corresponding to each Gaussian distribution function. The reference voiceprint vector registered for the object identification is obtained according to the model parameters corresponding to each Gaussian distribution function.

[0084] Similarly, the reference voiceprint vector of the registration object is also obtained from a Gaussian mixture model of the registration object, which is modeled based on a general background model and an audio signal originating from the registration object. The computer device can determine the reference voiceprint vector registered for the object identifier according to model parameters of the obtained Gaussian mixture model.

[0085] In an embodiment, the method further comprises: after logging in the audio interaction client with the object identifier, prompting for voiceprint extraction; in response to a voiceprint extraction confirmation operation, collecting a sound source signal of the registration object; and after obtaining the reference voiceprint vector registered for the object identifier using the sound source signal of the registration object, saving the reference voiceprint vector registered for the object identifier in the audio interaction client.

[0086] In the embodiment, after logging in the audio interaction client with the object identifier, the registration object is prompted for voiceprint extraction. For example, after logging in the audio interaction client with the object identifier, the audio interaction client can immediately prompt the registration object for voiceprint extraction to protect the sound of other objects except the registration object. For another example, after logging in the audio interaction client with the object identifier, when the registration object creates a new online conference through the audio interaction client, the registration object can be prompted for voiceprint extraction. Alternatively, before or after the audio interaction is started, the audio interaction client can support starting or stopping the protection of the sound of other objects except the registration object. The computer device can collect a sound source signal of the registration object in response to a voiceprint extraction confirmation operation triggered by the registration object, and obtain the reference voiceprint vector registered for the object identifier using the sound source signal of the registration object in the above manner. After obtaining the reference voiceprint vector registered for the object identifier, the reference voiceprint vector registered for the object identifier is saved in the audio interaction client, so as to facilitate subsequent voiceprint comparison and improve the comparison efficiency.

[0087] In an embodiment, the voice-changing processing on the audio signal to obtain a changed audio signal comprises: pitch processing on the audio signal to obtain a pitch-processed audio signal; and speed processing on the pitch-processed audio signal to obtain the changed audio signal.

[0088] The pitch processing can be implemented by upsampling or downsampling the frequency of the audio signal. For example, when performing linear interpolation on the audio signal, a suitable interpolation factor can be selected according to actual requirements to adjust the frequency, so as to change the pitch of the sound. The speed processing can be implemented by compressing or expanding the pitch-processed audio signal in the time domain. For example, the audio signal is divided into audio frames with a preset time length (e.g., 20 ms), each audio frame is stretched or compressed in the time dimension, and then the audio frames are synthesized in the time sequence to restore or adjust the audio time length, so as to obtain the changed audio signal.

[0089] Optionally, the audio signal is pitch-shifting processed to obtain a pitch-shifting processed audio signal, including: converting the audio signal into a frequency domain signal; sampling the frequency domain signal according to frequency to obtain a sampling frequency and a corresponding sampling amplitude; converting the frequency domain signal with the sampling frequency and the corresponding sampling amplitude into a time domain signal, and taking the obtained time domain signal as the pitch-shifting processed audio signal.

[0090] Since different people sing the same sound, the difference is mainly reflected in the frequency distribution, especially the difference in fundamental frequency, so resampling the frequency of the audio signal can realize the pitch-shifting of the sound, and by linear interpolation of the audio signal, the frequency of the speech signal can be scaled to realize the pitch-shifting of the sound. The sampling frequency here is the frequency sampled, and according to the obtained sampling frequency and the corresponding sampling amplitude, the computer device can generate a time domain signal as the pitch-shifting processed audio signal, which can ensure that the sound pitch changes greatly and protect the user's voice information. Wherein, sampling the frequency domain signal according to frequency can be upsampling or downsampling.

[0091] Optionally, before converting the audio signal into a frequency domain signal, the method further includes: performing pitch detection on the audio signal to determine a pitch period of the audio signal; determining a pitch period range to which the pitch period of the audio signal belongs, and obtaining a sampling rate corresponding to the pitch period range to which the pitch period of the audio signal belongs according to a pre-set corresponding relationship between the pitch period range and the sampling rate; sampling the frequency domain signal according to frequency to obtain a sampling frequency and a corresponding sampling amplitude, including: sampling the frequency domain signal according to frequency using the sampling rate to obtain a sampling frequency and a corresponding sampling amplitude.

[0092] Wherein, the pitch detection refers to the process of determining the pitch period in the audio signal, and in speech signal processing, the pitch period is an important parameter for describing the periodicity of the basic frequency (i.e. fundamental frequency) of vocal cord vibration. The pitch period range refers to the interval range divided according to the pitch period characteristics of different audio signals, which is used for subsequent determination of the corresponding sampling rate to avoid too large difference between the pitch and the original speed after pitch conversion. The sampling rate refers to the rate used when sampling the frequency domain signal according to frequency, which is determined according to the pitch period range to which the pitch period of the audio signal belongs, to ensure the natural and harmonious of the audio after voice conversion.

[0093] In the speech signal processing, the pitch period is an important parameter, which describes the basic frequency of vocal cord vibration, i.e. the periodicity of the fundamental frequency. For example, assuming that the up-sampling factor is P and the down-sampling factor is Q, the up-sampling process is to interpolate P-1 sampling points in the original frequency domain signal, so that the pitch period becomes P times of the original, the spectrum is compressed to 1 / P of the original, the time length is P times of the original, the pitch is lowered to 1 / P of the original, and the speech speed is 1 / P of the original. The down-sampling process is to extract every Q-1 points, so that the pitch period length becomes 1 / Q of the original, the spectrum is expanded to Q times of the original, the time length is 1 / Q of the original, the pitch is raised to Q times of the original, and the speech speed is Q times of the original. In order to avoid too large differences in pitch and speech speed after pitch conversion, causing unnatural speech, in this embodiment, the computer device can determine the pitch period range to which the pitch period of the audio signal belongs, and determine the corresponding sampling rate according to the pre-set correspondence between the pitch period range and the sampling rate, and sample the frequency domain signal according to the determined sampling rate to obtain the sampling frequency and the corresponding sampling amplitude.

[0094] Optionally, the step of performing speed conversion on the pitch-converted audio signal comprises: dividing the pitch-converted audio signal into different audio frames, each audio frame having a preset time length; stretching each audio frame in the time dimension, and then synthesizing the stretched audio frames in chronological order to obtain the variable audio signal.

[0095] Since the time length of the pitch-converted audio signal has also changed, the time length needs to be restored. Specifically, the computer device can perform compression or expansion processing on the pitch-converted audio signal in the time domain. Specifically, if the time length of the pitch-converted audio signal is compressed, the computer device can divide the audio signal into different frames, stretch each frame, and then recombine the frames to obtain the final variable audio signal. If the time length of the pitch-converted audio signal is expanded, the computer device can divide the audio signal into different frames, compress each frame, and then recombine the frames to obtain the final variable audio signal.

[0096] In one embodiment, the method of performing voice conversion on the audio signal to obtain a variable audio signal comprises: inputting the audio signal into a voice conversion model to extract features to obtain speech recognition features of the audio signal; inputting a sound source signal of a preset object into the voice conversion model to extract features to obtain speech recognition features of the sound source signal; and mapping the audio signal to the sound of the preset object through the voice conversion model according to the speech recognition features of the sound source signal and the speech recognition features of the audio signal to obtain the variable audio signal.

[0097] The voice conversion model refers to an artificial intelligence model trained by samples and capable of converting a voice to a specified timbre. The voice conversion model can be used for voice conversion processing of an audio signal and mapping the audio signal to a preset object voice.

[0098] In this embodiment, the computer device can perform voice conversion processing on the audio signal by using the trained voice conversion model. The voice conversion model is an artificial intelligence model trained by samples and capable of converting a voice to a specified timbre. The computer device can input the obtained audio signal into the voice conversion model to extract features, thereby obtaining voice recognition features of the audio signal. The voice recognition features can capture linguistic information of the audio signal. The computer device can input the sound source signal of the preset object into the voice conversion model to extract features, thereby obtaining voice recognition features of the sound source signal. The voice recognition features capture timbre information of the sound source signal. The computer device can map the audio signal to the voice of the preset object by using the voice conversion model according to the voice recognition features of the audio signal and the voice recognition features of the sound source signal, thereby obtaining a converted audio signal. The sound source signal of the preset object can be a synthesized voice signal of a virtual object, a current registered object, or any other default preset sound signal. That is, the computer device can convert the voices of other persons except the registered object to the voice of a specified object by using the voice conversion model, thereby protecting the voices of the other persons. The specified object can be the registered object. The voice conversion method based on the model has good effects in timbre similarity and naturalness, and can ensure natural and harmonious converted audio signals.

[0099] In one embodiment, in a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, the audio signal is subjected to voice conversion processing to obtain a converted audio signal, and the converted audio signal is sent to the audio interaction terminal, including: in a case where it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, the audio signal is subjected to voice conversion processing to obtain a converted audio signal; a sound source identifier indicating that the speaking voice does not come from the registered object is added to the converted audio signal, and the converted audio signal with the added sound source identifier is sent to the audio interaction terminal to prompt that the current speaking voice does not come from the registered object.

[0100] For example, in one application scenario, the audio interaction terminal can prompt the audio interaction terminal user that the current speaking voice does not come from the registered object when the audio interaction terminal extracts a sound source identifier in the received variable voice signal indicating that the speaking voice does not come from the registered object, such as can prompt "current is not XX speaking", wherein "XX" represents a registered object that does not need to be protected by voice. In this way, although the audio interaction terminal user cannot identify through voice that the current local terminal is spoken by which other object except the registered object, it can also be known that the current is not the registered object speaking, and no one will feel unusual, which can improve the audio interaction experience of the audio interaction terminal user.

[0101] As shown in FIG. 7, it is a schematic diagram of a voice protection module in one embodiment, including four modules of voiceprint registration, speech activity detection, voiceprint comparison and voice processing. Among them, the voiceprint registration is used to record the voiceprint vector of the registered object as the reference voiceprint vector, the speech activity detection is used to detect whether the real-time collected audio signal in the audio interaction process contains human voice, the voiceprint comparison is used to extract the voiceprint vector when the audio signal contains human voice and compare it with the reference voiceprint vector registered for the object identifier, and the voice processing is used to perform voice processing on the audio signal when the audio signal contains human voice of other objects except the registered object.

[0102] For example, in a multi-person conference, the reference voiceprint vector of the host is first recorded by the voiceprint registration module, and after the host records a segment of audio in the voiceprint registration module, the voiceprint registration module will generate the reference voiceprint vector of the host. When the collected audio signal is detected by the speech activity detection and found not to contain speech signal, it is considered not to involve privacy, no voiceprint recognition is performed, and it is directly sent to the encoding module. When it contains speech signal, voiceprint recognition is performed, the calculated parameter vector is compared with the voiceprint vector of the host saved by the voiceprint registration module, when they are close, it is considered to be the voice of the host, otherwise it is considered to be the voice of other users, then the voiceprint recognition module includes the information in the speech data frame and sends it to the voice processing module. The voice processing module processes the sound according to the identifier in the speech frame, when the speech frame does not contain valid human voice, no processing is performed, and the speech is sent to the encoding module. When it contains speech frame, it needs to be seen whether it is the voice of the host, if it is the voice of the host, then no processing is performed and it is sent to the encoding module, if it is not the voice of the host, then the voice needs to be processed.

[0103] As shown in FIG. 8, it is a detailed step diagram of sound protection in a detailed embodiment. Referring to FIG. 8, the audio signal is collected, the real-time collected audio signal is subjected to speech activity detection, the detection result indicates that the human voice is included, then further voiceprint recognition is performed, otherwise the detection result indicates that the human voice is not included, then no sound protection processing is needed, if the voiceprint recognition result indicates that the voice of other person is included, then voice changing processing is performed, otherwise no voice changing processing is needed. In this way, the identity information of other users around the registered object can be protected without manual operation of the user, meanwhile the normal audio interaction is not affected, and the accuracy is high and misjudgment is not easy to occur.

[0104] It should be understood that, although each step in the flowchart involved in each of the above embodiments is shown in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other sequences. Moreover, at least part of the steps in the flowchart involved in each of the above embodiments can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution sequence of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0105] Based on the same inventive concept, the present embodiment also provides an audio processing device for implementing the above-mentioned audio processing method. The problem-solving implementation scheme provided by the device is similar to the implementation scheme described in the above method, so the specific limitations in one or more audio processing device embodiments provided below can refer to the limitations of the audio processing method described above, which will not be repeated here.

[0106] In an exemplary embodiment, as shown in FIG. 9, an audio processing device 900 is provided, which includes an audio interaction initiation module 902, a comparison module 904 and a sending module 906, wherein:

[0107] The audio interaction initiation module 902 is configured to collect an audio signal in real time during the process of audio interaction with an audio interaction terminal in the identity of an object identifier;

[0108] The comparison module 904 is configured to calculate a real-time voiceprint vector of the real-time collected audio signal, compare the calculated real-time voiceprint vector with the reference voiceprint vector registered for the object identifier, and obtain a comparison result;

[0109] The sending module 906 is configured to send the audio signal to the audio interaction terminal in the case that it is determined according to the comparison result that the current speaking voice of the audio signal is derived from the registered object.

[0110] The sending module 906 is configured to, in a case where it is determined according to the comparison result that the current speech sound of the audio signal does not come from the registered object, perform voice conversion processing on the audio signal to obtain a converted audio signal, and send the converted audio signal to the audio interaction terminal.

[0111] In an embodiment, the comparison module 904 is further configured to perform activity detection on the real-time collected audio signal to obtain a detection result, and in a case where the detection result indicates that the audio signal contains speech sound, perform the steps of calculating a real-time voiceprint vector of the real-time collected audio signal, and comparing the calculated real-time voiceprint vector with the reference voiceprint vector registered for the object identifier; and in a case where the detection result indicates that the audio signal does not contain speech sound, continue to collect the audio signal and perform activity detection on the real-time collected audio signal.

[0112] In an embodiment, the comparison module 904 is further configured to divide the real-time collected audio signal into a plurality of audio frames, convert each audio frame into a frequency domain signal, calculate a speech sound energy corresponding to speech spectrum in the audio frame according to the frequency domain signal, calculate a total energy of the audio frame according to the frequency domain signal, obtain a detection result that the audio signal contains speech sound when a proportion of the speech sound energy to the total energy is greater than or equal to a set threshold, and obtain a detection result that the audio signal does not contain speech sound when the proportion of the speech sound energy to the total energy is less than the set threshold.

[0113] In an embodiment, the comparison module 904 is further configured to obtain a general background model trained based on speech samples from different speakers, adjust model parameters of the general background model using the real-time collected audio signal through an adaptive algorithm to obtain a Gaussian mixture model corresponding to the real-time collected audio signal, the Gaussian mixture model being obtained by linearly combining a plurality of Gaussian distribution functions according to model parameters corresponding to each Gaussian distribution function, obtain a real-time voiceprint vector of the real-time collected audio signal according to the model parameters corresponding to each Gaussian distribution function, obtain the reference voiceprint vector registered for the object identifier, and compare the real-time voiceprint vector of the real-time collected audio signal with the reference voiceprint vector registered for the object identifier to obtain a comparison result.

[0114] In an embodiment, the comparison module 904 is further configured to adjust model parameters of the general background model using the sound source signal of the registered object through an adaptive algorithm to obtain a Gaussian mixture model corresponding to the sound source signal of the registered object, the Gaussian mixture model being obtained by linearly combining a plurality of Gaussian distribution functions according to model parameters corresponding to each Gaussian distribution function, and obtain the reference voiceprint vector registered for the object identifier according to the model parameters corresponding to each Gaussian distribution function.

[0115] In an embodiment, the comparison module 904 is further configured to, after logging in to the audio interaction client using the object identifier, prompt for voiceprint extraction; in response to a voiceprint extraction confirmation operation, collect a sound source signal of the registered object; and after obtaining the reference voiceprint vector registered for the object identifier using the sound source signal of the registered object, save the reference voiceprint vector registered for the object identifier in the audio interaction client.

[0116] In an embodiment, the audio processing apparatus 900 further includes:

[0117] The voice changing module is configured to sequentially perform pitch processing and speed processing on the audio signal to obtain a changed audio signal.

[0118] In an embodiment, the voice changing module is further configured to convert the audio signal into a frequency domain signal; sample the frequency domain signal according to frequency to obtain a sampling frequency and a corresponding sampling amplitude; and convert the frequency domain signal with the sampling frequency and the corresponding sampling amplitude of the sampling frequency into a time domain signal, and use the obtained time domain signal as the audio signal after pitch processing.

[0119] In an embodiment, the voice changing module is further configured to, before converting the audio signal into a frequency domain signal, perform pitch detection on the audio signal to determine a pitch period of the audio signal; determine a pitch period range to which the pitch period of the audio signal belongs, and according to a preset correspondence between pitch period ranges and sampling rates, obtain a sampling rate corresponding to the pitch period range to which the pitch period of the audio signal belongs; and sample the frequency domain signal according to frequency using the sampling rate to obtain a sampling frequency and a corresponding sampling amplitude.

[0120] In an embodiment, the voice changing module is further configured to divide the audio signal after pitch processing into different audio frames, and the length of each audio frame is a preset length; stretch each audio frame in the time dimension, and then synthesize the stretched audio frames in chronological order to obtain a changed audio signal.

[0121] In an embodiment, the voice changing module is further configured to input the audio signal into a voice changing speech model to extract features to obtain speech recognition features of the audio signal; input a sound source signal of a preset object into the voice changing speech model to extract features to obtain speech recognition features of the sound source signal; and according to the speech recognition features of the sound source signal and the speech recognition features of the audio signal, map the audio signal to the sound of the preset object through the voice changing speech model to obtain a changed audio signal.

[0122] In an embodiment, the sending module 906 is further configured to, in a case where it is determined according to the comparison result that the current speech sound of the audio signal does not come from the registered object, perform voice changing processing on the audio signal to obtain a changed audio signal, add a sound source identifier indicating that the speech sound does not come from the registered object in the changed audio signal, and send the changed audio signal with the sound source identifier to the audio interaction terminal to prompt that the current speech sound does not come from the registered object.

[0123] The audio processing apparatus 900 described above, in the process of audio interaction with the audio interaction terminal in the identity of the object identifier, collects audio signals in real time, calculates a real-time voiceprint vector of the real-time collected audio signals after starting audio signal collection, compares the calculated real-time voiceprint vector with a reference voiceprint vector registered for the object identifier to obtain a comparison result, sends the audio signal to the audio interaction terminal in a case where it is determined according to the comparison result that the current speech sound of the audio signal comes from the registered object, and performs voice changing processing on the audio signal to obtain a changed audio signal in a case where it is determined according to the comparison result that the current speech sound of the audio signal does not come from the registered object, and sends the changed audio signal to the audio interaction terminal. In this way, in a scenario where multiple people share the audio interaction initiated by the object identifier, the voice information of the other members except the registered object can be prevented from being directly exposed to the audio interaction terminal in the process of audio interaction, and the audio interaction initiated by the object identifier can also not be affected, the voice information of the other people can be protected in the process of audio interaction, and the personal information of the user can be protected.

[0124] Each module in the audio processing apparatus 900 described above can be realized by software, hardware, and a combination thereof in whole or in part. Each module described above can be embedded in or independent of a processor in a computer device in hardware form, or stored in a memory in the computer device in software form to be called and executed by a processor to perform the operations corresponding to each module.

[0125] In an example embodiment, a computer device, which can be a server, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 10. The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program is executed by the processor to implement an audio processing method.

[0126] In an example embodiment, a computer device, which can be a terminal, is provided, and an internal structure diagram of the computer device can be as shown in FIG. 11. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected through a system bus, and the communication interface, the display unit, and the input device are connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals in a wired or wireless manner. The wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC), or other technologies. The computer program is executed by the processor to implement an audio processing method. The display unit of the computer device is configured to form a visually visible picture, which can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or can be a key, a trackball, or a touchpad arranged on the shell of the computer device, or can be an external keyboard, a touchpad, a mouse, or the like.

[0127] Those skilled in the art can understand that the structures shown in FIG. 10 and FIG. 11 are only block diagrams of part of the structures related to the solutions of the present application, and do not constitute a limitation on the computer device to which the solutions of the present application are applied. The specific computer device can include more or fewer components than those shown in the figures, or combine certain components, or have a different arrangement of components.

[0128] In an example embodiment, a computer device is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the audio processing method provided by the embodiments of the present application when executing the computer program, for example:

[0129] In the process of audio interaction with the audio interaction terminal in the identity of the object identifier, the audio signal is collected in real time;

[0130] The real-time voiceprint vector of the real-time collected audio signal is calculated, and the calculated real-time voiceprint vector is compared with the reference voiceprint vector of the registered object corresponding to the object identifier, to obtain a comparison result;

[0131] In the case where it is determined according to the comparison result that the current speaking voice of the audio signal is derived from the registered object, the audio signal is sent to the audio interaction terminal;

[0132] In the case where it is determined according to the comparison result that the current speaking voice of the audio signal is not derived from the registered object, the audio signal is processed to obtain a changed audio signal, and the changed audio signal is sent to the audio interaction terminal.

[0133] In an example embodiment, a computer readable storage medium is provided, which stores a computer program, and the computer program is executed by a processor to implement the steps of the audio processing method provided by the embodiments of the present application.

[0134] In an example embodiment, a computer program product is provided, including a computer program, and the computer program is executed by a processor to implement the steps of the audio processing method provided by the embodiments of the present application.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0136] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above-mentioned embodiment methods. Any reference to memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. The non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. The volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration but not limitation, the RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc. The database involved in the embodiments provided in the present application can include at least one of a relational database and a non-relational database. The non-relational database can include a distributed database based on a block chain, etc., without being limited thereto. The processor involved in the embodiments provided in the present application can be a general-purpose processor, a central processing unit, a graphics processing unit, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., without being limited thereto.

[0137] In summary, the present application provides an audio processing method and device, computer equipment, computer readable storage medium and computer program product. When interacting with an audio interaction terminal in the identity of an object identifier, real-time audio signals are collected, real-time voiceprint vectors are calculated and compared with reference voiceprint vectors registered for the object identifier. If the current speaking voice is from the registered object, the audio signals are directly sent to the audio interaction terminal. If the current speaking voice is not from the registered object, the audio signals are processed to obtain variable audio signals which are then sent to the audio interaction terminal. This can avoid exposing the voice information of members other than the registered object to the audio interaction terminal in a multi-person shared object identifier initiated audio interaction scenario, protecting the voice information of other personnel and user personal information.

[0138] Further, before calculating the real-time voiceprint vector, the real-time collected audio signals are subjected to activity detection. If the detection result indicates that the audio signals contain speaking voices, the calculation of the real-time voiceprint vector and the comparison are performed. If the detection result indicates that the audio signals do not contain speaking voices, the audio signal collection and activity detection are continued, avoiding subsequent sound protection processing steps for audio signals that do not contain human voices, reducing resource waste and improving audio transmission efficiency.

[0139] Further, when the real-time collected audio signals are subjected to activity detection, they are divided into multiple audio frames, each audio frame is converted into a frequency domain signal, the speaking voice energy corresponding to the speaking spectrum in the audio frame and the total energy of the audio frame are calculated based on the frequency domain signal. When the proportion of the speaking voice energy to the total energy is greater than or equal to a set threshold, it is determined that the audio signals contain speaking voices. When the proportion is less than the set threshold, it is determined that the audio signals do not contain speaking voices. The current real-time collected audio signals can be effectively, quickly and accurately detected to determine whether they contain speaking voices.

[0140] Further, when the real-time voiceprint vector is calculated and compared with the reference voiceprint vector, a general background model trained based on different speaker voice samples is obtained. The model parameters of the general background model are adjusted by an adaptive algorithm using the real-time collected audio signals to obtain a Gaussian mixture model corresponding to the real-time collected audio signals. The model is linearly combined by multiple Gaussian distribution functions according to their respective model parameters. The real-time voiceprint vector of the real-time collected audio signals is obtained according to the model parameters of each Gaussian distribution function. Finally, the real-time voiceprint vector is compared with the reference voiceprint vector registered for the object identifier to obtain a comparison result, which can accurately determine whether the real-time collected audio signals are from the registered object.

[0141] Further, when the reference voiceprint vector registered for the object identifier is acquired, the model parameters of the general background model are adjusted by an adaptive algorithm using the sound source signal of the registered object to obtain a Gaussian mixture model corresponding to the sound source signal of the registered object, and the reference voiceprint vector registered for the object identifier is obtained according to the model parameters corresponding to each Gaussian distribution function.

[0142] Further, after the object identifier logs in the audio interaction client, voiceprint extraction is prompted, and in response to a voiceprint extraction confirmation operation, the sound source signal of the registered object is collected. After the reference voiceprint vector registered for the object identifier is obtained using the sound source signal of the registered object, the reference voiceprint vector is saved in the audio interaction client, so as to facilitate subsequent voiceprint comparison and improve comparison efficiency.

[0143] Further, when the audio signal is processed to obtain a variable sound audio signal, the audio signal is first processed to obtain a variable pitch audio signal, and then the variable pitch audio signal is processed to obtain a variable speed audio signal. The frequency and time length are adjusted to realize voice change, and the sound pitch change is ensured to be more natural and harmonious.

[0144] Further, when the audio signal is processed to obtain a variable pitch audio signal, the audio signal is converted into a frequency domain signal, the frequency domain signal is sampled according to frequency to obtain a sampling frequency and a corresponding sampling amplitude, and the frequency domain signal with these parameters is converted into a time domain signal as the variable pitch audio signal. The sound pitch change is ensured to be large, and the user's voice information is protected.

[0145] Further, before the audio signal is converted into a frequency domain signal, the pitch of the audio signal is detected to determine the pitch period, the pitch period range to which the pitch period belongs is determined, the corresponding sampling rate of the range is obtained according to the pre-set corresponding relationship between the pitch period range and the sampling rate, and the frequency domain signal is sampled according to frequency using the sampling rate to obtain a sampling frequency and a corresponding sampling amplitude. After the pitch conversion, the difference between the pitch and the original speed is avoided to be too large, and the natural and harmonious sound after voice change is ensured.

[0146] Further, when the variable pitch audio signal is processed to obtain a variable speed audio signal, the variable pitch audio signal is divided into different audio frames, each audio frame has a preset time length, each audio frame is stretched in the time dimension, and the stretched audio frames are synthesized in chronological order to obtain a variable speed audio signal, and the time length of the variable pitch audio signal is restored.

[0147] Further, when the audio signal is processed to obtain a transformed audio signal, the audio signal is input into the voice transformation model to extract features to obtain voice recognition features of the audio signal, the sound source signal of the preset object is input into the voice transformation model to extract features to obtain voice recognition features of the sound source signal, and the audio signal is mapped to the sound of the preset object to obtain the transformed audio signal through the voice transformation model based on the two voice recognition features. The voice transformation based on the model has good effects in timbre similarity and naturalness, and can ensure voice transformation while making the transformed audio signal natural and harmonious.

[0148] Further, when it is determined according to the comparison result that the current speaking voice of the audio signal does not come from the registered object, after the audio signal is processed to obtain a transformed audio signal, a sound source identifier indicating that the speaking voice does not come from the registered object is added to the transformed audio signal, and the transformed audio signal with the added sound source identifier is sent to the audio interaction terminal to prompt that the current speaking voice does not come from the registered object, thereby improving the audio interaction experience of the user of the audio interaction terminal.

[0149] The technical features of the above embodiments can be combined in any manner. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictions, they should be considered as the scope of the present disclosure.

[0150] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent of the present application should be subject to the appended claims.

Claims

1. An audio processing method executed by a computer device, the method comprising: collecting an audio signal in real time in a process of audio interaction with an audio interaction terminal in an identity of an object identifier; calculating a real-time voiceprint vector of the audio signal collected in real time, comparing the calculated real-time voiceprint vector with a reference voiceprint vector of a registered object registered for the object identifier to obtain a comparison result; in a case where it is determined according to the comparison result that a current speaking voice of the audio signal is derived from the registered object, sending the audio signal to the audio interaction terminal; and in a case where it is determined according to the comparison result that the current speaking voice of the audio signal is not derived from the registered object, performing voice conversion processing on the audio signal to obtain a converted audio signal, and sending the converted audio signal to the audio interaction terminal.

2. The method of claim 1, further comprising: detecting an activity level of the audio signal collected in real time to obtain a detection result; in a case where the detection result indicates that the audio signal contains a speaking voice, performing the steps of calculating a real-time voiceprint vector of the audio signal collected in real time, and comparing the calculated real-time voiceprint vector with a reference voiceprint vector of a registered object registered for the object identifier; in a case where the detection result indicates that the audio signal does not contain a speaking voice, continuing to collect the audio signal and detecting an activity level of the audio signal collected in real time.

3. The method of claim 2, wherein the detecting an activity level of the audio signal collected in real time to obtain a detection result comprises: dividing the audio signal collected in real time into a plurality of audio frames; converting each audio frame into a frequency domain signal; calculating a speaking voice energy corresponding to a speaking spectrum in the audio frame according to the frequency domain signal; calculating a total energy of the audio frame according to the frequency domain signal; when a proportion of the speaking voice energy to the total energy is greater than or equal to a set threshold, obtaining a detection result that the audio signal contains a speaking voice; when the proportion of the speaking voice energy to the total energy is less than the set threshold, obtaining a detection result that the audio signal does not contain a speaking voice.

4. The method of any one of claims 1 to 3, wherein the calculating a real-time voiceprint vector of the audio signal collected in real time, and comparing the calculated real-time voiceprint vector with a reference voiceprint vector of a registered object registered for the object identifier to obtain a comparison result comprises: obtaining a universal background model trained based on speech samples from different speakers, adjusting model parameters of the universal background model by an adaptive algorithm using the audio signal collected in real time to obtain a Gaussian mixture model corresponding to the audio signal collected in real time, the Gaussian mixture model being obtained by linearly combining a plurality of Gaussian distribution functions according to model parameters corresponding to each Gaussian distribution function; obtaining a real-time voiceprint vector of the audio signal collected in real time according to the model parameters corresponding to each Gaussian distribution function. ​ Obtaining a reference voiceprint vector of the registered object registered for the object identifier, comparing a real-time voiceprint vector of a real-time audio signal collected in real time with the reference voiceprint vector of the registered object registered for the object identifier, and obtaining a comparison result.

5. The method of claim 4, wherein the obtaining the reference voiceprint vector of the registered object registered for the object identifier comprises: adjusting model parameters of the general background model by using the sound source signal of the registered object through an adaptive algorithm to obtain a Gaussian mixture model corresponding to the sound source signal of the registered object, the Gaussian mixture model being obtained by linearly combining a plurality of Gaussian distribution functions according to model parameters corresponding to each Gaussian distribution function; obtaining the reference voiceprint vector of the registered object registered for the object identifier according to the model parameters corresponding to each Gaussian distribution function.

6. The method of claim 5, further comprising: prompting for voiceprint extraction after logging in to the audio interaction client with the object identifier; collecting the sound source signal of the registered object in response to a voiceprint extraction confirmation operation; saving the reference voiceprint vector of the registered object registered for the object identifier in the audio interaction client after obtaining the reference voiceprint vector of the registered object registered for the object identifier by using the sound source signal of the registered object.

7. The method of any one of claims 1 to 6, wherein the voice-altering processing of the audio signal to obtain an altered audio signal comprises: pitch-altering processing of the audio signal to obtain a pitch-altered audio signal; speed-altering processing of the pitch-altered audio signal to obtain the altered audio signal.

8. The method of claim 7, wherein the pitch-altering processing of the audio signal to obtain the pitch-altered audio signal comprises: converting the audio signal into a frequency domain signal; sampling the frequency domain signal according to frequency to obtain a sampling frequency and a corresponding sampling amplitude; converting the frequency domain signal with the sampling frequency and the corresponding sampling amplitude into a time domain signal, and taking the obtained time domain signal as the pitch-altered audio signal.

9. The method of claim 8, wherein before the converting of the audio signal into the frequency domain signal, the method further comprises: pitch detection of the audio signal to determine a pitch period of the audio signal; determination of a pitch period range to which the pitch period of the audio signal belongs, and obtaining a sampling rate corresponding to the pitch period range to which the pitch period of the audio signal belongs according to a pre-set correspondence between pitch period ranges and sampling rates; the sampling of the frequency domain signal according to frequency to obtain a sampling frequency and a corresponding sampling amplitude comprises sampling the frequency domain signal according to frequency by using the sampling rate to obtain a sampling frequency and a corresponding sampling amplitude.

10. The method of claim 8 or 9, wherein the speed-altering processing of the pitch-altered audio signal comprises: dividing the pitch-altered audio signal into different audio frames, each audio frame having a preset time length. After stretching each audio frame in the time dimension, the stretched audio frames are sequentially synthesized to obtain a transformed audio signal.

11. The method of any one of claims 1-10, wherein the transforming the audio signal to obtain a transformed audio signal comprises: inputting the audio signal into a transformed speech voice model to extract a speech recognition feature of the audio signal; inputting a sound source signal of a preset object into the transformed speech voice model to extract a speech recognition feature of the sound source signal; mapping, by the transformed speech voice model, the audio signal to a sound of the preset object according to the speech recognition feature of the sound source signal and the speech recognition feature of the audio signal to obtain the transformed audio signal.

12. The method of any one of claims 1-11, wherein in a case where it is determined, according to the comparison result, that the current speaking sound of the audio signal does not come from the registered object, the audio signal is transformed to obtain a transformed audio signal, and the transformed audio signal is sent to the audio interaction terminal, comprising: transforming the audio signal to obtain a transformed audio signal in a case where it is determined, according to the comparison result, that the current speaking sound of the audio signal does not come from the registered object; adding a sound source identifier indicating that the speaking sound does not come from the registered object in the transformed audio signal, and sending the transformed audio signal with the sound source identifier to the audio interaction terminal to prompt that the current speaking sound does not come from the registered object.

13. An audio processing apparatus, comprising: an audio interaction initiation module configured to collect an audio signal in real time in a process of audio interaction with an audio interaction terminal in an identity of an object identifier; a comparison module configured to calculate a real-time voiceprint vector of the audio signal collected in real time, compare the calculated real-time voiceprint vector with a reference voiceprint vector of a registered object corresponding to the object identifier to obtain a comparison result; a sending module configured to send the audio signal to the audio interaction terminal in a case where it is determined, according to the comparison result, that the current speaking sound of the audio signal comes from the registered object; and the sending module is configured to transform the audio signal to obtain a transformed audio signal in a case where it is determined, according to the comparison result, that the current speaking sound of the audio signal does not come from the registered object, and send the transformed audio signal to the audio interaction terminal.

14. A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements steps of the method in any one of claims 1-12 when executing the computer program.

15. A computer readable storage medium, having a computer program stored thereon, wherein the computer program is executed by a processor to implement steps of the method in any one of claims 1-12.

16. A computer program product, comprising a computer program, wherein the computer program is executed by a processor to implement steps of the method in any one of claims 1-12. ​

Images