Audio processing method and device, electronic equipment and readable storage medium

Abstract

The invention discloses an audio processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of audio processing, and the method comprises the steps: separating a voice emitted by a sound source object from a to-be-processed audio as a first audio; performing voice recognition on the first audio to obtain a text corresponding to the first audio; extracting sound features of the sound source object; translating a text corresponding to the first audio into a target text of a target language; and obtaining a target audio based on the sound features of the sound source object and the target text. The reality sense of the obtained target audio is improved, and the split sense caused by the difference between the sound features of the target audio and the sound features of the sound source object is reduced.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and more specifically, to an audio processing method, device, electronic device and readable storage medium. Background Art

[0002] At present, with the development of electronic information technology, audio can be obtained and translated to obtain audio in another language. However, the audio in another language obtained by translating audio is currently less authentic and has a strong sense of separation from the audio before translation. Summary of the invention

[0003] The present application proposes an audio processing method, device, electronic device and readable storage medium.

[0004] In a first aspect, an embodiment of the present application provides an audio processing method, comprising: separating the speech emitted by a sound source object from the audio to be processed as a first audio; performing speech recognition on the first audio to obtain a text corresponding to the first audio; extracting the sound features of the sound source object; translating the text corresponding to the first audio into a target text in a target language; and obtaining a target audio based on the sound features of the sound source object and the target text.

[0005] In the second aspect, the embodiment of the present application also provides an audio processing device, including: an audio determination unit, a speech recognition unit, an extraction unit, a translation unit, and a target audio determination unit. Among them, the audio determination unit is used to separate the speech emitted by the sound source object from the audio to be processed as the first audio; the speech recognition unit is used to perform speech recognition on the first audio to obtain the text corresponding to the first audio; the extraction unit is used to extract the sound features of the sound source object; the translation unit is used to translate the text corresponding to the first audio into a target text in a target language; the target audio determination unit is used to obtain the target audio based on the sound features of the sound source object and the target text.

[0006] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: one or more processors; a memory; and one or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method described in the first aspect.

[0007] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, in which a program code is stored. The program code can be called by a processor to execute the method described in the first aspect above.

[0008] The audio processing method, device, electronic device and readable storage medium provided in the present application first separate the speech emitted by the sound source object from the audio to be processed as the first audio; further perform speech recognition on the first audio to obtain the text corresponding to the first audio; then extract the sound features of the sound source object; translate the text corresponding to the first audio into the target text of the target language; finally obtain the target audio based on the sound features of the sound source object and the target text. If the translated audio is generated by the sound corresponding to the preset sound features, the generated translated audio will have a strong sense of separation from the translated audio and a low sense of reality. In the present application, the target audio corresponding to the translated target text is generated based on the sound with the same sound features as the sound source object in the original audio to be processed, rather than generating the target audio through some preset sound features, thereby improving the realism of the target audio and reducing the sense of separation caused by the difference between the sound features of the target audio and the sound features of the sound source object.

[0009] Other features and advantages of the embodiments of the present application will be described in the subsequent description, and partly become apparent from the description, or be understood by practicing the embodiments of the present application. The purposes and other advantages of the embodiments of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 An application scenario diagram of the audio processing method provided by an embodiment of the present application is shown;

[0012] Figure 2 A method flow chart of an audio processing method provided by an embodiment of the present application is shown;

[0013] Figure 3 A method flow chart of an audio processing method provided by another embodiment of the present application is shown;

[0014] Figure 4 A method flow chart of an audio processing method provided by another embodiment of the present application is shown;

[0015] Figure 5 A method flow chart of an audio processing method provided by yet another embodiment of the present application is shown;

[0016] Figure 6A method flow chart of an audio processing method provided by another embodiment of the present application is shown;

[0017] Figure 7 The structure block diagram of the audio processing device provided by the embodiment of the present application is shown;

[0018] Figure 8 A structural block diagram of an electronic device provided in an embodiment of the present application is shown;

[0019] Fig. 9 A structural block diagram of a computer-readable storage medium provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] In order to make those skilled in the art better understand the present application scheme, the technical scheme in the present application embodiment will be clearly and completely described below in conjunction with the drawings in the present application embodiment. Obviously, the described embodiment is only a part of the present application embodiment, rather than all the embodiments. The components of the present application embodiment usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the present application for protection, but merely represents the selected embodiment of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0021] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0022] At present, with the development of electronic information technology, audio can be obtained and translated to obtain audio in another language. However, the audio in another language obtained by translating audio is currently less realistic and has a strong sense of disconnection from the audio before translation. How to improve the authenticity of the translated audio and reduce the sense of disconnection between the translated audio and the audio before translation is an urgent problem to be solved.

[0023] At present, the speech of the speaking object can be converted into text through speech recognition, and then the text is translated and displayed. For example, the translated text can be displayed on the display screen of the electronic device in the form of bullet screen. In addition, the translated text is generated into corresponding audio through preset sound.

[0024] However, the inventors discovered during research that generating corresponding audio from translated text using preset sounds makes the translated audio less realistic and there is a greater sense of disconnection between the translated audio and the voice of the object that makes the sound.

[0025] Therefore, in order to solve or partially solve the above problems, the present application provides an audio processing method, device, electronic device and readable storage medium.

[0026] See also Figure 1 , Figure 1 An application scenario diagram of the audio processing method provided in an embodiment of the present application is shown, namely, an audio processing scenario 100 , which includes an electronic device 110 and a user 120 .

[0027] exist Figure 1 The electronic device 110 shown in the figure is a smart phone, and the user 120 can use the electronic device 110. The electronic device 110 can run an operating system, so that the user 120 can execute the audio processing method through the electronic device 110. Exemplarily, the audio processing method may include translating audio in one language into audio in another language, and may also include translating audio in one language into sign language in another language. For an introduction to the audio processing method, please refer to the subsequent embodiments.

[0028] See also Figure 2 , Figure 2 A method flow chart of an audio processing method provided by an embodiment of the present application is shown. The audio processing method can be applied to Figure 1 The electronic device in the audio processing scenario shown may specifically use a processor of the electronic device as an execution subject for executing the audio processing method. The audio processing method may include steps S110 to S150.

[0029] Step S110: separating the speech emitted by the sound source object from the audio to be processed as the first audio.

[0030] The audio to be processed may be audio data acquired by the electronic device, which may be audio data collected by a microphone or other device, or may be audio data received by the electronic device, or may be audio data contained in multimedia data received by the electronic device.

[0031] Exemplarily, the subject is in a speaking state, so that the subject's speech can be collected through the microphone of the electronic device to generate audio data as the audio to be processed. In another exemplary embodiment, the user runs a conference application through the electronic device, so that the electronic device can obtain multimedia data through the conference application, and then obtain the audio data included in the multimedia data as the audio to be processed. Among them, the multimedia data can be collected and sent by other electronic devices participating in the same conference as the electronic device. In another exemplary embodiment, the electronic device can run an online video application, so that it can play online videos, and then use the audio data of the online video as the audio to be processed.

[0032] Optionally, the audio to be processed may also be the audio data obtained by the electronic device after the audio data is acquired, and the background sound is separated from the audio data to obtain the human voice audio in the audio data, as the audio to be processed. For a detailed introduction, please refer to the subsequent embodiments.

[0033] It can be understood that the audio to be processed includes speech sounds, and the object that emits the speech sounds is the sound source object. The sounds emitted by different sound source objects may have different voiceprint features. Therefore, the audio to be processed may be subjected to voiceprint analysis to obtain the voiceprint features of the sound source object in the audio to be processed. The voiceprint features include the sound characteristics of the corresponding sound source object, which can be used to uniquely characterize the identity information of the sound source object.

[0034] It should be noted that there may be only one sound source object in the audio to be processed. For example, when a user is recorded through an electronic device, the user is the sound source object. The audio to be processed may also include multiple sound source objects. For example, when multiple users conduct a video conference through various electronic devices, the audio data contained in the data obtained based on the video conference can be used as the audio to be processed, and each user contained in the audio to be processed can be used as a sound source object. In addition, the voiceprint features corresponding to each sound source object are different.

[0035] Optionally, the electronic device may obtain the audio to be processed and jump to step S110 when an audio processing request is obtained. Exemplarily, the audio processing request may be a translation request. The user may generate an audio processing request through a translation application running on the electronic device. For example, when the electronic device runs a translation application, the display interface may display a prompt text such as "Request Translation" and may also display an operation control corresponding to the prompt text, so that the user may generate an audio processing request by manipulating the operation control.

[0036] It is understandable that in addition to the sound emitted by the sound source object, the audio to be processed may also contain background audio, such as background noise. Therefore, after obtaining the voiceprint feature of the sound source object, the sound emitted by the sound source object can be searched in the audio to be processed as the first audio, thereby separating the voice emitted by the sound source object from the audio to be processed as the first audio.

[0037] Exemplarily, if the audio to be processed includes background noise and sound emitted by a sound source object, the sound source emitted by the sound source object can be determined as the first audio.

[0038] In another exemplary embodiment, if the audio to be processed includes multiple sound source objects, the first audio corresponding to each sound source object can also be determined in the audio to be processed.

[0039] For example, if the audio to be processed contains background noise, sound emitted by sound source object A1, sound emitted by sound source object A2, and sound emitted by sound source object A3. Then the sound emitted by sound source object A1 can be used as the first audio corresponding to sound source object A1; the sound emitted by sound source object A2 can be used as the first audio corresponding to sound source object A2; and the sound emitted by sound source object A3 can be used as the first audio corresponding to sound source object A3. In some embodiments, the sound emitted by the sound source object can be determined by speaker separation technology as the first audio corresponding to the sound source object. For a detailed description, please refer to the subsequent embodiments.

[0040] It should be noted that the audio to be processed includes content that needs to be translated, for example, the first audio needs to be translated. Translating the first audio may include translating the first audio into corresponding audio data, and may also include translating the text corresponding to the first audio to obtain the translated text.

[0041] Step S120: performing speech recognition on the first audio to obtain text corresponding to the first audio.

[0042] Furthermore, after obtaining the first audio, the text corresponding to the first audio can be obtained. In some embodiments, the text corresponding to the first audio can be obtained by performing speech recognition on the first audio. For example, the text corresponding to the first audio can be obtained by automatic speech recognition (Automatic Speech Recognition, ASR) technology. Among them, the automatic speech recognition technology can convert the speech into the text content corresponding to the speech by analyzing and processing the speech. For example, it can specifically include preprocessing, feature extraction, acoustic model modeling, language model modeling, decoding and post-processing steps. Optionally, the automatic speech recognition technology can also use complex neural network structures, such as recurrent neural networks (RNN), long short-term memory networks (LSTM), etc., to construct acoustic models and language models. Thereby, the recognition accuracy and efficiency of the automatic speech recognition technology can be significantly improved.

[0043] The text corresponding to the first audio may be represented in the form of a character string, thereby facilitating subsequent translation operations on the text corresponding to the first audio.

[0044] Step S130: extracting the sound features of the sound source object.

[0045] Among them, sound features refer to various attributes of sound. For example, sound features may include information such as timbre, intonation, and speech speed. The sound features of the sound source object can be extracted, so that the target audio can be generated in combination with the sound features later, which can reduce the sense of disconnection caused by the difference between the sound features of the target audio and the sound features of the sound source object.

[0046] The sound emitted by the sound source object can be feature extracted to obtain the sound features emitted by the sound source object. Exemplarily, the feature extraction can include steps such as short-time Fourier transform, Mel spectrum filtering, Mel frequency cepstrum analysis, linear prediction coding, and constant Q transform.

[0047] Optionally, before extracting features from the sound emitted by the sound source object, the sound emitted by the sound source object may be preprocessed. Exemplarily, the preprocessing may include noise reduction or sampling quantization.

[0048] Step S140: translating the text corresponding to the first audio into a target text in a target language.

[0049] Furthermore, after obtaining the text corresponding to the first audio, the text corresponding to the first audio can be translated into a target text, wherein the language of the target text is the target language.

[0050] For some embodiments, the target language can be pre-set. For example, it can be pre-set that no matter what language the text corresponding to the first audio is in, the text corresponding to the first audio is translated into a target text in the target language. For example, the target language can be Chinese, English, etc.

[0051] For other embodiments, before translating the text corresponding to the first audio into the target text of the target language, the electronic device may also display a prompt message through the display screen. Wherein, the prompt message is used to prompt the user to determine the target language. Exemplarily, the prompt message may include text information of "Please select the target language to be translated to", and may also include text information in multiple languages, such as "Chinese", "English", "German", "Japanese" and "French", etc., and operation controls corresponding to the text information in each language. Thus, the user can manipulate the operation control corresponding to the language based on the prompt information to select the language corresponding to the manipulated operation control, so as to use the language as the target language. For example, the user manipulates the operation control corresponding to "German" based on the prompt information, so as to use German as the target language. Wherein, manipulating the operation control may include touching the operation control on the display screen, and may also include selecting the operation control by keyboard and mouse manipulation. The embodiment of the present application does not specifically limit how to manipulate the operation control.

[0052] Among them, the text corresponding to the first audio can be translated into a target text in a target language based on an existing language translation model, and the language translation model can be a neural network model.

[0053] Step S150: obtaining target audio based on the sound features of the sound source object and the target text.

[0054] Furthermore, after obtaining the target text, since the text corresponding to the translated first audio is obtained from the sound emitted by the sound source object, if the target text is only displayed to the user in the form of text display, for example, the target text is displayed on the display screen of the electronic device in the form of a bullet screen, the translation effect will be poor, reducing the user experience. Therefore, after obtaining the target text, the target audio can also be obtained based on the sound characteristics of the sound source object and the target text.

[0055] From the above introduction, it can be known that the sound features can be used to characterize the various properties of the sound emitted by the sound source object. Therefore, the target audio obtained through the sound features of the sound source object and the target text is actually an audio that simulates the sound features of the sound source object, so that the sound features of the obtained target audio are the same or approximately the same as the sound features of the sound source object. Therefore, when the target audio is subsequently displayed to the user, for example, when the target audio is played through an electronic device, from the user's perspective, it is like hearing the sound source object making a sound in the target language, achieving a better translation effect and reducing the sense of separation between the target audio and the first audio.

[0056] It is understandable that the first audio corresponding to the sound source object, the candidate text corresponding to the first audio, the target text corresponding to the candidate text, and the generated target audio correspond to each other. That is, when there are multiple sound source objects in the audio to be processed, each sound source object can correspond to a target audio, and the sound features of each target audio are the same or approximately the same as the sound features of the sound source object corresponding to the target audio.

[0057] In some implementations, the corresponding audio may be first generated based on the target text, and then the audio may be adjusted according to the sound characteristics of the sound source object to obtain the target object. For a detailed description, please refer to the subsequent embodiments.

[0058] The audio processing method provided in the embodiment of the present application first separates the speech emitted by the sound source object from the audio to be processed as the first audio; further performs speech recognition on the first audio to obtain the text corresponding to the first audio; then extracts the sound features of the sound source object; translates the text corresponding to the first audio into the target text of the target language; and finally obtains the target audio based on the sound features of the sound source object and the target text. If the translated audio is generated by the sound corresponding to the preset sound features, the generated translated audio will have a strong sense of separation from the translated audio and a low sense of reality. In the present application, the target audio corresponding to the translated target text is generated based on the sound with the same sound features as the sound source object in the original audio to be processed, rather than generating the target audio through some preset sound features, thereby improving the sense of reality of the target audio and reducing the sense of separation caused by the difference between the sound features of the target audio and the sound features of the sound source object.

[0059] See also Figure 3 , Figure 3 A method flow chart of an audio processing method provided by an embodiment of the present application is shown. The audio processing method can be applied to Figure 1The electronic device in the audio processing scenario shown may specifically use a processor of the electronic device as an execution subject for executing the audio processing method. The audio processing method may include steps S210 to S250.

[0060] Step S210: searching for sounds emitted by all sound source objects in the audio to be processed as the second audio.

[0061] Step S220: Based on the voiceprint features of each of the sound source objects, the sound emitted by each of the sound source objects is determined in the second audio by using speaker separation technology as the first audio corresponding to the sound source object.

[0062] For some embodiments, there may be multiple sound source objects. Thus, the audio to be processed may include sounds emitted by multiple sound source objects, and may also include background sounds, such as noise. Therefore, the sounds emitted by all sound source objects may be first searched in the audio to be processed as the second audio, that is, the second audio only includes the sounds emitted by the sound source objects. Exemplarily, the sound emitted by all sound source objects may be searched in the audio to be processed as the second audio by using voice separation technology. Among them, voice separation technology is generally used to extract human voice from mixed audio, and usually also includes separating human voice from other audio components such as background music and noise.

[0063] Then, the speaker separation technology can be used to determine the sound emitted by each of the sound source objects in the second audio based on the voiceprint features of each of the sound source objects as the first audio corresponding to the sound source object. The audio to be processed can be subjected to voiceprint analysis to obtain the voiceprint features of the sound source objects in the audio to be processed.

[0064] Speaker separation technology is a technology that separates the voices of multiple speakers in audio. Unlike vocal separation (separating vocals from background music or noise), speaker separation focuses on distinguishing and extracting the voices of different speakers. It should be noted that the speaker can be regarded as a sound source object. Therefore, by combining the vocal separation technology and the speaker separation technology, the separated second audio is separated again to distinguish the audio data of different speakers, and the first audio corresponding to each sound source object is obtained, thereby improving the accuracy of the subsequent generation of the target audio corresponding to each sound source object. Among them, vocal separation technology and speaker separation technology are both artificial intelligence-based technologies.

[0065] Step S230: Perform speech recognition on the first audio to obtain text corresponding to the first audio.

[0066] Step S240: translating the text corresponding to the first audio into a target text in a target language.

[0067] Among them, step S230 and step S240 have been introduced in detail in the above embodiments and will not be repeated here.

[0068] Step S250: Generate target audio corresponding to the target text and having the sound characteristics of the sound source object.

[0069] For some embodiments, first, a fourth audio corresponding to the target text may be generated. It is understandable that the content represented by the fourth audio is the same as the content represented by the target text, that is, the fourth audio represents the content of the target text in the form of audio data. Exemplarily, the fourth audio corresponding to the target text may be generated with a preset sound feature.

[0070] Furthermore, the sound features of the fourth audio are adjusted to the sound features of the sound source object to obtain the target audio, so that the sound features of the obtained target audio are the same or approximately the same as the sound features of the sound source object, that is, the target audio corresponds to the target text and has the sound features of the sound source object.

[0071] The sound feature of the fourth audio may be adjusted to the sound feature of the sound source object by using voice cloning technology, wherein the voice cloning technology is a technology for generating synthetic speech by imitating and copying the voice feature of a speaker.

[0072] In some embodiments, the sound cloning technique can be implemented by a neural network model based on deep learning. Therefore, optionally, the target audio can also be obtained directly based on the sound features of the sound source object and the target text through the model. Specifically, the sound features of the sound source object and the target text can be used as input parameters of the model to obtain the output parameters of the model as the target voice.

[0073] It can be understood that in the embodiment of the present application, there are multiple sound source objects, so that multiple target audios can be obtained, and each target audio matches the corresponding sound source object.

[0074] The audio processing method provided in the embodiment of the present application searches for the sounds emitted by all the sound source objects in the audio to be processed as the second audio; then based on the voiceprint features of each of the sound source objects, the sound emitted by each of the sound source objects is determined in the second audio by speaker separation technology as the first audio corresponding to the sound source object. Among them, the sounds emitted by all the sound source objects can be searched from the audio to be processed by voice separation technology, and the sounds emitted by each of the sound source objects can be determined in the second audio based on the voiceprint features of each of the sound source objects by speaker separation technology as the first audio corresponding to the sound source object. Thus, by combining the voice separation technology and the speaker separation technology, the separated second audio is separated again to distinguish the audio data of different speakers, and the first audio corresponding to each sound source object is obtained, thereby improving the accuracy of the subsequent generation of the target audio corresponding to each sound source object. In addition, a target audio corresponding to the target text and having the sound characteristics of the sound source object can be generated through voice cloning technology, so that the sound characteristics of the obtained target audio are the same or approximately the same as the sound characteristics of the sound source object, thereby improving the realism of the target audio and reducing the sense of fragmentation.

[0075] See also Figure 4 , Figure 4 A method flow chart of an audio processing method provided by an embodiment of the present application is shown. The audio processing method can be applied to Figure 1 The electronic device in the audio processing scenario shown may specifically use a processor of the electronic device as an execution subject for executing the audio processing method. The audio processing method may include steps S310 to S390.

[0076] Step S310: Acquire the third audio.

[0077] Step S320: performing background sound separation on the third audio to obtain human voice audio in the third audio as the audio to be processed.

[0078] As can be seen from the above introduction, the audio to be processed may include background sound, such as noise, etc. Therefore, in some embodiments, the background sound can also be separated from the acquired third audio, so as to obtain the human voice audio in the third audio as the audio to be processed. Among them, the third audio can be audio data acquired by the electronic device, and the audio data can be audio data collected by a microphone or other device, or audio data received by the electronic device, etc., or audio data included in the multimedia data received by the electronic device, etc. For a detailed introduction, please refer to the above embodiment, which will not be repeated here.

[0079] By separating the background sound from the third audio, the human voice audio in the third audio can be obtained, and the background audio in the third audio can also be obtained, and then the human voice audio in the third audio can be used as the audio to be processed.

[0080] In some implementations, the third audio may be separated from the background sound by using a human voice separation technology, thereby obtaining human voice audio and background audio.

[0081] Step S330: Separate the speech emitted by the sound source object from the audio to be processed as the first audio.

[0082] Step S340: Perform speech recognition on the first audio to obtain text corresponding to the first audio.

[0083] Step S350: extracting the sound features of the sound source object.

[0084] Step S360: Translate the text corresponding to the first audio into a target text in a target language.

[0085] Among them, step S330 to step S360 have been introduced in detail in the above embodiments and will not be repeated here.

[0086] Step S370: Generate a fourth audio corresponding to the target text.

[0087] Step S380: adjusting the sound feature of the fourth audio to the sound feature of the sound source object, to obtain a fifth audio matching the sound source object.

[0088] Step S390: Fusing the background audio in the third audio with the fifth audio to obtain a target audio that matches the sound source object.

[0089] As can be seen from the above introduction, the audio to be processed may include background sound and sound from the sound source object. Therefore, if the target audio is obtained based only on the sound features of the sound source object and the target text, the target audio obtained only restores the sound features of the sound source object, but does not restore the background audio, so the target audio will still have a certain sense of fragmentation.

[0090] Therefore, for some implementations, after the target text is acquired, the fourth audio corresponding to the target text may be generated first. The detailed introduction may refer to the above-mentioned embodiment, which will not be repeated here.

[0091] Then, the sound feature of the fourth audio can be adjusted to the sound feature of the sound source object to obtain the fifth audio matching the sound source object. For the relevant introduction of adjusting the sound feature of the fourth audio to the sound feature of the sound source object, please refer to the above embodiment, which will not be repeated here. In order to further process the audio after the sound feature is adjusted, the audio after the sound feature is adjusted can be used as the fifth audio matching the sound source object.

[0092] Furthermore, the background audio in the third audio can be fused with the fifth audio to obtain the target audio matching the sound source object. Thus, the obtained target audio not only restores the sound characteristics of the sound source object, but also restores the background audio in the audio to be processed, thereby improving the immersion of the obtained target audio and further reducing the sense of fragmentation.

[0093] The audio processing method provided in the embodiment of the present application obtains human voice audio and background audio by separating background sound. Subsequently, the background audio in the third audio is fused with the fifth audio to obtain the target audio matching the sound source object. Thus, the obtained target audio not only restores the sound characteristics of the sound source object, but also restores the background audio in the audio to be processed, thereby improving the immersion of the obtained target audio and further reducing the sense of separation.

[0094] See also Figure 5 , Figure 5 A method flow chart of an audio processing method provided by an embodiment of the present application is shown. The audio processing method can be applied to Figure 1 The electronic device in the audio processing scenario shown may specifically use a processor of the electronic device as an execution subject for executing the audio processing method. The audio processing method may include steps S410 to S490.

[0095] Step S410: Separate the speech emitted by the sound source object from the audio to be processed as the first audio.

[0096] Step S420: Perform speech recognition on the first audio to obtain text corresponding to the first audio.

[0097] Step S430: extracting the sound features of the sound source object.

[0098] Step S440: translating the candidate text into a target text in a target language.

[0099] Among them, step S410 to step S440 have been introduced in detail in the above embodiments and will not be repeated here.

[0100] Step S450: searching the video to be processed for a video frame including the sound source object as a candidate video matching the sound source object.

[0101] Step S460: Determine the appearance features of the sound source object based on the candidate video.

[0102] Step S470: Generate a first sign language video using sign language expression through target human skeleton information based on the appearance features, wherein the first sign language video is used to represent the target text.

[0103] In some embodiments, the audio to be processed is the audio corresponding to the video to be processed. Exemplarily, if the electronic device obtains multimedia data, the multimedia data includes the audio to be processed and the video to be processed corresponding to the audio to be processed. For example, in the application scenario of video conferencing, the multimedia data of the meeting can be obtained, and the multimedia data of the meeting includes a video related to the meeting as the video to be processed, and the audio corresponding to the video as the audio to be processed, the audio to be processed includes the first audio emitted by the sound source object, and the video to be processed can include the video picture of the sound source object. For another example, in the scene of a live performance, the video picture of the performer can be collected by a camera as the video to be processed, and the audio data of the performer can be collected by a microphone as the audio to be processed, wherein the performer can be used as the sound source object.

[0104] Thus, a video frame including the sound source object can be searched in the video to be processed as a candidate video matching the sound source object. The video frame can include multiple frames, so the candidate video can be formed by multiple frames.

[0105] It is understandable that the video to be processed may include multiple sound source objects, so that the video screen corresponding to each sound source object can be searched in the video to be processed to obtain a candidate video matching each sound source object. For some implementations, the video to be processed can be used to perform character recognition based on artificial intelligence character recognition technology to obtain a candidate video matching each sound source object.

[0106] Furthermore, after obtaining the candidate video, since the candidate video includes the video screen of the sound source object, the appearance features of the sound source object can be determined based on the candidate video. Thus, a first sign language video expressing the sign language through the target human skeleton information can be generated based on the appearance features, wherein the first sign language video is used to represent the target text.

[0107] It should be noted that the target human skeleton information is the skeleton information obtained based on the appearance features of the sound source object, that is, the target human skeleton information includes the appearance features of the sound source object.

[0108] Therefore, the target text is represented by the first sign language video that uses the target human skeleton information for sign language expression, which can meet the needs of different users. For example, for a hearing-impaired user, the user cannot know the target text by listening to the target audio. In the embodiment of the present application, the target text can be represented by the first sign language video, which meets the needs of hearing-impaired users. Moreover, the first sign language video uses the target human skeleton information for sign language expression. Compared with the sign language expression through preset character images, such as cartoon characters, animation characters, etc., the sign language expression through the target human skeleton information in the embodiment of the present application makes the sign language expression more consistent with the appearance characteristics of the sound source object in the video to be processed, has a better expression effect, and also reflects the humanistic care for hearing-impaired users. At the same time, the hearing-impaired user can quickly understand the content of the target text through the first sign language video, and the efficiency of obtaining information is higher than directly watching the target text.

[0109] It should be noted that, in the case where there are multiple sound source objects, the target text corresponding to each sound source object can be matched with the first sign language video corresponding to the sound source object.

[0110] In some implementations, generating a first sign language video using the target human skeleton information for sign language expression based on the appearance feature may be first using preset human skeleton information for sign language expression, and then adjusting the preset human skeleton information for sign language expression using the appearance feature, thereby finally obtaining the first sign language video. Specifically, step S470 may also include step S471 and step S472.

[0111] Step S471: Generate a second sign language video that uses preset human skeleton information to express the sign language corresponding to the target text.

[0112] Step S472: adjusting the preset human skeleton information in the second sign language video by using the appearance features to obtain the first sign language video.

[0113] First, a second sign language video can be generated that uses preset human skeleton information to express the sign language corresponding to the target text. Exemplarily, the preset human skeleton information can include human skeleton information of a preset cartoon image. It can be understood that the second sign language video that uses preset human skeleton information to express the sign language corresponding to the target text is also used to represent the target text.

[0114] Exemplarily, a second sign language video can be generated by a sign language generation model, which uses preset human skeleton information to express the sign language corresponding to the target text. Specifically, for example, the second sign language video can be generated by the SignLLM model. Among them, the SignLLM model is a comprehensive multilingual model for generating sign language, which can convert text or voice prompts into videos expressed in sign language. By collecting sign language videos from all over the world, and then converting them into a format that is easy to train machine learning models through processing and annotation, a multilingual sign language dataset Prompt2Sign is constructed. The model can be trained with the dataset Prompt2Sign to obtain the SignLLM model, so that the SignLLM model has higher performance, and thus the embodiment of the present application has a higher efficiency in obtaining the first video, and the accuracy of the obtained first video for sign language expression is also higher. In the embodiment provided in the present application, the preset human skeleton and the target text can be used as input parameters of the SignLLM model to obtain the second sign language video output by the SignLLM model.

[0115] Furthermore, the preset human skeleton information in the second sign language video is adjusted by the appearance feature, so as to obtain the first sign language video. The adjusted preset human skeleton can be the target human skeleton information.

[0116] Exemplarily, the appearance in the preset human skeleton information may be replaced by appearance features, thereby obtaining the target human skeleton information.

[0117] Optionally, corresponding target human skeleton information can also be generated directly based on the appearance features of the sound source object. Thus, after generating a second sign language video that uses the preset human skeleton information to perform sign language expression corresponding to the target text, the action in the second sign language video that uses the preset human skeleton information to perform sign language expression can be transferred to the target human skeleton information to obtain the first sign language video.

[0118] Among them, the motion transfer technology can be used to transfer the motion expressed by the preset human skeleton information in the second sign language video to the target human skeleton information to obtain the first sign language video. The motion transfer technology can be used to transfer the motion or posture of the skeleton information of an object to the skeleton information of another object. It usually involves extracting motion features from the human skeleton information of the object, and then applying the motion features to the skeleton information of the object to which it is transferred.

[0119] Therefore, the motion features of the target human skeleton information in the first sign language video obtained in the embodiment of the present application are the same or similar to the motion features of the preset human skeleton information in the second sign language video, so as to realize the sign language expression through the target human skeleton information to represent the target text.

[0120] In other embodiments, the first sign language video expressing the sign language through the target human skeleton information is generated based on the appearance feature, or the first sign language video is generated directly based on the target human skeleton information. Specifically, step S470 may also include step S474 and step S475.

[0121] Step S474: adjusting the preset human skeleton information according to the appearance features to obtain the target human skeleton information.

[0122] Step S475: Generate a video of the sign language expression corresponding to the target text using the target human skeleton information as the first sign language video.

[0123] Specifically, the preset human skeleton information may be adjusted firstly according to the appearance features to obtain the target human skeleton information, thereby directly generating the first sign language video based on the target human skeleton information.

[0124] Similar to the above steps, the first sign language video can also be directly generated by the SignLLM model. Specifically, after obtaining the target human skeleton information, the target human skeleton information and the target text can be used as input parameters of the SignLLM model to obtain the first sign language video output by the SignLLM model.

[0125] Optionally, taking the aforementioned live performance scenario as an example, the parameters of the SignLLM model may be fine-tuned in combination with the personality characteristics of the performer, so that the generated first sign language video is more consistent with the performer's personality.

[0126] Step S480: Obtain target audio based on the sound features of the sound source object and the target text.

[0127] Among them, step S480 has been introduced in detail in the above embodiment and will not be repeated here.

[0128] Step S490: merging the target audio matched with the sound source object and the first sign language video matched with the sound source object to obtain target multimedia data matched with the sound source object.

[0129] After obtaining the target audio and the first sign language video, the target audio and the first sign language video may be further fused to obtain target multimedia data.

[0130] It should be noted that what is fused is the target audio corresponding to the sound source object and the first sign language video corresponding to the sound source object, and the obtained target multimedia data is also the target multimedia data matched by the sound source object.

[0131] Optionally, in some implementations, the multimedia data corresponding to the sound source object and the alternative video corresponding to the sound source object may be fused again. Exemplarily, the first sign language video in the multimedia data may be scaled and superimposed on the alternative video for display, so as to display the first sign language video in a partial screen area of ​​the alternative video, for example, displaying the first sign language video in the lower left corner of the alternative video. At the same time, the target audio in the multimedia data is used as the audio corresponding to the alternative video, so as to realize the fusion of the multimedia data corresponding to the sound source object and the alternative video corresponding to the sound source object.

[0132] Exemplarily, a user can run a translation application through an electronic device, thereby collecting video data through the camera of the electronic device as video to be processed, and collecting audio data through the microphone of the electronic device as audio to be processed, and then based on the audio processing methods introduced in the aforementioned embodiments, realize the fusion of the multimedia data corresponding to the sound source object and the alternative video corresponding to the sound source object, thereby achieving the effect of real-time sign language translation of audio in real life, providing an excellent experience.

[0133] In the embodiments provided by the present application, after translating the candidate text into the target text of the target language, the video screen including the sound source object can also be searched in the video to be processed as an alternative video matching the sound source object; the appearance features of the sound source object are determined based on the alternative video; and a first sign language video expressing the target text through sign language using the target human skeleton information is generated based on the appearance features, wherein the first sign language video is used to represent the target text. The motion features of the target human skeleton information expressed through sign language in the first sign language video obtained in the embodiment of the present application are the same or similar to the motion features of the preset human skeleton information expressed through sign language in the second sign language video, so as to achieve the characterization of the target text through sign language expression using the target human skeleton information.

[0134] See also Figure 6 , Figure 6 A method flow chart of an audio processing method provided by an embodiment of the present application is shown. The audio processing method can be applied to Figure 1 The electronic device in the audio processing scenario shown may specifically use a processor of the electronic device as an execution subject for executing the audio processing method. The audio processing method may include steps S510 to S5140.

[0135] Step S510: Acquire an audio processing request.

[0136] The user can generate an audio processing request through the electronic device, so that the electronic device can obtain the audio processing request. In the embodiment provided in the present application, the audio processing request can be a translation request. The method for the user to generate an audio processing request can be described in the above embodiment, which will not be repeated here.

[0137] Step S520: Acquire the audio to be processed.

[0138] Step S530: Obtain the video to be processed.

[0139] The electronic device can obtain the audio to be processed and the video to be processed, wherein the video to be processed and the audio to be processed are matched. The method for obtaining the audio to be processed and the video to be processed can refer to the introduction of the above embodiment, which will not be repeated here.

[0140] Step S540: vocal separation.

[0141] Step S550: Speaker separation.

[0142] After step S520, the process may jump to step S540 to perform human voice separation on the audio to be processed, and specifically, human voice separation may be performed on the audio to be processed by human voice separation technology. Speaker separation may also be further performed, and specifically, speaker separation may be performed on the audio to be processed after human voice separation by speaker separation technology. It should be noted that the speaker is the sound source object.

[0143] Step S560: Obtain background audio.

[0144] After the human voice is separated from the audio to be processed in step S540, background audio can also be obtained.

[0145] Step S570: Obtain the first audio of each sound source object.

[0146] Step S580: Automatic speech recognition to obtain the target text corresponding to each sound source object.

[0147] After speaker separation, the first audio of each sound source object can be obtained by combining the sound features of the sound source object. After obtaining the first audio, the target text corresponding to each sound source object can also be obtained by combining the automatic speech recognition technology. For a detailed introduction, please refer to the above embodiment, which will not be repeated here.

[0148] Step S590: the fifth audio corresponding to each sound source object.

[0149] A fourth audio corresponding to the target text may be generated; then the sound feature of the fourth audio is adjusted to the sound feature of the sound source object to obtain a fifth audio matching the sound source object. For a detailed description, please refer to the above embodiment, which will not be repeated here.

[0150] Step S5100: The background audio is merged with the fifth audio corresponding to each sound source object to obtain the target audio matching the sound source object.

[0151] The background audio in step S560 is then fused with the fifth audio corresponding to each sound source object to obtain the target audio matching the sound source object.

[0152] Step S5110: Obtain candidate videos matching each sound source object.

[0153] After step S530, candidate videos matching each sound source object may be obtained from the video to be processed.

[0154] Step S5120: Generate a second sign language video that uses preset human skeleton information to express the sign language corresponding to the target text.

[0155] Then, a second sign language video may be generated that uses the preset human skeleton information to perform sign language expression corresponding to the target text. Specifically, the second video may be generated using the SignLLM model.

[0156] Step S5130: adjusting the preset human skeleton information in the second sign language video by using appearance features to obtain the first sign language video.

[0157] Among them, step S5130 has been introduced in detail in the above embodiment and will not be repeated here.

[0158] Step S5140: Fusing the multimedia data corresponding to the sound source object with the candidate video corresponding to the sound source object.

[0159] Finally, the multimedia data corresponding to the sound source object can be fused with the candidate video corresponding to the sound source object. The multimedia data is the data obtained by fusion of the target audio matched by the sound source object and the first sign language video matched by the sound source object. The multimedia data matches the sound source object.

[0160] See also Figure 7 , Figure 7 The structure block diagram of the audio processing device provided in the embodiment of the present application is shown. Figure 7 The illustrated audio processing apparatus 700 includes an audio determination unit 710 , a speech recognition unit 720 , an extraction unit 730 , a translation unit 740 , and a target audio determination unit 750 .

[0161] The audio determination unit 710 is used to separate the speech emitted by the sound source object from the audio to be processed as the first audio.

[0162] Optionally, the audio determination unit 710 can also be used to search for sounds emitted by all sound source objects in the audio to be processed as the second audio; based on the voiceprint features of each sound source object, determine the sound emitted by each sound source object in the second audio through speaker separation technology as the first audio corresponding to the sound source object.

[0163] Optionally, the audio determination unit 710 may also be configured to obtain a third audio; perform background sound separation on the third audio to obtain human voice audio in the third audio as the audio to be processed.

[0164] The speech recognition unit 720 is used to perform speech recognition on the first audio to obtain text corresponding to the first audio.

[0165] The extraction unit 730 is used to extract the sound features of the sound source object.

[0166] The translation unit 740 is used to translate the text corresponding to the first audio into a target text in a target language.

[0167] Optionally, the translation unit 740 can also be used to search for a video frame including the sound source object in the video to be processed as an alternative video matching the sound source object; determine the appearance features of the sound source object based on the alternative video; and generate a first sign language video that expresses sign language through target human skeleton information based on the appearance features, wherein the first sign language video is used to represent the target text.

[0168] Optionally, the translation unit 740 can also be used to generate a second sign language video that uses preset human skeleton information to express the sign language corresponding to the target text; and adjust the preset human skeleton information in the second sign language video through the appearance features to obtain the first sign language video.

[0169] Optionally, the translation unit 740 can also be used to adjust the preset human skeleton information through the appearance features to obtain the target human skeleton information; and generate a video of the sign language expression corresponding to the target text through the target human skeleton information as a first sign language video.

[0170] Optionally, the translation unit 740 may also be configured to fuse the target audio that matches the sound source object and the first sign language video that matches the sound source object to obtain target multimedia data that matches the sound source object.

[0171] The target audio determination unit 750 is configured to obtain the target audio based on the sound features of the sound source object and the target text.

[0172] Optionally, the target audio determination unit 750 may also be configured to generate a target audio corresponding to the target text and having the sound features of the sound source object.

[0173] Optionally, the target audio determination unit 750 can also be used to generate a fourth audio corresponding to the target text; adjust the sound features of the fourth audio to the sound features of the sound source object to obtain a fifth audio matching the sound source object; and fuse the background audio in the third audio with the fifth audio to obtain the target audio matching the sound source object.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0175] In several embodiments provided in the present application, the coupling between the units may be electrical, mechanical or other forms of coupling. In addition, each functional unit in each embodiment of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0176] See also Figure 8 , Figure 8 The block diagram of a structure of an electronic device provided in an embodiment of the present application is shown. The electronic device 110 may be a smart phone, a desktop computer, a vehicle-mounted computer, a server or a tablet computer, etc. The electronic device 110 in the present application may include one or more of the following components: a processor 111, a memory 112 and one or more application programs, wherein the processor 111 is electrically connected to the memory 112, and the one or more program configurations are used to execute the methods described in the aforementioned embodiments.

[0177] The processor 111 may include one or more processing cores. The processor 111 uses various interfaces and lines to connect various parts of the entire electronic device 110, and executes various functions and processes data of the electronic device 110 by running or executing instructions, programs, code sets or instruction sets stored in the memory 112, and calling data stored in the memory 112. Optionally, the processor 111 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 111 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and computer programs; the GPU is responsible for rendering and drawing display content; the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 111, and it can be implemented separately through a communication chip. Specifically, the method described in the above embodiment can be executed by one or more processors 111.

[0178] For some embodiments, the memory 112 may include a random access memory (RAM) or a read-only memory (ROM). The memory 112 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 112 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function, instructions for implementing the following various method embodiments, etc. The data storage area may also store data created by the electronic device 110 during use, etc.

[0179] See also Fig. 9 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 900 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0180] The computer-readable storage medium 900 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer-readable storage medium 900 includes a non-transitory computer-readable storage medium. The computer-readable storage medium 900 has storage space for program code 910 that performs any method step in the above method. These program codes can be read from or written to one or more computer program products. The program code 910 can be compressed, for example, in an appropriate form.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio processing method, characterized in that: include: Separating the speech emitted by the sound source object from the audio to be processed as the first audio; Performing speech recognition on the first audio to obtain text corresponding to the first audio; Extracting sound features of the sound source object; Translating the text corresponding to the first audio into a target text in a target language; A target audio is obtained based on the sound features of the sound source object and the target text.

2. The method according to claim 1, characterized in that There are multiple sound source objects, and separating the speech emitted by the sound source objects from the audio to be processed as the first audio includes: Searching for sounds emitted by all sound source objects in the audio to be processed as the second audio; Based on the voiceprint features of each of the sound source objects, the sound emitted by each of the sound source objects is determined in the second audio by using speaker separation technology as the first audio corresponding to the sound source object.

3. The method according to claim 1, characterized in that: The obtaining the target audio based on the sound feature of the sound source object and the target text comprises: A target audio corresponding to the target text and having the sound features of the sound source object is generated.

4. The method according to claim 1, characterized in that: Before separating the speech emitted by the sound source object from the audio to be processed as the first audio, the method further includes: Get the third audio; The background sound is separated from the third audio to obtain the human voice audio in the third audio as the audio to be processed.

5. The method according to claim 4, characterized in that The obtaining the target audio based on the sound feature of the sound source object and the target text comprises: Generating a fourth audio corresponding to the target text; Adjusting the sound feature of the fourth audio to the sound feature of the sound source object to obtain a fifth audio matching the sound source object; The background audio in the third audio is fused with the fifth audio to obtain the target audio that matches the sound source object.

6. The method according to claim 1, characterized in that The audio to be processed is the audio corresponding to the video to be processed, and after translating the text corresponding to the first audio into the target text in the target language, the method further includes: Searching for a video frame including the sound source object in the video to be processed as a candidate video matching the sound source object; Determining the appearance features of the sound source object based on the candidate video; A first sign language video is generated based on the appearance features to express the target text in sign language using the target human skeleton information, wherein the first sign language video is used to represent the target text.

7. The method according to claim 6, characterized in that The step of generating a first sign language video expressing the sign language through target human skeleton information based on the appearance feature includes: Generate a second sign language video that uses preset human skeleton information to express the sign language corresponding to the target text; The preset human skeleton information in the second sign language video is adjusted using the appearance features to obtain the first sign language video.

8. The method according to claim 6, characterized in that The step of generating a first sign language video expressing the sign language through target human skeleton information based on the appearance feature includes: The preset human skeleton information is adjusted according to the appearance features to obtain the target human skeleton information; A video of the sign language expression corresponding to the target text using the target human skeleton information is generated as a first sign language video.

9. The method according to claim 6, characterized in that The method further comprises: The target audio matched with the sound source object and the first sign language video matched with the sound source object are fused to obtain target multimedia data matched with the sound source object.

10. An audio processing device, characterized in that: include: An audio determination unit, used to separate the speech emitted by the sound source object from the audio to be processed as the first audio; a speech recognition unit, configured to perform speech recognition on the first audio to obtain a text corresponding to the first audio; An extraction unit, used to extract the sound features of the sound source object; A translation unit, configured to translate the text corresponding to the first audio into a target text in a target language; The target audio determination unit is used to obtain the target audio based on the sound characteristics of the sound source object and the target text.

11. An electronic device, characterized in that: include: one or more processors; Memory; One or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1-9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, and the program code can be called by a processor to execute the method according to any one of claims 1 to 9.

Images