Audio data processing method and device, equipment, medium and product

Abstract

The invention provides an audio data processing method and device, equipment, a medium and a product. The method comprises the following steps: acquiring an external human voice signal and an ear canal sound signal through earphone equipment, performing audio negative superposition on the ear canal sound signal through a loudspeaker signal to obtain a physical conduction human voice signal, and preprocessing the physical conduction human voice signal and the external human voice signal to obtain an ear return human voice signal, and generating an uplink audio file according to the ear return voice signal and the accompaniment synchronization signal, transmitting the uplink audio file to the terminal equipment when receiving a transmission instruction sent by the terminal equipment, so that the terminal equipment verifies the uplink synchronization signal according to the accompaniment synchronization signal, and sending the accompaniment synchronization signal to the terminal equipment after the verification is passed. And generating a target audio mixing file according to the uplink audio file and the corresponding accompaniment signal and outputting the target audio mixing file. According to the method provided by the invention, the ear return listening and recording effects of the user are enhanced, the audio quality of the karaoke of the user and the audio processing efficiency are improved, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to audio processing technology, and in particular to a method, device, equipment, medium and product for processing audio data. Background Art

[0002] With the development of technology, users no longer need to use special karaoke headphones to sing karaoke, they can sing karaoke by just wearing headphones.

[0003] The prior art involves singing karaoke by wearing TWS (True Wireless Stereo) headphones. During the karaoke process, the user hears the accompaniment and vocals fed back to the human ear by the headphones, thereby singing karaoke in real time based on the accompaniment and vocals heard. At the same time, the sound is picked up by the mobile phone microphone so that the picked-up audio can be edited to obtain a target audio file.

[0004] However, since wearing headphones can easily cause physical noise reduction effects and / or occlusion effects, there is a large difference between the sound produced by the user when singing karaoke with headphones and the actual sound produced. In addition, there is a transmission delay in the process of sound pickup by the mobile phone microphone, which causes the human voice and accompaniment heard by the human ear to be out of sync, thereby affecting the effect of sound pickup and recording and reducing the user experience. Summary of the invention

[0005] The embodiments of the present application provide methods, devices, equipment, media and products for processing audio data to improve the audio quality of users' karaoke and the efficiency of audio processing, enhance the effects of users' in-ear listening and recording, and improve user experience.

[0006] In a first aspect, an embodiment of the present application provides a method for processing audio data, which is applied to a headphone device, comprising:

[0007] Acquire an external human voice signal and an ear canal sound signal; perform audio negative superposition processing on the ear canal sound signal based on the speaker signal of the headphone device to obtain a physically conducted human voice signal, where the physically conducted human voice signal is the human voice heard by the user, and the external human voice signal is the actual sound emitted by the user;

[0008] Preprocessing physically conducted human voice and external human voice signals to obtain ear-return human voice signals;

[0009] Generate an uplink audio file based on the accompaniment synchronization signal and the ear-return human voice signal sent by the terminal device, where the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal;

[0010] When receiving a transmission instruction sent by a terminal device, an uplink audio file is transmitted to the terminal device so that the terminal device determines an uplink synchronization signal corresponding to the uplink audio file, and verifies the uplink synchronization signal based on a preset synchronization signal. After the verification is passed, a target mixing file is generated and output according to the uplink audio file and the corresponding accompaniment signal.

[0011] Optionally, preprocessing the physically conducted human voice and the external human voice signal to obtain an ear-return human voice signal includes:

[0012] Perform noise reduction on physically conducted vocal signals and external vocal signals to obtain clean vocal signals;

[0013] Perform spatial processing on the clean vocal signal to obtain a real-time in-ear return vocal signal.

[0014] Optionally, generating an uplink audio file based on the accompaniment synchronization signal and the ear-return human voice signal sent by the terminal device includes:

[0015] By performing signal extraction processing on the accompaniment synchronization signal, a preset synchronization signal is obtained;

[0016] According to the preset sequence information of each accompaniment audio frame in the preset synchronization signal, the real-time clean vocal signal and the ear-return vocal signal are aligned and stored;

[0017] When the current accompaniment audio frame is determined to be the end audio frame according to the preset synchronization signal, the currently generated clean vocal signal and the ear-return vocal signal are aligned and stored to generate an uplink audio file. The uplink audio file includes all clean vocal signals and ear-return vocal signals stored after the previous end audio frame.

[0018] Optionally, after obtaining the ear-return human voice signal, the method includes:

[0019] The audio positive superposition processing is performed based on the ear-return human voice signal and the accompaniment synchronization signal sent by the terminal device to obtain the ear-return audio, which is used to update the real-time speaker signal in the ear canal.

[0020] In a second aspect, an embodiment of the present application provides a method for processing audio data, which is applied to a terminal device, including:

[0021] Acquire an accompaniment signal and a preset synchronization signal, generate an accompaniment synchronization signal and send it to the headphone device, so that the headphone device generates an uplink audio file according to the accompaniment synchronization signal and the ear return human voice signal, the ear return human voice signal is obtained after the headphone device pre-processes the physically conducted human voice and the obtained external human voice signal, and the physically conducted human voice is obtained after the headphone device performs audio negative superposition processing on the obtained ear canal sound signal based on the speaker signal;

[0022] Sending a transmission instruction to the headphone device, so that the headphone device sends the uplink audio file to the terminal device based on the transmission instruction;

[0023] When an uplink audio file sent by the headphone device is received, an uplink synchronization signal corresponding to the uplink audio file is determined, and the uplink synchronization signal is verified based on a preset synchronization signal. After the verification is passed, a target mixing file is generated and output according to the uplink audio file and the corresponding accompaniment signal.

[0024] Optionally, performing audio processing on the uplink audio file to obtain a processed uplink audio file;

[0025] Perform signal separation processing on the processed uplink audio file to obtain an uplink synchronization signal and a processed clean human voice signal and an ear-return human voice signal;

[0026] A target dry sound file is generated according to the processed clean vocal signal and accompaniment signal; and a target wet sound file is generated according to the in-ear return vocal signal and accompaniment signal.

[0027] In a third aspect, an embodiment of the present application provides an audio data processing device, which is applied to a headphone device, including:

[0028] The first acquisition module is used to acquire an external human voice signal and an ear canal sound signal; based on the speaker signal of the headphone device, the ear canal sound signal is subjected to audio negative superposition processing to obtain a physically conducted human voice signal, where the physically conducted human voice signal is the human voice heard by the user, and the external human voice signal is the actual sound emitted by the user;

[0029] A processing module, used for pre-processing physically conducted human voice and external human voice signals to obtain an ear-return human voice signal;

[0030] The processing module is further used to generate an uplink audio file based on the accompaniment synchronization signal and the ear-return human voice signal sent by the terminal device, where the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal;

[0031] The processing module is also used to transmit the uplink audio file to the terminal device when receiving the transmission instruction sent by the terminal device, so that the terminal device determines the uplink synchronization signal corresponding to the uplink audio file, verifies the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generates a target mixing file according to the uplink audio file and the corresponding accompaniment signal and outputs it.

[0032] Optionally, the processing module is further used to perform noise reduction processing on the physically conducted human voice and the external human voice signal to obtain a clean human voice signal;

[0033] Perform spatial processing on the clean vocal signal to obtain a real-time in-ear return vocal signal.

[0034] Optionally, the processing module is further used to obtain a preset synchronization signal by performing signal extraction processing on the accompaniment synchronization signal;

[0035] According to the preset sequence information of each accompaniment audio frame in the preset synchronization signal, the real-time clean vocal signal and the ear-return vocal signal are aligned and stored;

[0036] When the current accompaniment audio frame is determined to be the end audio frame according to the preset synchronization signal, the currently generated clean vocal signal and the ear-return vocal signal are aligned and stored to generate an uplink audio file. The uplink audio file includes all clean vocal signals and ear-return vocal signals stored after the previous end audio frame.

[0037] Optionally, the processing module is also used to perform audio positive superposition processing based on the ear return vocal signal and the accompaniment synchronization signal sent by the terminal device after obtaining the ear return vocal signal, so as to obtain the ear return audio, and the ear return audio is used to update the real-time speaker signal in the ear canal.

[0038] In a fourth aspect, an embodiment of the present application provides an audio data processing device, which is applied to a terminal device, including:

[0039] A second acquisition module is used to acquire an accompaniment signal and a preset synchronization signal, generate an accompaniment synchronization signal and send it to the headphone device, so that the headphone device generates an uplink audio file according to the accompaniment synchronization signal and the ear return human voice signal, the ear return human voice signal is obtained after the headphone device pre-processes the physically conducted human voice and the acquired external human voice signal, and the physically conducted human voice is obtained after the headphone device performs audio negative superposition processing on the acquired ear canal sound signal based on the speaker signal;

[0040] A sending module, used for sending a transmission instruction to the earphone device, so that the earphone device sends the uplink audio file to the terminal device based on the transmission instruction;

[0041] The receiving module is used to determine the uplink synchronization signal corresponding to the uplink audio file when receiving the uplink audio file sent by the headphone device, verify the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generate a target mixing file according to the uplink audio file and the corresponding accompaniment signal and output it.

[0042] Optionally, the receiving module is further used to perform audio processing on the uplink audio file to obtain a processed uplink audio file;

[0043] Perform signal separation processing on the processed uplink audio file to obtain an uplink synchronization signal and a processed clean human voice signal and an ear-return human voice signal;

[0044] A target dry sound file is generated according to the processed clean vocal signal and accompaniment signal; and a target wet sound file is generated according to the in-ear return vocal signal and accompaniment signal.

[0045] In a fifth aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0046] Memory stores computer-executable instructions;

[0047] The processor executes the computer-executable instructions stored in the memory, so that the processor executes various possible implementations of the first aspect and / or the second aspect as described above.

[0048] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement various possible implementations of the first aspect and / or the second aspect as described above.

[0049] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements various possible implementations of the first aspect and / or the second aspect as described above.

[0050] The audio data processing method, device, equipment, medium and product provided in the embodiments of the present application obtain external human voice signals and ear canal sound signals through earphone equipment, perform audio negative superposition processing on the ear canal sound signals with the speaker signal of the earphone equipment, obtain physical conduction human voice signals, pre-process the physical conduction human voice and external human voice signals, obtain the ear return human voice signal, and generate an uplink audio file according to the ear return human voice signal and the accompaniment synchronization signal, wherein the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal. Upon receiving the transmission instruction sent by the terminal device, the uplink audio file is transmitted to the terminal device so that the terminal device verifies the uplink synchronization signal according to the preset synchronization signal. After the verification is passed, the target mixed file is generated and output according to the uplink audio file and the corresponding accompaniment signal. The present application improves the audio quality of the user's karaoke and the efficiency of audio processing, enhances the effect of the user's ear return listening and recording, and improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0052] Figure 1 A schematic diagram of an application scenario of the audio data processing method provided in this application;

[0053] Figure 2 Schematic diagram of the process of processing audio data provided by this application Figure 1 ;

[0054] Figure 3The structure of the method for processing audio data provided by this application is shown in FIG. Figure 1 ;

[0055] Figure 4 The structure of the method for processing audio data provided by this application is shown in FIG. Figure 2 ;

[0056] Figure 5 Schematic diagram of the process of processing audio data provided by this application Figure 2 ;

[0057] Figure 6 The structure of the audio data processing device provided in this application is shown as follows: Figure 1 ;

[0058] Figure 7 The structure of the audio data processing device provided in this application is shown as follows: Figure 2 ;

[0059] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0060] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0061] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0062] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0063] Users use dedicated karaoke live broadcast earphones to sing karaoke, but dedicated karaoke live broadcast earphones have poor compatibility with terminal devices, and usually require the use of multiple devices at the same time for wired and / or wireless connection, thereby occupying the mobile phone charging port during the karaoke process, reducing the user's karaoke experience.

[0064] With the development of technology, users no longer need to use special karaoke headphones to sing karaoke, they can sing karaoke by just wearing headphones.

[0065] Figure 1 A schematic diagram of an application scenario of the audio data processing method provided in this application, such as Figure 1 As shown, the specific application scenario of the present application includes a terminal device 10, an earphone device 20 and a user 30. The earphone device is communicatively connected to the terminal device. When the connection is successful, the user opens the karaoke software by operating the terminal device, and the accompaniment of the target song is played in the user's ear canal through the earphone device. The user sings along with the accompaniment, and the microphone of the terminal device collects the human voice emitted by the user, and the collected human voice and accompaniment are fused to obtain the target audio file generated by the user's karaoke.

[0066] However, when users wear earphones to sing karaoke, in addition to hearing the accompaniment played by the earphones, they can also hear their own singing voices. Because the user wears earphones, it will cause physical noise reduction and / or occlusion effects, and there is a transmission delay in the mobile phone microphone during the sound pickup process, so the singing sound heard by the user is quite different from the sound effect of the user's actual singing. This further causes the human voice and accompaniment in the target audio file obtained by the terminal device after the collected human voice and accompaniment are not synchronized, so a large amount of manual editing and repair of the target audio is required, which reduces the processing efficiency and quality of the audio data. In addition, during the karaoke process, it is difficult for users to accurately determine the current pitch based on the human voice that is physically transmitted to the ear, which further affects the subsequent karaoke effect and reduces the user experience during the karaoke process.

[0067] The audio data processing method provided by the present application obtains an external human voice signal and an ear canal sound signal through a microphone of an earphone device, determines a loudspeaker signal to be played through the earphone device, performs audio negative superposition processing on the ear canal sound signal based on the loudspeaker signal to obtain a physically conducted human voice signal, obtains an ear return human voice signal by preprocessing the physically conducted human voice signal and the external human voice signal, and upon receiving an accompaniment synchronization signal sent by a terminal device, performs signal separation processing on the accompaniment synchronization signal to obtain a preset synchronization signal, generates an uplink audio file according to the preset synchronization signal and the ear return human voice signal, and sends the uplink audio file to a terminal device upon receiving a transmission instruction. The terminal device performs signal separation processing on the uplink audio file after receiving the file to obtain an uplink synchronization signal, verifies the uplink synchronization signal according to a preset synchronization signal, and after the verification, generates a target mixed file with the uplink audio file and the accompaniment signal and outputs it, thereby improving the synchronization between the human voice and the accompaniment in the target mixed file, and further improving the efficiency and quality of processing the audio data. In addition, by performing audio positive superposition on the accompaniment synchronization signal and the ear-return human voice signal, the corresponding ear-return audio is obtained to update the speaker signal played by the current headphone device, thereby improving the quality of the human voice heard by the user and further improving the user's subsequent karaoke effect.

[0068] The audio data processing method provided in this application is intended to solve the above technical problems in the prior art.

[0069] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0070] Figure 2 Schematic diagram of the process of processing audio data provided by this application Figure 1 ,like Figure 2 As shown, the method includes:

[0071] S201. Acquire an external human voice signal and an ear canal sound signal, and perform audio negative superposition processing on the ear canal sound signal based on a speaker signal of an earphone device to obtain a physically conducted human voice signal.

[0072] More specifically, an external human voice signal and an ear canal sound signal are obtained; audio negative superposition processing is performed on the ear canal sound signal based on the speaker signal of the headphone device to obtain a physically conducted human voice signal, the physically conducted human voice signal is the human voice heard by the user, and the external human voice signal is the actual sound emitted by the user.

[0073] Optionally, an external human voice signal is obtained through the external microphone of the headphone device, an ear canal sound signal in the human ear is obtained through the in-ear microphone of the headphone device, and a speaker signal played in the ear canal is collected through the headphone device, and the collected speaker signal and the ear canal sound signal in the human ear are subjected to audio negative superposition processing, and after the audio negative superposition, a physically conducted human voice signal heard by the human ear is obtained.

[0074] In one possible embodiment, Figure 3 The structure of the method for processing audio data provided by this application is shown in FIG. Figure 1 ,like Figure 3 As shown, the headphone device collects the external human voice signal emitted by the user during karaoke through the external microphone, collects the ear canal sound signal heard in the ear canal of the human ear through the in-ear microphone, collects the speaker signal played by the headphone device through the speaker, filters the speaker signal, and negatively superimposes it on the ear canal sound signal to obtain the physical conduction human voice signal heard in the ear canal of the human ear. The physical conduction human voice signal is the actual sound signal of the user singing karaoke heard by the user when wearing the headphones. This embodiment negatively superimposes the ear canal sound signal heard in the ear canal of the user through the speaker signal to obtain the actual karaoke sound heard in the ear canal of the user, thereby achieving accurate acquisition of the actual karaoke sound heard in the ear canal of the user.

[0075] S202: Preprocess the physically conducted human voice and the external human voice signal to obtain an ear-return human voice signal.

[0076] More specifically, the real-time physically conducted human voice and the real-time collected external human voice signal are preprocessed to obtain the corresponding real-time ear-return human voice signal, which is the sound and image space information suitable for the song.

[0077] Optionally, the physically conducted human voice and the external human voice signal are subjected to noise reduction processing to obtain a clean human voice signal; and the clean human voice signal is subjected to spatial processing to obtain a real-time ear-return human voice signal.

[0078] In a possible embodiment, Figure 3 As shown, the external vocal signal collected by the external microphone at the current moment and the physically conducted vocal signal after the audio negative superposition processing at the current moment are respectively sent to the audio processing unit 1, and the above signals are subjected to noise reduction processing by the audio processing unit 1 to obtain a clean vocal signal after noise reduction. The noise reduction processing method includes but is not limited to phase synchronization and volume adjustment. The clean vocal signal is sent to the audio processing unit 2, and the clean vocal signal is spatially processed by the audio processing unit 2 to obtain an ear return vocal signal suitable for the corresponding song. The spatial processing method includes but is not limited to reverberation processing and delay processing.

[0079] Exemplarily, before the clean vocal signal is spatially processed, the clean vocal signal is finely processed, and after obtaining a clean vocal signal that is comfortable to listen to after fine processing, the clean vocal signal is spatially processed to obtain an ear-return vocal signal, wherein the fine processing method includes but is not limited to EQ (Equalizer) processing, compression processing, harmonic excitation processing, sibilance elimination processing, etc. This embodiment performs noise reduction processing and spatial processing on the external vocal signal and the physically conducted vocal signal heard by the human ear in sequence to obtain sound and image spatial information suitable for the song, which is convenient for the subsequent fusion of vocals and accompaniment, thereby further improving the quality of the fused audio, avoiding repeated processing of audio data, and improving the processing efficiency of audio data.

[0080] Optionally, the clean vocal signal output by the audio processing unit 1 and the ear-return vocal signal output by the audio processing unit 2 are sent to the audio uplink unit.

[0081] S203: Generate an uplink audio file based on the accompaniment synchronization signal and the ear-return human voice signal sent by the terminal device.

[0082] More specifically, an uplink audio file is generated based on the accompaniment synchronization signal and the ear-return human voice signal sent by the terminal device, and the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal.

[0083] Optionally, a preset synchronization signal is obtained by performing signal extraction processing on the accompaniment synchronization signal; the real-time clean vocal signal and the ear-return vocal signal are aligned and stored according to the preset sequence information of each accompaniment audio frame in the preset synchronization signal; when the current accompaniment audio frame is determined to be the end audio frame according to the preset synchronization signal, the currently generated clean vocal signal and the ear-return vocal signal are aligned and stored to generate an uplink audio file, and the uplink audio file includes all clean vocal signals and ear-return vocal signals stored after the previous end audio frame.

[0084] In a possible embodiment, Figure 3 As shown, the accompaniment synchronization signal sent by the terminal device is received by the headphone device, and the accompaniment synchronization signal is extracted and processed by the audio processing unit 3 in the headphone device to obtain the corresponding preset synchronization signal and send it to the audio uplink unit, and the preset sequence information corresponding to the accompaniment audio frame at the current moment is determined by the audio uplink unit of the headphone device according to the preset synchronization signal. If the current preset sequence information indicates the starting audio frame of the target song accompaniment, the clean vocal signal and the ear-returned vocal signal obtained at the current moment are aligned with the starting audio frame of the target song accompaniment, and the first audio data block is generated and stored.

[0085] Exemplarily, if the current preset sequence information indicates the end audio frame of the target song accompaniment, the clean vocal signal and the ear-return vocal signal obtained at the current moment are aligned with the end audio frame of the target song accompaniment, and the nth audio data block is generated and stored, where n is a positive integer greater than 1. Based on the above-mentioned end audio frame, the first audio database to the nth audio data block are generated into an uplink audio file according to the order indicated by the preset synchronization signal, and stored in the database of the headphone device. This embodiment improves the synchronization between the vocals and the accompaniment by performing an alignment operation on the clean vocal signal and the ear-return vocal signal on the headphone device side.

[0086] S204: When receiving a transmission instruction sent by the terminal device, transmit the uplink audio file to the terminal device.

[0087] More specifically, when a transmission instruction is received from the terminal device, an uplink audio file is transmitted to the terminal device so that the terminal device determines the uplink synchronization signal corresponding to the uplink audio file, and verifies the uplink synchronization signal based on a preset synchronization signal. After the verification is passed, a target mixing file is generated and output according to the uplink audio file and the corresponding accompaniment signal.

[0088] Optionally, an accompaniment signal and a preset synchronization signal are obtained through a terminal device, and an accompaniment synchronization signal is generated and sent to the headphone device, so that the headphone device generates an uplink audio file according to the accompaniment synchronization signal and the ear-return human voice signal. The ear-return human voice signal is obtained after the headphone device pre-processes the physically conducted human voice and the obtained external human voice signal. The physically conducted human voice is obtained after the headphone device performs audio negative superposition processing on the obtained ear canal sound signal based on the speaker signal of the headphone device.

[0089] Optionally, a transmission instruction is sent to the headphone device so that the headphone device sends the uplink audio file to the terminal device based on the transmission instruction. When the terminal device receives the uplink audio file sent by the headphone device, an uplink synchronization signal corresponding to the uplink audio file is determined, and the uplink synchronization signal is verified based on a preset synchronization signal. After the verification is passed, a target mixing file is generated according to the uplink audio file and the corresponding accompaniment signal and output.

[0090] In one possible embodiment, Figure 4 The structure of the method for processing audio data provided by this application is shown in FIG. Figure 2 ,like Figure 4 As shown, the terminal device obtains the accompaniment signal, determines the preset synchronization signal, processes the accompaniment signal and the preset synchronization signal through the audio downlink unit, obtains the accompaniment synchronization signal, and sends it to the headphone device.

[0091] Optionally, audio processing is performed on the uplink audio file to obtain a processed uplink audio file; signal separation processing is performed on the processed uplink audio file to obtain an uplink synchronization signal and a processed clean vocal signal and an in-ear return vocal signal; a target dry sound file is generated based on the processed clean vocal signal and the in-ear return vocal signal; and a target wet sound file is generated based on the target dry sound file and the corresponding accompaniment signal.

[0092] For example, Figure 4 As shown, when the terminal device receives an uplink audio file, the audio processing unit 4 performs audio modification on the uplink audio file, and sends the modified uplink audio file to the signal separation unit, and the signal separation unit separates and processes the modified uplink audio file to obtain an uplink synchronization signal corresponding to the uplink audio file, as well as a modified clean vocal signal and an ear-return vocal signal, and sends the uplink synchronization signal, the modified clean vocal signal and the ear-return vocal signal to the final mixing unit. The final mixing unit receives a preset synchronization signal, verifies the uplink synchronization signal according to the preset synchronization signal, and when the verification is consistent, determines that the uplink synchronization signal has been verified, and then performs the final mixing step according to user needs.

[0093] For example, Figure 4 As shown, when it is determined that the user needs to obtain a target dry sound file, the preset synchronization signal and accompaniment signal are obtained through the final mixing unit of the terminal device, and the target dry sound file is generated according to the preset synchronization signal, the accompaniment signal and the modified clean vocal signal. When it is determined that the user needs to obtain a target wet sound file, the accompaniment signal and the preset synchronization signal are obtained through the final mixing unit of the terminal device, and the target wet sound file is generated according to the preset synchronization signal, the accompaniment signal and the modified ear return vocal signal. This embodiment outputs the corresponding dry sound file or wet sound file according to user needs, enhances the processing quality of audio data, further improves the processing efficiency of audio data, and enhances the user's karaoke experience.

[0094] The audio data processing method provided in the embodiment of the present application processes the external human voice signal and the human voice signal inside the ear canal through the headphone device to obtain a clean human voice signal and an in-ear return human voice signal, and performs final mixing processing on the clean human voice signal or the in-ear return human voice signal according to user needs to obtain a target dry sound file or a target wet sound file, thereby improving the synchronization of the files obtained after the final mixing processing, improving the efficiency and quality of user audio processing, and further enhancing the user experience.

[0095] Optionally, audio positive superposition processing is performed based on the ear-return human voice signal and the accompaniment synchronization signal sent by the terminal device to obtain the ear-return audio, and the ear-return audio is used to update the real-time speaker signal in the ear canal.

[0096] For example, Figure 3As shown, the ear return audio is obtained by performing audio positive superposition processing on the ear return human voice signal generated by the audio processing unit 2 of the headphone device and the received accompaniment synchronization signal, and the real-time ear return audio is used to update the speaker signal of the speaker and play it. This embodiment enhances the clarity of the human voice heard by the user's ears during karaoke, improves the synchronization of the human voice and the accompaniment, and further enhances the user experience.

[0097] Figure 5 Schematic diagram of the process of processing audio data provided by this application Figure 2 ,like Figure 5 As shown, in this embodiment Figure 2 Based on the embodiment, a method for processing audio data is described in detail, and the method includes:

[0098] S501: The earphone device obtains an external human voice signal and an internal human voice signal of an ear canal.

[0099] S502: The headphone device determines a speaker signal, and generates a physically conducted voice signal based on the speaker signal and a voice signal inside the ear canal.

[0100] More specifically, the headphone device determines the speaker signal emitted by the speaker, and negatively superimposes the speaker signal on the human voice signal inside the ear canal, thereby canceling the speaker signal played by the headphone device in the human voice signal inside the ear canal, and obtaining the physically conducted human voice signal heard inside the ear canal of the human ear.

[0101] S503: The headphone device performs denoising processing on the physically conducted human voice signal and the external human voice signal to obtain a clean human voice signal.

[0102] More specifically, the headphone device performs denoising on the physically conducted vocal signal and the collected external vocal signal, and fuses them after the denoising to obtain a clean vocal signal.

[0103] S504: The headphone device performs spatial processing on the clean vocal signal to obtain an ear-return vocal signal.

[0104] More specifically, the headphone device further performs spatial processing on the obtained clean vocal signal to obtain an ear-return vocal signal that can be better integrated with the accompaniment.

[0105] S505: The terminal device sends an accompaniment synchronization signal to the headphone device.

[0106] S506: The headphone device generates an ear-return audio according to the ear-return human voice signal and the accompaniment synchronization signal, and updates the speaker signal.

[0107] More specifically, after receiving the accompaniment synchronization signal, the headphone device positively superimposes the accompaniment synchronization signal audio onto the ear return vocal signal to obtain the ear return audio, and feeds the obtained ear return audio back to the speaker of the headphone device so that the speaker plays the updated speaker signal.

[0108] S507: The headphone device determines a preset synchronization signal according to the accompaniment synchronization signal, and generates an uplink audio file according to the preset synchronization signal, the ear-return vocal signal, and the clean vocal signal.

[0109] More specifically, the headphone device extracts and processes the accompaniment synchronization signal to obtain a preset synchronization signal, and aligns and stores the ear return vocal signal and the clean vocal signal according to the preset synchronization signal. When the preset synchronization signal indicates the end of the audio frame, the ear return vocal signal and the clean vocal signal at the current moment are aligned and stored, and all the ear return vocal signals and the clean vocal signals after the last end audio frame are aligned and stored in the order indicated by the preset synchronization signal to generate an uplink audio file.

[0110] S508: The terminal device sends a transmission instruction to the headset device.

[0111] S509: The headset device sends an uplink audio file to the terminal device.

[0112] More specifically, after receiving the transmission instruction sent by the terminal device, the headphone device sends the generated uplink audio file to the terminal device.

[0113] S510: The terminal device verifies the uplink audio file.

[0114] More specifically, after receiving the uplink audio file, the terminal device separates the uplink audio file to obtain an uplink synchronization signal, verifies the uplink synchronization signal through a preset synchronization signal, and if the verification signal sequence is consistent, it is determined that the uplink audio file has passed the verification.

[0115] S511. When the verification is passed, the terminal device generates a target file according to the user's requirements.

[0116] More specifically, when it is determined that the uplink audio file verification has passed, the terminal device generates a corresponding target file according to user requirements.

[0117] The audio data processing method provided in the embodiment of the present application obtains external human voice signals and human voice signals inside the ear canal through an earphone device, obtains a more accurate clean human voice signal and a more comfortable ear return human voice signal after processing, updates the speaker signal with the ear return human voice signal, and after obtaining the accompaniment synchronization signal, aligns and stores the ear return human voice signal and the clean human voice signal to obtain an uplink audio file, thereby enhancing the human voice quality. After the terminal device obtains the audio uplink signal, the uplink synchronization signal in the audio uplink signal is verified to improve the reliability of the audio uplink signal. After the verification is passed, a target dry sound file and / or a target wet sound file are generated according to user needs to meet the user's needs for obtaining different types of audio and further improve the user experience.

[0118] Figure 6 The structure of the audio data processing device provided in this application is shown as follows: Figure 1 , used in headphone devices, such as Figure 6 As shown, the audio data processing device 60 provided in this embodiment includes:

[0119] The first acquisition module 601 is used to acquire an external human voice signal and an ear canal sound signal; based on the speaker signal of the earphone device, the ear canal sound signal is subjected to audio negative superposition processing to obtain a physically conducted human voice signal, where the physically conducted human voice signal is the human voice heard by the user, and the external human voice signal is the actual sound emitted by the user;

[0120] The processing module 602 is used to pre-process the physically conducted human voice and the external human voice signal to obtain an ear-return human voice signal;

[0121] The processing module 602 is further used to generate an uplink audio file based on the accompaniment synchronization signal and the ear-return human voice signal sent by the terminal device, where the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal;

[0122] The processing module 602 is also used to transmit the uplink audio file to the terminal device when receiving the transmission instruction sent by the terminal device, so that the terminal device determines the uplink synchronization signal corresponding to the uplink audio file, verifies the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generates a target mixing file according to the uplink audio file and the corresponding accompaniment signal and outputs it.

[0123] Optionally, the processing module 602 is further used to perform noise reduction processing on the physically conducted human voice and the external human voice signal to obtain a clean human voice signal;

[0124] Perform spatial processing on the clean vocal signal to obtain a real-time in-ear return vocal signal.

[0125] Optionally, the processing module 602 is further configured to obtain a preset synchronization signal by performing signal extraction processing on the accompaniment synchronization signal;

[0126] According to the preset sequence information of each accompaniment audio frame in the preset synchronization signal, the real-time clean vocal signal and the ear-return vocal signal are aligned and stored;

[0127] When the current accompaniment audio frame is determined to be the end audio frame according to the preset synchronization signal, the currently generated clean vocal signal and the ear-return vocal signal are aligned and stored to generate an uplink audio file. The uplink audio file includes all clean vocal signals and ear-return vocal signals stored after the previous end audio frame.

[0128] Optionally, the processing module 602 is also used to perform audio positive superposition processing based on the ear return vocal signal and the accompaniment synchronization signal sent by the terminal device after obtaining the ear return vocal signal, so as to obtain the ear return audio, and the ear return audio is used to update the real-time speaker signal in the ear canal.

[0129] The audio data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0130] Figure 7 The structure of the audio data processing device provided in this application is shown as follows: Figure 2 , applied to terminal devices, such as Figure 7 As shown, the audio data processing device 70 provided in this embodiment includes:

[0131] The second acquisition module 701 is used to acquire an accompaniment signal and a preset synchronization signal, generate an accompaniment synchronization signal and send it to the headphone device, so that the headphone device generates an uplink audio file according to the accompaniment synchronization signal and the ear return human voice signal, the ear return human voice signal is obtained after the headphone device pre-processes the physically conducted human voice and the obtained external human voice signal, and the physically conducted human voice is obtained after the headphone device performs audio negative superposition processing on the obtained ear canal sound signal based on the speaker signal;

[0132] The sending module 702 is used to send a transmission instruction to the earphone device, so that the earphone device sends the uplink audio file to the terminal device based on the transmission instruction;

[0133] The receiving module 703 is used to determine the uplink synchronization signal corresponding to the uplink audio file when receiving the uplink audio file sent by the headphone device, verify the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generate a target mixing file according to the uplink audio file and the corresponding accompaniment signal and output it.

[0134] Optionally, the receiving module 703 is further configured to perform audio processing on the uplink audio file to obtain a processed uplink audio file;

[0135] Perform signal separation processing on the processed uplink audio file to obtain an uplink synchronization signal and a processed clean human voice signal and an ear-return human voice signal;

[0136] A target dry sound file is generated according to the processed clean vocal signal and accompaniment signal; and a target wet sound file is generated according to the in-ear return vocal signal and accompaniment signal.

[0137] The audio data processing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be described in detail here.

[0138] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 also includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0139] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0140] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0141] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0142] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0143] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0144] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0145] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0146] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0147] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0148] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0149] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0150] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0151] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0152] It should be noted that the audio data processing method of the present application can be applicable to different network standards, for example, it can be applicable to Global System of Mobile communication (GSM), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division-Synchronous Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE) system and future 5G and other network standards. Optionally, the above communication system can be a system in the scenario of Ultra-Reliable and Low Latency Communications (URLLC) transmission in a 5G communication system.

[0153] The above-mentioned terminal equipment can be a wireless terminal or a wired terminal. The wireless terminal can be a device that provides voice and / or other business data connectivity to users, a handheld device with wireless connection function, or other processing equipment connected to a wireless modem. The wireless terminal can communicate with one or more core network devices via a radio access network (Radio Access Network, referred to as RAN). The wireless terminal can be a mobile terminal, such as a mobile phone (or "cellular" phone) and a computer with a mobile terminal. For example, it can be a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device that exchanges language and / or data with the radio access network. For another example, the wireless terminal can also be a personal communication service (Personal Communication Service, referred to as PCS) phone, a cordless phone, a Session Initiation Protocol (Session Initiation Protocol, referred to as SIP) phone, a wireless local loop (Wireless Local Loop, referred to as WLL) station, a personal digital assistant (Personal Digital Assistant, referred to as PDA) and other devices. The wireless terminal may also be referred to as a system, a subscriber unit, a subscriber station, a mobile station, a mobile station, a remote station, a remote terminal, an access terminal, a user terminal, a user agent, a user device or user equipment, without limitation. Optionally, the terminal device may also be a smart watch, a tablet computer or other device.

[0154] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0155] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0156] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing audio data, characterized in that: Applicable to headphone devices, including: Acquire an external human voice signal and an ear canal sound signal; perform audio negative superposition processing on the ear canal sound signal based on the speaker signal of the headphone device to obtain a physically conducted human voice signal, wherein the physically conducted human voice signal is the human voice heard by the user, and the external human voice signal is the actual sound emitted by the user; Preprocessing the physically conducted human voice and the external human voice signal to obtain an ear-return human voice signal; Generate an uplink audio file based on the accompaniment synchronization signal sent by the terminal device and the ear-return human voice signal, wherein the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal; When receiving a transmission instruction sent by a terminal device, the uplink audio file is transmitted to the terminal device so that the terminal device determines the uplink synchronization signal corresponding to the uplink audio file, verifies the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generates a target mixing file according to the uplink audio file and the corresponding accompaniment signal and outputs it.

2. The method according to claim 1, characterized in that Preprocessing the physically conducted human voice and the external human voice signal to obtain an ear-returned human voice signal includes: Performing noise reduction processing on the physically conducted human voice and the external human voice signal to obtain a clean human voice signal; The clean vocal signal is spatially processed to obtain a real-time in-ear return vocal signal.

3. The method according to claim 2, characterized in that The uplink audio file is generated based on the accompaniment synchronization signal sent by the terminal device and the ear-return human voice signal, including: By performing signal extraction processing on the accompaniment synchronization signal, a preset synchronization signal is obtained; According to the preset sequence information of each accompaniment audio frame in the preset synchronization signal, the clean vocal signal and the ear-return vocal signal obtained in real time are aligned and stored; When the current accompaniment audio frame is determined to be the end audio frame according to the preset synchronization signal, the currently generated clean vocal signal and the ear-return vocal signal are aligned and stored to generate an uplink audio file, and the uplink audio file includes all clean vocal signals and ear-return vocal signals stored after the previous end audio frame.

4. The method according to claim 1, characterized in that: After obtaining the ear-return human voice signal, the method includes: Audio positive superposition processing is performed based on the ear-return human voice signal and the accompaniment synchronization signal sent by the terminal device to obtain the ear-return audio, and the ear-return audio is used to update the real-time speaker signal in the ear canal.

5. A method for processing audio data, characterized in that: Applied to terminal equipment, including: Acquire an accompaniment signal and a preset synchronization signal, generate an accompaniment synchronization signal and send it to a headphone device, so that the headphone device generates an uplink audio file according to the accompaniment synchronization signal and an ear-returned human voice signal, wherein the ear-returned human voice signal is obtained after the headphone device pre-processes the physically conducted human voice and the obtained external human voice signal, and the physically conducted human voice is obtained after the headphone device performs audio negative superposition processing on the obtained ear canal sound signal based on the speaker signal; Sending a transmission instruction to the headphone device, so that the headphone device sends the uplink audio file to the terminal device based on the transmission instruction; When an uplink audio file sent by the headphone device is received, an uplink synchronization signal corresponding to the uplink audio file is determined, and the uplink synchronization signal is verified based on a preset synchronization signal. After the verification is passed, a target mixing file is generated according to the uplink audio file and the corresponding accompaniment signal and output.

6. The method according to claim 5, characterized in that The method further comprises: Performing audio processing on the uplink audio file to obtain a processed uplink audio file; Performing signal separation processing on the processed uplink audio file to obtain an uplink synchronization signal and a processed clean human voice signal and an ear-return human voice signal; A target dry sound file is generated according to the processed clean vocal signal and the accompaniment signal; and a target wet sound file is generated according to the ear-return vocal signal and the accompaniment signal.

7. An audio data processing device, characterized in that: Applicable to headphone devices, including: A first acquisition module is used to acquire an external human voice signal and an ear canal sound signal; based on the speaker signal of the earphone device, the ear canal sound signal is subjected to audio negative superposition processing to obtain a physically conducted human voice signal, wherein the physically conducted human voice signal is the human voice heard by the user, and the external human voice signal is the actual sound emitted by the user; A processing module, used for preprocessing the physically conducted human voice and the external human voice signal to obtain an ear-return human voice signal; The processing module is further used to generate an uplink audio file based on the accompaniment synchronization signal sent by the terminal device and the ear-return human voice signal, wherein the accompaniment synchronization signal is generated by the terminal device according to the accompaniment signal and the preset synchronization signal; The processing module is also used to transmit the uplink audio file to the terminal device when receiving a transmission instruction sent by the terminal device, so that the terminal device determines the uplink synchronization signal corresponding to the uplink audio file, verifies the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generates a target mixing file according to the uplink audio file and the corresponding accompaniment signal and outputs it.

8. An audio data processing device, characterized in that: Applied to terminal equipment, including: a second acquisition module, for acquiring an accompaniment signal and a preset synchronization signal, generating an accompaniment synchronization signal and sending it to a headphone device, so that the headphone device generates an uplink audio file according to the accompaniment synchronization signal and an ear-returned vocal signal, wherein the ear-returned vocal signal is obtained after the headphone device pre-processes the physically conducted vocal signal and the acquired external vocal signal, and the physically conducted vocal signal is obtained after the headphone device performs audio negative superposition processing on the acquired ear canal sound signal based on the speaker signal; A sending module, configured to send a transmission instruction to the earphone device, so that the earphone device sends the uplink audio file to the terminal device based on the transmission instruction; The receiving module is used to determine the uplink synchronization signal corresponding to the uplink audio file when receiving the uplink audio file sent by the headphone device, verify the uplink synchronization signal based on the preset synchronization signal, and after the verification is passed, generate a target mixing file according to the uplink audio file and the corresponding accompaniment signal and output it.

9. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1-4 or 5-6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1-4 or 5-6 when executed by a processor.

11. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 4 or 5 to 6 when being executed by a processor.

Images