A method and electronic device for audio synthesis in a singing software based on an Android system

Abstract

The application provides a method for audio synthesis in singing software based on an Android system, and belongs to the fields of mobile application development and streaming media technology, and comprises the following steps: S1, acquiring an audio source file and decoding to obtain left and right channel audio files; S2, playing accompaniment audio and recording sound; a double-valve start-stop synchronization method is used to make the audio track lengths of the accompaniment data stream and the recording data stream consistent; S3, constant-volume segmentation is performed on the accompaniment data stream and the recording data stream; S4, noise reduction and echo elimination processing are performed on the recording data stream by a front-end algorithm processing unit, and a processed recording data stream is obtained; S5, a responsive responsibility chain recording and mixing synchronization mechanism is started, the accompaniment data stream and the recording data stream are mixed, and a mixed audio file is output; S6, the singing mode is exited, user attribute configuration processing is performed by a rear-end algorithm processing unit, and an audio file in a set format is output; the application can quickly output a mixed audio file after singing is finished, and ensures the synchronization of accompaniment and vocals.

Description

Technical Field

[0001] This invention relates to the fields of mobile application development and streaming media technology, specifically to a method and electronic device for audio synthesis in Android system singing software. Background Technology

[0002] Nowadays, Android is one of the most popular smartphone operating systems. With the improvement of cultural living standards, users have a strong demand for multimedia applications such as audio and video. Karaoke software on the Android platform allows users to sing anytime, anywhere, satisfying the desire of the general public to have their own good voice.

[0003] Currently, on the Android system, audio recording and synthesis involves multiple system modules: Pulse Code Modulation (PCM) speech acquisition, algorithm processing, encoding and compression, decoding, and speech playback. Speech processing includes algorithms such as mixing, noise cancellation, echo cancellation, voice enhancement, and Audio Gain Control (AGC). Audio encoding and decoding include both hardware and software methods. Encoding aims to compress file sizes to a smaller extent while preserving the original sound quality as much as possible, such as MP3. Decoding extracts data to provide PCM data to the media codec for playback and other operations.

[0004] Therefore, the steps for singing on an Android smartphone are as follows: First, the source audio file is downloaded from a file server. This file can be a separate PCM format file containing the accompaniment audio and vocal audio. However, to optimize download resource size and reduce bandwidth burden, most use MP3 or other encoded audio files. After decoding, the output consists of the original vocals in the left channel and the accompaniment in the right channel, both in PCM format, facilitating channel switching and mixing of the accompaniment and vocals during singing. Second, when singing begins, the right channel accompaniment is played using the Android system's underlying playback component AudioTrack, and the recording component AudioRecord... The system is responsible for synchronously recording the user's voice to obtain the user's vocal PCM audio file. Then, after the singing ends, the right channel accompaniment PCM audio file in the source file and the vocal PCM audio file recorded by the user through the microphone (MIC) are mixed using an algorithm. Since the sampling rate, number of channels, and sampling precision are consistent, the mixing can be achieved as a linear superposition of the sampled audio data of the corresponding signals. Finally, the mixed PCM audio file is processed by noise reduction, echo cancellation, audio enhancement and voice changing, audio gain, etc., and then transcoded by the PCM encoder to obtain MP3 or other format target audio files for easy storage, playback, and other customized needs.

[0005] However, existing technologies have the following problems:

[0006] 1. After the user finishes singing, the PCM vocal audio file recorded by the MIC needs to be mixed with the right channel accompaniment file to obtain a mix. The mix is ​​then processed with noise reduction, echo cancellation, audio gain, and voice enhancement before being further encoded and compressed to obtain the target audio file. The whole process is very time-consuming. Users cannot obtain and play the target audio immediately after singing and must wait a long time before their needs are met, resulting in a poor user experience.

[0007] 2. During the generation of the target audio file, there are frequent persistent I / O operations on the PCM file stream, which consumes a lot of memory, poses a risk of memory overflow, and further affects the performance of the APP itself.

[0008] 3. While playing the backing track audio, a callback function is used to extract the backing track data stream in real time. When the backing track audio is first played, there is a delay in the system from calling the playback API to the playback of the sound and the extraction of the data stream. However, the delay from the start of recording to the extraction of the recording data stream through the callback function is lower than the delay of extracting the backing track data stream. This will cause the backing track and recording to start playing simultaneously, but the recording data stream will start storing earlier and will be longer than the backing track data stream (that is, the audio track length of the recording data stream is longer than the audio track length of the backing track data stream). This will cause the timing of the vocals to lag behind the backing track in the subsequent mixing stage, when mixing is based on the first frame data node. Summary of the Invention

[0009] This invention addresses the shortcomings of existing technologies by providing a method and electronic device for audio synthesis based on Android system singing software, which can quickly output a mixed file after singing and ensure synchronization between the accompaniment and the vocals.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] A method for audio synthesis based on Android system singing software includes the following steps:

[0012] S1. Obtain the audio source file and decode it to obtain the left and right channel audio files, namely the accompaniment audio file source_left.pcm and the vocal audio file source_right.pcm;

[0013] S2. Start singing mode; play the accompaniment audio file and record the sound, obtain the accompaniment data stream palyerData and the recording data stream recordData through the callback function; and use the dual-valve start-stop synchronization method to make the track duration of the accompaniment data stream palyerData and the recording data stream recordData consistent.

[0014] S3. Perform capacity-limited segmentation on the accompaniment data stream palyerData and the recording data stream recordData obtained in step S2;

[0015] S4. The pre-processing algorithm unit performs noise reduction and echo cancellation processing on the recording data stream recordData after the fixed-capacity segmentation in step S3 to obtain the processed recording data stream denoiseRecordData.

[0016] S5. Enable the responsive chain of responsibility recording and mixing synchronization mechanism, mix the accompaniment data stream palyerData after the fixed-capacity segmentation in step S3 and the processed recording data stream denoiseRecordData obtained in step S4, and output the mixing file mix.pcm.

[0017] S6. Exit singing mode, start background service, extract the mixing file mix.pcm obtained in step S5 through post-algorithm processing unit, perform user attribute configuration processing, and output the final audio file in the set format.

[0018] Preferably, the dual-valve start / stop synchronization method described in step S2 is as follows: Independent thread valves are set in both the recording thread and the playback thread. The recording thread is used to record sound, and the playback thread is used to play the accompaniment audio file and call back the accompaniment data. The thread valves in both the recording and playback threads are closed by default, preventing the callback function from sending back data. When both the recording and playback threads receive the first frame of data obtained by the callback function, both thread valves open simultaneously, allowing the callback function to send back data. When the singing mode ends, both thread valves close simultaneously.

[0019] Preferably, the volume-fixed segmentation in step S3 is performed according to the set PCM data granularity length.

[0020] Preferably, the PCM data granularity length is an even multiple of 1024.

[0021] Preferably, the pre-processing algorithm unit in step S4 includes a preprocessing module, a noise reduction actuator module, and an echo cancellation actuator module; the preprocessing module is used to convert the recorded data stream recordData after the fixed-capacity segmentation in step S3 from byte to short data type; the noise reduction actuator module removes noise data from the data converted by the preprocessing module using an adaptive gain averaging spectral subtraction algorithm; and the echo cancellation actuator module removes echo data from the data converted by the preprocessing module using an adaptive filtering algorithm.

[0022] Preferably, the processing procedure of the pre-processing algorithm unit is as follows:

[0023] S401. Initialize the three modules; set the relevant parameters for each module;

[0024] S402. The recording data stream recordData after the fixed-capacity segmentation in step S3 is converted into a format by the preprocessing module.

[0025] S403. The data converted in step S402 is processed by both the noise reduction actuator module and the echo cancellation actuator module.

[0026] S404. The data processed in step S403 is reversed to obtain the processed recording data stream denoiseRecordData, and the data stream is called back through the JNI interface.

[0027] Preferably, the responsive chain-of-responsibility recording and mixing synchronization mechanism described in step S5 includes a recording chain-of-responsibility pool, an accompaniment chain-of-responsibility pool, and a mixing chain-of-responsibility pool. The recording chain-of-responsibility pool is used to sequentially pass the processed recording data stream denoiseRecordData obtained in step S4 to the mixing chain-of-responsibility pool. The accompaniment chain-of-responsibility pool is used to sequentially pass the accompaniment data stream palyerData after being segmented and sized in step S3 to the mixing chain-of-responsibility pool. The mixing chain-of-responsibility pool receives and buffers the data streams passed from the recording chain-of-responsibility pool and the accompaniment chain-of-responsibility pool based on responsive reception, and mixes the two data streams with the same input order sequentially using an adaptive mixing weighting method.

[0028] Preferably, the post-processing algorithm unit includes an actuator, an audio gain module, a voice changing module, and an encoding module. The actuator is used to extract the mixing file mix.pcm; the audio gain module uses an adaptive digital gain strategy to gain the audio data extracted by the actuator; the voice changing module uses Fmod to change the voice of the audio data extracted by the actuator; and the encoding module encodes and converts the audio data processed by the audio gain module or the voice changing module, and outputs a final audio file in a set format.

[0029] A computer-readable storage medium storing a computer program that causes a computer to perform the audio synthesis method in the Android system singing software described above.

[0030] An electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the audio synthesis method described above in Android system singing software.

[0031] The beneficial effects of this invention are:

[0032] 1. A dual-valve start-stop synchronization method is used, which sets an independent thread valve for each of the recording and playback threads to control the input of data streams. The two thread valves will only be opened when the first frame of data streams into both threads. If the first frame of data is not received by either thread, both thread valves will be closed, preventing the data extracted by the callback function from being input. In this way, the extra data extracted by the recording that starts earlier than the accompaniment will not be input. The two thread valves will only be opened when the callback function of the lagging playback thread also extracts the first frame of data, allowing data to be input. This ensures that the duration of the accompaniment and the recording track is synchronized, and there will be no situation where the vocals lag behind the accompaniment during subsequent mixing.

[0033] 2. By using a pre-processing algorithm unit to perform noise reduction and echo cancellation on the recording data stream during singing, and a responsive chain-of-responsibility recording-mixing synchronization mechanism to mix the recording data stream and accompaniment data stream during singing, the system achieves simultaneous singing and mixing. This fully utilizes the time slice resources during the singing stage, greatly improving post-processing efficiency. It avoids the need to extract the singing file into memory for noise reduction, echo cancellation, and mixing after singing, and then save it as a mixed file (this process involves frequent and persistent I / O operations (such as writing, reading, and remixing audio files), which consumes a lot of memory, is time-consuming, and poses a risk of memory overflow). After the user finishes singing, most of the mixing is already completed, and the final mixed file can be obtained in a very short time. This not only improves the user experience but also avoids unnecessary I / O operations, avoids the risk of memory overflow, and improves the performance of the app itself.

[0034] 3. By first dividing the recording data stream and the accompaniment data stream into fixed-capacity segments before proceeding with subsequent processing, the amount of data processed per session is reduced, thus improving processing efficiency. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating the present invention;

[0036] Figure 2 This is a schematic diagram of the processing structure of the responsive chain of responsibility recording and synchronization mechanism. Detailed Implementation

[0037] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0038] like Figure 1As shown, this invention provides a method for audio synthesis based on Android system singing software.

[0039] This invention embodiment is a complete application unit, including audio source file download, decoding and separation, and recording and playback synchronization (that is, ensuring that the accompaniment and recording start simultaneously, keeping the duration of both tracks consistent, and preventing either from starting prematurely). The operation sequence is as follows:

[0040] Step S1: Obtain the audio file. This step specifically includes the following sub-steps:

[0041] Step S101: Download the source MP3 audio file (Note that MP3 audio file is only the case in this embodiment, and is not limited to this format in practice; it can be any audio file format).

[0042] Since storing PCM audio files on computer-readable storage media consumes too much storage resources and excessive bandwidth usage is not conducive to terminal downloads, the audio source uses encoded format files, including but not limited to MP3 format; at the same time, the source MP3 format audio stored on the terminal media is encrypted with AES, and is decrypted before being loaded by the playback component when separated.

[0043] Specifically, in this embodiment of the invention, the encrypted source MP3 audio file is downloaded to a specified storage directory via the HTTP protocol.

[0044] Step S102: Decode and separate the left and right channel PCM audio.

[0045] Specifically, the decoder preprocessing unit completes the AES decryption of the source MP3, and further, the high-precision MPEG audio decoder completes the PCM decoding. The splitter outputs the left and right channel audio files source_left.pcm and source_right.pcm, where source_left.pcm is the accompaniment audio file and source_right.pcm is the vocal audio file.

[0046] Step S2: Activate singing mode. This step includes the following sub-steps:

[0047] Step S201: The player reads the left channel accompaniment and prepares for recording.

[0048] Specifically, the PCM stream of the accompaniment (i.e., the accompaniment audio file source_left.pcm) extracted in step S102 is passed to the audio playback component. At the same time, the recording component is initialized, and the sampling rate and bit depth parameters and the callback function for extracting the bit stream are set.

[0049] Step S202: Start playing the accompaniment.

[0050] Specifically, a custom player based on the Android system playback component AudioTrack extension is used to play the source_left.pcm accompaniment audio file, and the accompaniment data stream palyerData is extracted in real time through a callback function, which is a data byte[] array.

[0051] Step S203: Turn on MIC recording.

[0052] Specifically, the AudioRecorder component is used to record sound, and the recording data stream recordData is extracted in the callback function.

[0053] It should be noted that steps 202 and 203 are not actually in any particular order; the order can be reversed or performed simultaneously. They are separated only for ease of description and understanding.

[0054] Step S204: The dual-valve start-stop synchronization method controls the recording and playback synchronization.

[0055] Specifically, independent thread valves are set in both the recording and playback threads. These thread valves are closed by default, preventing callback functions from sending back data. The two thread valves open simultaneously only when both receive the first frame data from the callback function, allowing the callback function to send back data. In other words, if only one (or none) thread valve receives the first frame data from the callback function, both thread valves remain closed. They only open simultaneously when both thread valves receive the first frame data from the callback function. When the singing mode ends, both thread valves close simultaneously.

[0056] This avoids a situation where either the recording thread or the playback thread takes a long time to start, causing the other thread to acquire audio data first. This would result in inconsistent audio track lengths between the two threads. During mixing, the mixing process begins after aligning with the first frame of data. If the track lengths are inconsistent, the audio data acquired first (i.e., the one with the longer track length) will lag behind after mixing. Normally, the recording thread can acquire audio data earlier, resulting in vocal lag. The dual-valve start-stop synchronization method of this invention solves this problem, ensuring that the track lengths of the recorded audio data and the accompaniment audio data are consistent.

[0057] Step S3: Perform fixed-size segmentation on the accompaniment data stream palyerData and the recording data stream recordData obtained in step S2.

[0058] Set the PCM data granularity length for mixing to 4096 (the specific value can be adjusted according to actual needs). Divide the accompaniment data stream `palyerData` and the recording data stream `recordData` into segments according to this granularity length. Assuming the total length of these two data streams is `TotalSize`, the number of segments `n` to be divided into for both streams is determined by the following formula.

[0059]

[0060] In other words, when TotalSize and length are divisible, the number of segments n is the quotient of the two; when TotalSize and length are not divisible, the number of segments n is the integer part of the quotient plus 1, which is the last extra part (the part whose length does not reach the data granularity length after all the previous segments) that is segmented separately.

[0061] Step S4: The recording data stream recordData after the fixed-capacity segmentation in step S3 is subjected to noise reduction and echo cancellation processing by the pre-processing algorithm unit to obtain the processed recording data stream denoiseRecordData.

[0062] Specifically, the pre-processing unit includes a preprocessing module, a noise reduction actuator module, and an echo cancellation actuator module. The preprocessing module is used to convert the recorded data stream recordData after the fixed-size segmentation in step S3 into a byte-to-short data type. The noise reduction actuator module is used to remove noise data from the data converted by the preprocessing module. The echo cancellation actuator module is used to remove echo data from the data converted by the preprocessing module.

[0063] The specific process is as follows:

[0064] S401. Initialize the three modules; set the relevant parameters for each module;

[0065] S402. Start the asynchronous thread pool. The preprocessing module performs format conversion processing on the MIC audio (that is, the recording data stream recordData after the fixed-size segmentation in step S3). The MIC audio bitstream data byte[] array needs to be converted from byte to short data type in the Java layer.

[0066] S403. The data converted in step S402 is processed by both the noise reduction actuator module and the echo cancellation actuator module.

[0067] The noise reduction method, specifically the following steps: Noise reduction is achieved by subtracting the noise spectrum from the noisy speech spectrum using an adaptive gain averaging spectral subtraction algorithm. In the gain function, a linear phase is introduced to avoid the problem of non-causal filtering, and the gain function is smoothed over time.

[0068] The echo cancellation method involves the following steps: an adaptive filtering algorithm is used to estimate the multipath echo, and the filter coefficients are continuously optimized. When the estimated value approaches the true value, the echo estimate is subtracted from the MIC input signal to complete the echo cancellation algorithm processing.

[0069] S404. The data processed in step S403 is reversed to obtain the processed recording data stream denoiseRecordData, and the data stream is called back through the JNI interface.

[0070] The pre-processing unit executor is initialized through modules. It converts the continuously received recording data stream recordData into short[] stream data and processes it through the underlying C / C++ algorithm of the JNI interface processing function. Finally, it performs a data type conversion from short[] to byte[], which means reversing the format of the short[] stream data processed by the noise reduction executor module and the echo cancellation executor module, and returning the processed recording data stream denoiseRecordData through the JNI interface.

[0071] Step S5: Enable the responsive chain of responsibility recording and mixing synchronization mechanism, mix the accompaniment data stream palyerData after the fixed-capacity segmentation in step S3 and the processed recording data stream denoiseRecordData obtained in step S4, and output the mixing file mix.pcm.

[0072] Specifically, such as Figure 2 As shown, the responsive chain-of-responsibility recording and mixing synchronization mechanism includes a recording chain-of-responsibility pool, a backing track chain-of-responsibility pool, and a mixing chain-of-responsibility pool. The recording chain-of-responsibility pool is used to sequentially send the processed recording data stream denoiseRecordData obtained in step S4 to the mixing chain-of-responsibility pool. The backing track chain-of-responsibility pool is used to sequentially send the backing track data stream palyerData after being segmented and sized in step S3 to the mixing chain-of-responsibility pool. The mixing chain-of-responsibility pool receives and buffers the data streams sent by the recording chain-of-responsibility pool and the backing track chain-of-responsibility pool based on responsiveness, and then mixes them.

[0073] The recording responsibility pool and the accompaniment responsibility pool respectively include a recording publisher and an accompaniment publisher; the mixing responsibility pool includes a buffer pool and a mixing subscriber.

[0074] The I / O-based reactive chain-of-responsibility recording and mixing synchronization mechanism works as follows: the callback functions in the recording publisher and accompaniment publisher reactively publish the extracted audio bitstream data to the mixing subscriber. I / O processing is temporarily stored in a buffer pool to effectively prevent new data from overwriting data to be executed. The mixing thread performs mixing operations through reactive subscription, which ensures that the recording and accompaniment data are effectively processed by the chain of responsibility, allowing the execution logic to run in different threads, thus improving the system's I / O throughput efficiency.

[0075] The recording responsibility pool and the accompaniment responsibility pool sequentially transmit data streams to the mixing responsibility pool in the order they are received. The buffer pool within the mixing responsibility pool groups two sets of data with the same input order into one group, and then mixes the data in the same group sequentially. For example, the data transmitted from the recording and accompaniment responsibility pools to the mixing responsibility pool are numbered sequentially. Data streams with the same number from the two responsibility pools to the buffer pool are grouped together, and the mixing responsibility pool then mixes the two data streams (recording data stream and accompaniment data stream) with the same number to ensure that the accompaniment and vocals match at the same time.

[0076] The mixing responsibility pool is used to receive data streams from the recording responsibility pool and the accompaniment responsibility pool, and mixes the two synchronously input (as mentioned above, those with the same number) data streams using an adaptive mixing weighting method. The mixing processing method is implemented through the JNI layer, and the underlying C / C++ algorithm improves the mixing efficiency. Specifically, the mixing uses an adaptive mixing weighting method with attenuation factors. The maximum value of the overflow boundary is set to 2^15-1, and the minimum value is set to -2^15. When audio data overflows, the attenuation factor decreases, and when there is no data overflow, the attenuation factor gradually increases. The purpose is to make the changes in audio data relatively smooth, and to ensure that the sound intensity does not decay rapidly by smoothing the attenuation factor, thus ensuring a small degree of distortion.

[0077] The final output is the mix file mix.pcm.

[0078] Step S6: Exit singing mode, start the background Service, extract the mixing file mix.pcm obtained in step S5 through the post-processing algorithm unit, perform user attribute configuration processing, and output the final audio file in the set format; specifically including the following steps:

[0079] Step S601: Singing ends, exit singing mode, and the work is played back for preview (optional step, preview is optional).

[0080] Specifically, users can trigger a preview operation and use a custom playback component based on the Android system's AudioTrack component extension to preview the final mix file mix.pcm.

[0081] Step S602: Post-processing algorithm;

[0082] Specifically, the post-processing algorithm unit executes the audio processing action based on the user's customized selection through the operation interface. This post-processing algorithm unit includes an actuator, an audio gain module, a voice changing module, and an encoding module. (Note that in this embodiment, the post-processing algorithm unit includes both an audio gain module and a voice changing module to configure the audio data attributes; however, in practice, it is not limited to these two modules and can be set according to actual attribute configuration requirements.) The actuator extracts the mixing file mix.pcm; the audio gain module uses an adaptive digital gain strategy to gain the audio; the voice changing module uses Fmod to change the voice of the audio file; and the encoding module performs encoding conversion on the audio file processed by the audio gain module or the voice changing module.

[0083] The executor extracts the mix.pcm mixing component to complete the audio enhancement function. The attribute configuration includes algorithms for audio gain and voice enhancement. Specifically, it uses the underlying C / C++ algorithms of the JNI interface processing functions to complete the program implementation.

[0084] Furthermore, the audio gain adopts an adaptive digital gain strategy, adjusting the amplitude of the sampling points of the discrete digital signal and regulating it based on a feedback mechanism. It dynamically adjusts the digital gain parameters according to the effect of the preceding processing to optimize the gain effect.

[0085] Furthermore, the method for implementing voice changing is based on Fmod. The source file is compiled into a library file, libchange.so, which is then triggered by the user to meet customized requirements.

[0086] The specific attribute configuration to be executed and how many to execute depends on the user's customization selection in the operation interface.

[0087] Encode the audio data after attribute configuration:

[0088] Specifically, the raw PCM data processed by the algorithm (the audio gain is configured using an adaptive digital gain strategy or Fmod or other attributes to match the algorithm) is resampled and encoded. The FFmpeg library is installed, along with the matching and necessary LAME decoding library. The sampling frequency is 44100Hz, the format is PCM_16-bit, the necessary parameters are set, the MP3 encoder is obtained and opened, the resampler context is configured, the source PCM file is opened, frames are read in a loop, the encoding conversion begins, the final audio MP3 format file is transcoded and output, and finally the resources are released, and the transcoding is complete.

[0089] Step 603 (optional step, can be performed or not, for example, only store the audio file locally, without uploading): Upload the audio file of the work to the server for storage;

[0090] Specifically, the final encoded MP3 file is uploaded to a file server for storage via the HTTP protocol.

[0091] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0092] This embodiment also provides a computer-readable storage medium storing a computer program that enables a computer to execute any of the above-described methods for audio synthesis in Android system singing software.

[0093] This embodiment also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements any of the above-described methods for audio synthesis based on Android system singing software.

[0094] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0095] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for audio synthesis based on Android system singing software, characterized in that, Includes the following steps: S1. Obtain the audio source file and decode it to obtain the left and right channel audio files, namely the accompaniment audio file source_left.pcm and the vocal audio file source_right.pcm; S2. Start singing mode; play the accompaniment audio file and record the sound, obtain the accompaniment data stream playerData and the recording data stream recordData through the callback function; and use the dual-valve start-stop synchronization method to make the track duration of the accompaniment data stream playerData and the recording data stream recordData consistent. S3. Perform fixed-capacity segmentation on the accompaniment data stream playerData and the recording data stream recordData obtained in step S2; S4. The pre-processing algorithm unit performs noise reduction and echo cancellation processing on the recording data stream recordData after the fixed-capacity segmentation in step S3 to obtain the processed recording data stream denoiseRecordData. S5. Enable the responsive chain of responsibility recording and mixing synchronization mechanism, mix the accompaniment data stream playerData after the fixed-capacity segmentation in step S3 and the processed recording data stream denoiseRecordData obtained in step S4, and output the mixing file mix.pcm. S6. Exit singing mode, start background service, extract the mixing file mix.pcm obtained in step S5 through the post-algorithm processing unit, perform user attribute configuration processing, and output the final audio file in the set format. The dual-valve start / stop synchronization method described in step S2 is as follows: Independent thread valves are set in both the recording thread and the playback thread. The recording thread is used to record sound, and the playback thread is used to play the accompaniment audio file and call back the accompaniment data. The thread valves in both the recording and playback threads are closed by default, preventing the callback function from sending back data. When both the recording and playback threads receive the first frame of data obtained by the callback function, both thread valves open simultaneously, allowing the callback function to send back data. When the singing mode ends, both thread valves close simultaneously.

2. The method for audio synthesis in Android system singing software according to claim 1, characterized in that: The fixed-volume segmentation described in step S3 is performed according to the set PCM data granularity length.

3. The method for audio synthesis in Android system singing software according to claim 2, characterized in that: The PCM data granularity length is an even multiple of 1024.

4. The method for audio synthesis in Android system singing software according to claim 1, characterized in that: The pre-processing unit described in step S4 includes a preprocessing module, a noise reduction actuator module, and an echo cancellation actuator module; the preprocessing module is used to convert the recorded data stream recordData after the fixed-size segmentation in step S3 into a byte-to-short data type. The noise reduction actuator module removes noise data from the data converted by the preprocessing module using an adaptive gain averaging spectral subtraction algorithm; the echo cancellation actuator module removes echo data from the data converted by the preprocessing module using an adaptive filtering algorithm.

5. The method for audio synthesis in Android system singing software according to claim 4, characterized in that: The processing procedure of the pre-processing algorithm unit is as follows: S401. Initialize the three modules; set the relevant parameters for each module; S402. The recording data stream recordData after the fixed-capacity segmentation in step S3 is converted into a format by the preprocessing module. S403. The data converted in step S402 is processed by both the noise reduction actuator module and the echo cancellation actuator module. S404. The data processed in step S403 is reversed to obtain the processed recording data stream denoiseRecordData, and the data stream is called back through the JNI interface.

6. The method for audio synthesis in Android system singing software according to claim 1, characterized in that: The responsive chain-of-responsibility recording and mixing synchronization mechanism described in step S5 includes a recording chain-of-responsibility pool, an accompaniment chain-of-responsibility pool, and a mixing chain-of-responsibility pool. The recording chain-of-responsibility pool is used to sequentially pass the processed recording data stream denoiseRecordData obtained in step S4 to the mixing chain-of-responsibility pool. The accompaniment chain-of-responsibility pool is used to sequentially pass the accompaniment data stream playerData after being segmented and sized in step S3 to the mixing chain-of-responsibility pool. The mixing chain-of-responsibility pool receives and buffers the data streams passed from the recording chain-of-responsibility pool and the accompaniment chain-of-responsibility pool based on responsive reception, and mixes the two data streams with the same input order sequentially using an adaptive mixing weighting method.

7. The method for audio synthesis in Android system singing software according to claim 1, characterized in that: The post-processing unit includes an actuator, an audio gain module, a voice changing module, and an encoding module. The actuator is used to extract the mixing file mix.pcm. The audio gain module uses an adaptive digital gain strategy to gain the audio data extracted by the actuator. The voice changing module uses Fmod to change the voice of the audio data extracted by the actuator. The encoding module encodes and converts the audio data processed by the audio gain module or the voice changing module, and outputs the final audio file in the set format.

8. A computer-readable storage medium storing a computer program, characterized in that: The computer program causes the computer to perform the audio synthesis method in the Android system singing software as described in any one of claims 1-7.

9. An electronic device, characterized in that, include: The device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the method for audio synthesis in an Android system singing software as described in any one of claims 1-7.

Images