Cloud mobile phone real-time multi-channel audio mixing processing method and device based on end-cloud collaboration, computer equipment and storage medium
Through the method of additive fusion and normalization processing on the cloud mobile phone, the problem that the mixing method in the existing technology cannot flexibly control the sampling rate, bit width and channel number is solved, and multiple channels are realized to adapt to multiple business scenarios. Audio processing reduces the load on cloud phones and improves processing efficiency.
Patent Information
- Application Number
- CN202411383431.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-30
- Publication Date
- 2025-01-07
AI Technical Summary
The mixing method of existing cloud mobile phones cannot flexibly control the sampling rate, bit width and channel number of mixing output, and mainly depends on audio sorting, and has limited applicable business scenarios.
Using the method based on end-to-cloud collaboration, the client collects basic audio data and sends it to the cloud mobile phone. The cloud mobile phone performs additive fusion and normalization processing, mixes sounds according to weights, without relying on audio sorting, and flexibly adjusts the mixing processing parameters. .
It realizes flexible processing of multiple audio data, adapts to the needs of different business scenarios, reduces the load of cloud mobile phones, and avoids resampling and channel conversion processes.
Smart Images
Figure CN119276860A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cloud mobile phones, and more specifically to a method, device, computer equipment and storage medium for real-time multi-channel audio mixing processing of cloud mobile phones based on end-cloud collaboration. Background Art
[0002] Cloud phones are a type of smart phone based on cloud computing technology. In the prior art, the main way to mix audio on cloud phones is to decode the received audio data and play it with AudioTrack, which is actually multi-channel mixing implemented by AndroidAudioMixer. This method cannot flexibly control important parameters such as the sampling rate, bit width, and number of channels of the mixed output, and it mainly mixes the received audio after sorting it, so it is applicable to fewer business scenarios. Summary of the invention
[0003] In order to address the deficiencies in the prior art, the present invention provides a cloud phone real-time multi-channel audio mixing processing method, device, computer equipment and storage medium based on end-cloud collaboration. Based on a cloud phone and a client system, basic audio data can be obtained from multiple clients, and the basic audio data is not sorted, but additively fused and normalized according to the weight of each basic audio data, which is more suitable for the needs of different business scenarios.
[0004] In order to achieve the above object, the specific scheme adopted by the present invention is:
[0005] The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration includes the following steps:
[0006] The client uses the microphone to collect basic audio data and sends it to the cloud phone;
[0007] The cloud phone processes the basic audio data one by one to obtain sampling buffer data;
[0008] The cloud phone performs weighted fusion on the sample buffer data to obtain fused sample data;
[0009] The cloud phone normalizes the fused sampled data to obtain output audio data;
[0010] The cloud phone sends the output audio data to the client, which parses and plays it.
[0011] Preferably, the specific method for the client to collect basic audio data using a microphone includes:
[0012] The client uses a microphone to collect raw audio data;
[0013] The client encodes the original audio data based on a preset encoding method to generate basic audio data.
[0014] Preferably, the specific method for the cloud phone to process the basic audio data to obtain the sample buffer data is:
[0015] The cloud phone processes the basic audio data based on the preset mixing processing parameters;
[0016] Alternatively, the cloud phone selects one of all the basic audio data as the standard audio data, and uses the parameters of the standard audio data as mixing processing parameters to process the remaining basic audio data.
[0017] Preferably, the specific method for the cloud phone to process the basic audio data based on the mixing processing parameters includes:
[0018] The cloud phone determines whether the sampling rate of the basic audio data to be processed matches the sampling rate in the mixing processing parameters;
[0019] If they match, the basic audio data is converted into sample buffer data;
[0020] If there is no match, the basic audio data is resampled and converted into sample buffer data.
[0021] Preferably, the cloud phone performs a similarity pre-adjustment on the sampling buffer data before performing weighted fusion on the sampling buffer data.
[0022] Preferably, the method for the cloud phone to pre-adjust the similarity of the sample buffer data includes:
[0023] The cloud phone performs bit width conversion on the sample buffer data until the bit width of all sample buffer data is consistent;
[0024] The cloud phone converts the number of channels of the sample buffer data until the number of channels of all the sample buffer data is consistent.
[0025] Preferably, when the cloud phone performs weighted fusion on the sample buffer data, the range of the weight assigned to the fused sample data is 0-10.
[0026] A cloud phone real-time multi-channel audio mixing processing device based on end-cloud collaboration is used to implement the above-mentioned cloud phone real-time multi-channel audio mixing processing method based on end-cloud collaboration, and the device includes:
[0027] A data transmission module, used to send basic audio data from the client to the cloud phone, and to send output audio data from the cloud phone to the client;
[0028] The human-computer interaction module is used to obtain the mixing processing parameters and send them to the cloud phone;
[0029] The weight acquisition module is used to obtain the weight value and send it to the cloud phone.
[0030] Computer equipment, including:
[0031] a memory storing a computer program;
[0032] The processor is used to execute a computer program to implement the above-mentioned cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration.
[0033] The storage medium stores a computer program, which, when executed, implements the above-mentioned cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration.
[0034] The present invention is based on a cloud phone and a client system, and can obtain basic audio data from multiple clients, and can flexibly adjust mixing processing parameters for mixing; after the mixing is completed, the present invention does not need to perform processes such as resampling and channel number conversion, and the load of the cloud phone is lower; the present invention does not sort the basic audio data, but additively fuses and normalizes the basic audio data based on the weights, which is more suitable for the needs of different business scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1 It is a schematic diagram of the collaborative mode between the cloud phone and the client in the present invention;
[0037] Figure 2 is a flow chart of the mixing processing method of the present invention;
[0038] Figure 3 It is a schematic diagram of the specific process of the mixing process of the present invention. DETAILED DESCRIPTION
[0039] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0040] See also Figures 1 to 3 The present invention first provides a cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration, including S1 to S5.
[0041] S1. The client uses a microphone to collect basic audio data and sends it to the cloud phone. The specific method of the client using a microphone to collect basic audio data includes S1 to S12.
[0042] S11, the client uses a microphone to collect raw audio data. After collecting the raw audio data, the client can perform certain processing on the raw audio data to improve the quality of the raw audio data. Specific methods may include denoising, voice separation, and background sound reduction, which belong to the existing technology in this field and will not be repeated here.
[0043] S12, the client encodes the original audio data based on a preset encoding method to generate basic audio data. The original audio data can be encoded using various conventional encoding methods in the art, such as PCM, WAV, MP3, and AAC. In order to ensure the quality of the original audio data, in the present invention, the original audio data is encoded using PCM.
[0044] S2. The cloud phone processes the basic audio data one by one to obtain the sample buffer data. The specific method for the cloud phone to process the basic audio data to obtain the sample buffer data is: the cloud phone processes the basic audio data based on the preset mixing processing parameters; or, the cloud phone selects one of the basic audio data as the standard audio data from all the basic audio data, and uses the parameters of the standard audio data as the mixing processing parameters to process the remaining basic audio data. The specific selection method can be the following two.
[0045] The first method is that if among all the basic audio data, the sampling rate, bit width, number of channels and other data of some basic audio data exceed the preset mixing processing parameters, then its parameters are used as the actual mixing processing parameters; if the three parameters of all the basic audio data do not reach the preset mixing processing parameters, then the preset mixing processing parameters are used to process the basic audio data.
[0046] The second method is to analyze the parameters of all basic audio data. If the three parameters of most of the basic audio data are consistent, these parameters are used as actual mixing processing parameters. If the number of basic audio data with consistent three parameters is small, the basic audio data is processed with preset mixing processing parameters.
[0047] More specifically, the specific method for the cloud phone to process the basic audio data based on the mixing processing parameters includes: the cloud phone determines whether the sampling rate of the basic audio data to be processed matches the sampling rate in the mixing processing parameters; if it matches, the basic audio data is converted into sampling buffer data; if it does not match, the basic audio data is resampled and converted into sampling buffer data. The specific method of resampling can be implemented by calling the ffmpeg swr_convert resampling function in the cloud phone, which belongs to the conventional technology in this field and will not be repeated here. When resampling, if the sampling rate of the basic audio data is Fin, the sampling rate in the mixing processing parameters is Fout, and the number of mono sampling points is Flen, the length of the PCM data required to be input for each mixing can be calculated according to Fout*Flen / Fin.
[0048] Afterwards, the cloud phone performs similarity pre-adjustment on the sampling buffer data before performing weighted fusion on the sampling buffer data.
[0049] The method for the cloud phone to pre-adjust the similarity of the sampling buffer data includes: the cloud phone converts the bit width of the sampling buffer data until the bit width of all the sampling buffer data is consistent; the cloud phone converts the number of channels of the sampling buffer data until the number of channels of all the sampling buffer data is consistent.
[0050] Through the above processing of the sampling rate, bit width and number of channels, all the sampling buffer data can be kept with the same parameters, which is beneficial to subsequent processing.
[0051] S3. The cloud phone performs weighted fusion on the sample buffer data to obtain fused sample data. When the cloud phone performs weighted fusion on the sample buffer data, the weight assigned to the fused sample data ranges from 0 to 10. The specific value of the weight can be determined based on the weight of the client corresponding to the original audio data. If the weight of the client is larger, that is, the original audio data collected by the client is more important, then the corresponding sample buffer data has a larger weight. Conversely, if the weight of the client is smaller, that is, the original audio data collected by the client is not very important, then the corresponding sample buffer data has a smaller weight. In the default state, the weights are all set to 1.
[0052] S4. The cloud phone normalizes the fused sampled data to obtain output audio data. Normalization is mainly used to further ensure the quality of the output audio data. Specific methods may include removing popping sounds, etc., so that the output audio data is smoother and more delicate. It may also process sound effects so that the output audio data can produce better effects when played.
[0053] S5. The cloud phone sends the output audio data to the client, which parses and plays it.
[0054] In a preferred embodiment of the present invention, after obtaining the output audio data, the output audio data can also be sliced to obtain multiple audio data segments. Further, each client has different playback permissions, and a client with a higher playback permission can play more audio data segments, and a client with a lower playback permission can play less audio data segments. On this basis, the audio data segments are reorganized according to the playback permissions of the client, and the reorganized audio data segments are sent to the client. Because the playback permissions of the client are associated with the audio data segments, there may be discontinuity between two adjacent audio data segments that need to be reorganized. If they are directly spliced, it is easy to cause discomfort when the client plays. In order to avoid this problem, when the audio data segments are reorganized, a segmented prompt tone can be added between two adjacent audio data segments to avoid the problem of discomfort caused by the large difference between the audio data segments during listening. Obviously, if the two adjacent audio data segments that need to be reorganized are continuous in themselves, there is no need to add a segmented prompt tone.
[0055] In the above embodiment, the playback permissions of different clients should be pre-set and stored in the cloud phone, so that the cloud phone can reorganize all the audio data segments according to the playback defects and send the reorganized audio data segments to the client.
[0056] On the other hand, the obtained audio data fragments can also be sent to the client after encryption, and the cloud phone only saves the output audio data, thereby realizing distributed backup storage of the output audio data, avoiding data loss, and facilitating subsequent viewing.
[0057] The present invention further provides a cloud mobile phone real-time multi-channel audio mixing processing device based on end-cloud collaboration, which is used to implement the above-mentioned cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration. The device includes a data transmission module, a human-computer interaction module and a weight acquisition module.
[0058] The data transmission module is used to send basic audio data from the client to the cloud phone, and to send output audio data from the cloud phone to the client.
[0059] The human-computer interaction module is used to obtain the mixing processing parameters and send them to the cloud phone.
[0060] The weight acquisition module is used to obtain the weight value and send it to the cloud phone.
[0061] Those skilled in the art can clearly understand that, for the convenience and simplicity of description, the specific working process of the system and device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.
[0062] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0063] Computer equipment, including memory and processor.
[0064] A memory storing a computer program.
[0065] The processor is used to execute a computer program to implement the above-mentioned cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration.
[0066] The present invention finally provides a storage medium storing a computer program, which, when executed, implements the above-mentioned cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration.
[0067] Based on the storage medium, the technical solution of the present disclosure, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.
[0068] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.
[0069] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A cloud phone real-time multi-channel audio mixing processing method based on end-cloud collaboration, characterized in that: The steps include: The client uses the microphone to collect basic audio data and sends it to the cloud phone; The cloud phone processes the basic audio data one by one to obtain sampling buffer data; The cloud phone performs weighted fusion on the sample buffer data to obtain fused sample data; The cloud phone normalizes the fused sampled data to obtain output audio data; The cloud phone sends the output audio data to the client, which parses and plays it.
2. The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration as claimed in claim 1, characterized in that: The specific methods for the client to collect basic audio data using a microphone include: The client uses a microphone to collect raw audio data; The client encodes the original audio data based on a preset encoding method to generate basic audio data.
3. The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration as claimed in claim 1, characterized in that: The specific method for the cloud phone to process the basic audio data to obtain the sample buffer data is: The cloud phone processes the basic audio data based on the preset mixing processing parameters; Alternatively, the cloud phone selects one of all the basic audio data as the standard audio data, and uses the parameters of the standard audio data as mixing processing parameters to process the remaining basic audio data.
4. The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration as claimed in claim 3 is characterized in that: The specific method for the cloud phone to process basic audio data based on the mixing processing parameters includes: The cloud phone determines whether the sampling rate of the basic audio data to be processed matches the sampling rate in the mixing processing parameters; If they match, the basic audio data is converted into sample buffer data; If there is no match, the basic audio data is resampled and converted into sample buffer data.
5. The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration as claimed in claim 1, characterized in that: Before the cloud phone performs weighted fusion on the sample buffer data, it performs a similarity pre-adjustment on the sample buffer data.
6. The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration as claimed in claim 5, characterized in that: The method for the cloud phone to pre-adjust the similarity of the sample buffer data includes: The cloud phone performs bit width conversion on the sample buffer data until the bit width of all sample buffer data is consistent; The cloud phone converts the number of channels of the sample buffer data until the number of channels of all the sample buffer data is consistent.
7. The method for real-time multi-channel audio mixing processing of a cloud phone based on end-cloud collaboration as claimed in claim 1, characterized in that: When the cloud phone performs weighted fusion on the sample buffer data, the weight range of the fused sample data is 0-10.
8. A cloud mobile phone real-time multi-channel audio mixing processing device based on end-cloud collaboration, characterized in that: The device is used to implement the cloud phone real-time multi-channel audio mixing processing method based on terminal-cloud collaboration as described in any one of claims 1 to 7, the device comprising: a data transmission module, used to send basic audio data from the client to the cloud phone, and used to send output audio data from the cloud phone to the client; The human-computer interaction module is used to obtain the mixing processing parameters and send them to the cloud phone; The weight acquisition module is used to obtain the weight value and send it to the cloud phone.
9. Computer device, characterized in that include: a memory storing a computer program; A processor is used to execute a computer program to implement a cloud phone real-time multi-channel audio mixing processing method based on end-cloud collaboration as described in any one of claims 1-7.
10. A storage medium, characterized in that: A computer program is stored, and when the computer program is executed, a cloud mobile phone real-time multi-channel audio mixing processing method based on end-cloud collaboration as described in any one of claims 1-7 is implemented.