Audio control method and device, medium and equipment

Abstract

The embodiment of the invention provides an audio control method and device, a medium and equipment. The method comprises the following steps: acquiring background audio data output by an application layer and transmitted through a first transmission path; acquiring acquired audio data received by the audio acquisition device and transmitted through a second transmission path, wherein the second transmission path does not pass through the application layer included in the first transmission path; according to the collected audio data and the background audio data, carrying out sound mixing processing on the collected audio data and the background audio data to obtain target audio data; and transmitting the target audio data to an audio playing device for playing. According to the audio control method provided by the embodiment of the invention, the background audio data and the acquired audio data are transmitted by adopting different transmission paths respectively, so that logic layers through which the acquired audio data need to pass in a traditional transmission mode are reduced; therefore, the data transmission delay of the acquired audio data and the target audio data obtained through audio mixing processing in the transmission process is reduced.

Description

Technical Field

[0001] The present application relates to the field of audio control technology, and in particular to the field of audio control technology, and in particular to an audio control method, device, medium and equipment. Background Art

[0002] In the scenario where multi-audio mixing processing is required, for example, when a user uses a karaoke application on a mobile phone, the karaoke application first collects the human voice produced by the user's singing through the built-in or external microphone of the mobile phone, and then mixes the human voice with the background music built into the karaoke application. The karaoke data obtained by the mixing process is then passed through the complex flow of the underlying logic of the mobile phone system, and finally the complete audio that combines the human voice and background music is output through the speakers of the mobile phone or other external devices. For details, please refer to Figure 1 .

[0003] However, from Figure 1 It can be seen from the data transmission flow chart that human voice needs to go through a long transmission path when reaching the karaoke application. At the same time, after the human voice and background music are mixed in the karaoke application to form karaoke data, the karaoke data is sent to the speaker for output. In this process, the karaoke data also needs to go through a long transmission path, resulting in a large data transmission delay in the entire data transmission process. The final impact is that it takes a long time for the user to input the human voice through the microphone and hear the complete audio from the speaker, that is, there is a significant delay, which brings a poor product experience to the user. Summary of the invention

[0004] The present application provides an audio control method, device, medium and equipment. The audio control method provided in the present application is used to solve the problem of poor user experience caused by large data delay in the current audio control method.

[0005] An embodiment of the present application provides an audio control method, including:

[0006] Acquire background audio data output by the application layer and transmitted through the first transmission path;

[0007] Acquire collected audio data received by the audio collection device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path;

[0008] According to the collected audio data and the background audio data, mixing processing is performed on the collected audio data and the background audio data to obtain target audio data including the collected audio data and the background audio data;

[0009] The target audio data is transmitted to an audio playback device for playback.

[0010] Furthermore, in the audio control method described in the embodiment of the present application, the first transmission path includes a transmission path that sequentially connects the application layer, the hardware abstraction layer, and the data processor, and the obtaining of background audio data output by the application layer and transmitted through the first transmission path includes:

[0011] Controlling the application layer to output background audio data, and transmitting the background audio data to a hardware abstraction layer that provides a data transmission interface for the application layer and the data processor respectively;

[0012] The hardware abstraction layer is controlled to transmit the background audio data to the data processor.

[0013] Further, in the audio control method described in the embodiment of the present application, the second transmission path includes a transmission path that sequentially connects the audio acquisition device, the hardware abstraction layer, and the data processor, and the acquiring of the collected audio data received by the audio acquisition device and transmitted through the second transmission path includes:

[0014] Controlling the audio acquisition device to receive the collected audio data, and transmitting the collected audio data to the hardware abstraction layer, and dividing the collected audio data into a plurality of data units according to a preset division rule through the hardware abstraction layer;

[0015] The hardware abstraction layer is controlled to transmit the plurality of data units to the data processor according to a preset data transmission frequency.

[0016] Furthermore, in the audio control method described in the embodiment of the present application, before controlling the hardware abstraction layer to transmit the plurality of data units to the data processor according to a preset data transmission frequency, the method further includes:

[0017] The hardware abstraction layer is controlled to perform reverberation processing on the collected audio data to obtain the collected audio data after the reverberation processing.

[0018] Furthermore, in the audio control method described in the embodiment of the present application, the mixing process of the collected audio data and the background audio data to obtain the target audio data including the collected audio data and the background audio data includes:

[0019] Performing mixing processing on the collected audio data and the background audio data through the data processor; or

[0020] Controlling the hardware abstraction layer to first adjust the sampling rate and bit width of the collected audio data and the background audio data into a preset format, and then controlling the hardware abstraction layer to perform mixing processing on the collected audio data and the background audio data adjusted into the preset format;

[0021] Target audio data including the collected audio data and background audio data is obtained.

[0022] Furthermore, in the audio control method described in the embodiment of the present application, the mixing of the collected audio data and the background audio data by the data processor includes:

[0023] The data processor is controlled to perform sound effect processing on the background audio data and then send it to the mixer, and the collected audio data skips the sound effect post-processing and is directly sent to the mixer to mix with the background audio data after the sound effect processing.

[0024] Furthermore, in the audio control method described in the embodiment of the present application, before obtaining the target audio data including the collected audio data and the background audio data, the method further includes:

[0025] Controlling the data processor to adjust the volume of the collected audio data and the background audio data so that the volume values ​​of the collected audio data and the background audio data are coordinated;

[0026] The controlling the data processor to adjust the volume of the collected audio data and the background audio data includes:

[0027] Respectively acquiring a plurality of the collected audio data sampling points and the background audio data sampling points located in the same sampling time period;

[0028] Respectively calculating a first average value of a plurality of the collected audio data sampling points and a second average value of a plurality of the background audio data sampling points;

[0029] The first average value is compared with the second average value, and the volume value of the collected audio data or the background audio data is adjusted accordingly according to the comparison result.

[0030] Accordingly, another aspect of the present application embodiment further provides an audio control device, including:

[0031] A first acquisition module, used for acquiring background audio data output by the application layer and transmitted through a first transmission path;

[0032] A second acquisition module, used to acquire collected audio data received by the audio acquisition device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path;

[0033] A mixing processing module, used for performing mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data, so as to obtain target audio data including the collected audio data and the background audio data;

[0034] The audio playing module is used to transmit the target audio data to the audio playing device for playing.

[0035] Correspondingly, another aspect of the embodiment of the present application further provides a storage medium, wherein the storage medium stores a plurality of instructions, and the instructions are suitable for loading by a processor to execute the audio control method as described above.

[0036] Correspondingly, another aspect of an embodiment of the present application further provides an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the audio control method as described above.

[0037] The embodiments of the present application provide an audio control method, device, medium and equipment, which obtains background audio data output by the application layer and transmitted through a first transmission path; obtains collected audio data received by an audio collection device and transmitted through a second transmission path, wherein the second transmission path does not pass through the application layer included in the first transmission path; performs mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data to obtain target audio data including the collected audio data and the background audio data; and transmits the target audio data to an audio playback device for playback. Utilizing the audio control method provided by the embodiments of the present application, by transmitting the background audio data and the collected audio data using different transmission paths respectively, the application layers that the collected audio data needs to pass through in the traditional transmission method are reduced, thereby reducing the data transmission delay of the collected audio data and the target audio data obtained after the mixing processing during the transmission process. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0039] Figure 1 The figure is a flow chart of the mixing process in the prior art.

[0040] Figure 2 A flowchart of an audio control method provided in an embodiment of the present application.

[0041] Figure 3Schematic diagram of the mixing process in the embodiment of the present application.

[0042] Figure 4 Another flowchart of the audio control method provided in an embodiment of the present application.

[0043] Figure 5 Another flowchart of the audio control method provided in an embodiment of the present application.

[0044] Figure 6 A schematic diagram of the structure of an audio control device provided in an embodiment of the present application.

[0045] Figure 7 Another structural schematic diagram of the audio control device provided in an embodiment of the present application.

[0046] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0047] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0048] An embodiment of the present application provides an audio control method, which transmits background audio data and collected audio data using different transmission paths respectively, thereby reducing the logical layers that the collected audio data needs to pass through in the traditional transmission method, thereby reducing the data transmission delay of the collected audio data and the target audio data obtained after mixing processing during the transmission process.

[0049] The term "and / or" in this application can be a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this application generally indicates that the associated objects before and after are in an "or" relationship.

[0050] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable in appropriate circumstances, so that the embodiments described here can be implemented in a sequence other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, the process, method, system, product or television set comprising a series of steps or modules need not be limited to the steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or television sets. The naming or numbering of the steps in the present application does not mean that the steps in the method flow must be performed in the time / logic sequence indicated by the naming or numbering, and the process steps that have been named or numbered can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of modules in this application is a logical division. There may be other division methods when it is implemented in actual applications. For example, multiple modules can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection between modules can be electrical or other similar forms, which are not limited in this application. In addition, the modules or submodules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the present application.

[0051] The audio control method involved in the embodiment of the present application is mainly applied to electronic devices, which may be karaoke equipment, televisions, smart phones, tablet computers, laptops, desktop computers, and servers, etc., which are not limited here. Optionally, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or an IoT cloud (Internet of Things cloud) that provides the ability to store, process and manage data generated by an IoT television, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, which are not limited here.

[0052] For ease of understanding, the specific process in the embodiments of the present application is described below. Figure 2 , Figure 2A flowchart of an embodiment of an audio control method provided in an embodiment of the present application.

[0053] exist Figure 2 In the illustrated embodiment, the method specifically comprises the following steps:

[0054] S101, obtaining background audio data output by an application layer and transmitted through a first transmission path.

[0055] It should be explained that the application layer refers to the highest layer in the computer network model (such as the five-layer model of Android or the four-layer model of TCP / IP), which is located at the top of the network and directly faces the user. The main function of the application layer is to provide network applications and services, allowing users to directly use the network for communication and data exchange. In the present embodiment, the application layer is illustrated by taking the karaoke application as an example. When the user uses the karaoke application for entertainment, the karaoke application can provide the user with background audio data, such as accompaniment audio, original audio or other audio data as background sound, which is not limited here. And the background audio data is transferred to the subsequent logic layer of the computer network model through the first transmission path for corresponding processing, and finally mixed with the human voice audio data entered by the user to form a complete audio that combines human voice and background music.

[0056] S102: Acquire collected audio data received by an audio collection device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path.

[0057] It should be explained that the audio acquisition device refers to a device used to collect audio data generated by an external sound source, such as a built-in or external microphone on a mobile phone. The collected audio data refers to audio data generated by an external sound source, such as human voices, musical instrument sounds, or other audio data used as foreground sounds, which are not limited here. It should be noted that the collected audio data needs to be distinguished from the background audio data in this solution. The collected audio data is equivalent to the foreground audio data of the background audio data. In the scenario where multi-audio mixing processing is required, the collected audio data and the background audio data are used together to complete the multi-audio mixing processing.

[0058] In this embodiment, taking the example of a user singing using a karaoke application on a mobile phone, the karaoke application first collects the human voice produced by the user's singing through the built-in or external microphone of the mobile phone (i.e., collects audio data), and then transfers the human voice through the second transmission path to the subsequent logical layer of the computer network model for corresponding processing, and finally mixes it with the background sound provided by the karaoke application (i.e., background audio data) to form a complete audio that combines the human voice and background sound.

[0059] It should be noted that, in this embodiment, the second transmission path used for the collected audio data does not pass through the application layer included in the first transmission path used for the background audio data. By eliminating the application layer that the collected audio data needs to pass through in the traditional data transmission method, the transmission path of the collected audio data is shortened.

[0060] S103, performing mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data to obtain target audio data including the collected audio data and the background audio data.

[0061] In this embodiment, the process of collecting audio data and background audio data for mixing processing is modified from being executed by the traditional application layer to being executed by other logical layers in the computer network model, thereby reducing the data transmission delay of the collected audio data and the target audio data obtained after the mixing processing during the transmission process by shortening the transmission path of the collected audio data without affecting the mixing processing.

[0062] S104: Transmit the target audio data to an audio playback device for playback.

[0063] It should be explained that the audio playback device refers to a device for outputting target audio data, such as a speaker or a sound box or a speaker or other device capable of outputting audio, which is not limited here. In this embodiment, the target audio data takes a speaker as an example, and the target audio data obtained by mixing the collected audio data and the background audio data is output through the speaker, so that the user can hear the complete audio that combines the human voice and the background sound.

[0064] In some embodiments, the first transmission path includes a transmission path sequentially connecting the application layer, the hardware abstraction layer, and the data processor.

[0065] In this embodiment, the first transmission path includes a transmission path connecting the application layer, the hardware abstraction layer and the data processor. In other embodiments, the first transmission path may also include a system framework layer and a system kernel layer.

[0066] Based on the above introduction to the nodes involved in the first transmission path, in this embodiment, the acquisition of background audio data output by the application layer and transmitted through the first transmission path specifically includes the following steps:

[0067] Controlling the application layer to output background audio data, and transmitting the background audio data to a hardware abstraction layer that provides a data transmission interface for the application layer and the data processor respectively;

[0068] The control hardware abstraction layer transfers the background audio data to the data processor.

[0069] It needs to be explained that the hardware abstraction layer (Hal) is an interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware. It hides the hardware interface details of a specific platform and provides a virtual hardware platform for the operating system, making it hardware-independent and portable on multiple platforms. The functional simulation of the user device interface by the hardware abstraction layer is mainly completed by the virtual driver module, including the sending and receiving of data packets and the preprocessing of protocol messages, providing standard API functions for the upper-layer protocol software, while the interface management of the user device is managed, configured and monitored by the device management module of the upper-layer network management software. The internal communication module runs on the internal Ethernet network, coordinates the functional interfaces between the modules, and ensures real-time and reliable data transmission between the slave processing unit and the master processing unit.

[0070] The main role of the data processor in this embodiment is to receive the background audio data sent by the hardware abstraction layer, and after mixing with the collected audio data to obtain the target audio data, send the target audio data to the audio playback device for output. It should be noted that this solution cancels the traditional method of collecting audio data and background audio data in the application layer, and modifies the process to be performed in the data processor. In order to further improve the data processing capability of the data processor and replace the application layer, in this embodiment, a digital processor (DSP) is preferably selected as the data processor. The unique hardware multiplier of the digital processor can be used to quickly implement various digital signal processing algorithms. Without affecting the original mixing process, it can also further improve the efficiency of data processing and reduce the delay of data transmission.

[0071] The kernel layer is the innermost layer of the operating system. It is the first layer of software that runs directly on the hardware. The kernel is responsible for managing the basic resources of the computer and providing basic services. It should be noted that the kernel layer is mainly used to provide underlying system services, such as memory management, process management, etc. In some simple data processing scenarios (such as karaoke audio data processing), since the hardware abstraction layer can directly interact with the data processor, and the data processor is responsible for the processing of audio data, the intervention of the kernel layer is not required.

[0072] The system framework layer (Framework) is located above the system kernel layer. It provides a series of services and interfaces for applications, allowing applications to efficiently access the resources provided by the operating system. The role of the system framework layer is to simplify application development, provide consistent services and interfaces, and isolate the direct interaction between applications and the underlying hardware and operating system. It should be noted that in some simple data processing scenarios, such as karaoke audio data can be processed and output directly from the microphone through the hardware abstraction layer, and the system framework layer is usually used to manage communication between applications and the abstraction of system resources. For simple audio streams directly from hardware (such as microphones) to speakers, the system framework layer can be omitted.

[0073] In some embodiments, the second transmission path includes a transmission path sequentially connecting the audio acquisition device, the hardware abstraction layer and the data processor.

[0074] In this embodiment, if Figure 3 As shown, the second transmission path includes a transmission path connecting the audio acquisition device, the hardware abstraction layer and the data processor. In other embodiments, such as Figure 4 As shown, the second transmission path may also include a system framework layer and a system kernel layer, etc.

[0075] Based on the above introduction to the nodes involved in the second transmission path, in this embodiment, Figure 5 As shown, the S102 specifically includes the following steps:

[0076] S1021, controlling the audio acquisition device to receive the collected audio data, and transmitting the collected audio data to the hardware abstraction layer, and dividing the collected audio data into a plurality of data units according to a preset division rule through the hardware abstraction layer;

[0077] S1022, controlling the hardware abstraction layer to transmit a plurality of data units to the data processor according to a preset data transmission frequency.

[0078] In this embodiment, the preset segmentation rule refers to the requirement of cutting the collected audio data according to the system performance, for example, cutting once every 5ms, and ensuring that the data volume corresponding to each cut data unit is small, thereby improving the speed of the collected audio data during transmission and indirectly reducing the data transmission delay. At the same time, the hardware abstraction layer can also be controlled to transmit a number of data units to the data processor according to the preset data transmission frequency (for example, once every 5ms), ensuring that the data processor can process the received data units in a timely manner.

[0079] In some embodiments, before controlling the hardware abstraction layer to transmit the plurality of data units to the data processor according to a preset data transmission frequency, the method further includes:

[0080] The hardware abstraction layer is controlled to perform reverberation processing on the collected audio data to obtain the collected audio data after the reverberation processing.

[0081] In this embodiment, the collected audio data is reverberated by controlling the hardware abstraction layer, so that the target audio data finally output through the speaker has a reverberation effect, thereby improving the user experience. It should be noted that the reverberation processing can be implemented based on a reverberator, and the reverberator can be added to the data processor or to the hardware abstraction layer.

[0082] In some embodiments, the mixing process of the collected audio data and the background audio data to obtain the target audio data including the collected audio data and the background audio data includes:

[0083] Performing mixing processing on the collected audio data and the background audio data through the data processor; or

[0084] Controlling the hardware abstraction layer to first adjust the sampling rate and bit width of the collected audio data and the background audio data into a preset format, and then controlling the hardware abstraction layer to perform mixing processing on the collected audio data and the background audio data adjusted into the preset format;

[0085] Target audio data including the collected audio data and background audio data is obtained.

[0086] In the present embodiment, the occurrence node of the mixing process can be selected according to whether the data processor has the mixing capability. When the data processor has the mixing and SRC (re-sampling) capabilities, the sampling rate and bit width of the collected audio data and the background audio data can be maintained, and the mixing process can be directly transmitted to the data processor without the need for mixing pre-processing in advance. When the data processor does not have the mixing and SRC capabilities, the sampling rate and bit width of the collected audio data and the background audio data can be adjusted respectively at the hardware abstraction layer, and after these necessary parameters are adjusted to be consistent with the preset format required by the system, the collected audio data and the background audio data adjusted to the preset format can be mixed to process the target audio data, and then the target audio data can be transmitted to the data processor for subsequent processing.

[0087] In some embodiments, the mixing of the collected audio data and the background audio data by the data processor includes:

[0088] The data processor is controlled to perform sound effect processing on the background audio data and then send it to the mixer, and the collected audio data skips the sound effect post-processing and is directly sent to the mixer to mix with the background audio data after the sound effect processing.

[0089] In this embodiment, the sound effect processing of background audio data (such as noise reduction or improving audio resolution, etc.) originally performed at the application layer is changed to be performed on the data processor, and the excellent data processing capabilities of the data processor are utilized. It is possible to process the sound effects of the background audio data more efficiently than the application layer, and cancel the sound effect processing of the collected audio data, thereby further reducing the delay in data transmission.

[0090] In some embodiments, before obtaining the target audio data including the collected audio data and the background audio data, the method further includes:

[0091] Controlling the data processor to adjust the volume of the collected audio data and the background audio data so that the volume values ​​of the collected audio data and the background audio data are coordinated;

[0092] In this embodiment, by comparing the volume values ​​of the background audio data and the collected audio data, it is determined whether the volume of the background audio data or the collected audio data needs to be adjusted, so that the background audio data and the collected audio data in the final output target audio data can be played in a coordinated manner.

[0093] In some embodiments, controlling the data processor to adjust the volume of the collected audio data and the background audio data specifically includes the following steps:

[0094] Respectively obtain a plurality of collected audio data sampling points and background audio data sampling points located in the same sampling time period;

[0095] Calculate first average values ​​of a plurality of collected audio data sampling points and second average values ​​of a plurality of background audio data sampling points respectively;

[0096] The first average value is compared with the second average value, and the volume value of the collected audio data or the background audio data is adjusted accordingly according to the comparison result.

[0097] It should be explained that the collected audio data sampling points and the background audio data sampling points refer to digital signals converted from analog signals during the transmission process, so the collected audio data sampling points and the background audio data sampling points exist in digital form.

[0098] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, which will not be described one by one here.

[0099] In specific implementation, the present application is not limited by the execution order of the various steps described. If no conflict occurs, some steps can be performed in other orders or simultaneously.

[0100] As can be seen from the above, the audio control method provided in the embodiment of the present application obtains background audio data output by the application layer and transmitted through a first transmission path; obtains collected audio data received by an audio collection device and transmitted through a second transmission path, wherein the second transmission path does not pass through the application layer included in the first transmission path; performs mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data to obtain target audio data containing the collected audio data and the background audio data; and transmits the target audio data to an audio playback device for playback. Utilizing the audio control method provided in the embodiment of the present application, by transmitting the background audio data and the collected audio data using different transmission paths respectively, the logical layers that the collected audio data needs to pass through in the traditional transmission method are reduced, thereby reducing the data transmission delay of the collected audio data and the target audio data obtained after the mixing processing during the transmission process.

[0101] An embodiment of the present application further provides an audio control device, which can be integrated into an electronic device.

[0102] See also Figure 6 , Figure 6 The audio control device 30 may include:

[0103] A first acquisition module 31, used to acquire background audio data output by the application layer and transmitted through a first transmission path;

[0104] A second acquisition module 32, used to acquire the collected audio data received by the audio acquisition device and transmitted through a second transmission path, wherein the second transmission path does not pass through the application layer included in the first transmission path;

[0105] A mixing processing module 33 is used to perform mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data to obtain target audio data containing the collected audio data and the background audio data;

[0106] The audio playing module 34 is used to transmit the target audio data to an audio playing device for playing.

[0107] In some embodiments, the first transmission path includes a transmission path that sequentially connects the application layer, the hardware abstraction layer and the data processor, and the first acquisition module 31 is used to control the application layer to output background audio data, and transmit the background audio data to the hardware abstraction layer that provides a data transmission interface for the application layer and the data processor respectively; and control the hardware abstraction layer to transmit the background audio data to the data processor.

[0108] In some embodiments, the second transmission path includes a transmission path that sequentially connects the audio acquisition device, the hardware abstraction layer and the data processor, and the second acquisition module 32 is used to control the audio acquisition device to receive collected audio data, and transmit the collected audio data to the hardware abstraction layer, and divide the collected audio data into a number of data units according to a preset segmentation rule through the hardware abstraction layer; control the hardware abstraction layer to transmit the several data units to the data processor according to a preset data transmission frequency.

[0109] In some embodiments, the device further includes a reverberation processing module, which is used to control the hardware abstraction layer to perform reverberation processing on the collected audio data to obtain the collected audio data after reverberation processing.

[0110] In some embodiments, the mixing processing module 33 is used to perform mixing processing on the collected audio data and the background audio data through the data processor; or control the hardware abstraction layer to first adjust the sampling rate and bit width of the collected audio data and the background audio data to a preset format, and then control the hardware abstraction layer to perform mixing processing on the collected audio data and the background audio data adjusted to the preset format; and obtain the target audio data containing the collected audio data and the background audio data.

[0111] In some embodiments, the mixing processing module 33 is used to control the data processor to perform sound effect processing on the background audio data and then send it to the mixer, and skip the sound effect post-processing on the collected audio data and directly send it to the mixer for mixing with the background audio data after the sound effect processing.

[0112] In some embodiments, the device also includes a volume adjustment module, which is used to control the data processor to adjust the volume of the collected audio data and the background audio data so that the volume values ​​of the collected audio data and the background audio data are coordinated; the volume adjustment module is used to respectively obtain multiple collected audio data sampling points and background audio data sampling points located in the same sampling time period; respectively calculate a first average value of multiple collected audio data sampling points and a second average value of multiple background audio data sampling points; compare the first average value with the second average value, and adjust the volume value of the collected audio data or the background audio data accordingly according to the comparison result.

[0113] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.

[0114] As can be seen from the above, the audio control device 30 provided in the embodiment of the present application, wherein the first acquisition module 31 is used to acquire background audio data output by the application layer and transmitted through the first transmission path; the second acquisition module 32 is used to acquire collected audio data received by the audio acquisition device and transmitted through the second transmission path, wherein the second transmission path does not pass through the application layer included in the first transmission path; the mixing processing module 22 is used to perform mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data, and obtain target audio data containing the collected audio data and the background audio data; the audio playback module 34 is used to transmit the target audio data to the audio playback device for playback. By transmitting the background audio data and the collected audio data respectively through different transmission paths, the logical layers that the collected audio data needs to pass through in the traditional transmission method are reduced, thereby reducing the data transmission delay of the collected audio data and the target audio data obtained after the mixing processing during the transmission process.

[0115] An embodiment of the present application further provides an audio control device, which can be integrated into an electronic device.

[0116] In specific implementation, the above modules can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same entity or several entities.

[0117] See also Figure 7 , Figure 7 Another structural diagram of an audio control device provided in an embodiment of the present application, the audio control device 30 includes a memory 120, one or more processors 180, and one or more applications, wherein the one or more applications are stored in the memory 120 and configured to be executed by the processor 180; the processor 180 may include a first acquisition module 31, a second acquisition module 32, a mixing processing module 33, and an audio playback module 34. For example, the structure and connection relationship of the above components may be as follows:

[0118] The memory 120 can be used to store applications and data. The applications stored in the memory 120 include executable codes. The applications can be composed of various functional modules. The processor 180 executes various functional applications and data processing by running the applications stored in the memory 120. In addition, the memory 120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 120 may also include a memory controller to provide the processor 180 with access to the memory 120.

[0119] The processor 180 is the control center of the device, and uses various interfaces and lines to connect various parts of the entire terminal, and executes various functions of the device and processes data by running or executing applications stored in the memory 120, and calling data stored in the memory 120, so as to monitor the device as a whole. Optionally, the processor 180 may include one or more processing cores; preferably, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs.

[0120] Specifically in this embodiment, the processor 180 will load the executable code corresponding to the process of one or more application programs into the memory 120 according to the following instructions, and the processor 180 will run the application programs stored in the memory 120 to implement various functions:

[0121] A first acquisition instruction, used to acquire background audio data output by the application layer and transmitted through a first transmission path;

[0122] A second acquisition instruction is used to acquire the collected audio data received by the audio acquisition device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path;

[0123] A mixing processing instruction, used for performing mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data, so as to obtain target audio data including the collected audio data and the background audio data;

[0124] The audio playback instruction is used to transmit the target audio data to the audio playback device for playback.

[0125] In some embodiments, the first transmission path includes a transmission path that sequentially connects the application layer, the hardware abstraction layer and the data processor, and the first acquisition instruction is used to control the application layer to output background audio data and transmit the background audio data to the hardware abstraction layer that provides a data transmission interface for the application layer and the data processor respectively; and control the hardware abstraction layer to transmit the background audio data to the data processor.

[0126] In some embodiments, the second transmission path includes a transmission path that sequentially connects the audio acquisition device, the hardware abstraction layer and the data processor, and the second acquisition instruction is used to control the audio acquisition device to receive collected audio data and transmit the collected audio data to the hardware abstraction layer, and divide the collected audio data into a number of data units according to a preset segmentation rule through the hardware abstraction layer; control the hardware abstraction layer to transmit the several data units to the data processor according to a preset data transmission frequency.

[0127] In some embodiments, the program further includes a reverberation processing instruction for controlling the hardware abstraction layer to perform reverberation processing on the collected audio data to obtain the collected audio data after reverberation processing.

[0128] In some embodiments, the mixing processing instruction is used to perform mixing processing on the collected audio data and the background audio data through the data processor; or to control the hardware abstraction layer to first adjust the sampling rate and bit width of the collected audio data and the background audio data to a preset format, and then control the hardware abstraction layer to perform mixing processing on the collected audio data and the background audio data adjusted to the preset format; and obtain the target audio data containing the collected audio data and the background audio data.

[0129] In some embodiments, the mixing processing instruction is used to control the data processor to perform sound effect processing on the background audio data and then send it to the mixer, and to skip the sound effect post-processing on the collected audio data and directly send it to the mixer for mixing with the background audio data after the sound effect processing.

[0130] In some embodiments, the program also includes a volume adjustment instruction for controlling the data processor to adjust the volume of the collected audio data and the background audio data so that the volume values ​​of the collected audio data and the background audio data are coordinated; the volume adjustment instruction is used to respectively obtain multiple collected audio data sampling points and background audio data sampling points located in the same sampling time period; respectively calculate a first average value of multiple collected audio data sampling points and a second average value of multiple background audio data sampling points; compare the first average value with the second average value, and adjust the volume value of the collected audio data or the background audio data accordingly according to the comparison result.

[0131] The present application also provides an electronic device. Figure 8 , Figure 8 The electronic device 1200 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, and the electronic device can be used to implement the audio control method provided in the above embodiment. The electronic device 1200 can be a television, a smart phone, or a tablet computer.

[0132] like Figure 8As shown, the electronic device 1200 may include an RF (Radio Frequency) circuit 110, a memory 120 including one or more (only one is shown in the figure) computer-readable storage media, an input unit 130, a display unit 140, a sensor 150, an audio circuit 160, a transmission module 170, a processor 180 including one or more (only one is shown in the figure) processing cores, and a power supply 190. Those skilled in the art will understand that Figure 8 The structure of the electronic device 1200 shown in the figure does not constitute a limitation on the electronic device 1200, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0133] The RF circuit 110 is used to receive and send electromagnetic waves, realize the mutual conversion between electromagnetic waves and electrical signals, and thus communicate with a communication network or other televisions. The RF circuit 110 may include various existing circuit elements for performing these functions, such as an antenna, a radio frequency transceiver, a digital signal processor, an encryption / decryption chip, a user identity module (SIM) card, a memory, etc. The RF circuit 110 can communicate with various networks such as the Internet, an intranet, a wireless network, or communicate with other televisions via a wireless network.

[0134] The memory 120 can be used to store software programs and modules, such as the program instructions / modules corresponding to the audio control method in the above embodiment. The processor 180 executes various functional applications and data processing by running the software programs and modules stored in the memory 120. The vibration reminder mode can be automatically selected according to the current scene of the electronic device to perform audio control, which can not only ensure that scenes such as meetings are not disturbed, but also ensure that users can perceive incoming calls, thereby improving the intelligence of the electronic device. The memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 120 may further include a memory remotely arranged relative to the processor 180, and these remote memories may be connected to the electronic device 1200 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0135] The input unit 130 can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control. Specifically, the input unit 130 may include a touch-sensitive surface 131 and other input devices 132. The touch-sensitive surface 131, also known as a touch display screen or touch pad, can collect user touch operations on or near it (such as operations performed by users using fingers, styluses, or any other suitable objects or accessories on or near the touch-sensitive surface 131), and drive corresponding connection devices according to a pre-set program. Optionally, the touch-sensitive surface 131 may include a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch position, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into touch point coordinates, and then sends it to the processor 180, and can receive and execute commands sent by the processor 180. In addition, the touch-sensitive surface 131 may be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch-sensitive surface 131, the input unit 130 may further include other input devices 132. Specifically, the other input devices 132 may include, but are not limited to, one or more of a physical keyboard, a function key (such as a volume control key, a switch key, etc.), a trackball, a mouse, a joystick, and the like.

[0136] The display unit 140 can be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device 1200, which can be composed of graphics, text, icons, videos and any combination thereof. The display unit 140 may include a display panel 141. Optionally, the display panel 141 may be configured in the form of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc. Further, the touch-sensitive surface 131 may cover the display panel 141. When the touch-sensitive surface 131 detects a touch operation on or near it, it is transmitted to the processor 180 to determine the type of touch event. Subsequently, the processor 180 provides corresponding visual output on the display panel 141 according to the type of touch event. Although in Figure 8 In the embodiment, the touch-sensitive surface 131 and the display panel 141 are implemented as two independent components to realize the input and output functions. However, in some embodiments, the touch-sensitive surface 131 and the display panel 141 can be integrated to realize the input and output functions.

[0137] The electronic device 1200 may also include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 141 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 141 and / or the backlight when the electronic device 1200 is moved to the ear. As a type of motion sensor, the gravity acceleration sensor can detect the magnitude of acceleration in each direction (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc. that can also be configured in the electronic device 1200, they will not be repeated here.

[0138] The audio circuit 160, the speaker 161, and the microphone 162 can provide an audio interface between the user and the electronic device 1200. The audio circuit 160 can transmit the received audio data to the speaker 161 after converting the received audio data into an electrical signal, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then the audio data is processed by the output processor 180, and then sent to another terminal through the RF circuit 110, or the audio data is output to the memory 120 for further processing. The audio circuit 160 may also include an earplug jack to provide communication between an external headset and the electronic device 1200.

[0139] The electronic device 1200 can help users send and receive emails, browse web pages, and access streaming media through the transmission module 170 (such as a Wi-Fi module), which provides users with wireless broadband Internet access. Figure 8 The transmission module 170 is shown, but it is understandable that it is not an essential component of the electronic device 1200 and can be omitted as needed without changing the essence of the invention.

[0140] The processor 180 is the control center of the electronic device 1200. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 120, and calling data stored in the memory 120, it executes various functions of the electronic device 1200 and processes data, thereby monitoring the mobile phone as a whole. Optionally, the processor 180 may include one or more processing cores; in some embodiments, the processor 180 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 180.

[0141] The electronic device 1200 also includes a power supply 190 for supplying power to various components. In some embodiments, the power supply can be logically connected to the processor 180 through a power management system, so that the power management system can realize functions such as managing discharge and power consumption. The power supply 190 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0142] Although not shown, the electronic device 1200 may also include a camera (such as a front camera, a rear camera), a Bluetooth module, etc., which will not be described in detail here. Specifically in this embodiment, the display unit 140 of the electronic device 1200 is a touch screen display, and the electronic device 1200 also includes a memory 120, and one or more programs, wherein one or more programs are stored in the memory 120, and are configured to be executed by one or more processors 180. One or more programs include instructions for performing the following operations:

[0143] A first acquisition instruction, used to acquire background audio data output by the application layer and transmitted through a first transmission path;

[0144] A second acquisition instruction is used to acquire the collected audio data received by the audio acquisition device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path;

[0145] A mixing processing instruction, used for performing mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data, so as to obtain target audio data including the collected audio data and the background audio data;

[0146] The audio playback instruction is used to transmit the target audio data to the audio playback device for playback.

[0147] In some embodiments, the first transmission path includes a transmission path that sequentially connects the application layer, the hardware abstraction layer and the data processor, and the first acquisition instruction is used to control the application layer to output background audio data and transmit the background audio data to the hardware abstraction layer that provides a data transmission interface for the application layer and the data processor respectively; and control the hardware abstraction layer to transmit the background audio data to the data processor.

[0148] In some embodiments, the second transmission path includes a transmission path that sequentially connects the audio acquisition device, the hardware abstraction layer and the data processor, and the second acquisition instruction is used to control the audio acquisition device to receive collected audio data and transmit the collected audio data to the hardware abstraction layer, and divide the collected audio data into a number of data units according to a preset segmentation rule through the hardware abstraction layer; control the hardware abstraction layer to transmit the several data units to the data processor according to a preset data transmission frequency.

[0149] In some embodiments, the program further includes a reverberation processing instruction for controlling the hardware abstraction layer to perform reverberation processing on the collected audio data to obtain the collected audio data after reverberation processing.

[0150] In some embodiments, the mixing processing instruction is used to perform mixing processing on the collected audio data and the background audio data through the data processor; or to control the hardware abstraction layer to first adjust the sampling rate and bit width of the collected audio data and the background audio data to a preset format, and then control the hardware abstraction layer to perform mixing processing on the collected audio data and the background audio data adjusted to the preset format; and obtain the target audio data containing the collected audio data and the background audio data.

[0151] In some embodiments, the mixing processing instruction is used to control the data processor to perform sound effect processing on the background audio data and then send it to the mixer, and to skip the sound effect post-processing on the collected audio data and directly send it to the mixer for mixing with the background audio data after the sound effect processing.

[0152] In some embodiments, the program also includes a volume adjustment instruction for controlling the data processor to adjust the volume of the collected audio data and the background audio data so that the volume values ​​of the collected audio data and the background audio data are coordinated; the volume adjustment instruction is used to respectively obtain multiple collected audio data sampling points and background audio data sampling points located in the same sampling time period; respectively calculate a first average value of multiple collected audio data sampling points and a second average value of multiple background audio data sampling points; compare the first average value with the second average value, and adjust the volume value of the collected audio data or the background audio data accordingly according to the comparison result.

[0153] The embodiment of the present application also provides an electronic device, which may be a smart phone, a television, a computer, or other device.

[0154] As can be seen from the above, the embodiment of the present application provides an electronic device 1200, and the electronic device 1200 performs the following steps:

[0155] Acquire background audio data output by the application layer and transmitted through the first transmission path;

[0156] Acquire collected audio data received by the audio collection device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path;

[0157] According to the collected audio data and the background audio data, mixing processing is performed on the collected audio data and the background audio data to obtain target audio data including the collected audio data and the background audio data;

[0158] The target audio data is transmitted to an audio playback device for playback.

[0159] An embodiment of the present application further provides a storage medium, wherein a computer program is stored in the storage medium. When the computer program is executed on a computer, the computer executes the audio control method described in any one of the above embodiments.

[0160] It should be noted that, for the audio control method described in the present application, ordinary testers in the field can understand that all or part of the process of implementing the audio control method described in the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, such as stored in the memory of an electronic device, and executed by at least one processor in the electronic device, and the execution process may include the process of the embodiment of the audio control method. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM, ReadOnly Memory), a random access memory (RAM, Random Access Memory), etc.

[0161] For the audio control device of the embodiment of the present application, each functional module can be integrated into a processing chip, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk.

[0162] The audio control method, device, medium and equipment provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application; at the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An audio control method, characterized in that: include: Acquire background audio data output by the application layer and transmitted through the first transmission path; Acquire collected audio data received by the audio collection device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path; Performing mixing processing on the collected audio data and the background audio data to obtain target audio data including the collected audio data and the background audio data; The target audio data is transmitted to an audio playback device for playback.

2. The audio control method according to claim 1, characterized in that: The first transmission path includes a transmission path that sequentially connects the application layer, the hardware abstraction layer, and the data processor, and the obtaining of background audio data output by the application layer and transmitted through the first transmission path includes: Controlling the application layer to output background audio data, and transmitting the background audio data to a hardware abstraction layer that provides a data transmission interface for the application layer and the data processor respectively; The hardware abstraction layer is controlled to transmit the background audio data to the data processor.

3. The audio control method according to claim 2, characterized in that: The second transmission path includes a transmission path sequentially connecting the audio acquisition device, the hardware abstraction layer and the data processor, and the acquisition of the collected audio data received by the audio acquisition device and transmitted through the second transmission path includes: Controlling the audio acquisition device to receive the collected audio data, and transmitting the collected audio data to the hardware abstraction layer, and dividing the collected audio data into a plurality of data units according to a preset division rule through the hardware abstraction layer; The hardware abstraction layer is controlled to transmit the plurality of data units to the data processor according to a preset data transmission frequency.

4. The audio control method according to claim 3, characterized in that: Before controlling the hardware abstraction layer to transmit the plurality of data units to the data processor according to a preset data transmission frequency, the method further includes: The hardware abstraction layer is controlled to perform reverberation processing on the collected audio data to obtain the collected audio data after the reverberation processing.

5. The audio control method according to claim 3, characterized in that: The mixing process of the collected audio data and the background audio data to obtain the target audio data including the collected audio data and the background audio data includes: Performing mixing processing on the collected audio data and the background audio data through the data processor; or Controlling the hardware abstraction layer to first adjust the sampling rate and bit width of the collected audio data and the background audio data into a preset format, and then controlling the hardware abstraction layer to perform mixing processing on the collected audio data and the background audio data adjusted into the preset format; Target audio data including the collected audio data and background audio data is obtained.

6. The audio control method according to claim 5, characterized in that: The mixing of the collected audio data and the background audio data by the data processor includes: The data processor is controlled to perform sound effect processing on the background audio data and then send it to the mixer, and the collected audio data skips the sound effect post-processing and is directly sent to the mixer to mix with the background audio data after the sound effect processing.

7. The audio control method according to claim 1, characterized in that: Before obtaining the target audio data including the collected audio data and the background audio data, the method further includes: Controlling the data processor to adjust the volume of the collected audio data and the background audio data so that the volume values ​​of the collected audio data and the background audio data are coordinated; The controlling the data processor to adjust the volume of the collected audio data and the background audio data includes: Respectively acquiring a plurality of the collected audio data sampling points and the background audio data sampling points located in the same sampling time period; Respectively calculating a first average value of a plurality of the collected audio data sampling points and a second average value of a plurality of the background audio data sampling points; The first average value is compared with the second average value, and the volume value of the collected audio data or the background audio data is adjusted accordingly according to the comparison result.

8. An audio control device, characterized in that: The audio control device comprises: A first acquisition module, used for acquiring background audio data output by the application layer and transmitted through a first transmission path; A second acquisition module, used to acquire collected audio data received by the audio acquisition device and transmitted through a second transmission path, wherein the second transmission path does not pass through an application layer included in the first transmission path; A mixing processing module, used for performing mixing processing on the collected audio data and the background audio data according to the collected audio data and the background audio data, so as to obtain target audio data including the collected audio data and the background audio data; The audio playing module is used to transmit the target audio data to the audio playing device for playing.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the audio control method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The system comprises a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the audio control method according to any one of claims 1 to 7.

Images