Audio data processing method and device

Abstract

The invention provides an audio data processing method and device, the device is connected with an audio device, and the method comprises the following steps: obtaining original audio data; wherein the original audio data belongs to a three-dimensional cyanine color tone; determining the format of the original audio data according to at least one of an audio output mode of the audio equipment or a connection mode of the equipment and the audio equipment; wherein the audio output mode is one of a source code output mode and a non-source code output mode, and the connection mode is one of audio return channel (ARC) connection or enhanced audio return channel (eARC) connection; sending the original audio data to the audio equipment in a determined format; wherein the audio equipment is used for processing and / or playing the original audio data. According to the method, the effect of processing and / or playing the three-dimensional cyanine color sound by the equipment through the external equipment can be improved, so that the playing effect of the three-dimensional cyanine color sound is improved.

Description

Technical Field

[0001] The present application relates to the field of device technology, and in particular to an audio data processing method and device. Background Art

[0002] An audio device can be used as an external extended sound output device for a device (such as a smart screen). Usually, in addition to connecting the device, the external audio device can also be connected to at least one audio playback device (such as a speaker) for playing audio. The audio device can process (such as amplifying audio signals) the audio data from the device and drive the at least one connected audio playback device to produce sound based on the processed audio data.

[0003] Currently, there are almost no audio devices that can support decoding of audio data belonging to 3D vivid sound. Therefore, many devices use audio devices as extended sound output devices and play 3D vivid sound without sound. Currently, some devices can convert the audio data of 3D vivid sound into data of set channels and then send it to the audio device for processing. Although sound output can be achieved, the playback effect of this method is fixed, so the immersive auditory effect is poor. Summary of the invention

[0004] The present application provides an audio data processing method and device for improving the playback effect of three-dimensional colorful sound played through an external device.

[0005] In a first aspect, the present application provides an audio data processing method, which is applied to a device, and the device is connected to an audio device, the method comprising: obtaining original audio data; wherein the original audio data belongs to three-dimensional colorful sound; determining the format of the original audio data according to at least one of the audio output mode of the audio device or the connection mode between the device and the audio device; wherein the audio output mode is one of a source code output mode and a non-source code output mode, and the connection mode is one of an audio return channel ARC connection or an enhanced audio return channel eARC connection; sending the original audio data to the audio device in a determined format; wherein the audio device is used to process and / or play the original audio data.

[0006] In this method, the device can flexibly and adaptively determine the more appropriate format for outputting audio data according to at least one of the audio output mode of the audio device or the connection method between the device and the audio device, and can output the audio data to the audio device in the corresponding format, and then the audio device can perform subsequent audio data processing. Based on this method, the device can output audio data in a more optimal format to the audio device as much as possible, thereby improving the efficiency of the audio device's subsequent audio playback control based on the audio data and the effect of audio playback. In various scenarios such as different types of audio devices connected to the device and different connection channels between the device and the external audio device, the above method can achieve the effect of processing audio data according to the scene, thereby improving the effect of subsequent audio playback based on audio data, so that users can listen to audio with a higher degree of immersion if the audio device capabilities support it, thereby improving the user's auditory experience.

[0007] In one possible design, the audio device is an audio amplifier device or a sound device.

[0008] In this method, the audio device can send audio data to an external audio playback device for playback, or the audio device itself can play audio according to the audio data. When the audio device processes or plays audio data from the device, the format of the audio data is a format supported by the audio device capabilities determined by the device. Therefore, the audio device can efficiently process or play the audio data, thereby improving the playback effect when playing audio according to the audio data.

[0009] In one possible design, determining the format of the original audio data according to at least one of the audio output mode of the audio device or the connection mode between the device and the audio device includes: when the audio output mode is the source code output mode, determining that the format of the original audio data is the source code format; when the audio output mode is the non-source code output mode and the connection mode is the ARC connection, determining that the format of the original audio data is a set channel format; when the audio output mode is the non-source code output mode and the connection mode is the eARC connection, determining that the format of the original audio data is a multi-channel format supported by the audio device.

[0010] In this method, when the output mode of the audio device is the source code output mode, the device can send the original audio data in the source code format to the audio device for subsequent processing, which can ensure that the audio is processed in a mode that meets the requirements of the audio device. When the connection mode between the device and the audio device is ARC connection, the transmission capacity of the connection channel between the device and the audio device is limited. When the connection mode between the device and the audio device is eARC connection, the transmission capacity of the connection channel between the device and the audio device is higher. Therefore, when the output mode of the audio device is not the source code output mode, the device can adaptively determine the format of the audio data supported by the connection channel between the device and the audio device according to the connection mode between the device and the audio device, which can ensure the successful transmission of audio data in a better format. In the above method, the format of the original audio data can be adaptively determined according to the actual scenario, and the flexibility and practicality are high.

[0011] In one possible design, the multi-channel format meets at least one of the following conditions: the number of channels of the multi-channel is less than or equal to the number of channels supported by the audio device; the channels contained in the multi-channel belong to the channels supported by the audio device; the azimuth information corresponding to the same channels in the multi-channel and the channels supported by the audio device is the same; wherein each azimuth information is used to indicate the azimuth of the audio of the corresponding channel when it is played; and the audio format of the audio in the multi-channel format belongs to the audio format supported by the audio device.

[0012] In this method, the audio format of the original audio data belongs to the format supported by the audio device, which can ensure that the audio device can process the original audio data, thereby ensuring the smooth execution of subsequent audio playback and obtaining a better audio playback effect.

[0013] In one possible design, before sending the original audio data to the audio device in a determined format, the method also includes: obtaining an audio capability descriptor of the audio device and / or a speaker allocation data block of the audio device; determining capability parameters of the audio device based on the audio capability descriptor of the audio device and / or the speaker allocation data block of the audio device; wherein the capability parameters include at least one of the following: the number of channels supported by the audio device, the channels supported by the audio device, the orientation information corresponding to the channels supported by the audio device, and the audio format supported by the audio device.

[0014] In this method, the device can more accurately determine the capability of the audio device based on the audio capability descriptor of the audio device and / or the speaker allocation data block of the audio device, so as to determine the format of the audio transmitted to the audio device according to the capability of the audio device.

[0015] In a possible design, the device includes an application, an audio hardware abstraction layer HAL unit and an audio sending unit; wherein the audio HAL unit is in the hardware abstraction layer of the device; the obtaining of original audio data includes: obtaining, through the audio HAL unit, the original audio data transmitted by the application through the application framework layer of the device; the sending of the original audio data to the audio device in a determined format includes: converting the original audio data into the determined format through the audio HAL unit; and sending the original audio data converted into the determined format to the audio device through the audio sending unit.

[0016] In this method, the application framework layer in the device can transparently transmit audio data, and the hardware abstraction layer in the device integrates audio data processing functions, which can perform processing such as format conversion of audio data with high efficiency. Based on this method, through the audio processing link composed of the application in the device, the application framework layer, the audio HAL unit in the hardware abstraction layer, and the audio sending unit, audio data can be processed and transmitted efficiently.

[0017] In one possible design, the device is connected to the audio device via a high-definition multimedia interface cable.

[0018] In a second aspect, the present application provides a device comprising a memory and one or more processors; wherein the memory is used to store program code, and the program code comprises instructions; when the instructions are executed by one or more processors, the device executes the method described in the above-mentioned first aspect or any possible design of the first aspect.

[0019] In a third aspect, the present application provides a readable storage medium storing a program, which, when executed on a device, enables the device to execute the method described in the first aspect or any possible design of the first aspect.

[0020] In a fourth aspect, the present application provides a program product, which includes a program or instructions. When the program or instructions are executed on a device, the device executes the method described in the first aspect or any possible design of the first aspect.

[0021] For the beneficial effects of the second to fourth aspects, please refer to the description of the beneficial effects of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 A schematic diagram of the hardware architecture of a device provided in an embodiment of the present application;

[0023] Figure 2 A schematic diagram of an application scenario provided in an embodiment of the present application;

[0024] Figure 3 A schematic diagram of the architecture of an audio processing link of a device provided in an embodiment of the present application;

[0025] Figure 4 A schematic diagram of an audio data processing method provided in an embodiment of the present application;

[0026] Figure 5 A schematic diagram of an audio data processing method provided in an embodiment of the present application;

[0027] Figure 6 A schematic diagram of the structure of a device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0029] In the description of the embodiments of the present application, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0030] To facilitate understanding, exemplary descriptions of concepts related to the present application are provided for reference.

[0031] 1) Device, which can be a device with communication function and audio processing function.

[0032] In some embodiments of the present application, the device includes a portable device, such as a mobile phone, a tablet computer, a wearable device with a wireless communication function (for example, a watch, a bracelet, a helmet, a headset, etc.), a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), smart home devices (for example, smart TVs, smart speakers, etc.), smart robots, workshop equipment, wireless devices in self-driving, wireless devices in remote medical surgery, wireless devices in smart grids, wireless devices in transportation safety, wireless devices in smart cities, or wireless devices in smart homes, flying devices (for example, smart robots, hot air balloons, drones, airplanes), etc.

[0033] In some embodiments of the present application, the device also includes a portable device with other functions such as a personal digital assistant and / or a music player. Exemplary embodiments of portable devices include but are not limited to: Or a portable device with other operating systems. The portable device also includes a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments of the present application, the device also includes a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0034] 2) Audio device: The audio device described in the embodiment of the present application is a device for amplifying audio signals. The audio device described in the embodiment of the present application can be used as an extended sound output device external to a signal source (such as a device, etc.), and can be used to amplify the audio electrical signal from the signal source to drive the audio playback device (such as a speaker, etc.) to emit sound (i.e., play audio), or can be used to amplify the audio electrical signal from the signal source and play audio.

[0035] In some embodiments of the present application, the audio device may be an audio power amplifier device (referred to as amplifier) ​​or a sound device.

[0036] It should be understood that in the embodiments of the present application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of the associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0037] Compared with traditional sound, three-dimensional sound (or three-dimensional audio, etc.) increases the sense of space and direction, and can provide an immersive listening experience to meet users' auditory needs for highly restored and highly immersive sound. Currently, a three-dimensional sound with a good immersive listening experience and a wide range of applications is Audio Vivid (also known as three-dimensional Audio Vivid). Audio Vivid is an audio that is encoded and decoded based on artificial intelligence (AI) technology. Audio Vivid supports mainstream three-dimensional sound encoding and is compatible with mono, stereo, and surround sound. It allows sound to be accurately placed and moved at any position in three-dimensional space, providing an ultimate immersive listening experience.

[0038] Devices can generally use external audio devices as extended sound output devices, but there are currently almost no audio devices that can support the playback of Jingcai sound. Therefore, when the device is connected to an audio device and plays Jingcai sound, if the audio source code is directly transmitted to the audio device for processing, there will be a problem of silence, that is, the sound cannot be played normally. Currently, some devices can convert the audio source code into data of the set channel and then transmit it to the audio device for processing. This method is relatively rigid, the playback effect is single, and some channel data will be lost during the conversion process, so the actual playback effect is poor.

[0039] Based on the above problems, in order to improve the playback effect of Jingcaisheng and improve the user's auditory experience, the embodiment of the present application provides an audio data processing method and device. In this solution, the device can combine the actual scene and flexibly and adaptively output the audio data in a better format supported by the audio device to the audio device, thereby improving the playback effect of the audio playback based on the audio data and improving the user's auditory experience. This solution can achieve a relatively ideal audio playback effect in different scenarios (for example, using audio devices manufactured by different manufacturers, and using different connection methods between the device and the audio device, etc.).

[0040] See below Figure 1 , the structure of the device to which the method provided in the embodiment of the present application is applicable is introduced.

[0041] like Figure 1As shown, the device 100 may include a processor 110, a memory 120, a high definition multimedia interface (HDMI) 130, a wired communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a sensor module 180, a button 190, an indicator 192, a display screen 194, etc.

[0042] Understandably, Figure 1 The device 100 shown is merely an example and does not constitute a limitation of the device, and the device may have more or fewer components than shown in the figure, may combine two or more components, or may have a different configuration of components. Figure 1 The various components shown in the EMBODIMENTS 2000 may be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application specific integrated circuits.

[0043] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the device 100. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0044] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0045] The execution of the audio data processing method provided in the embodiment of the present application can be controlled by the processor 110 or completed by calling other components. The processor 110 may include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute the audio data processing method provided in the embodiment of the present application. For example, part of the algorithm in the audio data processing method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain faster processing efficiency.

[0046] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. In some embodiments, the device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. The display screen 194 may be used to display information input by a user or information provided to a user and various graphical user interfaces (GUIs).

[0047] In the embodiment of the present application, the display screen 194 may be an integrated flexible display screen, or a spliced ​​display screen consisting of two rigid screens and a flexible screen located between the two rigid screens.

[0048] The memory 120 can be used to store program codes, which include instructions. The processor 110 executes various functional applications and data processing of the device 100 by running the instructions stored in the memory 120. Optionally, the memory 120 may include a program storage area and a data storage area. Among them, the program storage area can store codes of an operating system, an application program (such as functions corresponding to the audio data processing method, etc.). The data storage area can store data created during the use of the device 100, etc.

[0049] The memory 120 may also store one or more programs corresponding to the algorithm of the audio data processing method provided in the embodiment of the present application. The one or more programs are stored in the above-mentioned memory 120 and are configured to be executed by one or more processors 110, and the one or more programs include instructions, which can be used to execute the various steps in the following embodiments.

[0050] Optionally, the memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, etc.

[0051] The communication function of the device 100 can be implemented through the wired communication module 150, the wireless communication module 160, etc.

[0052] The wired communication module 150 can realize wired communication with other devices through physical transmission media such as cables. The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) network), Bluetooth (BT), etc. applied to the device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via an antenna, modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency, amplify it, and convert it into electromagnetic waves for radiation through the antenna.

[0053] In addition, the device 100 can implement audio functions through the audio module 170, the speaker 170A and the application processor. The device 100 can receive input from the key 190 and generate key signal input related to user settings and function control of the device 100. The indicator 192 in the device 100 can be an indicator light, which can be used to indicate the power on / off state, etc.

[0054] It should be understood that in actual applications, the device 100 may include Figure 1 The embodiments of the present application are not limited to the more or fewer components shown. The illustrated device 100 is only an example, and the device 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0055] The solution provided in the embodiment of the present application can be applied to a device that supports playing a Jingcai sound resource (such as an audio resource or a video resource, etc.). The solution provided in the embodiment of the present application can be applied to a scenario where a device is connected to an external audio device and plays a Jingcai sound resource.

[0056] Figure 2 The schematic diagram of an application scenario is shown as an example. Figure 2As shown in , in a possible scenario, the audio device can be an amplifier, and the application scenario can be a scenario in which a smart screen is connected to an external amplifier through an HDMI cable to play the sound of Jingcai. Optionally, the amplifier can be connected to at least one audio playback device (such as a speaker, etc.). Based on the solution provided in the embodiment of the present application, the device (i.e., the smart screen) can determine the optimal output mode of the audio output data supported by the amplifier according to the connection mode with the amplifier, the amplifier capability and other scene information, and perform audio processing on the audio data of Jingcai to be played according to the mode to obtain audio output data. The device can transfer the audio output data to the amplifier via an HDMI cable, and the amplifier can transfer the audio output data to the corresponding audio playback device for playback. The specific implementation method of the above process can refer to the relevant introduction in the following embodiments, which will not be described in detail here.

[0057] It should be noted that the above Figure 2 The scenario shown is an example and does not limit the application scenarios to which the solutions provided by the embodiments of the present application are applicable. The application will not list the application scenarios to which the solutions provided by the embodiments of the present application are applicable one by one.

[0058] Figure 3 The following is a schematic diagram of an audio processing link of a device. Figure 3 As shown in , the system architecture of a device may include an application layer, an application framework layer (Framework, FWK), a hardware abstract layer (hardwareabstract layer, HAL), etc. Figure 2 The audio processing link in the smart screen (shown in the figure) can be composed of an application in the application layer, an audio processing unit in the application framework layer, an audio HAL unit in the HAL layer, and an audio sending unit.

[0059] Among them, the application layer is the top layer of the operating system, including the native applications of the operating system, and may also include other applications. The application involved in the embodiment of the present application is referred to as application (APP), which is a software program that can realize one or more specific functions. Usually, multiple applications can be installed in a device, such as camera applications, information applications, etc. The applications mentioned below can be system applications installed when the device leaves the factory, or they can be third-party applications downloaded from the Internet or obtained from other devices by the user during the use of the device.

[0060] In some embodiments of the present application, the application in the application layer may be an application that supports Jingcai sound (e.g., a video application, a music application, etc.). The application may be used to transfer the original audio data to the hardware abstract layer (HAL) through the application framework layer for audio processing. The original audio data (also referred to as the original audio stream or the original audio data stream in the embodiments of the present application) is the encoded audio data.

[0061] The application framework layer is the API and programming framework of the application layer. The application framework layer may include some predefined functions and corresponding functional units.

[0062] In some embodiments of the present application, the audio processing unit in the application framework layer may include an audio framework (audio flinger) and an audio track (audio track). Wherein, the audio framework may be used to be responsible for the control and management related to audio services, such as the collection and output control of audio data, the transmission and control of audio data, the management of audio equipment, etc. The audio track is an instance of an API class of an audio framework, which may be used to transmit audio data. The audio data may be audio stream data, and each audio track may correspond to an audio stream. In the scheme provided in the present application embodiment, the audio track and the audio framework are used to transmit audio data. The audio framework may transmit audio data to the HAL layer by a direct output thread (direct thread) (i.e., a thread that does not perform any mixing process and directly outputs the received audio).

[0063] In some embodiments of the present application, the audio HAL unit in the HAL layer is used to perform audio processing on the audio data transmitted by the application framework layer. Specifically, the audio HAL unit can be used to decode and render audio data, obtain audio output data corresponding to the optimal audio output mode supported by the audio device, and send the obtained audio output data to the audio sending unit. Optionally, the audio HAL unit may include a decoding unit and a rendering unit, and the decoding unit may be used to decode the received audio data and send the decoded data to the rendering unit. The rendering unit may be used to convert the data from the decoding unit into audio output data of the optimal audio data output format (which can be determined according to the optimal audio output mode supported by the audio device) supported by the audio device. In some embodiments of the present application, the audio output data may be pulse code modulation (PCM) data.

[0064] The audio sending unit can be used to send audio output data from the HAL layer to the audio device connected to the device.

[0065] Figure 3The solid arrows shown in the figure can be used to indicate the flow direction of the encoded audio data (ie, original audio data (elementary streams, ES)), and the dotted arrows can be used to indicate the flow direction of the audio output data (ie, PCM data).

[0066] In some embodiments of the present application, the system architecture of the device may also include Figure 3 Layers or functional services not shown, etc. For example, the system architecture of the device may also include a runtime and a system library, a kernel layer, etc. Among them, the runtime can be responsible for the scheduling and management of the system. The system library consists of two parts: one part is the functional functions that need to be called by the Java language, and the other part is the core library of the system. The kernel layer provides core system services of the operating system, such as security, memory management, process management, network protocol stack and driver model, which are all implemented based on the kernel layer. The kernel layer also serves as an abstract layer between the hardware and software stacks. This layer has many device-related drivers, such as display drivers, audio drivers, etc.

[0067] It should be understood that the hierarchical architecture, audio processing links, functional services, etc. described above are only examples. In actual applications, the architecture may also include more or fewer functional modules (or services), or the functions of each module may be divided in other ways, or the functional modules may not be divided, but work as a whole.

[0068] The solution provided in this application is described in detail below in conjunction with specific embodiments.

[0069] See also Figure 4 , an audio data processing method provided by an embodiment of the present application may include:

[0070] S401: The device obtains original audio data, wherein the original audio data belongs to Jingcaiyin.

[0071] In some embodiments of the present application, the device can obtain original audio data belonging to the jingcai sound (also referred to as three-dimensional jingcai sound) in response to a received audio playback operation, wherein the audio playback operation can be used to indicate the playback of a target audio, and the target audio belongs to the jingcai sound. The original audio data obtained by the device is the original audio data of the target audio. Among them, the original audio data is encoded audio data. Optionally, the original audio data can be encoded audio stream data. Optionally, the original audio data can be audio data pre-stored by the device, or the original audio data can be audio data received by the device from other devices, or the original audio data can be audio data downloaded by the device from a server in real time.

[0072] In some embodiments of the present application, the audio playback operation may be an operation performed on an application interface of an application supporting Jingcaiyin in a device. In response to the operation, the application may obtain the original audio data according to the above method, and send the original audio data to a functional unit for processing audio data for subsequent processing.

[0073] Optionally, step S401 may be performed by Figure 3 The application and the audio HAL unit in the audio processing link shown are completed in collaboration. The application can be used to obtain the original audio data according to the above method, and transmit the original audio data to the audio HAL unit through the application framework layer; the audio HAL unit can obtain the original audio data by receiving the original audio data transmitted by the application through the application layer.

[0074] S402: The device determines whether the audio output mode of the current audio device is the source code mode. If so, execute step S403; otherwise, execute step S404.

[0075] In some embodiments of the present application, the audio device may support multiple audio output modes, such as source code output mode (i.e., a mode for directly outputting original audio data), PCM output mode (i.e., a mode for outputting PCM data), etc. The user may control the audio output mode adopted by the audio device, and the audio device may work according to the audio output mode selected by the user.

[0076] In some embodiments of the present application, when the device determines that the audio output mode of the audio device is the source code mode, the device can send the original audio data directly to the audio device according to the method described in step S403 below, so that the audio device performs audio processing and playback according to the source code mode.

[0077] In some embodiments of the present application, when the device determines that the audio output mode of the audio device is not the source code mode, the device can perform certain audio processing on the original audio data before sending it to the audio device, thereby adaptively determining and outputting the audio data in a suitable format to the audio device, thereby improving the effect of the subsequent audio device performing audio playback according to the audio data. The specific implementation of this method can refer to the description of the following steps S404 to S412.

[0078] Optionally, step S402 may be performed by Figure 3 The audio processing chain shown is performed in the Audio HAL unit.

[0079] S403: The device sends the original audio data to the audio device.

[0080] Optionally, step S403 may be performed by Figure 3 The audio processing chain shown is performed by the audio sending unit.

[0081] S404: The device determines whether the connection channel between the device and the audio device is ARC; if so, execute step S405; otherwise, execute step S408.

[0082] In the embodiment of the present application, the device and the audio device may be connected via an HDMI cable. The connection channel provided by the HDMI cable may be an audio return channel (ARC) or an enhanced audio return channel (eARC).

[0083] Among them, ARC allows devices to transmit audio to audio devices through HDMI cables without using dedicated audio cables. ARC can achieve two-way audio transmission between devices through a single HDMI cable, which can simplify the audio system architecture and save layout costs. eARC is an improved ARC that can transmit more complex multi-channel format audio in both directions. eARC's audio transmission capability, supported audio formats, number of channels and other aspects are better than ARC, and eARC is backward compatible with ARC.

[0084] When the connection channel provided by the HDMI cable is ARC, affected by the ARC capability limitation, the device can convert the original audio data into data of the set channel and output it to the audio device. Optionally, the set channel belongs to the channel supported by ARC. The specific implementation process of this method can refer to the description of the following steps S405 to S407.

[0085] When the connection channel provided by the HDMI cable is eARC, since eARC supports the transmission of all current audio formats, the device can perform audio processing according to the audio device capability, thereby converting the original audio data into a multi-channel data stream supported by the audio device capability and outputting it to the audio device. The specific implementation process of this method can refer to the description of the following steps S408 to S412.

[0086] Optionally, step S404 may be performed by Figure 3 The audio processing chain shown is performed in the Audio HAL unit.

[0087] S405: The device decodes the original audio data to obtain decoded audio data.

[0088] In some embodiments of the present application, the device can decode the original audio data through an integrated decoding library. The decoded audio data may include audio data and metadata, wherein the metadata includes information for describing the audio data (or attributes of the audio data).

[0089] Optionally, step S405 may be performed by Figure 3The audio processing chain shown is performed by the decoding unit in the audio HAL unit.

[0090] S406: The device converts the decoded audio data into PCM data of a set channel by rendering the decoded audio data.

[0091] In some embodiments of the present application, the set channel may include one or more pre-set (or configured) channels. For example, the set channel may be a main channel or left and right channels (ie, a left channel and a right channel).

[0092] Optionally, step S406 may be performed by Figure 3 The audio processing chain shown is performed by the rendering unit in the audio HAL unit.

[0093] S407: The device sends the obtained PCM data of the set channel to the audio device.

[0094] Optionally, step S407 may be performed by Figure 3 The audio processing chain shown is performed by the audio sending unit.

[0095] S408: When the device determines that the connection channel between the device and the audio device is eARC, the device obtains the SAD and SADB of the audio device.

[0096] Among them, the audio capability descriptor (short audio descriptor, SAD) includes information such as the channels (or number of channels) and audio formats supported by the audio device. Exemplarily, the audio format may include PCM, Dolby, initial audio chunk (initial audio chunk, IAC), etc. The speaker allocation data block (speaker allocation data block, SADB) includes the azimuth information corresponding to each channel supported by the audio device, and the azimuth information corresponding to each channel is used to indicate the position of the audio output device corresponding to the channel, and the audio output device corresponding to each channel is used to play the audio data of the channel. Exemplarily, the azimuth information in SADB may include the front right channel, the front left channel, the surround right channel, the surround left channel, etc. Based on the SAD and SADB of the audio device, the device can determine the maximum number of channels, channels, audio formats, and azimuth information corresponding to each channel supported by the audio device, and then obtain the maximum support capability information of the audio device.

[0097] In some embodiments of the present application, the device may obtain the SAD and SADB of the audio device from the audio device after establishing a connection with the audio device. Alternatively, the device may establish a connection with the audio device in advance, and obtain and store the SAD and SADB of the audio device from the audio device. During the audio processing process, the device may obtain the pre-stored SAD and SADB of the audio device when necessary (for example, when determining that the connection channel between the audio device and the audio device is eARC).

[0098] Optionally, step S408 may be performed by Figure 3 The audio processing chain shown is performed in the Audio HAL unit.

[0099] S409: The device determines the maximum support capability information of the audio device according to the SAD and SADB of the audio device, where the maximum support capability information includes at least one of the following: the maximum number of channels supported by the audio device, the orientation information of the channels supported by the audio device, and the audio format supported by the audio device.

[0100] In some embodiments of the present application, the device can determine the maximum number of channels and audio formats supported by the audio device based on SAD, and the device can determine the orientation information of each channel supported by the audio device based on SADB, and then obtain the maximum support capability information of the audio device, that is, the conditions that the audio data that the audio device can support processing to the greatest extent.

[0101] Optionally, step S409 may be performed by Figure 3 The audio processing chain shown is performed in the Audio HAL unit.

[0102] S410: The device decodes the original audio data to obtain decoded audio data.

[0103] Optionally, step S410 may be performed by Figure 3 The audio processing chain shown is performed by the decoding unit in the audio HAL unit.

[0104] S411: The device performs rendering processing on the decoded audio data to convert the decoded audio data into multi-channel PCM data that complies with the maximum support capability information of the audio device.

[0105] Among them, the multi-channel PCM data that complies with the maximum supported capability information of the audio device specifically refers to multi-channel PCM data that meets at least one of the following conditions: the number of channels is less than or equal to the maximum number of channels supported by the audio device, the orientation information of each channel is the orientation information of the channel supported by the audio device, and the audio format is an audio format supported by the audio device.

[0106] Optionally, step S411 may be performed by Figure 3The audio processing chain shown is performed by the rendering unit in the audio HAL unit.

[0107] S412: The device sends the obtained multi-channel PCM data to the audio device.

[0108] Optionally, step S412 may be performed by Figure 3 The audio processing chain shown is performed by the audio sending unit.

[0109] In some embodiments of the present application, after the device sends audio data (which may be the above-mentioned original audio data or PCM data of set channels or multi-channel PCM data) to the audio device, the audio device may play the audio according to the received audio data, or the audio device may output the audio data to at least one audio output device connected to the audio device according to the format and other characteristics of the audio data. The audio output device may play the received audio.

[0110] In the above method, the device can combine the scene information such as the output mode of the audio device, the connection channel type (ARC or eARC) between the device and the audio device, and the maximum supported capability information of the audio device to flexibly and adaptively determine the audio data format that is more suitable for output to the audio device in the usage scenario, and convert the original audio data into audio data of the corresponding format and output it to the audio device, and the audio device performs subsequent audio playback processing. The audio data in the optimal audio data format can be output to the audio device as much as possible, thereby improving the playback effect of the audio device when playing audio. In various scenarios such as different types of audio devices connected to the device and different connection channels between the device and the external audio device, the above method can achieve the effect of controlling audio playback according to the scene, especially the audio device capabilities, so that users can listen to audio with a higher degree of immersion if the audio device capabilities support it, thereby improving the user's auditory experience.

[0111] It should be noted that the specific implementation process provided in the above embodiment is only an example of the method process applicable to the embodiment of the present application. The execution order of each step can be adjusted accordingly according to actual needs, and other steps can be added or some steps can be reduced.

[0112] Based on the above embodiments and the same technical concept, the present application embodiment also provides an audio data processing method, such as Figure 5 As shown in , the method may include:

[0113] S501: The device obtains original audio data; wherein the original audio data belongs to three-dimensional colorful sound.

[0114] In some embodiments of the present application, the device may be a device capable of processing three-dimensional colorful sound, and the audio device may be an amplifier device or an audio device, etc.

[0115] In some embodiments of the present application, the device and the audio device may be connected via a high-definition multimedia interface cable, wherein the audio transmission channel provided by the high-definition multimedia interface cable may be ARC or eARC.

[0116] Illustratively, the device may be the device provided in the aforementioned embodiment, and the audio device may be the audio device provided in the aforementioned embodiment.

[0117] In some embodiments of the present application, the device can obtain the original audio data and perform subsequent processing in response to the received instruction to play the original audio data. For example, when the method provided in the embodiment of the present application is applied to the scenario described in the above embodiment where the smart screen is connected to an external amplifier through a high-definition multimedia interface line to play the 3D ...

[0118] Optionally, when the audio device is a sound device and the sound device integrates the function of an amplifier, the processing method of the sound device can refer to the processing method of the audio device described in the aforementioned embodiment, which will not be repeated here. Optionally, when the sound device integrates the function of an amplifier and the audio playback device described in the aforementioned embodiment, after the sound device receives the audio from the device, it can process the received audio and play it according to the method provided in the embodiment of the present application.

[0119] S502: The device determines the format of the original audio data according to at least one of an audio output mode of the audio device or a connection method between the device and the audio device; wherein the audio output mode is one of a source code output mode and a non-source code output mode, and the connection method is one of an audio return channel ARC connection or an enhanced audio return channel eARC connection.

[0120] Optionally, the ARC connection or the eARC connection may be provided via a high-definition multimedia interface cable, and the connection method between the device and the audio device may be determined based on the configuration of the high-definition multimedia interface cable between the device and the audio device.

[0121] Optionally, the audio output mode may be determined according to the configuration of the audio device, for example, according to the configuration performed by the user on the audio device.

[0122] In some embodiments of the present application, the device determines the format of the original audio data according to at least one of the audio output mode of the audio device or the connection mode between the device and the audio device. The method may be: when the audio output mode is the source code output mode, the format of the original audio data is determined to be the source code format; when the audio output mode is the non-source code output mode and the connection mode is the ARC connection, the format of the original audio data is determined to be the set channel format; when the audio output mode is the non-source code output mode and the connection mode is the eARC connection, the format of the original audio data is determined to be the multi-channel format supported by the audio device. Wherein, the multi-channel format supported by the audio device meets at least one of the following conditions: the number of channels of the multi-channel is less than or equal to the number of channels supported by the audio device; the channels contained in the multi-channel belong to the channels supported by the audio device; the azimuth information corresponding to the same channel in the multi-channel and the channels supported by the audio device is the same; wherein each azimuth information is used to indicate the azimuth of the audio of the corresponding channel when playing; the audio format of the audio in the multi-channel format belongs to the audio format supported by the audio device.

[0123] The non-source code output mode includes other modes except the source code output mode.

[0124] In some embodiments of the present application, setting the channel format means that the channel of the audio data is in the format of the set channel. The set channel may be one or more pre-set channels. For example, the set channel may be a main channel, and the set channel format is the main channel format. For another example, the set channel may include a left channel and a right channel, and the set channel format is the left and right channel format.

[0125] In some embodiments of the present application, the multi-channel format refers to a format in which the channels of the audio data include multiple channels, wherein the multiple channels meet at least one of the conditions met by the multi-channel format mentioned above.

[0126] In some embodiments of the present application, before sending the original audio data in a determined format to the audio device, the device may obtain in advance an audio capability descriptor SAD of the audio device and / or a speaker allocation data block SADB of the audio device from the audio device; and determine the capability parameters of the audio device based on the audio capability descriptor of the audio device and / or the speaker allocation data block of the audio device, so as to determine the format supported by the audio device based on the capability parameters of the audio device; wherein the capability parameters include at least one of the following: the number of channels supported by the audio device, the channels supported by the audio device, the orientation information corresponding to the channels supported by the audio device, and the audio format supported by the audio device.

[0127] Among them, as an optional implementation, the device can determine the number of channels supported by the audio device and the audio format supported by the audio device according to the SAD of the audio device, and the device can determine the channels supported by the audio device and the orientation information corresponding to the channels supported by the audio device according to the SADB of the audio device.

[0128] S503: The device sends the original audio data to the audio device in a determined format; wherein the audio device is used to process and / or play the original audio data.

[0129] In some embodiments of the present application, the device may include an application, an audio hardware abstraction layer HAL unit and an audio sending unit; wherein the audio HAL unit is in the hardware abstraction layer of the device. When the device obtains raw audio data, the raw audio data transmitted by the application through the application framework layer of the device can be obtained through the audio HAL unit. When the device sends the raw audio data to the audio device in a determined format, the raw audio data can be first converted into a determined format through the audio HAL unit, and then the raw audio data converted into the determined format can be sent to the audio device through the audio sending unit. Optionally, the application framework layer of the device may include an audio processing unit, which can be used to transmit audio data from the application to the audio HAL unit in the HAL layer.

[0130] Exemplarily, the application may be the application provided in the aforementioned embodiment, and the audio HAL unit may be the audio HAL unit described in the aforementioned embodiment. The specific structure of the device, the functions of the application and the audio HAL unit, and the related audio data processing method, etc., can all be implemented with reference to the relevant descriptions in the aforementioned embodiment, and will not be repeated here.

[0131] In the above method, the specific steps executed by the device can refer to the relevant introduction in the above embodiments, which will not be described in detail here.

[0132] Based on the above embodiments and the same technical concept, the present application also provides a device for implementing the audio data processing method provided in the present application. Figure 6 As shown in FIG. 6 , the device 600 may include: a memory 601 , one or more processors 602 , and one or more programs (not shown in the figure). The above components may be coupled via one or more communication buses 603 . Optionally, the device 600 may also include a display screen 604 .

[0133] Among them, one or more programs (codes) are stored in the memory 601, and the one or more programs include instructions; one or more processors 602 call the instructions stored in the memory 601, so that the device 600 executes the audio data processing method provided in the above-mentioned embodiment of the present application.

[0134] In a specific implementation, the memory 601 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The memory 601 may store an operating system (hereinafter referred to as system), such as ANDROID, IOS, WINDOWS, or an embedded operating system such as LINUX. The memory 601 may be used to store an implementation program of an embodiment of the present application. The memory 601 may also store a network communication program, which may be used to communicate with one or more devices.

[0135] The one or more processors 602 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application.

[0136] The display screen 604 can be used to display application interfaces and other related user interfaces, such as an audio / video playback control interface.

[0137] It should be noted that Figure 6 This is only one implementation of the device 600 provided in the embodiment of the present application. In actual applications, the device 600 may also include more or fewer components. For details, please refer to Figure 1 The specific structure and description shown are not limited here.

[0138] Based on the above embodiments and the same technical concept, an embodiment of the present application further provides a readable storage medium, which stores a program. When the program runs on a device, the device executes the method provided in the above embodiments.

[0139] Based on the above embodiments and the same technical concept, an embodiment of the present application also provides a program product, which includes a program or instructions. When the program or instructions are executed on a device, the device executes the method provided in the above embodiments.

[0140] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An audio data processing method, applied to a device, characterized in that: The device is connected to an audio device, and the method comprises: Acquire original audio data; wherein the original audio data belongs to three-dimensional colorful sound; Determine the format of the original audio data according to at least one of an audio output mode of the audio device or a connection mode between the device and the audio device; wherein the audio output mode is one of a source code output mode and a non-source code output mode, and the connection mode is one of an audio return channel ARC connection or an enhanced audio return channel eARC connection; The original audio data is sent to the audio device in a determined format; wherein the audio device is used to process and / or play the original audio data.

2. The method according to claim 1, characterized in that The audio device is an audio power amplifier device or a sound device.

3. The method according to claim 1 or 2, characterized in that The determining the format of the original audio data according to at least one of an audio output mode of the audio device or a connection mode between the device and the audio device comprises: When the audio output mode is the source code output mode, determining that the format of the original audio data is a source code format; When the audio output mode is the non-source code output mode and the connection mode is the ARC connection, determining that the format of the original audio data is a set channel format; When the audio output mode is the non-source code output mode and the connection mode is the eARC connection, it is determined that the format of the original audio data is a multi-channel format supported by the audio device.

4. The method according to claim 3, characterized in that The multi-channel format meets at least one of the following requirements: The number of channels of the multi-channel is less than or equal to the number of channels supported by the audio device; The channels included in the multiple channels belong to the channels supported by the audio device; The azimuth information corresponding to the same channel among the multiple channels and the channels supported by the audio device is the same; wherein each azimuth information is used to indicate the azimuth of the audio of the corresponding channel when it is played; The audio format of the audio in the multi-channel format belongs to the audio format supported by the audio device.

5. The method according to claim 4, characterized in that Before sending the original audio data to the audio device in the determined format, the method further comprises: Obtaining an audio capability descriptor of the audio device and / or a speaker allocation data block of the audio device; Determine capability parameters of the audio device according to the audio capability descriptor of the audio device and / or the speaker allocation data block of the audio device; wherein the capability parameters include at least one of the following: the number of channels supported by the audio device, the channels supported by the audio device, the orientation information corresponding to the channels supported by the audio device, and the audio format supported by the audio device.

6. The method according to any one of claims 1 to 5, characterized in that: The device comprises an application, an audio hardware abstraction layer HAL unit and an audio sending unit; wherein the audio HAL unit is in the hardware abstraction layer of the device; The obtaining of original audio data comprises: Acquire, through the audio HAL unit, the original audio data transmitted by the application through the application framework layer of the device; The sending the original audio data to the audio device in a determined format comprises: Converting the raw audio data into the determined format by the audio HAL unit; The original audio data converted into the determined format is sent to the audio device through the audio sending unit.

7. The method according to any one of claims 1 to 6, characterized in that: The device is connected to the audio device via a high-definition multimedia interface line.

8. A device, characterized in that The device comprises a memory and one or more processors; The memory is used to store program codes, and the program codes include instructions; when the instructions are executed by the one or more processors, the device executes the method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores a program, and when the program is executed on a device, the device executes the method according to any one of claims 1 to 7.

10. A program product, characterized in that The program product includes a program or an instruction. When the program or the instruction is executed on a device, the device is caused to execute the method according to any one of claims 1 to 7.

Images