Audio processing method and apparatus, terminal and storage medium

Abstract

The application provides an audio processing method and device, a terminal and a storage medium, and belongs to the technical field of computers. The method comprises the following steps: in response to receiving a program category of a target application program, determining an audio rendering parameter based on the program category, a space characteristic parameter of a target space where a terminal is located and a head characteristic parameter of a target object, the target application program being an application program currently running on the terminal, the target object being an object currently using the terminal, and the audio rendering parameter being used for rendering a space effect of audio; rendering audio output by the target application program based on the audio rendering parameter to obtain target audio, the target audio having a space effect; and outputting the target audio to the target object. The above scheme can improve the hearing experience of the target object while improving the efficiency of the target object listening to audio.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to an audio processing method, apparatus, terminal and storage medium. Background Technology

[0002] With the development of audio and video technology, users are increasingly seeking an immersive listening experience when using audio output devices such as wireless and wired headphones. Therefore, improving the spatial effect of the audio output from these devices is a technical problem that needs to be solved.

[0003] In related technologies, audio output devices typically use several spatial sound effect modes, such as those for sound theaters, cinemas, and concert halls, to simulate the spatial effects of audio in different scenarios. Users can choose the spatial sound effect mode according to their needs. The audio output device then outputs audio based on the user's selected spatial sound effect mode. However, this method requires manual selection by the user, which is cumbersome and reduces the efficiency of listening to audio. Summary of the Invention

[0004] This application provides an audio processing method, apparatus, terminal, and storage medium, which can improve the auditory experience of the target audience and the efficiency of the target audience listening to audio. The technical solution is as follows:

[0005] According to one aspect of the embodiments of this application, an audio processing method is provided, the method comprising:

[0006] In response to receiving the program category of the target application, audio rendering parameters are determined based on the program category, the spatial feature parameters of the target space where the terminal is located, and the header feature parameters of the target object. The target application is the application currently running on the terminal, the target object is the object currently using the terminal, and the audio rendering parameters are used to render spatial effects for the audio.

[0007] Based on the audio rendering parameters, the audio output by the target application is rendered to obtain the target audio, which has spatial effects;

[0008] The target audio is output to the target object.

[0009] According to another aspect of the embodiments of this application, an audio processing method is provided, the method comprising:

[0010] The grouping interface displays multiple groups, with each group corresponding to a program category.

[0011] In response to a grouping operation on a target application, the target application is added to the corresponding group;

[0012] In response to the completion of the grouping operation, a parameter acquisition interface is displayed. The parameter acquisition interface is used to set the spatial feature parameters of the target space where the terminal is located and the head feature parameters of the target object currently using the terminal.

[0013] In response to the establishment of a connection between the terminal and the audio output device, the spatial feature parameters and the head feature parameters are sent to the audio output device;

[0014] In response to switching to the target application, the program category of the target application is sent to the audio output device based on the group to which the target application belongs, so that the audio output device renders the audio output by the target application according to the program category, the spatial feature parameters, and the header feature parameters.

[0015] According to another aspect of the embodiments of this application, an audio processing apparatus is provided, the apparatus comprising:

[0016] The determination module is used to determine audio rendering parameters based on the program category of the target application, the spatial feature parameters of the target space where the terminal is located, and the header feature parameters of the target object in response to receiving the program category of the target application. The target application is the application currently running on the terminal, the target object is the object currently using the terminal, and the audio rendering parameters are used to render spatial effects for the audio.

[0017] The rendering module is used to render the audio output by the target application based on the audio rendering parameters to obtain the target audio, which has spatial effects;

[0018] The output module is used to output the target audio to the target object.

[0019] According to another aspect of the embodiments of this application, an audio processing apparatus is provided, the apparatus comprising:

[0020] The display module is used to display the group interface, which displays multiple groups, and each group corresponds to a program category.

[0021] A grouping module is used to add the target application to the corresponding group in response to a grouping operation on the target application;

[0022] The display module is also used to display a parameter acquisition interface in response to the completion of the grouping operation. The parameter acquisition interface is used to set the spatial feature parameters of the target space where the terminal is located and the head feature parameters of the target object currently using the terminal.

[0023] The sending module is configured to send the spatial feature parameters and the head feature parameters to the audio output device in response to the establishment of a connection between the terminal and the audio output device.

[0024] The sending module is further configured to, in response to switching to the target application, send the application category of the target application to the audio output device based on the group to which the target application belongs.

[0025] According to another aspect of the embodiments of this application, an audio output device is provided, the audio output device including a processor and a memory; the memory stores at least one piece of program code, the at least one piece of program code being executed by the processor to implement the audio processing method as described above.

[0026] According to another aspect of the embodiments of this application, a terminal is provided, the terminal including a processor and a memory; the memory stores at least one piece of program code, the at least one piece of program code being executed by the processor to implement the audio processing method as described above.

[0027] According to another aspect of the embodiments of this application, a chip is provided, the chip including programmable logic circuits and / or program instructions, which, when the chip is run on a terminal, is used to implement the audio processing method described above.

[0028] According to another aspect of the embodiments of this application, a computer-readable storage medium is provided, the storage medium storing at least one piece of program code, the at least one piece of program code being executed by a processor to implement the audio processing method as described above.

[0029] According to another aspect of the embodiments of this application, a computer program product is provided, which stores at least one piece of program code, the at least one piece of program code being executed by a processor to implement the audio processing method described above.

[0030] This application provides an audio processing scheme executed by an audio output device. Since the program category of the target application reflects the category of the audio output by the target application, the head feature parameters of the target object reflect the ear structure of the target object, and the spatial feature parameters of the target space reflect the spatial effect of audio propagation in different spaces, rendering the audio output by the target application using audio rendering parameters determined by the application category, the spatial feature parameters of the target space, and the head feature parameters of the target object allows the target audio output by applications of different program categories to have different spatial effects. Furthermore, it can adapt to the ear structure of different target objects, improving the auditory experience for different target objects. The target application's output audio can be adaptively rendered without the target object manually selecting a spatial sound effect mode, improving the spatial effect of the audio. While improving the auditory experience of the target object, it also improves the efficiency of the target object listening to audio. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0033] Figure 2 This is a flowchart of an audio processing method provided in an embodiment of this application;

[0034] Figure 3 This is a flowchart of another audio processing method provided in the embodiments of this application;

[0035] Figure 4 This is a schematic diagram of an audio rendering method provided in an embodiment of this application;

[0036] Figure 5 This is a flowchart of an audio processing method provided in an embodiment of this application;

[0037] Figure 6 This is a schematic diagram of a grouping interface provided in an embodiment of this application;

[0038] Figure 7 This is a schematic diagram of a parameter acquisition interface provided in an embodiment of this application;

[0039] Figure 8 This is a flowchart of an audio rendering method provided in an embodiment of this application;

[0040] Figure 9 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application;

[0041] Figure 10 This is a schematic diagram of another audio processing device provided in an embodiment of this application;

[0042] Figure 11 This is a schematic diagram of another audio processing device provided in an embodiment of this application;

[0043] Figure 12 This is a structural block diagram of a terminal provided in an embodiment of this application. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0045] In this article, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0046] It should be noted that all information (including but not limited to wireless audio device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the head feature parameters of the target object involved in this application were obtained with full authorization.

[0047] Figure 1 This is a schematic diagram of an implementation environment provided in an embodiment of this application. See also... Figure 1 The implementation environment includes a terminal 101 and an audio output device 102. The terminal 101 and the audio output device 102 are connected directly or indirectly via wired or wireless communication, which is not limited herein.

[0048] Optionally, the terminal 101 can be various types of terminals such as mobile phones, desktop computers, laptops, tablets, and smartwatches. The terminal 101 has various application categories installed, such as music, video, and audiobook applications. Users can log in to these applications through the terminal 101 and listen to music, watch videos, and listen to audiobooks.

[0049] Optionally, the audio output device 102 can be a device capable of outputting audio, such as wireless headphones, wired headphones, or over-ear headphones. The audio output device 102 is used to output audio from applications installed on the user's terminal 101. The audio output device 102 can also render the audio output by different applications according to their application categories, resulting in rendered audio with different spatial effects. The audio output device 102 is used to output the rendered audio to the user.

[0050] The audio processing method provided in this application can be applied to scenarios where users listen to music using an application. While a user is listening to music through an application categorized by music type, in response to the terminal 101 running the application, the terminal 101 sends the application's program category to the audio output device 102. The audio output device 102 can then render the audio output by the application according to its program category and output the rendered audio to the user, thereby enhancing the user's auditory experience.

[0051] Of course, the audio processing method provided in this application embodiment can also be applied to application scenarios where users use other program categories, and this application embodiment does not limit this.

[0052] Figure 2 This is a flowchart illustrating an audio processing method provided in an embodiment of this application. The method is executed by an audio output device; see [link to relevant documentation]. Figure 2 The method includes:

[0053] 201. In response to receiving the program category of the target application, the audio rendering parameters are determined based on the program category, the spatial feature parameters of the target space where the terminal is located, and the header feature parameters of the target object. The target application is the application currently running on the terminal, the target object is the object currently using the terminal, and the audio rendering parameters are used to render spatial effects for the audio.

[0054] In this embodiment, the audio output device can be a device capable of outputting audio, such as wireless headphones, wired headphones, or over-ear headphones. The terminal can be various types of terminals, such as mobile phones, desktop computers, laptops, tablets, and smartwatches. The audio output device and the terminal are connected via wired or wireless communication. Once a connection is established between the audio output device and the terminal, the audio output device can play audio output by applications installed on the terminal to the target user. The terminal has various application categories installed, including but not limited to music, video, and audiobooks.

[0055] When the target object uses the target application through a terminal, the terminal sends the application category of the target application to the audio output device. Accordingly, in response to receiving the application category, the audio output device can determine the application category. The audio output device acquires the spatial characteristic parameters of the target space where the terminal is located and the head characteristic parameters of the target object. The spatial characteristic parameters reflect the size and openness of the target space. The head characteristic parameters reflect the head shape and ear shape of the target object.

[0056] The audio output device determines the audio rendering parameters based on the target application's application category, the spatial characteristic parameters of the target space, and the header characteristic parameters of the target object. These audio rendering parameters are used to render the audio output by the target application, giving the rendered audio a spatial effect.

[0057] 202. Based on the audio rendering parameters, render the audio output by the target application to obtain the target audio, which has spatial effects.

[0058] In this embodiment, the audio output device acquires the audio output by the target application. The audio output device renders the audio using audio rendering parameters to obtain target audio with spatial effects.

[0059] 203. Output the target audio to the target object.

[0060] In this embodiment of the application, the audio output device plays target audio with spatial effects to the target object.

[0061] This application provides an audio processing method executed by an audio output device. Since the program category of the target application reflects the category of the audio output by the target application, the head feature parameters of the target object reflect the ear structure of the target object, and the spatial feature parameters of the target space reflect the spatial effect of audio propagation in different spaces, rendering the audio output by the target application using audio rendering parameters determined by the application category, the spatial feature parameters of the target space, and the head feature parameters of the target object allows the target audio output by applications of different program categories to have different spatial effects. Furthermore, it can adapt to the ear structure of different target objects, improving the auditory experience for different target objects. The method adaptively renders the audio output by the target application without requiring the target object to manually select a spatial sound effect mode, thus improving the spatial effect of the audio. While improving the auditory experience of the target object, it also improves the efficiency of the target object in listening to audio.

[0062] Figure 3 This is a flowchart of another audio processing method provided in an embodiment of this application. The method is executed by an audio output device; see [link to relevant documentation]. Figure 3 The method includes:

[0063] 301. In response to receiving the program category of the target application, determine the audio rendering parameters based on the program category, the spatial feature parameters of the target space where the terminal is located, and the header feature parameters of the target object. The target application is the application currently running on the terminal, the target object is the object currently using the terminal, and the audio rendering parameters are used to render spatial effects for the audio.

[0064] In this embodiment, the audio output device can be a device capable of outputting audio, such as wireless headphones, wired headphones, or over-ear headphones. The terminal can be various types of terminals, such as mobile phones, desktop computers, laptops, tablets, and smartwatches. The audio output device and the terminal are connected via wired or wireless communication. Once a connection is established between the audio output device and the terminal, the audio output device can play audio output by applications installed on the terminal to the target user. The terminal has various application categories installed, including but not limited to music, video, and audiobooks.

[0065] During the process of a target object using a target application through a terminal, the target application's running state includes running, started, and exited states. The following explanation uses the running state as an example. When the terminal is running the target application, the terminal sends the application's program category to the audio output device. Accordingly, in response to receiving the application's program category, the audio output device can determine the application's program category. The audio output device acquires the spatial characteristic parameters of the target space where the terminal is located and the head characteristic parameters of the target object. Based on the application's program category, the spatial characteristic parameters of the target space, and the head characteristic parameters of the target object, the audio output device determines the audio rendering parameters. The spatial characteristic parameters of the target space reflect its size and spaciousness. The head characteristic parameters of the target object reflect its head shape and ear shape. The audio rendering parameters are used to render the audio output by the target application, giving the rendered audio a spatial effect.

[0066] Optionally, the audio output device can acquire spatial characteristic parameters through the terminal. When the audio output device is connected to the terminal, the audio output device sends a first request to the terminal. The first request instructs the terminal to send a test audio signal to the target space. The audio output device collects the room impulse response generated by the propagation of the test audio signal in the target space through a microphone. The room impulse response includes direct sound, early reflections, and reverberation. Based on the energy-time curve of the collected room impulse response and parameters such as the ratio of direct sound to reverberation energy, the audio output device determines the spatial characteristic parameters of the target space, that is, the spatial dimensions of the target space, the wall material of the target space, and the distance between the terminal and the audio output device in the target space.

[0067] Optionally, the audio output device can acquire head feature parameters through the terminal. When the audio output device is connected to the terminal, the audio output device sends a second request to the terminal. The second request instructs the terminal to capture the head features of the target object using a camera and determine the head feature parameters of the target object based on these features. For example, the second request can instruct the terminal to capture a head image of the target object using a depth-sensing camera and to identify the head image of the target object using an artificial intelligence algorithm model to obtain head feature parameters such as the ear size and head size of the target object. The audio output device acquires the head feature parameters of the target object sent by the terminal.

[0068] Optionally, the audio output device can also obtain the application category of the target application through other means. Accordingly, the audio output device can identify the application category of the audio output by the current target application, such as music, video, and audiobooks, based on artificial intelligence algorithms. The audio output device uses the identified application category as the application category of the target application.

[0069] To more clearly illustrate the process by which the audio output device determines audio rendering parameters based on the aforementioned spatial feature parameters, head feature parameters, and program category, the following explanation uses program category and spatial feature parameters as examples to illustrate their respective impacts on audio rendering parameters. The impact of program category on audio rendering parameters is primarily due to the different audio requirements of different applications for the target audience. For example, for music applications, the target audience needs audio with a surround sound spatial effect, providing an immersive listening experience. Therefore, the audio output device determines the appropriate audio rendering parameters to achieve this surround sound spatial effect in the rendered audio. For video applications, the target audience needs clear dialogue in the output audio. Therefore, the audio output device determines the appropriate audio rendering parameters to improve the clarity of the dialogue. The impact of spatial feature parameters on audio rendering parameters is mainly due to the spatial effects created by audio propagation in different spaces. When the target space reflected by the spatial feature parameters is large and relatively open, the audio output device determines the appropriate audio rendering parameters. The aforementioned audio rendering parameters enable the rendered audio to have a spatial effect that allows it to propagate in relatively open spaces, such as having a certain degree of echo. When the target space reflected by the spatial characteristic parameters is small or confined, the audio output device determines the corresponding audio rendering parameters. These audio rendering parameters enable the rendered audio to have a spatial effect that allows it to propagate in confined spaces, such as having very clear rendered audio.

[0070] Optionally, the audio rendering parameters include an energy-time curve and the ratio of direct sound to reverberant sound energy. Spatial feature parameters include the spatial dimensions of the target space, the material of the walls of the target space, and the distance between the virtual sound source and the target object in the target space. Head feature parameters include the ear size and head size of the target object. Ear size includes the ear width and ear height of the target object. During the process of determining the audio rendering parameters, the audio output device can determine the energy-time curve and the ratio of direct sound to reverberant sound energy in the audio rendering parameters based on the program category, head feature parameters, and spatial feature parameters. In response to receiving the program category of the target application, the audio output device obtains a parameter library. The audio output device can obtain the parameter library locally, from a terminal connected to the audio output device, or from other terminals; this embodiment does not impose any limitations on this. The parameter library is used to store various energy-time curves and various direct sound to reverberant sound energy ratios. The parameter library stores energy-time curves and direct sound to reverberant sound energy ratios that match the program category, spatial feature parameters, and head feature parameters, respectively. The audio output device determines from the parameter library an energy-time curve and a direct-to-reverberation energy ratio that match at least one of the program category, spatial characteristic parameters, and head characteristic parameters.

[0071] The energy-time curve represents the decay of energy in a room's impulse response over time. It is related to the room's dimensions, wall materials, and the distance between the virtual sound source and the target object. For example, better sound attenuation from the wall materials results in a faster decay rate, while poorer attenuation leads to a slower decay. The room's impulse response includes direct sound, early reflections, and reverberation. Direct sound is the audio transmitted directly from the virtual sound source in the target space to the target object's ear. Early reflections are the audio transmitted after one or two reflections from the virtual sound source to the target object's ear. Reverberation is the audio transmitted after multiple reflections from the virtual sound source to the target object's ear. Direct sound is related to the distance between the virtual sound source and the target object. Reverberation is related to the dimensions of the target space, the wall materials, and the distance between the virtual sound source and the target object. The ratio of direct sound to reverberation energy reflects the distance between the target object and the virtual sound source. Since the energy of reverberation decreases faster than that of direct sound, the greater the distance between the target object and the virtual sound source, the larger the energy ratio of direct sound to reverberation; the closer the target object is to the virtual sound source, the smaller the energy ratio of direct sound to reverberation.

[0072] 302. The audio output device determines the head-related transfer function based on the head feature parameters of the target object. The head-related transfer function is used to represent the attenuation of the audio output by the target application as it propagates from the virtual sound source to the ear of the target object in the target space.

[0073] In this embodiment, after the audio output device acquires the head feature parameters of the target object, it determines a head-related transfer function (HRTF) that matches the target object's head feature parameters in the HRTF database, based on the target object's ear size, head size, and other head feature parameters. The HRTF database stores various head-related transfer functions and the head feature parameters that match each function. The head-related transfer function is a frequency domain transfer function that represents the position of the virtual sound source in the target space, the position of the target object's ear in the target space, and the attenuation of audio during transmission between the two. The audio output device can acquire the HRTF database locally or at the terminal; this embodiment does not impose any limitation on this.

[0074] 303. The audio output device renders the audio output by the target application based on the header-related transfer function and audio rendering parameters to obtain the target audio.

[0075] In this embodiment of the application, the audio output device processes the audio output by the target application according to the audio rendering parameters and the head-related function that matches the head parameters of the target object, so as to obtain the target audio with spatial effects.

[0076] Optionally, the audio output device transforms the head-related transfer function from the frequency domain to the time domain to obtain the room impulse response. The room impulse response simulates the impulse response of audio output by the target application propagating from a virtual sound source to the ear of the target object in the target space. The audio output device convolves the room impulse response with the audio output by the target application in the time domain to obtain the direct sound of the room impulse response. Based on the direct sound of the room impulse response and the energy ratio of the direct sound to the reverberation, the audio output device determines the reverberation of the room impulse response. Based on the direct sound, reverberation, and energy-time curves of the room impulse response, the audio output device determines the early reflections of the room impulse response. The audio output device synthesizes the direct sound, early reflections, and reverberation to obtain the target audio.

[0077] For example, Figure 4This is a schematic diagram of an audio rendering. The audio output device can process the audio xL(t) output by the target application through the left speaker 401 and the audio xR(t) output through the right speaker 402 through the following formulas (1) and (2) to obtain the BRIR (Binaural Room Impulse Response) of the audio reaching the left and right ears of the target object after the audio propagates in the target space.

[0078] yL(t)=xL(t)*hll(t)+xR(t)*hrl(t) (1)

[0079] yR(t)=xL(t)*hlr(t)+xR(t)*hrr(t) (2)

[0080] Where * represents convolution operation, and t represents the time series. xL(t) and xR(t) are the audio outputs of the left and right speakers, respectively. hll(t) and hlr(t) represent the room impulse responses of the left speaker to the left and right ears of the target, respectively. hrl(t) and hrr(t) represent the room impulse responses of the right speaker to the left and right ears of the target, respectively. yL(t) and yR(t) are the binaural room impulse responses of the sound source to the left and right ears of the target, that is, the audio with spatial effects that reaches the target's ears after propagating in space.

[0081] 304. The audio output device outputs the target audio to the target object.

[0082] In this embodiment of the application, the audio output device plays target audio with spatial effects to the target object.

[0083] This application provides an audio processing method executed by an audio output device. Since the program category of the target application reflects the category of the audio output by the target application, the head feature parameters of the target object reflect the ear structure of the target object, and the spatial feature parameters of the target space reflect the spatial effect of audio propagation in different spaces, rendering the audio output by the target application using audio rendering parameters determined by the application category, the spatial feature parameters of the target space, and the head feature parameters of the target object allows the target audio output by applications of different program categories to have different spatial effects. Furthermore, it can adapt to the ear structure of different target objects, improving the auditory experience for different target objects. The method adaptively renders the audio output by the target application without requiring the target object to manually select a spatial sound effect mode, thus improving the spatial effect of the audio. While improving the auditory experience of the target object, it also improves the efficiency of the target object in listening to audio.

[0084] Figure 5This is a flowchart illustrating an audio processing method provided in an embodiment of this application. The method is executed by a terminal; see [link to relevant documentation]. Figure 5 The method includes:

[0085] 501. Display the group interface. The group interface displays multiple groups, and each group corresponds to a program category.

[0086] In this embodiment, the terminal has multiple applications installed. The terminal can display applications of different application categories in different groups. The terminal displays a grouping interface with multiple groups. Each group corresponds to a specific application category. The terminal displays the applications of the application category corresponding to that group within each group. For example, see... Figure 6 The grouping interface shown has five groups: Music, Video, Audiobooks, Other, and Add. Each group displays applications belonging to the corresponding program category. The Add group is used by the target user to customize other program categories. For example, the target user can customize the Add group to be the Musical category. Accordingly, the terminal displays applications belonging to the Musical category in the Musical group.

[0087] Optionally, the terminal has a spatial audio interaction app installed. This app is used to group multiple applications installed on the terminal. Accordingly, the terminal can display the grouped interface through the audio interaction app.

[0088] 502. In response to a grouping operation on the target application, add the target application to the corresponding group.

[0089] In this embodiment, in response to a grouping operation of a target application, the terminal adds the target application to the corresponding group and displays it on the grouping interface. The grouping operation of the target application can be performed by the terminal. Accordingly, the terminal obtains the category of the target application and performs the grouping operation. Alternatively, the grouping operation of the target application can be performed by the target object. Accordingly, the target object performs the grouping operation of the target application as needed.

[0090] 503. In response to the completion of the grouping operation, the parameter acquisition interface is displayed. The parameter acquisition interface is used to set the spatial feature parameters of the target space where the terminal is located and the header feature parameters of the target object of the currently using terminal.

[0091] In this embodiment, after the grouping operation of the application is completed, the terminal displays a parameter acquisition interface. The parameter acquisition interface is used to acquire the spatial feature parameters of the target space where the terminal is located and the head feature parameters of the target object currently using the terminal. The spatial feature parameters include the spatial dimensions of the target space, the material of the walls of the target space, and the distance between the virtual sound source and the target object in the target space. The head feature parameters include the ear size and head size of the target object. The ear size includes the ear width and ear height of the target object.

[0092] Optionally, the terminal can obtain the spatial feature parameters and head feature parameters of the target object from the parameter acquisition interface. For example, see Figure 7 The parameter acquisition interface is shown. The target object can input head feature parameters such as ear width and ear height, as well as spatial feature parameters such as space type, space size, and sound source distance. The space type refers to the type of wall material, and the sound source distance is the distance between the virtual sound source and the target object. In response to the "Next" button on the parameter acquisition interface, the terminal acquires the head feature parameters and spatial feature parameters input by the target object on the interface.

[0093] 504. In response to the establishment of a connection between the terminal and the audio output device, send spatial feature parameters and head feature parameters to the audio output device.

[0094] In this embodiment, the terminal is capable of connecting to an audio output device. The audio output device is capable of outputting audio from an application installed on the terminal to a target object. The terminal acquires the connection status with the audio output device. In response to establishing a connection between the terminal and the audio output device, the terminal sends the acquired spatial feature parameters and header feature parameters to the audio output device.

[0095] Optionally, when a Bluetooth connection is established between the audio output device and the terminal, the terminal sends spatial feature parameters and head feature parameters to the audio output device via the Bluetooth proprietary protocol.

[0096] 505. In response to switching to the target application, based on the group to which the target application belongs, send the application category of the target application to the audio output device so that the audio output device renders the audio output by the target application according to the application category, spatial feature parameters, and header feature parameters.

[0097] In this embodiment, the terminal can detect the status of currently running applications, launched applications, and exited applications. In response to the terminal detecting that the currently running application has switched to the target application, the terminal obtains the group to which the target application belongs. Based on the group to which the target application belongs, the terminal determines the application's program category. The terminal sends the target application's program category to the audio output device. The audio output device can render the audio output by the target application based on the target application's program category and the previously obtained spatial feature parameters and header feature parameters, obtaining target audio with spatial effects.

[0098] To more clearly illustrate the interaction between the terminal and the audio output device, the following will combine... Figure 8 The flowchart shown illustrates the interaction between the terminal and the audio output device during the audio rendering process.

[0099] like Figure 8 As shown, the terminal detects running applications and their startup and exit statuses within the groups set by the terminal through the underlying interface. In response to the terminal detecting that a target application is running or has started, the terminal verifies the target application's group. The terminal determines the target application's program category based on the application's group. When the terminal is connected to the audio output device via Bluetooth, the terminal sends the target application's program category to the audio output device via the Bluetooth proprietary protocol. The audio output device renders the audio output by the target application based on its program category and room impulse response, obtaining target audio with spatial effects. The room impulse response can be simulated based on spatial and head feature parameters obtained from the audio output device.

[0100] This application provides an audio processing method executed by a terminal. The terminal groups installed applications according to program category in a grouping interface, providing a more intuitive display of applications of different categories. The terminal obtains spatial feature parameters and head feature parameters through a parameter acquisition interface, enabling it to accurately obtain the characteristics of the target space and the ear structure of the target object, thus rendering audio with spatial effects more accurately. When establishing a connection with the audio output device and switching to the target application, the terminal sends the spatial feature parameters, head feature parameters, and the program category of the target application to the audio output device. This allows the audio output device to render the audio output by the target application based on audio rendering parameters determined by the application category, the spatial feature parameters of the target space, and the head feature parameters of the target object, resulting in rendered audio with spatial effects. This improves the target object's auditory experience and also increases the efficiency of audio listening.

[0101] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0102] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0103] Figure 9 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application. See also... Figure 9 The device includes: a determination module 901, a rendering module 902, and an output module 903.

[0104] The determination module 901 is used to respond to the received program category of the target application, and determine the audio rendering parameters based on the program category, the spatial feature parameters of the target space where the terminal is located, and the header feature parameters of the target object. The target application is the application currently running on the terminal, the target object is the object currently using the terminal, and the audio rendering parameters are used to render spatial effects for the audio.

[0105] Rendering module 902 is used to render the audio output by the target application based on the audio rendering parameters to obtain the target audio, which has spatial effects;

[0106] Output module 903 is used to output target audio to the target object.

[0107] In some embodiments, the audio rendering parameters include an energy-time curve and the ratio of direct sound to reverberant sound energy; the spatial feature parameters include the spatial dimensions of the target space, the spatial wall material of the target space, and the distance between the virtual sound source and the target object in the target space; the head feature parameters include the ear size and head size of the target object.

[0108] The determination module 901 is used to obtain a parameter library, which stores various energy-time curves and various direct sound and reverberation energy ratios. It determines from the parameter library the energy-time curve and the direct sound and reverberation energy ratio that match at least one of the program category, spatial characteristic parameters, and head characteristic parameters. The energy-time curve is associated with the spatial dimensions, spatial wall materials, and the distance between the virtual sound source and the target object in the target space. The direct sound is associated with the distance between the virtual sound source and the target object in the target space. The reverberation is associated with the spatial dimensions, spatial wall materials, and the distance between the virtual sound source and the target object in the target space.

[0109] In some embodiments, Figure 10 This is a schematic diagram of another audio processing device provided in an embodiment of this application. See also... Figure 10 The device also includes:

[0110] The sending module 904 is used to send a first request to the terminal, the first request being used to instruct the terminal to send a test audio signal to the target space;

[0111] The acquisition module 905 is used to acquire the room impulse response generated by the propagation of the test audio signal in the target space. The room impulse response includes direct sound, early reflections and reverberation.

[0112] The spatial characteristic parameter determination module 906 is used to determine the spatial characteristic parameters of the target space based on the room impulse response.

[0113] In some embodiments, please continue to see Figure 10 The device also includes:

[0114] The sending module 904 is also used to send a second request to the terminal, the second request being used to instruct the terminal to collect the head features of the target object through the camera, and to determine the head feature parameters of the target object based on the head features;

[0115] The acquisition module 907 is used to acquire the header feature parameters of the target object sent by the terminal.

[0116] In some embodiments, the rendering module 902 is used to determine a head-related transfer function based on the head feature parameters of the target object. The head-related transfer function is used to represent the attenuation of the audio output by the target application from the virtual sound source to the ear of the target object in the target space. Based on the head-related transfer function and the audio rendering parameters, the rendering module 902 renders the audio output by the target application to obtain the target audio.

[0117] In some embodiments, the audio rendering parameters include an energy-time curve and a direct sound to reverberation energy ratio; the rendering module 902 is used to convert the head-related transfer function from the frequency domain to the time domain to obtain the room impulse response, which is used to simulate the impulse response of the audio output by the target application propagating from a virtual sound source to the ear of the target object in the target space; the room impulse response is convolved with the audio output by the target application in the time domain to obtain the direct sound of the room impulse response; the reverberation of the room impulse response is determined based on the direct sound of the room impulse response and the direct sound to reverberation energy ratio; the early reflections of the room impulse response are determined based on the direct sound, reverberation, and energy-time curve of the room impulse response; the direct sound, early reflections, and reverberation are synthesized to obtain the target audio.

[0118] This application provides an audio processing apparatus executed by an audio output device. Since the program category of the target application reflects the category of the audio output by the target application, the head feature parameters of the target object reflect the ear structure of the target object, and the spatial feature parameters of the target space reflect the spatial effect of audio propagation in different spaces, rendering the audio output by the target application using audio rendering parameters determined by the application category, the spatial feature parameters of the target space, and the head feature parameters of the target object allows the target audio output by applications of different program categories to have different spatial effects. Furthermore, it can adapt to the ear structure of different target objects, improving the auditory experience for different target objects. The target application's output audio can be adaptively rendered without the target object manually selecting a spatial sound effect mode, improving the spatial effect of the audio. While improving the auditory experience of the target object, it also improves the efficiency of the target object listening to audio.

[0119] It should be noted that the audio processing device provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the audio output device can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio processing device and the audio processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0120] Figure 11 This is a schematic diagram of another audio processing device provided in an embodiment of this application. See also... Figure 11 The device includes a display module 1101, a grouping module 1102, and a sending module 1103.

[0121] Display module 1101 is used to display the group interface, which displays multiple groups, and each group corresponds to a program category.

[0122] Grouping module 1102 is used to add the target application to the corresponding group in response to a grouping operation on the target application;

[0123] Display module 1101 is used to display a parameter acquisition interface in response to the completion of the grouping operation. The parameter acquisition interface is used to set the spatial feature parameters of the target space where the terminal is located and the header feature parameters of the target object of the currently used terminal.

[0124] The sending module 1103 is used to send spatial feature parameters and head feature parameters to the audio output device in response to the establishment of a connection between the terminal and the audio output device;

[0125] The sending module 1103 is also configured to, in response to switching to the target application, send the program category of the target application to the audio output device based on the group to which the target application belongs, so that the audio output device renders the audio output by the target application according to the program category, spatial feature parameters, and header feature parameters.

[0126] This application provides an audio processing device executed by a terminal. The terminal can group installed applications according to program category through a grouping interface, providing a more intuitive display of applications of different categories. The terminal can obtain spatial feature parameters and head feature parameters through a parameter acquisition interface, enabling it to accurately obtain the characteristics of the target space where the terminal is located and the ear structure of the target object, thus rendering audio with spatial effects more accurately for the target object. When the terminal establishes a connection with the audio output device and switches to the target application, it sends the spatial feature parameters, head feature parameters, and the program category of the target application to the audio output device. This allows the audio output device to render the audio output by the target application based on the audio rendering parameters determined by the application category, the spatial feature parameters of the target space, and the head feature parameters of the target object, resulting in rendered target audio with spatial effects. This improves the target object's auditory experience and also increases the efficiency of audio listening.

[0127] It should be noted that the audio processing device provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the terminal can be divided into different functional modules to complete all or part of the functions described above. In addition, the audio processing device and the audio processing method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0128] This application provides a terminal, which includes a processor and a memory; the memory stores at least one piece of program code, which is executed by the processor to implement the audio processing method provided in the above-described method embodiments.

[0129] Figure 12 This is a structural block diagram of a terminal provided in an embodiment of this application. In some embodiments, the terminal 1200 is a smartphone, tablet computer, wearable device, or other terminal capable of accessing a wireless local area network as a wireless station. The terminal 1200 in this application includes at least one or more of the following components: a processor 1210, a memory 1220, and at least two wireless links 1230.

[0130] In some embodiments, the processor 1210 includes one or more processing cores. The processor 1210 connects to various parts within the terminal 1200 using various interfaces and lines, and performs various functions and processes data of the terminal 1200 by running or executing program code stored in the memory 1220 and calling data stored in the memory 1220. In some embodiments, the processor 1210 is implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 1210 can integrate one or more of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural-network Processing Unit (NPU), and modem. Specifically, the CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display on the screen; the NPU is used to implement Artificial Intelligence (AI) functions; and the modem is used for wireless communication. It is understandable that the aforementioned modem could also be implemented as a separate chip without being integrated into the processor 1210.

[0131] In some embodiments, the processor 1210 is used to control the operating status of at least two wireless links 1230. Accordingly, the processor 1210 is a processor integrating a Wireless Fidelity (Wi-Fi) chip. This Wi-Fi chip is a chip with dual Wi-Fi processing capabilities. For example, the Wi-Fi chip is a dual-band dual-concurrent (DBDC) chip, or a dual-band simultaneous (DBS) chip, etc.

[0132] In some embodiments, memory 1220 includes random access memory (RAM), and in some embodiments, memory 1220 includes read-only memory (ROM). In some embodiments, memory 1220 includes non-transitory computer-readable storage medium. Memory 1220 can be used to store program code. Memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch functionality, sound playback functionality, image playback functionality, etc.), instructions for implementing the various method embodiments described below, etc.; the data storage area may store data created based on the use of terminal 1200 (such as audio data, phonebook, etc.).

[0133] In some embodiments, the memory 1220 stores reception schemes for different wireless links 1230 receiving beacon frames, as well as identifiers of access nodes connected to different wireless links 1230, identifiers of wireless links 1230, etc.

[0134] The at least two wireless links 1230 are used to connect different access points (APs). They receive downlink data from the APs. These different access points can be access points within the same router or access points within different routers.

[0135] In some embodiments, the terminal 1200 further includes a display screen. The display screen is a display component used to display a user interface. In some embodiments, the display screen is a touch-enabled display screen, allowing users to perform touch operations on the display screen using fingers, styluses, or any suitable object. In some embodiments, the display screen is typically located on the front panel of the terminal 1200. In some embodiments, the display screen is designed as a full-screen, curved screen, irregularly shaped screen, dual-sided screen, or foldable screen. In some embodiments, the display screen is also designed as a combination of a full-screen and a curved screen, or a combination of an irregularly shaped screen and a curved screen, etc., which are not limited in this embodiment.

[0136] In addition, those skilled in the art will understand that the structure of the terminal 1200 shown in the above figures does not constitute a limitation on the terminal 1200. The terminal 1200 may include more or fewer components than shown, or combine certain components, or have different component arrangements. For example, the terminal 1200 may also include components such as a microphone, speaker, input unit, sensor, audio circuit, module, power supply, and Bluetooth module, which will not be described in detail here.

[0137] This application also provides a computer-readable storage medium storing at least one piece of program code, which is loaded and executed by the processor to implement the audio processing methods shown in the above embodiments.

[0138] This application also provides a chip including programmable logic circuitry and / or program instructions, which, when running on a terminal, implements the audio processing methods shown in the above embodiments.

[0139] This application also provides a computer program product that stores at least one piece of program code for execution by a processor to implement the audio processing methods shown in the above embodiments.

[0140] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0141] Those skilled in the art will understand that all or part of the steps in the audio processing method of the above embodiments can be implemented by hardware, or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk. The above descriptions are merely optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An audio processing method, characterized in that, The method includes: Send a first request to the terminal, the first request being used to instruct the terminal to send a test audio signal to the target space; The room impulse response generated by the propagation of the test audio signal in the target space is collected, and the spatial characteristic parameters of the target space are determined based on the room impulse response; the spatial characteristic parameters include the spatial dimensions of the target space, the wall material of the target space, and the distance between the virtual sound source and the target object in the target space; A second request is sent to the terminal, the second request being used to instruct the terminal to collect head features of the target object through the camera, and to determine the head feature parameters of the target object based on the head features, thereby obtaining the head feature parameters of the target object sent by the terminal; the head feature parameters include the ear size and head size of the target object; Based on the program category of the target application, the spatial feature parameters of the target space where the terminal is located, and the head feature parameters of the target object, audio rendering parameters are determined. The target application is the application currently running on the terminal, and the target object is the object currently using the terminal. The audio rendering parameters are used to render spatial effects for the audio. The audio rendering parameters include energy-time curves and the ratio of direct sound to reverberant sound energy. Based on the head feature parameters of the target object, a head-related transfer function is determined. The head-related transfer function is used to represent the attenuation of the audio output by the target application from the virtual sound source to the ear of the target object in the target space. The head-related transfer function is transformed from the frequency domain to the time domain to obtain the room impulse response, which is used to simulate the impulse response of the audio output by the target application propagating from the virtual sound source to the ear of the target object in the target space; The room impulse response is convolved with the audio output by the target application in the time domain to obtain the direct sound of the room impulse response; The reverberation of the room impulse response is determined based on the direct sound and the ratio of the direct sound to the reverberation energy. Based on the direct sound, reverberation sound, and energy-time curve of the room impulse response, the early reflections of the room impulse response are determined; The direct sound, the early reflections, and the reverberation are synthesized to obtain the target audio, which has spatial effects. The target audio is output to the target object.

2. The method according to claim 1, characterized in that, The process of determining audio rendering parameters based on the program category, the spatial feature parameters of the target space where the terminal is located, and the head feature parameters of the target object includes: Obtain a parameter library, which stores various energy-time curves and various direct sound and reverberant sound energy ratios; The energy-time curve and the ratio of direct sound to reverberation sound energy are determined from the parameter library and matched with at least one of the program category, the spatial feature parameter, and the head feature parameter. The energy-time curve is associated with the spatial dimensions, the spatial wall material, and the distance between the virtual sound source in the target space and the target object. The direct sound is associated with the distance between the virtual sound source in the target space and the target object. The reverberation sound is associated with the spatial dimensions, the spatial wall material, and the distance between the virtual sound source in the target space and the target object.

3. An audio processing method, characterized in that, The method includes: The grouping interface displays multiple groups, with each group corresponding to a program category. In response to a grouping operation on a target application, the target application is added to the corresponding group; In response to the completion of the grouping operation, a parameter acquisition interface is displayed. The parameter acquisition interface is used to set the spatial feature parameters of the target space where the terminal is located and the head feature parameters of the target object currently using the terminal. In response to the establishment of a connection between the terminal and the audio output device, a first request is sent to the terminal, the first request being used to instruct the terminal to send a test audio signal to the target space; The room impulse response generated by the propagation of the test audio signal in the target space is collected, and the spatial characteristic parameters of the target space are determined based on the room impulse response; the spatial characteristic parameters include the spatial dimensions of the target space, the wall material of the target space, and the distance between the virtual sound source and the target object in the target space; A second request is sent to the terminal, the second request being used to instruct the terminal to collect head features of the target object through the camera, and to determine the head feature parameters of the target object based on the head features, thereby obtaining the head feature parameters of the target object sent by the terminal; the head feature parameters include the ear size and head size of the target object; Send the spatial feature parameters and the head feature parameters to the audio output device; In response to switching to the target application, the program category of the target application is sent to the audio output device based on the group to which the target application belongs, so that the audio output device renders the audio output by the target application according to the program category of the target application, the spatial feature parameters, and the header feature parameters.

4. An audio processing device, characterized in that, The device includes: The sending module is used to send a first request to the terminal, the first request being used to instruct the terminal to send a test audio signal to the target space; The acquisition module is used to acquire the room impulse response generated by the propagation of the test audio signal in the target space, and to determine the spatial characteristic parameters of the target space based on the room impulse response; the spatial characteristic parameters include the spatial dimensions of the target space, the wall material of the target space, and the distance between the virtual sound source and the target object in the target space; The sending module is further configured to send a second request to the terminal, the second request being configured to instruct the terminal to collect head features of the target object through the camera, and to determine head feature parameters of the target object based on the head features, thereby obtaining the head feature parameters of the target object sent by the terminal; the head feature parameters include the ear size and head size of the target object; The determination module is used to determine audio rendering parameters based on the program category of the target application, the spatial feature parameters of the target space where the terminal is located, and the header feature parameters of the target object. The target application is the application currently running on the terminal, and the target object is the object currently using the terminal. The audio rendering parameters are used to render spatial effects for the audio. The audio rendering parameters include energy-time curves and the ratio of direct sound to reverberant sound energy. The rendering module is used to determine a head-related transfer function based on the head feature parameters of the target object. The head-related transfer function represents the attenuation of audio output by the target application propagating from a virtual sound source to the ear of the target object in the target space. The head-related transfer function is converted from the frequency domain to the time domain to obtain a room impulse response. This room impulse response simulates the impulse response of audio output by the target application propagating from the virtual sound source to the ear of the target object in the target space. The room impulse response is convolved with the audio output by the target application in the time domain to obtain the direct sound of the room impulse response. Based on the direct sound and the energy ratio of the direct sound to the reverberation sound, the reverberation sound of the room impulse response is determined. Based on the direct sound, reverberation sound, and energy-time curve of the room impulse response, the early reflections of the room impulse response are determined. The direct sound, the early reflections, and the reverberation sound are synthesized to obtain the target audio, which has spatial effects. The output module is used to output the target audio to the target object.

5. An audio processing device, characterized in that, The device includes: The display module is used to display the group interface, which displays multiple groups, and each group corresponds to a program category. A grouping module is used to add the target application to the corresponding group in response to a grouping operation on the target application; The display module is also used to display a parameter acquisition interface in response to the completion of the grouping operation. The parameter acquisition interface is used to set the spatial feature parameters of the target space where the terminal is located and the head feature parameters of the target object currently using the terminal. A sending module is configured to, in response to the establishment of a connection between the terminal and the audio output device, send a first request to the terminal, the first request instructing the terminal to send a test audio signal to a target space; acquire the room impulse response generated by the propagation of the test audio signal in the target space, and determine the spatial characteristic parameters of the target space based on the room impulse response; the spatial characteristic parameters include the spatial dimensions of the target space, the wall material of the target space, and the distance between the virtual sound source and the target object in the target space; send a second request to the terminal, the second request instructing the terminal to acquire the head features of the target object through a camera, and determine the head characteristic parameters of the target object based on the head features, and acquire the head characteristic parameters of the target object sent by the terminal; the head characteristic parameters include the ear size and head size of the target object; and send the spatial characteristic parameters and the head characteristic parameters to the audio output device. The sending module is further configured to, in response to switching to the target application, send the application category of the target application to the audio output device based on the group to which the target application belongs, so that the audio output device renders the audio output by the target application according to the application category of the target application, the spatial feature parameters, and the header feature parameters.

6. An audio output device, characterized in that, The audio output device includes a processor and a memory; the memory stores at least one piece of program code, which is executed by the processor to implement the audio processing method as described in claim 1 or 2.

7. A terminal, characterized in that, The terminal includes a processor and a memory; the memory stores at least one piece of program code, which is executed by the processor to implement the audio processing method as described in claim 3.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is executed by a processor to implement the audio processing method as described in claim 1 or 2, or the audio processing method as described in claim 3.

9. A chip, characterized in that, The chip includes programmable logic circuits and / or program instructions, which, when the chip is run on a terminal, are used to implement the audio processing method as described in claim 1 or 2, or the audio processing method as described in claim 3.

10. A computer program product, comprising a computer program, characterized in that, The computer program product stores at least one piece of program code, which is executed by a processor to implement the audio processing method as described in claim 1 or 2, or the audio processing method as described in claim 3.

Images