Audio panning optimization method and apparatus, electronic device, and storage medium

Abstract

The present application is applicable to the technical field of audio, and provided thereby are an audio panning optimization method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring audio to be processed and rendering parameters corresponding to said audio; according to the rendering parameters, performing reverberation processing on the audio, so as to generate first reverberation audio corresponding to the audio to be processed; performing crosstalk elimination processing on the first reverberation audio, so as to generate optimized second reverberation audio; according to the rendering parameters, performing head-related transfer function (HRTF) rendering on the audio to be processed, so as to determine direct sound audio corresponding to said audio; and fusing the second reverberation audio and the direct sound audio, so as to generate optimized target audio. Therefore, the reverberation processing is performed, by means of a reverberation link, on the audio to be processed, and the HRTF rendering processing is performed on said audio by means of a direct sound link, respectively, and the crosstalk elimination processing is performed on reverberated audio, thereby improving the sound quality of external audio and the audio-visual experience of a user.

Description

Audio image optimization method, device, electronic device and storage medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 19, 2023, with application number 202310887727.5 and application name “Audio and audio-visual optimization method, device, electronic device and storage medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application belongs to the field of audio technology, and in particular relates to an audio sound and image optimization method, device, electronic device and computer-readable storage medium. Background Art

[0003] With the continuous development of mobile and portable devices, a good audio and video experience has become an essential attribute of high-end products. For example, people have increasingly higher requirements for the sound quality of mobile devices such as mobile phones and tablets, as well as audio equipment used in movie theaters, home theaters, and vehicles.

[0004] In related technologies, since external audio usually includes audio data from multiple channels, interference will occur between the audio data from multiple channels during the process of audio transmission from the device to the human ear, and the transmission environment will also interfere with the audio data, resulting in the user's actual audio quality being poor, affecting the user's audio and video experience.

[0005] Summary of the Invention

[0006] The embodiments of the present application provide an audio sound and image optimization method, device, electronic device and computer-readable storage medium, which can solve the problem that since external audio usually includes audio data of multiple channels, interference will occur between the audio data of multiple channels during the process of audio transmission from the device to the human ear, and the transmission environment will also interfere with the audio data, resulting in poor audio quality actually felt by the user, affecting the user's audio and video experience.

[0007] In a first aspect, an embodiment of the present application provides an audio sound image optimization method, comprising: obtaining audio to be processed and rendering parameters corresponding to the audio to be processed; performing reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed; performing crosstalk elimination processing on the first reverberation audio to generate an optimized second reverberation audio; performing head-related transfer function (HRTF) rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed; and performing fusion processing on the second reverberation audio and the direct sound audio to generate an optimized target audio.

[0008] In a possible implementation of the first aspect, the audio to be processed includes audio data of at least two sub-channels, the rendering parameters include room impulse responses corresponding to the respective sub-channels, and the first reverberated audio includes first sub-channel reverberated audio corresponding to the respective sub-channels. Accordingly, performing reverberation processing on the audio to be processed based on the rendering parameters to generate the first reverberated audio corresponding to the audio to be processed includes:

[0009] The audio data of each sub-channel is convolved with the corresponding room impulse response to generate each first sub-channel reverberation audio.

[0010] Optionally, in another possible implementation of the first aspect, the audio type of the to-be-processed audio is multi-channel audio; accordingly, before performing reverberation processing on the to-be-processed audio according to the rendering parameters to generate first reverberated audio corresponding to the to-be-processed audio, the method further includes:

[0011] Downmixing the audio to be processed to generate a left-channel audio to be processed and a right-channel audio to be processed corresponding to the audio to be processed;

[0012] The left channel audio to be processed and the right channel audio to be processed are determined to be audio data of each sub-channel included in the audio to be processed.

[0013] Optionally, in another possible implementation of the first aspect, the second reverberation audio includes second sub-channel reverberation audio corresponding to each sub-channel; accordingly, performing crosstalk cancellation processing on the first reverberation audio to generate optimized second reverberation audio includes:

[0014] Obtain the crosstalk cancellation filter coefficients corresponding to each sub-channel;

[0015] According to the crosstalk cancellation filter coefficient corresponding to each sub-channel, crosstalk cancellation processing is performed on each first sub-channel reverberation audio to generate optimized reverberation audio for each second sub-channel.

[0016] Optionally, in another possible implementation of the first aspect, the rendering parameters further include a crosstalk attenuation factor corresponding to each sub-channel, a crosstalk delay error corresponding to each sub-channel, and a crosstalk cancellation loss function; accordingly, obtaining the crosstalk cancellation filter coefficient corresponding to each sub-channel includes:

[0017] Obtain the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each sub-channel, and use it as the reference frequency corresponding to each sub-channel;

[0018] Determine the crosstalk cancellation order corresponding to each sub-channel according to the reference frequency corresponding to each sub-channel, each crosstalk attenuation factor, each crosstalk delay error and the crosstalk cancellation loss function;

[0019] According to the crosstalk cancellation order corresponding to each sub-channel, the crosstalk cancellation filter coefficient corresponding to each sub-channel is determined respectively.

[0020] Optionally, in another possible implementation of the first aspect, before fusing the second reverberation audio and the second direct sound audio to generate the optimized target audio, the process further includes:

[0021] The second reverberation audio is attenuated according to the attenuation coefficient to perform distance control on the second reverberation audio.

[0022] Optionally, in another possible implementation of the first aspect, the audio to be processed includes audio data of at least two sub-channels, and the rendering parameters further include azimuth rendering parameters; accordingly, performing HRTF rendering on the audio to be processed based on the rendering parameters to determine a first direct sound audio corresponding to the audio to be processed includes:

[0023] Determine the target HRTF rendering parameters corresponding to each sub-channel according to the azimuth rendering parameters;

[0024] According to the target HRTF rendering parameters corresponding to each sub-channel, HRTF rendering is performed on the audio data of each sub-channel to determine the direct sound audio corresponding to each sub-channel.

[0025] Optionally, in yet another possible implementation of the first aspect, the audio type of the to-be-processed audio is stereo audio, the sub-channels include a left channel and a right channel, and the sub-channel audio includes the to-be-processed left channel audio and the to-be-processed right channel audio; accordingly, before determining the target HRTF rendering parameters corresponding to each sub-channel based on the azimuth rendering parameters, the method further includes:

[0026] Perform center channel extraction on the left channel audio to be processed and the right channel audio to be processed to determine the center channel audio to be processed corresponding to the audio to be processed;

[0027] The left channel audio to be processed, the right channel audio to be processed, and the center channel audio to be processed are determined to be audio data of respective sub-channels included in the audio to be processed.

[0028] Optionally, in yet another possible implementation of the first aspect, the azimuth rendering parameters include a height rendering parameter and a width rendering parameter, and the sub-channels include at least a center channel, a left channel, and a right channel; accordingly, determining the target HRTF rendering parameters corresponding to each sub-channel based on the azimuth rendering parameters includes:

[0029] Determine the target HRTF rendering parameters corresponding to the center channel based on the height rendering parameters;

[0030] According to the width rendering parameter, the target HRTF rendering parameters corresponding to the left channel and the right channel are determined.

[0031] Optionally, in another possible implementation of the first aspect, determining the target HRTF rendering parameters corresponding to each sub-channel based on the azimuth rendering parameters includes:

[0032] According to the azimuth rendering parameters and the preset HRTF data, the sound source position of each sub-channel is calibrated to generate the target HRTF rendering parameters corresponding to each sub-channel.

[0033] Optionally, in another possible implementation of the first aspect, the rendering parameters further include a center channel enhancement parameter, and the sub-channels include a center channel; accordingly, after performing HRTF rendering on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel to determine the direct sound audio corresponding to each sub-channel, the method further includes:

[0034] The direct sound audio corresponding to the center channel is enhanced according to the center channel enhancement parameters.

[0035] Optionally, in yet another possible implementation of the first aspect, the sub-channels include at least a center channel, a left channel, and a right channel; accordingly, before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the method further includes:

[0036] Crosstalk cancellation is performed on the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel.

[0037] Optionally, in yet another possible implementation of the first aspect, the sub-channels further include a left surround channel and a right surround channel; accordingly, before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the method further includes:

[0038] Crosstalk cancellation is performed on the direct sound audio corresponding to the left surround channel and the direct sound audio corresponding to the right surround channel.

[0039] Optionally, in another possible implementation of the first aspect, before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the process further includes:

[0040] Downmixing is performed based on the direct sound audio corresponding to each sub-channel.

[0041] In a second aspect, an embodiment of the present application provides an audio sound and image optimization device, including: a first acquisition module, used to obtain the audio to be processed and the rendering parameters corresponding to the audio to be processed; a first reverberation module, used to perform reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed; a first crosstalk elimination module, used to perform crosstalk elimination processing on the first reverberation audio to generate an optimized second reverberation audio; a first rendering module, used to perform HRTF rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed; a first fusion module, used to perform fusion processing on the second reverberation audio and the direct sound audio to generate an optimized target audio.

[0042] In a possible implementation of the second aspect, the audio to be processed includes audio data of at least two sub-channels, the rendering parameters include room impulse responses corresponding to each sub-channel, and the first reverberation audio includes first sub-channel reverberation audio corresponding to each sub-channel. Accordingly, the first reverberation module includes:

[0043] The first generating unit is configured to perform convolution processing on the audio data of each sub-channel and the corresponding room impulse response to respectively generate each first sub-channel reverberation audio.

[0044] Optionally, in another possible implementation of the second aspect, the audio type of the audio to be processed is multi-channel audio; accordingly, the apparatus further includes:

[0045] A first downmixing module is configured to perform downmixing on the audio to be processed to generate a left channel audio to be processed and a right channel audio to be processed corresponding to the audio to be processed;

[0046] The first determining module is configured to determine the left channel audio to be processed and the right channel audio to be processed as audio data of each sub-channel included in the audio to be processed.

[0047] Optionally, in yet another possible implementation of the second aspect, the second reverberation audio includes second sub-channel reverberation audio corresponding to each sub-channel; accordingly, the first crosstalk cancellation module includes:

[0048] A first acquiring unit, configured to acquire a crosstalk cancellation filter coefficient corresponding to each sub-channel;

[0049] The first crosstalk cancellation unit is configured to perform crosstalk cancellation processing on each first sub-channel reverberation audio according to a crosstalk cancellation filter coefficient corresponding to each sub-channel, so as to generate optimized reverberation audio for each second sub-channel.

[0050] Optionally, in yet another possible implementation of the second aspect, the rendering parameters further include a crosstalk attenuation factor corresponding to each sub-channel, a crosstalk delay error corresponding to each sub-channel, and a crosstalk cancellation loss function; accordingly, the first acquisition unit is specifically configured to:

[0051] Obtain the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each sub-channel, and use it as the reference frequency corresponding to each sub-channel;

[0052] Determine the crosstalk cancellation order corresponding to each sub-channel according to the reference frequency corresponding to each sub-channel, each crosstalk attenuation factor, each crosstalk delay error and the crosstalk cancellation loss function;

[0053] According to the crosstalk cancellation order corresponding to each sub-channel, the crosstalk cancellation filter coefficient corresponding to each sub-channel is determined. Optionally, in another possible implementation of the second aspect, the above-mentioned device further includes:

[0054] The first attenuation module is configured to perform attenuation processing on the second reverberation audio according to the attenuation coefficient, so as to perform distance control on the second reverberation audio.

[0055] Optionally, in another possible implementation of the second aspect, the audio to be processed includes audio data of at least two sub-channels, and the rendering parameters further include azimuth rendering parameters; accordingly, the first rendering module includes:

[0056] A first determining unit, configured to determine target HRTF rendering parameters corresponding to each sub-channel according to the azimuth rendering parameters;

[0057] The second determining unit is configured to perform HRTF rendering on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel, so as to determine the direct sound audio corresponding to each sub-channel.

[0058] Optionally, in yet another possible implementation of the second aspect, the audio type of the audio to be processed is stereo audio, the sub-channel includes a left channel and a right channel, and the sub-channel audio includes the left channel audio to be processed and the right channel audio to be processed; accordingly, the first rendering module further includes:

[0059] A third determining unit is configured to perform center channel extraction on the left channel audio to be processed and the right channel audio to be processed to determine the center channel audio to be processed corresponding to the audio to be processed;

[0060] The fourth determining unit is configured to determine the left channel audio to be processed, the right channel audio to be processed, and the center channel audio to be processed as audio data of respective sub-channels included in the audio to be processed.

[0061] Optionally, in yet another possible implementation of the second aspect, the azimuth rendering parameter includes a height rendering parameter and a width rendering parameter, and the sub-channels include at least a center channel, a left channel, and a right channel; accordingly, the first determining unit is specifically configured to:

[0062] Determine the target HRTF rendering parameters corresponding to the center channel based on the height rendering parameters;

[0063] According to the width rendering parameter, the target HRTF rendering parameters corresponding to the left channel and the right channel are determined.

[0064] Optionally, in another possible implementation of the second aspect, the first determining unit is further configured to:

[0065] According to the azimuth rendering parameters and the preset HRTF data, the sound source position of each sub-channel is calibrated to generate the target HRTF rendering parameters corresponding to each sub-channel.

[0066] Optionally, in another possible implementation of the second aspect, the rendering parameters further include a center channel enhancement parameter, and the sub-channels include a center channel; accordingly, the first rendering module further includes:

[0067] The first enhancement unit is configured to enhance the direct sound audio corresponding to the center channel according to the center channel enhancement parameter.

[0068] Optionally, in yet another possible implementation of the second aspect, the sub-channels include at least a center channel, a left channel, and a right channel; accordingly, the apparatus further includes:

[0069] The second crosstalk cancellation module is used to perform crosstalk cancellation processing on the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel.

[0070] Optionally, in yet another possible implementation of the second aspect, the sub-channels further include a left surround channel and a right surround channel; accordingly, the apparatus further includes:

[0071] The third crosstalk cancellation module is configured to perform crosstalk cancellation processing on the direct sound audio corresponding to the left surround sound channel and the direct sound audio corresponding to the right surround sound channel.

[0072] Optionally, in another possible implementation of the second aspect, the apparatus further includes:

[0073] The second downmixing module is configured to perform downmixing processing according to the direct sound audio corresponding to each sub-channel.

[0074] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the aforementioned audio and video optimization method when executing the computer program.

[0075] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the audio and video optimization method as described above.

[0076] In a fifth aspect, an embodiment of the present application provides a computer program product, which, when executed on an electronic device, enables the electronic device to execute the audio and video optimization method as described above.

[0077] Compared with the prior art, the embodiments of the present application have the following advantages: reverberation processing and HRTF rendering processing are performed on the processed audio through the reverberation link and the direct sound link respectively, and crosstalk elimination is performed on the reverberation audio, so as to enhance the spatial sense and distance sense of the audio through reverberation processing, and enhance the positioning sense, force sense and clarity of the audio through HRTF rendering processing, and reduce the mutual interference between multiple channels through crosstalk elimination, thereby improving the sound quality of the external audio and improving the user's audio-visual experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0079] FIG1 is a flow chart of an audio and video optimization method according to an embodiment of the present invention;

[0080] FIG2 is a schematic diagram of a setting interface corresponding to audio and video optimization provided by an embodiment of the present application;

[0081] FIG3 is a schematic diagram of a room impulse response provided by an embodiment of the present application;

[0082] FIG4 is a schematic diagram of a human ear positioning principle provided by an embodiment of the present application;

[0083] FIG5 is a schematic diagram of a simplified stereo crosstalk model provided in an embodiment of the present application;

[0084] FIG6 is a diagram showing a relationship between the crosstalk delay Δt and the distance D provided in an embodiment of the present application;

[0085] FIG7 is a schematic diagram showing the relationship between the crosstalk delay Δt and the angle θ according to an embodiment of the present application;

[0086] FIG8 is a schematic diagram showing the relationship between the crosstalk cancellation order and the cancellation error at a specific frequency according to an embodiment of the present application;

[0087] FIG9 is a flow chart of a crosstalk cancellation algorithm for expanding the sweet spot provided by an embodiment of the present application;

[0088] FIG10 is a flow chart of an audio and video optimization method according to another embodiment of the present application;

[0089] FIG11 is a flow chart of an overall solution of an audio and video optimization method provided in yet another embodiment of the present application;

[0090] FIG12 is a schematic structural diagram of an audio and video optimization device provided in an embodiment of the present application;

[0091] FIG13 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0092] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0093] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0094] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0095] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0096] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0097] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0098] The audio and video optimization method, device, electronic device, storage medium and computer program provided in this application are described in detail below with reference to the accompanying drawings.

[0099] FIG1 shows a flow chart of an audio sound image optimization method provided in an embodiment of the present application.

[0100] Step 101: Obtain audio to be processed and rendering parameters corresponding to the audio to be processed.

[0101] It should be noted that the audio and video optimization method of the embodiment of the present application can be performed by the audio and video optimization device of the embodiment of the present application. The audio and video optimization device of the embodiment of the present application can be configured in any electronic device to perform the audio and video optimization method of the embodiment of the present application. For example, the audio and video optimization device of the embodiment of the present application can be configured in electronic devices such as mobile phones, computers, wearable devices, speakers, flat-screen TVs (such as smart screens, etc.) to achieve the user's audio and video experience in various application scenarios.

[0102] The audio to be processed may refer to the audio that currently needs to be played outward through the speaker of the electronic device.

[0103] The rendering parameters may include parameters set by the user when playing the audio to be processed, or may include pre-set default rendering parameters.

[0104] In an embodiment of the present application, when the electronic device obtains audio that needs to be played through a speaker, the audio can be determined as unprocessed audio; and after obtaining the audio to be processed, the corresponding rendering parameters can be obtained to facilitate subsequent rendering processing of the audio to be processed to improve the audio quality of the audio to be processed. For example, when the audio sound and image optimization method of the present application is applied to a mobile phone, if the operating environment of the mobile phone does not have an external playback scene (such as no headphones of any type are connected and the media volume is not 0), then when the user plays music through the audio software, the audio currently played by the audio software can be determined as the audio to be processed; or, when the user plays a video through the video software, the audio corresponding to the video can be determined as the audio to be processed, and so on.

[0105] As a possible implementation method, the rendering parameters used in the embodiment of the present application to optimize the audio and video can be pre-experimented and measured in the usage environment, and selected according to the rendering effect actually required, and stored in the electronic device as default rendering parameters. When it is necessary to optimize the audio and video of the processed audio, the stored default rendering parameters can be directly obtained from the electronic device; or, the user can also be allowed to set the rendering parameters according to his own usage needs, so that the rendering effect after the audio and video optimization of the processed audio can meet the user's personalized usage needs, and thus the corresponding rendering parameters can be determined according to the user's setting operations in the corresponding setting interface for optimizing the audio and video of the processed audio.

[0106] As a possible implementation, the rendering parameters in the embodiments of this application may include only pre-set default rendering parameters, only user-defined rendering parameters, or both. In practice, the type and method of obtaining rendering parameters may be determined based on actual needs and specific application scenarios, and this embodiment of the application does not limit this.

[0107] As a possible implementation method, a control for turning on audio and image optimization can also be provided to the user in the corresponding setting interface, so that the user can choose whether to turn on the audio and image optimization function according to their own needs. In the embodiment of the present application, when the user turns on the audio and image optimization function, the audio to be processed can be processed using the audio and image optimization method of the embodiment of the present application.

[0108] For example, when the rendering parameters include user-set rendering parameters, a setting interface 200 as shown in FIG. 2 may be provided so that the user can set corresponding rendering parameters according to his or her own personalized needs through the setting interface 200 when needed.

[0109] As shown in Figure 2, a schematic diagram of a setting interface 200 corresponding to an audio image optimization provided in an embodiment of the present application is shown, wherein 210 is a spatial audio control, which can be used to turn on or off the audio image optimization function of the present application; 220 is a vocal enhancement control, which can be used to turn on or off the vocal enhancement function in the audio image optimization, and the audio image optimization method of the embodiment of the present application can determine the rendering parameters corresponding to the vocal enhancement according to whether the vocal enhancement control 220 is turned on; 230 is an azimuth rendering control, which can be used to turn on the azimuth rendering function in the audio image optimization; 231 is a height rendering control, 2311 is a sliding component 2311 corresponding to the height rendering control 231, and the height rendering intensity can be adjusted by the position of the sliding component 2311 in the height rendering control 231; 32 is the width rendering control, 2321 is the sliding component 2321 corresponding to the width rendering control 232, and the width rendering intensity can be adjusted by the position of the sliding component 2321 in the width rendering control 232; the audio and video rendering method of the embodiment of the present application can determine the azimuth rendering parameters according to whether the azimuth rendering control 230 is turned on, the position of the sliding component 2311 corresponding to the height rendering control 231, and the position of the sliding component 2321 corresponding to the width rendering control 232. The azimuth rendering parameters can include height rendering parameters and width rendering parameters. The height rendering parameter can be determined according to the position of the sliding component 2311 in the height rendering control 231, and the width rendering parameter can be determined according to the position of the sliding component 2321 in the width rendering control 232.

[0110] As a possible implementation, before subsequent processing is performed on the audio to be processed, a pre-gain may be performed on the audio to be processed so that the amplitude of the audio to be processed does not exceed a processable range in subsequent steps.

[0111] Step 102 : performing reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed.

[0112] Reverberation is a physical phenomenon and an acoustic property. When sound waves propagate indoors, they are reflected by obstacles like walls, ceilings, and floors, and each reflection is absorbed by the obstacle. Consequently, after the sound source ceases, the sound waves undergo multiple reflections and absorptions before eventually disappearing. This is why the human ear perceives a mixture of sound waves persisting for a period of time after the sound source ceases (a phenomenon known as the persistence of sound even after the indoor sound source ceases). This phenomenon is called reverberation, and this period is called the reverberation time.

[0113] It is understandable that the sound waves that the human ear ultimately hears are the result of a series of interactions between the initial sound waves emitted by the sound source and the room, and this interaction process can be represented by the room impulse response. The room impulse responses corresponding to different rooms may be different, and the sound waves that the human ear ultimately hears can be represented by the convolution result of the initial sound waves emitted by the sound source and the room impulse response. It can be seen that if a specific reverberation effect is superimposed on the audio, the human ear will feel the specific sense of space and distance corresponding to the reverberation effect. Therefore, in an embodiment of the present application, the audio to be processed can be reverberated according to the rendering parameters to enhance the sense of space and distance of the audio to be processed.

[0114] As a possible implementation method, the rendering parameters may include a room impulse response corresponding to the reverberation processing, so that when the audio to be processed is subjected to reverberation processing, the room impulse response can be obtained from the rendering parameters, and the audio to be processed is convolved with the room impulse response, and the convolution result is used as the first reverberation audio, so that the first reverberation audio has a better sense of space and distance.

[0115] As shown in Figure 3, this is a schematic diagram of a room impulse response, where the horizontal axis of the coordinate system in Figure 3 represents time (t), in seconds (s), and the vertical axis represents the pulse amplitude. It can be seen that the room impulse response shown in Figure 3 includes the impulse response corresponding to the direct sound and the impulse response corresponding to the reverberation.

[0116] It should be noted that the room impulse response used for reverberation processing can be a room impulse response selected through experiments based on actual needs and specific application scenarios to achieve better reverberation effects for the audio, and stored in the electronic device as a default rendering parameter so that the user can feel a better sense of space and distance when receiving audio with optimized sound and image.

[0117] Furthermore, in actual application scenarios, the audio played in electronic devices is usually stereo audio or multi-channel audio containing multiple channels. Therefore, when performing reverberation processing on the audio to be processed, the audio data of each channel can be reverberated separately to further enhance the reverberation effect. That is, in a possible implementation of the embodiment of the present application, the above-mentioned audio to be processed may include audio data of at least two sub-channels, the above-mentioned rendering parameters may include the room impulse response corresponding to each sub-channel, and the above-mentioned first reverberation audio may include the first sub-channel reverberation audio corresponding to each sub-channel; accordingly, the above-mentioned step 102 may include:

[0118] The audio data of each sub-channel is convolved with the corresponding room impulse response to generate each first sub-channel reverberation audio.

[0119] As a possible implementation method, the room impulse response corresponding to each sub-channel can be determined experimentally as a rendering parameter based on the sub-channels that the audio to be processed may contain in the actual application scenario. Therefore, when performing reverberation processing on the audio to be processed, for a sub-channel, the audio data of the sub-channel can be convolved with the room impulse response corresponding to the sub-channel to generate the first sub-channel reverberation audio corresponding to the sub-channel; and so on, the first sub-channel reverberation audio corresponding to each sub-channel can be generated, thereby completing the reverberation processing of the audio to be processed.

[0120] As an example, when the audio type of the audio to be processed is stereo audio, the audio to be processed may include audio data of two sub-channels, namely, the left channel audio to be processed and the right channel audio to be processed. The above-mentioned rendering parameters may include a first room impulse response corresponding to the left channel and a second room impulse response corresponding to the right channel. The above-mentioned first reverberation audio may include a first left channel reverberation audio and a first right channel reverberation audio. Therefore, the left channel audio to be processed may be convolved with the first room impulse response to generate the first left channel reverberation audio, and the right channel audio to be processed may be convolved with the second room impulse response to generate the first right channel reverberation audio.

[0121] The left channel audio to be processed may refer to the left channel audio contained in the audio to be processed; and the right channel audio to be processed may refer to the right channel audio contained in the audio to be processed.

[0122] The first room impulse response may refer to a room impulse response for performing reverberation processing on the left-channel audio to be processed; and the second room impulse response may refer to a room impulse response for performing reverberation processing on the right-channel audio to be processed.

[0123] Among them, the first left-channel reverberation audio may refer to the reverberation audio generated after the left-channel audio to be processed is subjected to reverberation processing; the first right-channel reverberation audio may refer to the reverberation audio generated after the right-channel audio to be processed is subjected to reverberation processing.

[0124] As a possible implementation method, for stereo audio, it can include audio data of two channels, left channel audio and right channel audio. Therefore, when the audio type of the audio to be processed is stereo audio, the audio to be processed can include left channel audio to be processed and right channel audio to be processed. Therefore, the first room impulse response corresponding to the left channel and the second room impulse response corresponding to the right channel can be preset in advance in the rendering parameters; thus, when the audio to be processed is reverberated, the left channel audio to be processed can be convolved with the first room impulse response, and the convolution result can be used as the first left channel reverberated audio; and the right channel audio to be processed can be convolved with the second room impulse response, and the convolution result can be used as the first right channel reverberated audio. The process of reverberating the left channel audio to be processed and the right channel audio to be processed can be expressed by the following formula: L out1 =L in *I L R out1 =R in *I R

[0125] Among them, L out1 For the first left channel reverberation audio, L in For the left channel audio to be processed, I L is the first room impulse response; R out1 is the first right channel reverberation audio, R in For the right channel audio to be processed, I R is the impulse response of the second room.

[0126] As a possible implementation, multi-channel audio can include not only left and right channels but also audio data from other channels, such as a center channel, left surround channel, and right surround channel. Therefore, when the audio type to be processed is multi-channel audio, the room impulse response for each sub-channel can be preset in the rendering parameters. The audio data of each sub-channel can then be convolved with the corresponding room impulse response to achieve reverberation processing for the audio data of each sub-channel.

[0127] Furthermore, since multi-channel audio typically contains a large amount of channel data, if the audio data of each channel is reverberated separately, not only will the computational complexity of the reverberation be greatly increased, consuming computing resources, but the reverberation effect will not be significantly improved. Therefore, when the audio to be processed is multi-channel audio, the reverberation processing can be performed after downmixing the audio to be processed, so as to reduce the computational complexity of the reverberation and save computing resources while ensuring the reverberation effect. That is, in a possible implementation of the embodiment of the present application, before the above step 102, the following may also be included:

[0128] Downmixing the audio to be processed to generate a left-channel audio to be processed and a right-channel audio to be processed corresponding to the audio to be processed;

[0129] The left channel audio to be processed and the right channel audio to be processed are determined to be audio data of each sub-channel included in the audio to be processed.

[0130] In an embodiment of the present application, only the first room impulse response corresponding to the left channel and the second room impulse response corresponding to the right channel may be preset in the rendering parameters to simplify the reverberation processing process and save computing resources. Therefore, when the audio type of the audio to be processed is stereo audio, that is, the audio to be processed includes audio data of two channels, namely, the left channel audio to be processed and the right channel audio to be processed, the audio data to be processed can be transparently transmitted, that is, the left channel audio to be processed and the right channel audio to be processed can be directly reverberated separately according to the above-mentioned method; when the audio type of the audio to be processed is multi-channel audio, that is, the sub-channels contained in the audio to be processed may include, in addition to the left channel and the right channel, the center channel, the left surround channel, the right surround channel and other channels, so that the downmix coefficients corresponding to each sub-channel in the channel to be processed can be obtained according to the downmix table, and the audio to be processed can be downmixed to convert the audio to be processed from multiple channels into audio data of two channels, and then the left channel data to be processed and the right channel data to be processed generated after downmixing are reverberated separately according to the above-mentioned method of performing reverberation processing on the left channel data to be processed and the right channel data to be processed.

[0131] As an example, assuming that the audio to be processed is 5.1-channel audio, that is, the sub-channels contained in the audio to be processed are: center channel, left channel, right channel, left surround channel and right surround channel, then the audio to be processed can be downmixed using the following formula to generate the left channel data to be processed and the right channel data to be processed corresponding to the audio to be processed:

[0132] Among them, L in The left channel audio to be processed after downmixing, L' in The audio data of the left channel contained in the audio to be processed, C' in Ls' is the center channel audio data contained in the audio data to be processed. in The audio data of the left surround channel included in the audio to be processed; R in The right channel audio to be processed after downmixing, R' in The audio data of the right channel contained in the audio to be processed, Rs' in The audio data of the right surround channel included in the audio to be processed.

[0133] Step 103: Perform crosstalk cancellation processing on the first reverberation audio to generate an optimized second reverberation audio.

[0134] In an embodiment of the present application, since the audio to be processed may contain audio data of multiple sub-channels, there will be crosstalk between the sub-channels, thereby affecting the audio quality. Taking the left and right channels as an example, under ideal conditions, it is expected that the left channel audio is received by the left ear of the person, and the right channel audio is received by the right ear of the person. However, in an actual propagation environment, the left channel audio will also be received by the right ear of the person, and the right channel audio will also be received by the left ear of the person, thereby generating crosstalk between the channels. Therefore, after performing reverberation processing on the audio to be processed and generating a first reverberation audio corresponding to the audio to be processed, the first reverberation audio can also be subjected to crosstalk elimination processing to generate an optimized second reverberation audio, thereby further improving the audio quality of the reverberation audio.

[0135] Furthermore, when the audio to be processed includes audio data from multiple sub-channels, the crosstalk cancellation filter coefficients corresponding to each sub-channel may be first determined, and then crosstalk cancellation processing may be performed on the first reverberated audio based on the crosstalk cancellation filter coefficients corresponding to each sub-channel. That is, in one possible implementation of this embodiment of the present application, step 103 may include:

[0136] Obtain the crosstalk cancellation filter coefficients corresponding to each sub-channel;

[0137] According to the crosstalk cancellation filter coefficient corresponding to each sub-channel, crosstalk cancellation processing is performed on each first sub-channel reverberation audio to generate optimized reverberation audio for each second sub-channel.

[0138] As a possible implementation method, the crosstalk cancellation filter coefficients corresponding to each sub-channel can be pre-configured in the rendering parameters, and then the crosstalk cancellation filter coefficients corresponding to the sub-channel can be matrix multiplied with the first sub-channel reverberation audio corresponding to the sub-channel and the crosstalk channel of the sub-channel to generate an optimized second sub-channel reverberation audio corresponding to the sub-channel.

[0139] It should be noted that the crosstalk channel of a sub-channel may refer to a sub-channel that causes crosstalk to the audio data of that sub-channel. For example, the crosstalk channel of the left channel may be the right channel, while the crosstalk channel of the right channel may be the left channel; the crosstalk channel of the left surround channel may be the right surround channel, while the crosstalk channel of the right surround channel may be the left surround channel.

[0140] As an example, when the sub-channels of the audio to be processed include a left channel and a right channel, the following formula can be used to express the process of performing crosstalk cancellation on the first sub-channel reverberation audio corresponding to the left channel (i.e., the aforementioned first left channel reverberation audio) and the first sub-channel reverberation audio corresponding to the right channel (i.e., the aforementioned first right channel reverberation audio):

[0141] Among them, L out2 For the second left channel reverberation audio, L out1 is the first left channel reverberation audio, F 11 、F 12 is the crosstalk elimination filter coefficient corresponding to the left channel, R out2 is the second right channel reverberation audio, R out1 is the first right channel reverberation audio, F 21 、F 22 is the crosstalk elimination filter coefficient corresponding to the right channel.

[0142] As an example, when the sub-channels of the audio to be processed include a left surround channel and a right surround channel, the same method as the above formula can be used to perform crosstalk cancellation processing on the first sub-channel reverberation audio corresponding to the left surround channel to generate the second sub-channel reverberation audio corresponding to the left surround channel; and the first sub-channel reverberation audio corresponding to the right surround channel can be performed crosstalk cancellation processing to generate the second sub-channel reverberation audio corresponding to the right surround channel. No further details will be given here.

[0143] It should be noted that if the multi-channel audio is downmixed before the first reverberation audio is generated as described above, that is, the first reverberation audio only includes the first left channel reverberation audio and the first right channel reverberation audio, then the second reverberation audio also only includes the second left channel reverberation audio and the second right channel reverberation audio.

[0144] Furthermore, since the crosstalk cancellation signal superimposed on the audio signal during crosstalk cancellation will also generate crosstalk, high-order crosstalk cancellation processing can be performed on the audio signal to further improve the effectiveness of crosstalk cancellation. That is, in a possible implementation of the embodiment of the present application, the above rendering parameters may also include the crosstalk attenuation factor corresponding to each sub-channel, the crosstalk delay error corresponding to each sub-channel, and the crosstalk cancellation loss function; accordingly, the above crosstalk cancellation filter coefficients corresponding to each sub-channel may include:

[0145] Obtain the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each sub-channel, and use it as the reference frequency corresponding to each sub-channel;

[0146] Determine the crosstalk cancellation order corresponding to each sub-channel according to the reference frequency corresponding to each sub-channel, each crosstalk attenuation factor, each crosstalk delay error and the crosstalk cancellation loss function;

[0147] According to the crosstalk cancellation order corresponding to each sub-channel, the crosstalk cancellation filter coefficient corresponding to each sub-channel is determined respectively.

[0148] The following diagrams specifically illustrate the causes of crosstalk generation, the principles of crosstalk cancellation, and the reason why the crosstalk cancellation order needs to be determined in real time in this application:

[0149] As shown in FIG4 , it is a schematic diagram of the principle of ear positioning, where 410 is a speaker, 420 is the user's left ear, 430 is the user's right ear, 440 is the propagation path between the speaker 410 and the left ear 420, 450 is the propagation path between the speaker 410 and the right ear 430, x(t) is the original audio signal emitted by the speaker 410, and h L (t) is the response function of the propagation path 440, h R (t) is the response function of the propagation path 450, x L (t) is the audio signal after the original audio signal x(t) is transmitted to the left ear 420 through the propagation path 440, x R (t) is the audio signal after the original audio signal x(t) is transmitted to the left ear 430 via propagation path 450. The human ear's localization of sound sources relies primarily on interaural time difference, sound intensity difference, and timbre difference. For external stereo speakers (including left and right channel audio), ideally, the left channel audio signal should reach the left ear and the right channel audio signal should reach the right ear. However, due to the existence of crosstalk paths, the left and right channel audio signals are often received by both ears simultaneously. This phenomenon is called audio crosstalk. Taking the left channel audio signal as an example, the signal from the left channel audio signal reaching the right ear is crosstalk sound, and the corresponding path is the crosstalk path.

[0150] It should be noted that, taking the left-channel audio signal as an example, in actual scenarios, the left-channel audio will also be received by the user's right ear, thereby causing crosstalk to the right-channel audio signal. Therefore, if the right-channel audio signal is to be crosstalk eliminated, the left-channel audio signal needs to be inverted, attenuated, and delayed to generate an equal-amplitude inverted signal of the left-channel crosstalk sound at the right ear. This equal-amplitude inverted signal is then superimposed on the right-channel signal to cancel the crosstalk sound of the left-channel audio signal at the right ear, thereby achieving first-order crosstalk elimination of the right-channel audio signal. Since the inverted signal itself will also cause crosstalk, the above process can be iterated to achieve inverted cancellation of second-order and higher-order crosstalk sounds.

[0151] As shown in Figure 5, it is a schematic diagram of a simplified stereo crosstalk model, where l LLis the propagation path from the left speaker 510 to the left ear 530, l LR is the crosstalk path from the left speaker 510 to the right ear 540, l RR is the propagation path from the right speaker 520 to the right ear 540, l RL is the crosstalk path from the right speaker 520 to the left ear 530. The distance between the speakers is 2K, the width of the head is 2R, the vertical distance between the head and the speaker is D, and the angle between the head and the speaker is θ. The crosstalk delay t can be determined by the following formula:

[0152] Where c is the speed of sound waves in air.

[0153] However, the user's head position is not fixed when watching or listening to music, and the viewing or listening distance of different users varies from person to person. Therefore, the delay t is not a constant value, and the propagation path l RR and crosstalk path l LR Therefore, if a fixed delay time t, a fixed attenuation ratio, and a fixed crosstalk order are used to cancel crosstalk on the audio signal, the effect of crosstalk cancellation will be greatly reduced as long as the user's head position changes slightly.

[0154] As shown in Figures 6 and 7, Figure 6 shows the relationship between the crosstalk delay Δt and distance D, with the horizontal axis representing distance D in meters and the vertical axis representing delay Δt in milliseconds. Figure 7 shows the relationship between the crosstalk delay Δt and angle θ, with the horizontal axis representing angle θ in degrees (degrees) and the vertical axis representing delay Δt in milliseconds. As shown in Figure 6, when distance D is in the range of 0.4 to 0.6 meters, the crosstalk delay fluctuates by approximately 55 milliseconds; as shown in Figure 7, when angle θ is in the range of 85 to 95 degrees, the crosstalk delay fluctuates by 140 milliseconds. This indicates that the crosstalk delay fluctuates significantly, and this fluctuation can cause severe spectral coloration at high frequencies, significantly impacting the effectiveness of crosstalk cancellation.

[0155] Therefore, through the above analysis, in the embodiment of the present application, in order to reduce the impact of changes in the user's head position on the crosstalk cancellation effect, the order of crosstalk cancellation can be adjusted to maximize the stability of crosstalk cancellation, expand the sweet spot of crosstalk cancellation, and reduce spectral coloration, so that when the user's head position changes, the effect of crosstalk cancellation will not be reduced.

[0156] The following is a detailed description of the method for determining the crosstalk cancellation order and the crosstalk cancellation filter coefficients in the embodiment of the present application:

[0157] The crosstalk channel corresponding to a sub-channel may refer to a sub-channel that will cause crosstalk to the audio data of that sub-channel. For example, the crosstalk channel corresponding to the left channel may be the right channel, and correspondingly, the crosstalk channel corresponding to the right channel may be the left channel. For another example, the crosstalk channel corresponding to the left surround channel may be the right surround channel, and correspondingly, the crosstalk channel corresponding to the right surround channel may be the left surround channel.

[0158] The crosstalk attenuation factor corresponding to the sub-channel may refer to the ratio of the attenuation of the audio signal on the propagation path of the sub-channel to the attenuation of the crosstalk channel corresponding to the sub-channel on the crosstalk path. For example, taking Figure 5 as an example, assuming that the sub-channel is the right channel and the crosstalk channel is the left channel, the crosstalk attenuation factor corresponding to the right channel is the attenuation of the audio signal on the propagation path l. RR The attenuation on the crosstalk path l LR The ratio of the attenuation on .

[0159] The crosstalk delay error corresponding to a sub-channel can refer to a pre-set fluctuation range of the crosstalk delay of the crosstalk channel corresponding to the sub-channel used during crosstalk cancellation. For example, using Figure 5 as an example, assuming the sub-channel is the right channel and the crosstalk channel is the left channel, and the crosstalk delay of the left channel on the right channel at the current fixed position is t, when the distance between the user's head and the left and right speakers fluctuates within a certain range, the corresponding variation range of the crosstalk delay t is set to [t-Δt / 2, t+Δt / 2], then the crosstalk delay error corresponding to the right channel is Δt.

[0160] It should be noted that the specific values ​​of the crosstalk attenuation factors corresponding to each sub-channel are not limited in the embodiments of the present application. In actual use, according to actual needs and specific application scenarios, a statistical analysis can be performed on the distance and angle between the user's head and the left and right speakers of the electronic device when the user uses the electronic device to determine the distance fluctuation range and angle fluctuation range between the user's head and the left and right speakers. Then, based on the distance fluctuation range and angle fluctuation range, the appropriate crosstalk attenuation factor and crosstalk delay error corresponding to each sub-channel are selected and stored in the electronic device as default rendering parameters.

[0161] As one possible implementation, since changes in the user's head position significantly affect the effectiveness of crosstalk cancellation in the high-frequency portion of the audio signal, a crosstalk cancellation loss function can be preset with the audio signal frequency, crosstalk attenuation factor, crosstalk delay error, and crosstalk cancellation order as variables. A statistical analysis of the distance and angle between the user's head and the speaker in actual usage scenarios is then performed to select an appropriate crosstalk attenuation factor and crosstalk delay error as default rendering parameters for each sub-channel. Before performing crosstalk cancellation on the first sub-channel reverberation audio corresponding to each sub-channel, the frequency of each first sub-channel reverberation audio can be determined, and the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each sub-channel can be determined as the reference frequency corresponding to that sub-channel. Subsequently, for each sub-channel, the reference frequency, crosstalk attenuation factor, crosstalk delay error, and different crosstalk cancellation orders corresponding to that sub-channel can be substituted into the crosstalk cancellation loss function to obtain the crosstalk cancellation order that minimizes the value of the crosstalk cancellation loss function, which is then determined as the crosstalk cancellation order corresponding to that sub-channel. Finally, the crosstalk cancellation filter coefficient corresponding to the sub-channel can be generated according to the crosstalk cancellation order corresponding to the sub-channel, and then the crosstalk cancellation filter coefficient corresponding to the sub-channel can be used to fuse the first sub-channel reverberation audio of the sub-channel with the first sub-channel reverberation audio of the crosstalk channel to generate the second sub-channel reverberation audio corresponding to the sub-channel.

[0162] As an example, the crosstalk cancellation loss function can be expressed as follows:

[0163] Where Obj is the crosstalk cancellation loss function, k is the crosstalk cancellation order, f is the reference frequency, α is the crosstalk attenuation factor, and Δt is the crosstalk delay error.

[0164] As an example, if the sub-channel includes a left channel and a right channel, that is, the first reverberation audio includes the first left channel reverberation audio and the first right channel reverberation audio, then the reference frequency corresponding to the left channel (that is, the frequency of the first right channel reverberation audio), the crosstalk attenuation factor, and the crosstalk delay error can be substituted into the above formula to determine the crosstalk cancellation order corresponding to the left channel; correspondingly, the reference frequency corresponding to the right channel (that is, the frequency of the first left channel reverberation audio), the crosstalk attenuation factor, and the crosstalk delay error can be substituted into the above formula to determine the crosstalk cancellation order corresponding to the right channel, and then the crosstalk cancellation filter coefficient matrix can be determined according to the crosstalk cancellation orders corresponding to the left channel and the crosstalk cancellation orders corresponding to the right channel. Furthermore, crosstalk cancellation can be performed on the first left channel reverberation audio and the first right channel reverberation audio according to the crosstalk cancellation filter coefficient matrix, thereby generating a second left channel reverberation audio corresponding to the first left channel reverberation audio, and generating a second right channel reverberation audio corresponding to the first right channel reverberation audio.

[0165] As an example, if the sub-channels also include a left surround channel and a right surround channel, that is, the first reverberation audio includes the first left surround channel reverberation audio and the first right surround channel reverberation audio, then the reference frequency corresponding to the left surround channel (that is, the frequency of the first right surround channel reverberation audio), the crosstalk attenuation factor, and the crosstalk delay error can be substituted into the above formula to determine the crosstalk cancellation order corresponding to the left surround channel. Correspondingly, the reference frequency corresponding to the right surround channel (that is, the frequency of the first left surround channel reverberation audio), the crosstalk attenuation factor, and the crosstalk delay error can be substituted into the above formula to determine the crosstalk cancellation order corresponding to the right surround channel. Then, based on the crosstalk cancellation orders corresponding to the left surround channel and the right surround channel, a crosstalk cancellation filter coefficient matrix is ​​determined. Then, based on the crosstalk cancellation filter coefficient matrix, crosstalk cancellation can be performed on the first left surround channel reverberation audio and the first right surround channel reverberation audio, thereby generating a second left surround channel reverberation audio corresponding to the first left surround channel reverberation audio, and generating a second right surround channel reverberation audio corresponding to the first right surround channel reverberation audio.

[0166] For example, as shown in Figure 8, a schematic diagram of the changing relationship between the crosstalk cancellation order and the cancellation error at a specific frequency is shown, where curve 810 shows the changing relationship between the crosstalk cancellation order and the cancellation error (the value of the crosstalk cancellation loss function) when a crosstalk delay error exists, and curve 820 shows the changing relationship between the crosstalk cancellation order and the cancellation error under ideal conditions, where the ideal condition refers to the positional relationship between the user's head and the speaker remaining fixed. As can be seen from Figure 8, at this specific frequency, the audio data is sampled at a sampling rate of 48K, and with a 4-sample error (Δt is 4 sampling points), the crosstalk cancellation order that minimizes the cancellation error is 3, which means that the crosstalk cancellation order for the audio data can be determined to be 3.

[0167] As a possible implementation method, since an audio signal is usually a composite signal of multiple frequency signals, that is, an audio signal usually has a certain bandwidth, and different crosstalk cancellation orders are applicable to different audio signal frequencies, when determining the filter coefficient corresponding to a sub-channel, a preset divider can be used to perform frequency division processing on the first sub-channel reverberation audio of the crosstalk channel corresponding to the sub-channel to divide the bandwidth of the first sub-channel reverberation audio of the crosstalk channel into N frequency bands. For each frequency band, the maximum frequency value within the frequency band can be used as a reference frequency and substituted into the above-mentioned crosstalk cancellation loss function to determine the crosstalk cancellation order of the first sub-channel reverberation audio within the frequency band. Then, based on the crosstalk cancellation orders of the first sub-channel reverberation audio of the crosstalk channel within each frequency band, the filter coefficient corresponding to the sub-channel is determined, thereby further expanding the sweet spot of crosstalk cancellation and further improving the effectiveness and stability of crosstalk cancellation.

[0168] As shown in FIG9 , it is a flow chart of a crosstalk cancellation algorithm for expanding the sweet spot. Assuming that the sub-channels of the audio to be processed include a left channel and a right channel, that is, the first reverberation audio includes a first left-channel reverberation audio and a first right-channel reverberation audio, L in FIG9 represents the processing chain corresponding to the left channel, and R represents the processing chain corresponding to the right channel, then the first left-channel reverberation audio can be input into the processing chain corresponding to the left channel in FIG9 , and according to the frequency range of the first left-channel reverberation audio, the first left-channel reverberation audio is divided into N frequency bands by a frequency divider (that is, the first left-channel reverberation audio is separated into N audio data according to the frequency), where N is an integer greater than or equal to 1; then, according to the maximum value of the N frequency bands corresponding to the first left-channel reverberation audio, the crosstalk cancellation orders corresponding to the N frequency bands are respectively determined (the crosstalk cancellation order of the j-th frequency band can be expressed as i j , where j is an integer greater than or equal to 1 and less than or equal to N); then, the audio data corresponding to the j-th frequency band is subjected to i-interference according to the crosstalk cancellation order corresponding to the j-th frequency band. j Finally, the crosstalk cancellation results of the N frequency bands are fused. Similarly, according to the same process as above, the first right channel reverberation audio is input into the processing chain corresponding to the right channel in Figure 9, and the crosstalk cancellation result corresponding to the first right channel reverberation audio can be obtained. After the first left channel reverberation audio and the second right channel reverberation audio are processed according to the process shown in Figure 9, the crosstalk cancellation filter coefficients corresponding to the left channel and the right channel can be generated.

[0169] It should be noted that if the sub-channels of the audio to be processed also include other channels besides the left channel and the right channel, the audio data of the other channels can also be processed through the process shown in Figure 9 to generate crosstalk elimination filter coefficients corresponding to the other channels. This embodiment of the present application does not limit this.

[0170] Step 104 : Perform HRTF rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed.

[0171] In an embodiment of the present application, after reverberation processing and crosstalk elimination are performed on the audio to be processed through the aforementioned steps, reverberation audio with better effect is generated, thereby improving the spatial sense and distance sense of the audio; however, since the audio ultimately received by the human ear includes two parts, direct sound and reverberation, HRTF rendering can also be performed on the audio to be processed to improve the audio quality of the direct sound, so that the user can experience a better sense of force, positioning and clarity through the direct sound audio corresponding to the audio to be processed, making the audio more layered.

[0172] As a possible implementation method, preset HRTF rendering parameters can be generated based on the sound source position that the user is expected to perceive, and stored in the electronic device as default rendering parameters. Therefore, when HRTF rendering is performed on the audio to be processed, the preset HRTF rendering parameters are directly obtained to perform HRTF rendering on the audio to be processed to generate direct sound audio corresponding to the audio to be processed.

[0173] As a possible implementation method, users can also set the sound source position they expect to feel according to their own needs, and can determine HRTF rendering parameters based on the user's setting information obtained, and perform HRTF rendering processing on the audio to be processed based on the determined HRTF rendering parameters to generate direct sound audio corresponding to the audio to be processed.

[0174] For example, a setting interface 200 as shown in Figure 2 can be provided to the user, so that the user can set the azimuth rendering parameters through the azimuth rendering control 230, height rendering control 231, and width rendering control 232 in the setting interface 200, and then determine the HRTF rendering parameters based on the set azimuth rendering parameters to perform HRTF rendering processing on the audio to be processed.

[0175] Step 105 : Fusing the second reverberation audio and the direct sound audio to generate an optimized target audio.

[0176] In an embodiment of the present application, after the processed audio is subjected to reverberation processing and HRTF rendering processing, the obtained second reverberation audio can be fused with the direct sound audio to generate an optimized target audio, thereby enhancing the spatial sense and sense of distance of the target audio, and enhancing the sense of force, positioning and clarity of the target audio, making the audio more layered.

[0177] Furthermore, in order to make the user feel a better sense of distance, the second reverberation audio may be subjected to wrestling processing to enhance the sense of distance of the audio. That is, in a possible implementation of the embodiment of the present application, before the above step 105, the following steps may also be included:

[0178] The second reverberation audio is attenuated according to the attenuation coefficient to perform distance control on the second reverberation audio.

[0179] As one possible implementation, since audio signals experience different attenuation over different propagation distances, the greater the propagation distance, the greater the attenuation of the audio signal. Therefore, in this embodiment of the present application, an attenuation coefficient can be set based on the desired user perception of distance. The attenuation coefficient can then be used to attenuate the second reverberation audio to achieve distance control of the second reverberation audio. The attenuated second reverberation audio can then be fused with the direct sound audio to achieve a better sense of distance for the resulting target audio.

[0180] As a possible implementation manner, after the second reverberation audio is fused with the direct sound audio to generate the target audio, post-gain processing may be performed on the target audio.

[0181] The audio image optimization method provided in the embodiment of the present application performs reverberation processing on the processed audio according to the rendering parameters to generate a first reverberation audio corresponding to the processed audio, and performs crosstalk elimination processing on the first reverberation audio to generate an optimized second reverberation audio. Then, according to the rendering parameters, the processed audio is rendered by HRTF to determine the direct sound audio corresponding to the processed audio, and then the second reverberation audio and the direct sound audio are fused to generate the optimized target audio. Thus, the processed audio is subjected to reverberation processing and HRTF rendering processing respectively through the reverberation link and the direct sound link, and crosstalk elimination is performed on the reverberation audio, so as to enhance the spatial sense and sense of distance of the audio through the reverberation processing, enhance the sense of positioning, sense of force and clarity of the audio through the HRTF rendering processing, and reduce the mutual interference between multiple channels through crosstalk elimination, thereby improving the sound quality of the external audio and improving the user's audio and video experience.

[0182] The audio image optimization method provided in the embodiment of the present application is further described below with reference to FIG10 .

[0183] FIG10 shows a flow chart of another audio image optimization method provided in an embodiment of the present application.

[0184] As shown in FIG10 , the audio image optimization method includes the following steps:

[0185] Step 1001: Acquire audio to be processed and rendering parameters corresponding to the audio to be processed, wherein the audio to be processed includes audio data of at least two sub-channels, and the rendering parameters include azimuth rendering parameters.

[0186] The sub-channels may include a center channel, a left channel, a right channel, a left surround channel, a right surround channel, etc. In actual use, the type of sub-channels included in the audio to be processed is related to the audio type of the audio to be processed, and this embodiment of the application does not limit this.

[0187] The azimuth rendering parameter may refer to a parameter representing the virtual source position corresponding to the processed audio. It should be noted that the virtual source position refers to the desired sound source position of the processed audio as perceived by the user; that is, after rendering the processed audio according to the azimuth rendering parameter, the user will perceive the rendered audio as emanating from the virtual source position.

[0188] The specific implementation process and principle of the above step 1001 can be referred to the detailed description of the above embodiment and will not be repeated here.

[0189] Step 1002 : performing reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed.

[0190] Step 1003: Perform crosstalk cancellation processing on the first reverberation audio to generate an optimized second reverberation audio.

[0191] The specific implementation process and principles of the above steps 1002-1003 can be referred to the detailed description of the above embodiment and will not be repeated here.

[0192] Step 1004: Determine target HRTF rendering parameters corresponding to each sub-channel according to the azimuth rendering parameters.

[0193] Among them, the HRTF rendering parameters corresponding to the sub-channel may refer to the rendering parameters used when performing HRTF rendering on the audio data of the sub-channel; the physical meaning of the HRTF rendering parameters is the filter matrix corresponding to the transmission of audio from any position in space to the human ear.

[0194] In the embodiment of the present application, since the audio data of different channels can enable the user to perceive different azimuth information of the sound source, for example, the audio data of the left and right channels can enable the user to perceive the width azimuth information of the sound source, and the audio data of the center channel can enable the user to perceive the height azimuth information of the sound source, the audio data of each sub-channel can be HRTF rendered separately so that the audio data of each sub-channel can respectively carry different azimuth information of the virtual source position. Therefore, the target HRTF rendering parameters corresponding to each sub-channel can be determined separately based on the correspondence between the azimuth rendering parameters and each sub-channel.

[0195] Furthermore, when the sub-channels include a center channel, a left channel, and a right channel, the height of the sound source can be represented by the audio data of the center channel, and the width of the sound source can be represented by the audio data of the left and right channels. That is, in one possible implementation of this embodiment of the present application, the above-mentioned azimuth rendering parameters may include a height rendering parameter and a width rendering parameter, and the above-mentioned sub-channels include at least a center channel, a left channel, and a right channel; accordingly, the above-mentioned step 1004 may include:

[0196] Determine the target HRTF rendering parameters corresponding to the center channel based on the height rendering parameters;

[0197] According to the width rendering parameter, the target HRTF rendering parameters corresponding to the left channel and the right channel are determined.

[0198] The height rendering parameter may be used to indicate the pitch angle range of the virtual source position corresponding to the audio to be processed.

[0199] The width rendering parameter may be used to indicate the horizontal azimuth angle range of the virtual source position corresponding to the audio to be processed.

[0200] As a possible implementation method, the HRTF filter matrix of the position corresponding to the pitch angle range included in the height rendering parameter can be determined, and the HRTF filter matrix can be determined as the target HRTF rendering parameter corresponding to the center channel; and the HRTF filter matrix of the position corresponding to the horizontal azimuth angle range included in the width rendering parameter can be determined, and the HRTF filter matrix can be determined as the target HRTF rendering parameter corresponding to the left channel and the right channel.

[0201] It should be noted that when the audio to be processed is multi-channel data, the sub-channels of the audio to be processed may include not only the center channel, left channel and right channel, but also the left surround channel, right surround channel and other channels. In this case, the target HRTF rendering parameters corresponding to the left surround channel, right surround channel and other channels can also be determined based on the width rendering parameters.

[0202] Furthermore, the embodiment of the present application can also pre-measure the HRTF filter matrices corresponding to multiple positions in space through experiments, and then generate HRTF rendering parameters corresponding to the actual azimuth rendering parameters based on the preset HRTF data. That is, in a possible implementation of the embodiment of the present application, the above step 1004 may include:

[0203] According to the azimuth rendering parameters and the preset HRTF data, the sound source position of each sub-channel is calibrated to generate the target HRTF rendering parameters corresponding to each sub-channel.

[0204] The preset HRTF data may include HRTF filter matrices corresponding to multiple spatial positions on a sphere centered on the human head. It should be noted that the preset HRTF data may include HRTF filter matrices corresponding to a large number of spatial positions, so that HRTF rendering parameters for a large number of spatial positions can be directly obtained through the preset HRTF data.

[0205] As a possible implementation method, since the preset HRTF data does not and does not need to include rendering parameters for an infinite number of spatial positions, the preset HRTF data may not include the HRTF filter matrix of the sound source position corresponding to the azimuthal rendering parameters. Therefore, if the HRTF filter matrix of the spatial position adjacent to the sound source position in the preset HRTF data is directly determined as the target HRTF rendering parameter, the inaccurate target HRTF rendering parameter will cause an acoustic parallax effect. Therefore, in an embodiment of the present application, if the preset HRTF data may not include the HRTF filter matrix of the sound source position, the adjacent spatial position of the sound source position in the preset HRTF data can be determined by the azimuthal rendering parameters and the preset HRTF data, and then interpolation is performed in the preset HRTF data according to the HRTF filter matrices of each adjacent spatial position to generate the HRTF filter matrix corresponding to the sound source position, that is, to generate the target HRTF rendering parameters corresponding to each sub-channel, thereby achieving calibration of the sound source position of each sub-channel, expanding the sweet spot of HRTF rendering, and improving the accuracy and stability of HRTF rendering.

[0206] It should be noted that when determining the target HRTF rendering parameters corresponding to the center channel based on the height rendering parameters, and determining the target HRTF rendering parameters corresponding to the left channel and the right channel based on the width rendering parameters, the above-mentioned preset HRTF data can also be used to calibrate the sound source position to generate more accurate target HRTF rendering parameters for each sub-channel.

[0207] Furthermore, when the audio type of the audio to be processed is stereo audio, since the stereo audio only includes left-channel audio and right-channel audio, the audio to be processed can be upmixed before HRTF rendering is performed on the audio to be processed to achieve HRTF rendering of the audio to be processed. That is, in a possible implementation of the embodiment of the present application, before the above step 1004, the following may also be included:

[0208] Perform center channel extraction on the left channel audio to be processed and the right channel audio to be processed to determine the center channel audio to be processed corresponding to the audio to be processed;

[0209] The left channel audio to be processed, the right channel audio to be processed, and the center channel audio to be processed are determined to be audio data of respective sub-channels included in the audio to be processed.

[0210] As a possible implementation method, when the audio to be processed is stereo audio, principal component analysis can be performed on the left channel audio and the right channel audio to be processed of the audio to be processed to determine the main eigenvector corresponding to the audio to be processed, and then the center channel extraction is performed on the audio to be processed based on the main eigenvector to determine the center channel audio to be processed corresponding to the audio to be processed, and then the left channel audio to be processed, the right channel audio to be processed, and the generated center channel audio to be processed contained in the audio to be processed can be used as audio data of each sub-channel for subsequent HRTF rendering processing.

[0211] As an example, the center channel can be extracted from the processed audio using the following formula:

[0212] in, is the center channel audio to be processed, For the audio to be processed, is the main eigenvector corresponding to the audio to be processed, For ambient sound.

[0213] Step 1005 : Perform HRTF rendering on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel to determine the direct sound audio corresponding to each sub-channel.

[0214] In an embodiment of the present application, after determining the target HRTF rendering parameters corresponding to each sub-channel, HRTF rendering processing can be performed on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel, thereby generating direct sound audio corresponding to each sub-channel.

[0215] Furthermore, since the audio data of the center channel mainly contains human voices, the direct sound of the center channel can also be enhanced to further improve the clarity of the audio. That is, in a possible implementation of the embodiment of the present application, after the above step 1005, the following steps may also be included:

[0216] The direct sound audio corresponding to the center channel is enhanced according to the center channel enhancement parameters.

[0217] As a possible implementation method, the center sound enhancement parameter can be a default rendering parameter determined in advance based on experiments, so that when the direct sound audio corresponding to the center channel is enhanced, the preset center sound enhancement parameter can be used to enhance the direct sound audio corresponding to the center channel to a preset level.

[0218] As a possible implementation method, the center sound enhancement parameters can also be set by the user according to actual needs. For example, the user can be allowed to determine whether to turn on vocal enhancement through the vocal enhancement control 220 in the setting interface 200 as shown in Figure 2; if the user does not turn on vocal enhancement, the direct sound audio corresponding to the center channel can be not enhanced; if the user turns on vocal enhancement, the center channel enhancement parameters can be used to enhance the direct sound audio corresponding to the center channel, so that the clarity of the audio can meet the user's current actual usage needs.

[0219] Step 1006 : Fusing the second reverberation audio and the direct sound audio to generate an optimized target audio.

[0220] In an embodiment of the present application, after the audio to be processed is subjected to reverberation processing and HRTF rendering processing, the second reverberation audio generated after the reverberation processing and the direct sound audio generated after the HRTF rendering processing can be fused to generate an optimized target audio.

[0221] Furthermore, to further improve audio quality, crosstalk cancellation can be performed on the direct sound generated after HRTF rendering. That is, in one possible implementation of this embodiment of the present application, the sub-channels may include at least a center channel, a left channel, and a right channel; accordingly, before step 1006, the following steps may also be performed:

[0222] Crosstalk cancellation is performed on the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel.

[0223] It should be noted that the method of performing crosstalk elimination processing on the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel, as well as the method of determining the crosstalk elimination filter coefficients, are the same as the method of performing crosstalk elimination processing on the left channel data to be processed and the right channel data to be processed in the processed audio, as well as the method of determining the crosstalk elimination filter coefficients in the above-mentioned embodiments. The specific implementation process and principles can be referred to the detailed description of the above-mentioned embodiments, and will not be repeated here.

[0224] Furthermore, when the audio to be processed is multi-channel data, the sub-channels in the audio to be processed may also include other channels such as the left surround channel and the right surround channel. Therefore, crosstalk cancellation processing may also be performed on the direct sound audio corresponding to the left surround channel and the right surround channel. That is, in one possible implementation of this embodiment of the present application, the sub-channels may also include the left surround channel and the right surround channel. Accordingly, before step 1006, the following steps may also be performed:

[0225] Crosstalk cancellation is performed on the direct sound audio corresponding to the left surround channel and the direct sound audio corresponding to the right surround channel.

[0226] It should be noted that the method of performing crosstalk elimination processing on the direct sound audio corresponding to the left surround channel and the direct sound audio corresponding to the right surround channel, as well as the method of determining the crosstalk elimination filter coefficients, are the same as the method of performing crosstalk elimination processing on the left channel data to be processed and the right channel data to be processed in the processed audio, as well as the method of determining the crosstalk elimination filter coefficients in the above-mentioned embodiments. The specific implementation process and principles can be referred to the detailed description of the above-mentioned embodiments, and will not be repeated here.

[0227] It is understandable that, since ideally, both ears of the user are expected to obtain the audio data of the center channel, there is no crosstalk channel in the center channel. Therefore, crosstalk cancellation processing may not be performed on the direct sound audio corresponding to the center channel.

[0228] Furthermore, if the sub-channels included in the second reverberation audio are different from the sub-channels included in the direct sound audio, the channels of the two may be unified before the fusion process. That is, in one possible implementation of the embodiment of the present application, before step 1006, the following steps may also be included:

[0229] Downmixing is performed based on the direct sound audio corresponding to each sub-channel.

[0230] As a possible implementation, if the audio to be processed is multi-channel data including not only left and right channels, and the audio to be processed is downmixed before being reverberated, that is, the second reverberated audio only includes the second left channel reverberated audio and the second right channel reverberated audio; in this case, the direct sound audio generated after the HRTF rendering processing of the audio to be processed includes not only the direct sound audio corresponding to the left and right channels, but also the direct sound audio corresponding to other sub-channels. Therefore, the direct sound audio corresponding to each sub-channel can be downmixed to convert the direct sound audio corresponding to each sub-channel into the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel; then, the second left channel reverberated audio can be fused with the direct sound audio corresponding to the left channel to generate the target left channel audio, and the second right channel reverberated audio can be fused with the direct sound audio corresponding to the right channel to generate the target right channel audio, that is, the optimized target audio includes the target left channel audio and the target right channel audio.

[0231] As a possible implementation method, if the audio type of the audio to be processed is stereo audio, that is, the audio to be processed includes a left channel and a right channel, and the second reverberation audio includes a second left channel reverberation audio and a second right channel reverberation audio; in this case, if the center channel is extracted from the audio to be processed before HRTF rendering is performed on the audio to be processed, the direct sound audio generated after HRTF rendering may include the direct sound audio corresponding to the center channel, the direct sound audio corresponding to the left channel, and the direct sound audio corresponding to the right channel; thus, in this case, the direct sound audio corresponding to the center channel, the left channel, and the right channel may also be downmixed to generate the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel; thereafter, the second left channel reverberation audio may be fused with the direct sound audio corresponding to the left channel to generate the target left channel audio, and the second right channel reverberation audio may be fused with the direct sound audio corresponding to the right channel to generate the target right channel audio, that is, the optimized target audio includes the target left channel audio and the target right channel audio.

[0232] The audio image optimization method provided in the embodiment of the present application performs reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed, and performs crosstalk cancellation processing on the first reverberation audio to generate an optimized second reverberation audio. Then, according to the azimuth rendering parameters, the target HRTF rendering parameters corresponding to each sub-channel are determined, and according to the target HRTF rendering parameters corresponding to each sub-channel, HRTF rendering is performed on the audio data of each sub-channel respectively to determine the direct sound audio corresponding to each sub-channel, and then the second reverberation audio and the direct sound audio are fused to generate the optimized target audio. Therefore, the audio to be processed is subjected to reverberation processing and HRTF rendering processing respectively through the reverberation link and the direct sound link, so as to enhance the spatial sense and distance sense of the audio through reverberation processing, and each sub-channel is rendered differently through the azimuth rendering parameters, and the HRTF rendering parameters of each sub-channel are rendered, thereby reducing the acoustic parallax effect of HRTF rendering, expanding the sweet spot of HRTF rendering, further improving the effectiveness and stability of HRTF rendering processing, further improving the sense of positioning, strength and clarity of the audio, and reducing the mutual interference between multiple channels by eliminating crosstalk between the reverberation audio and the direct sound audio after HRTF rendering, thereby further improving the sound quality of the external audio and further improving the user's audio and video experience.

[0233] The audio image optimization method provided in the embodiment of the present application is further described below with reference to FIG11 .

[0234] FIG11 shows a flow chart of an overall solution of an audio and video optimization method provided in an embodiment of the present application.

[0235] As shown in Figure 11, C, L, R, Ls and Rs in Figure 11 represent the center channel, left channel, right channel, left surround channel and right surround channel respectively. When the input audio to be processed is obtained, the audio to be processed can be pre-gained first, and then the audio to be processed after gain is reverberated and HRTF rendered through the reverberation link and the direct sound link respectively. In the reverberation link, if the audio to be processed is stereo audio, the audio to be processed can be transmitted through it; if the audio to be processed is multi-channel audio (such as 5.1-channel audio including center channel, left channel, right channel, left surround channel and right surround channel), the audio to be processed can be down-mixed; then the audio after transmission or down-mixing is subjected to reverberation rendering, crosstalk elimination and distance control in sequence to generate reverberant audio. In the direct sound link, if the audio to be processed is stereo audio, the stereo audio is upmixed into three-channel audio including the center channel, left channel, and right channel; if the audio to be processed is multi-channel audio (such as 5.1-channel audio), the audio to be processed is transparently transmitted; then, HRTF calibration and HRTF rendering are performed on each channel of the transparently transmitted or upmixed audio, and crosstalk elimination is performed on the left and right channel audio and the left and right surround channel audio after HRTF rendering, and the center channel audio after HRTF rendering is transparently transmitted; then, the audio of each channel after crosstalk elimination is downmixed. The reverberation audio output by the reverberation link and the direct sound audio output by the direct sound link are then fused; finally, the fused audio is post-gained and the audio generated after post-gain is output as the target audio.

[0236] It should be noted that other specific implementation processes and principles of the embodiments of the present application can refer to the detailed description of the above embodiments and will not be repeated here.

[0237] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0238] Corresponding to the audio and sound image optimization method described in the above embodiment, Figure 12 shows a structural block diagram of the audio and sound image optimization device provided in the embodiment of the present application. For the sake of convenience, only the parts related to the embodiment of the present application are shown.

[0239] 12 , the apparatus 1200 includes:

[0240] The first acquisition module 1201 is used to obtain the audio to be processed and the rendering parameters corresponding to the audio to be processed;

[0241] A first reverberation module 1202 is configured to perform reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed;

[0242] A first crosstalk cancellation module 1203 is configured to perform crosstalk cancellation processing on the first reverberation audio to generate an optimized second reverberation audio;

[0243] A first rendering module 1204 is configured to perform HRTF rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed;

[0244] The first fusion module 1205 is configured to perform fusion processing on the second reverberation audio and the direct sound audio to generate an optimized target audio.

[0245] In actual use, the audio and sound image optimization device provided in the embodiment of the present application can be configured in any electronic device to execute the aforementioned audio and sound image optimization method.

[0246] The audio sound image optimization device provided in the embodiment of the present application performs reverberation processing on the processed audio according to the rendering parameters to generate a first reverberation audio corresponding to the processed audio, and performs crosstalk elimination processing on the first reverberation audio to generate an optimized second reverberation audio. Then, according to the rendering parameters, the processed audio is subjected to HRTF rendering to determine the direct sound audio corresponding to the processed audio, and then the second reverberation audio and the direct sound audio are fused to generate an optimized target audio. Thus, the processed audio is subjected to reverberation processing and HRTF rendering processing respectively through the reverberation link and the direct sound link, and crosstalk elimination is performed on the reverberation audio, so as to enhance the spatial sense and sense of distance of the audio through the reverberation processing, enhance the sense of positioning, sense of force and clarity of the audio through the HRTF rendering processing, and reduce the mutual interference between multiple channels through crosstalk elimination, thereby improving the sound quality of the external audio and improving the user's audio and video experience.

[0247] In one possible implementation of the present application, the audio to be processed includes audio data of at least two sub-channels, the rendering parameters include room impulse responses corresponding to each sub-channel, and the first reverberation audio includes first sub-channel reverberation audio corresponding to each sub-channel. Accordingly, the first reverberation module 1202 includes:

[0248] The first generating unit is configured to perform convolution processing on the audio data of each sub-channel and the corresponding room impulse response to respectively generate each first sub-channel reverberation audio.

[0249] Furthermore, in another possible implementation of the present application, the audio type of the audio to be processed is multi-channel audio; accordingly, the apparatus 1200 further includes:

[0250] A first downmixing module is configured to perform downmixing on the audio to be processed to generate a left channel audio to be processed and a right channel audio to be processed corresponding to the audio to be processed;

[0251] The first determining module is configured to determine the left channel audio to be processed and the right channel audio to be processed as audio data of each sub-channel included in the audio to be processed.

[0252] Furthermore, in another possible implementation of the present application, the second reverberation audio includes second sub-channel reverberation audio corresponding to each sub-channel; accordingly, the first crosstalk cancellation module 1203 includes:

[0253] A first acquiring unit, configured to acquire a crosstalk cancellation filter coefficient corresponding to each sub-channel;

[0254] The first crosstalk cancellation unit is configured to perform crosstalk cancellation processing on each first sub-channel reverberation audio according to a crosstalk cancellation filter coefficient corresponding to each sub-channel, so as to generate optimized reverberation audio for each second sub-channel.

[0255] Furthermore, in another possible implementation of the present application, the rendering parameters further include a crosstalk attenuation factor corresponding to each sub-channel, a crosstalk delay error corresponding to each sub-channel, and a crosstalk cancellation loss function; accordingly, the first acquisition unit is specifically configured to:

[0256] Obtain the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each sub-channel, and use it as the reference frequency corresponding to each sub-channel;

[0257] Determine the crosstalk cancellation order corresponding to each sub-channel according to the reference frequency corresponding to each sub-channel, each crosstalk attenuation factor, each crosstalk delay error and the crosstalk cancellation loss function;

[0258] According to the crosstalk cancellation order corresponding to each sub-channel, the crosstalk cancellation filter coefficient corresponding to each sub-channel is determined respectively.

[0259] Furthermore, in another possible implementation of the present application, the apparatus 1200 further includes:

[0260] The first attenuation module is configured to perform attenuation processing on the second reverberation audio according to the attenuation coefficient, so as to perform distance control on the second reverberation audio.

[0261] Furthermore, in another possible implementation of the present application, the audio to be processed includes audio data of at least two sub-channels, and the rendering parameters further include azimuth rendering parameters; accordingly, the first rendering module 1204 includes:

[0262] A first determining unit, configured to determine target HRTF rendering parameters corresponding to each sub-channel according to the azimuth rendering parameters;

[0263] The second determining unit is configured to perform HRTF rendering on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel, so as to determine the direct sound audio corresponding to each sub-channel.

[0264] Furthermore, in another possible implementation of the present application, the audio type of the audio to be processed is stereo audio, the sub-channel includes a left channel and a right channel, and the sub-channel audio includes the left channel audio to be processed and the right channel audio to be processed; accordingly, the first rendering module 1204 further includes:

[0265] A third determining unit is configured to perform center channel extraction on the left channel audio to be processed and the right channel audio to be processed to determine the center channel audio to be processed corresponding to the audio to be processed;

[0266] The fourth determining unit is configured to determine the left channel audio to be processed, the right channel audio to be processed, and the center channel audio to be processed as audio data of respective sub-channels included in the audio to be processed.

[0267] Furthermore, in another possible implementation of the present application, the azimuth rendering parameters include a height rendering parameter and a width rendering parameter, and the sub-channels include at least a center channel, a left channel, and a right channel; accordingly, the first determining unit is specifically configured to:

[0268] Determine the target HRTF rendering parameters corresponding to the center channel based on the height rendering parameters;

[0269] According to the width rendering parameter, the target HRTF rendering parameters corresponding to the left channel and the right channel are determined.

[0270] Furthermore, in another possible implementation of the present application, the first determining unit is further configured to:

[0271] According to the azimuth rendering parameters and the preset HRTF data, the sound source position of each sub-channel is calibrated to generate the target HRTF rendering parameters corresponding to each sub-channel.

[0272] Furthermore, in another possible implementation of the present application, the rendering parameters further include a center channel enhancement parameter, and the sub-channels include a center channel; accordingly, the first rendering module 1204 further includes:

[0273] The first enhancement unit is configured to enhance the direct sound audio corresponding to the center channel according to the center channel enhancement parameter.

[0274] Furthermore, in another possible implementation of the present application, the sub-channels include at least a center channel, a left channel, and a right channel; accordingly, the apparatus 1200 further includes:

[0275] The second crosstalk cancellation module is used to perform crosstalk cancellation processing on the direct sound audio corresponding to the left channel and the direct sound audio corresponding to the right channel.

[0276] Furthermore, in another possible implementation of the present application, the sub-channels further include a left surround channel and a right surround channel; accordingly, the apparatus 1200 further includes:

[0277] The third crosstalk cancellation module is configured to perform crosstalk cancellation processing on the direct sound audio corresponding to the left surround sound channel and the direct sound audio corresponding to the right surround sound channel.

[0278] Furthermore, in another possible implementation of the present application, the apparatus 1200 further includes:

[0279] The second downmixing module is configured to perform downmixing processing according to the direct sound audio corresponding to each sub-channel.

[0280] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0281] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0282] In order to implement the above embodiments, the present application also proposes an electronic device.

[0283] FIG13 is a schematic structural diagram of an electronic device according to an embodiment of the present application.

[0284] As shown in FIG13 , the electronic device 1300 includes:

[0285] The memory 1310 and at least one processor 1320, a bus 1330 connecting different components (including the memory 1310 and the processor 1320), the memory 1310 stores a computer program, and when the processor 1320 executes the program, the audio and video optimization method described in the embodiment of the present application is implemented.

[0286] Bus 1330 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor, or a local bus using any of a variety of bus architectures. Examples of these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0287] The electronic device 1300 typically includes a variety of electronic device-readable media. These media can be any available media that can be accessed by the electronic device 1300, including volatile and non-volatile media, removable and non-removable media.

[0288] The memory 1310 may also include computer system readable media in the form of volatile memory, such as random access memory (RAM) 1340 and / or cache memory 1350. The electronic device 1300 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 1360 may be used to read and write non-removable, non-volatile magnetic media (not shown in FIG13 , commonly referred to as a “hard drive”). Although not shown in FIG13 , a disk drive for reading and writing to a removable non-volatile disk (such as a “floppy disk”) and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical media) may be provided. In these cases, each drive may be connected to the bus 1330 via one or more data medium interfaces. The memory 1310 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of each embodiment of the present application.

[0289] A program / utility 1380 having a set (at least one) of program modules 1370 may be stored, for example, in memory 1310. Such program modules 1370 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data, each of which, or some combination thereof, may include an implementation of a network environment. Program modules 1370 generally implement the functions and / or methods of the embodiments described herein.

[0290] Electronic device 1300 can also communicate with one or more external devices 1390 (e.g., a keyboard, pointing device, display 1391, etc.), one or more devices that enable a user to interact with electronic device 1300, and / or any device that enables electronic device 1300 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication can occur via input / output (I / O) interface 1392. Furthermore, electronic device 1300 can communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network such as the Internet) via network adapter 1393. As shown, network adapter 1393 communicates with other modules of electronic device 1300 via bus 1330. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with electronic device 1300, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0291] The processor 1320 executes various functional applications and data processing by running programs stored in the memory 1310 .

[0292] It should be noted that the implementation process and technical principles of the electronic device of this embodiment can be found in the aforementioned explanation of the audio and video optimization method of the embodiment of the present application, and will not be repeated here.

[0293] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0294] An embodiment of the present application provides a computer program product. When the computer program product is run on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0295] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program, when executed by the processor, can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may at least include: any entity or device capable of carrying the computer program code to the device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, a computer-readable medium cannot be an electric carrier signal or a telecommunication signal.

[0296] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0297] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0298] In the embodiments provided in the present application, it should be understood that the disclosed devices / electronic devices and methods can be implemented in other ways. For example, the device / electronic device embodiments described above are merely schematic. For example, the division of the modules or units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0299] The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. The above-mentioned embodiments are only used to illustrate the technical scheme of the present application, rather than to limit it; although the present application has been described in detail with reference to the aforementioned embodiments, ordinary technicians in this field should understand that they can still modify the technical schemes recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein; and these modifications or replacements do not deviate the essence of the corresponding technical schemes from the spirit and scope of the technical schemes of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An audio image optimization method, characterized in that: include: Obtaining the audio to be processed and the rendering parameters corresponding to the audio to be processed; According to the rendering parameters, reverberation processing is performed on the audio to be processed to generate a first reverberation audio corresponding to the audio to be processed; Performing crosstalk elimination processing on the first reverberation audio to generate optimized second reverberation audio; Performing head-related transfer function (HRTF) rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed; The second reverberation audio and the direct sound audio are fused to generate an optimized target audio.

2. The method according to claim 1, characterized in that The audio to be processed includes audio data of at least two sub-channels, the rendering parameters include room impulse responses corresponding to the sub-channels, the first reverberation audio includes first sub-channel reverberation audio corresponding to the sub-channels, and performing reverberation processing on the audio to be processed according to the rendering parameters to generate first reverberation audio corresponding to the audio to be processed includes: The audio data of each of the sub-channels is convolved with the corresponding room impulse response to generate each of the first sub-channel reverberation audios respectively.

3. The method according to claim 2, characterized in that The audio type of the audio to be processed is multi-channel audio, and before performing reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed, the method further includes: Performing downmixing processing on the audio to be processed to generate left channel audio to be processed and right channel audio to be processed corresponding to the audio to be processed; The left channel audio to be processed and the right channel audio to be processed are determined as audio data of each of the sub-channels included in the audio to be processed.

4. The method according to claim 2 or 3, characterized in that The second reverberation audio includes second sub-channel reverberation audio corresponding to each of the sub-channels, and the performing crosstalk elimination processing on the first reverberation audio to generate optimized second reverberation audio includes: Obtaining crosstalk cancellation filter coefficients corresponding to each of the sub-channels; According to the crosstalk cancellation filter coefficient corresponding to each of the sub-channels, crosstalk cancellation processing is performed on each of the first sub-channel reverberation audios to generate optimized reverberation audios for each of the second sub-channels.

5. The method according to claim 4, characterized in that The rendering parameters also include a crosstalk attenuation factor corresponding to each of the sub-channels, a crosstalk delay error corresponding to each of the sub-channels, and a crosstalk cancellation loss function. The obtaining of the crosstalk cancellation filter coefficients corresponding to each of the sub-channels includes: Obtain the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each of the sub-channels as the reference frequency corresponding to each of the sub-channels; Determine the crosstalk cancellation order corresponding to each of the sub-channels according to the reference frequency corresponding to each of the sub-channels, each of the crosstalk attenuation factors, each of the crosstalk delay errors, and the crosstalk cancellation loss function; According to the crosstalk cancellation order corresponding to each of the sub-channels, the crosstalk cancellation filter coefficient corresponding to each of the sub-channels is determined respectively.

6. The method according to any one of claims 1 to 5, characterized in that: Before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the method further includes: The second reverberation audio is attenuated according to the attenuation coefficient to perform distance control on the second reverberation audio.

7. The method according to any one of claims 1 to 6, characterized in that: The audio to be processed includes audio data of at least two sub-channels, the rendering parameters further include azimuth rendering parameters, and performing HRTF rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed includes: Determining target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters; According to the target HRTF rendering parameters corresponding to each of the sub-channels, HRTF rendering is performed on the audio data of each of the sub-channels to determine the direct sound audio corresponding to each of the sub-channels.

8. The method according to claim 7, characterized in that The audio type of the audio to be processed is stereo audio, the sub-channel includes a left channel and a right channel, the audio data of the sub-channel includes the left channel audio to be processed and the right channel audio to be processed, and before determining the target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters, the method further includes: Performing center channel extraction on the left channel audio to be processed and the right channel audio to be processed to determine the center channel audio to be processed corresponding to the audio to be processed; The left channel audio to be processed, the right channel audio to be processed, and the center channel audio to be processed are determined as audio data of each of the sub-channels included in the audio to be processed.

9. The method according to claim 7 or 8, characterized in that The azimuth rendering parameters include a height rendering parameter and a width rendering parameter, the sub-channels include at least a center channel, a left channel, and a right channel, and determining target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters includes: Determining target HRTF rendering parameters corresponding to the center channel according to the height rendering parameters; According to the width rendering parameter, target HRTF rendering parameters corresponding to the left channel and the right channel are determined.

10. The method according to any one of claims 7 to 9, characterized in that: The step of determining target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters includes: The sound source position of each of the sub-channels is calibrated according to the azimuth rendering parameters and the preset HRTF data to generate target HRTF rendering parameters corresponding to each of the sub-channels.

11. The method according to any one of claims 7 to 10, characterized in that: The rendering parameters further include center channel enhancement parameters, the sub-channels include center channels, and after performing HRTF rendering on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel to determine the direct sound audio corresponding to each sub-channel, the method further includes: The direct sound audio corresponding to the center channel is enhanced according to the center channel enhancement parameter.

12. The method according to any one of claims 7 to 11, characterized in that: The sub-channels include at least a center channel, a left channel, and a right channel. Before the second reverberation audio and the direct sound audio are fused to generate an optimized target audio, the method further includes: Crosstalk cancellation processing is performed on the direct sound and audio corresponding to the left channel and the direct sound and audio corresponding to the right channel.

13. The method according to claim 12, characterized in that The sub-channels further include a left surround channel and a right surround channel. Before the second reverberation audio and the direct sound audio are fused to generate the optimized target audio, the method further includes: A crosstalk cancellation process is performed on the direct sound audio corresponding to the left surround sound channel and the direct sound audio corresponding to the right surround sound channel.

14. The method according to any one of claims 7 to 13, characterized in that: Before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the method further includes: Downmixing is performed according to the direct sound audio corresponding to each of the sub-channels.

15. An audio sound image optimization device, characterized in that: include: A first acquisition module, used to acquire the audio to be processed and the rendering parameters corresponding to the audio to be processed; A first reverberation module, configured to perform reverberation processing on the audio to be processed according to the rendering parameters, so as to generate a first reverberation audio corresponding to the audio to be processed; A first crosstalk elimination module, configured to perform crosstalk elimination processing on the first reverberation audio to generate an optimized second reverberation audio; A first rendering module, configured to perform HRTF rendering on the audio to be processed according to the rendering parameters, so as to determine a direct sound audio corresponding to the audio to be processed; The first fusion module is used to perform fusion processing on the second reverberation audio and the direct sound audio to generate an optimized target audio.

16. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 14 is implemented.

17. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

18. An audio image optimization method, characterized in that: include: Obtaining audio to be processed and rendering parameters corresponding to the audio to be processed, wherein the audio to be processed includes audio data of at least two sub-channels, and the rendering parameters include a crosstalk attenuation factor corresponding to each of the sub-channels, a crosstalk delay error corresponding to each of the sub-channels, and a crosstalk cancellation loss function; According to the rendering parameters, reverberation processing is performed on the audio to be processed to generate first reverberation audio corresponding to the audio to be processed, wherein the first reverberation audio includes first sub-channel reverberation audio corresponding to each of the sub-channels; Performing crosstalk cancellation processing on the first reverberation audio to generate optimized second reverberation audio, wherein the crosstalk cancellation order used when performing crosstalk cancellation processing on the first reverberation audio is determined according to the frequencies of the first sub-channel reverberation audios and the crosstalk attenuation factors, the crosstalk delay errors, and the crosstalk cancellation loss function; Performing head-related transfer function (HRTF) rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed; The second reverberation audio and the direct sound audio are fused to generate an optimized target audio.

19. The method according to claim 18, characterized in that The rendering parameters include room impulse responses corresponding to the sub-channels, and performing reverberation processing on the audio to be processed according to the rendering parameters to generate first reverberation audio corresponding to the audio to be processed, including: The audio data of each of the sub-channels is convolved with the corresponding room impulse response to generate each of the first sub-channel reverberation audios respectively.

20. The method of claim 19, wherein: The audio type of the audio to be processed is multi-channel audio, and before performing reverberation processing on the audio to be processed according to the rendering parameters to generate a first reverberation audio corresponding to the audio to be processed, the method further includes: Performing downmixing processing on the audio to be processed to generate left channel audio to be processed and right channel audio to be processed corresponding to the audio to be processed; The left channel audio to be processed and the right channel audio to be processed are determined as audio data of each of the sub-channels included in the audio to be processed.

21. The method according to claim 19 or 20, characterized in that The second reverberation audio includes second sub-channel reverberation audio corresponding to each of the sub-channels, and the performing crosstalk elimination processing on the first reverberation audio to generate optimized second reverberation audio includes: Obtaining crosstalk cancellation filter coefficients corresponding to each of the sub-channels; According to the crosstalk cancellation filter coefficient corresponding to each of the sub-channels, crosstalk cancellation processing is performed on each of the first sub-channel reverberation audios to generate optimized reverberation audios for each of the second sub-channels.

22. The method according to claim 21, characterized in that The obtaining of the crosstalk cancellation filter coefficients corresponding to each of the sub-channels includes: Obtain the frequency of the first sub-channel reverberation audio of the crosstalk channel corresponding to each of the sub-channels as the reference frequency corresponding to each of the sub-channels; Determine the crosstalk cancellation order corresponding to each of the sub-channels according to the reference frequency corresponding to each of the sub-channels, each of the crosstalk attenuation factors, each of the crosstalk delay errors, and the crosstalk cancellation loss function; According to the crosstalk cancellation order corresponding to each of the sub-channels, the crosstalk cancellation filter coefficient corresponding to each of the sub-channels is determined respectively.

23. The method according to any one of claims 18 to 22, characterized in that: Before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the method further includes: The second reverberation audio is attenuated according to the attenuation coefficient to perform distance control on the second reverberation audio.

24. The method according to any one of claims 18 to 23, characterized in that: The audio to be processed includes audio data of at least two sub-channels, the rendering parameters further include azimuth rendering parameters, and performing HRTF rendering on the audio to be processed according to the rendering parameters to determine the direct sound audio corresponding to the audio to be processed includes: Determining target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters; According to the target HRTF rendering parameters corresponding to each of the sub-channels, HRTF rendering is performed on the audio data of each of the sub-channels to determine the direct sound audio corresponding to each of the sub-channels.

25. The method of claim 24, wherein: The audio type of the audio to be processed is stereo audio, the sub-channel includes a left channel and a right channel, the audio data of the sub-channel includes the left channel audio to be processed and the right channel audio to be processed, and before determining the target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters, the method further includes: Performing center channel extraction on the left channel audio to be processed and the right channel audio to be processed to determine the center channel audio to be processed corresponding to the audio to be processed; The left channel audio to be processed, the right channel audio to be processed, and the center channel audio to be processed are determined as audio data of each of the sub-channels included in the audio to be processed.

26. The method according to claim 24 or 25, characterized in that The azimuth rendering parameters include a height rendering parameter and a width rendering parameter, the sub-channels include at least a center channel, a left channel, and a right channel, and determining target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters includes: Determining target HRTF rendering parameters corresponding to the center channel according to the height rendering parameters; According to the width rendering parameter, target HRTF rendering parameters corresponding to the left channel and the right channel are determined.

27. The method according to any one of claims 24 to 26, characterized in that: The step of determining target HRTF rendering parameters corresponding to each of the sub-channels according to the azimuth rendering parameters includes: The sound source position of each of the sub-channels is calibrated according to the azimuth rendering parameters and the preset HRTF data to generate target HRTF rendering parameters corresponding to each of the sub-channels.

28. The method according to any one of claims 24 to 27, characterized in that: The rendering parameters further include center channel enhancement parameters, the sub-channels include center channels, and after performing HRTF rendering on the audio data of each sub-channel according to the target HRTF rendering parameters corresponding to each sub-channel to determine the direct sound audio corresponding to each sub-channel, the method further includes: The direct sound audio corresponding to the center channel is enhanced according to the center channel enhancement parameter.

29. The method according to any one of claims 24 to 28, characterized in that: The sub-channels include at least a center channel, a left channel, and a right channel. Before the second reverberation audio and the direct sound audio are fused to generate an optimized target audio, the method further includes: Crosstalk cancellation processing is performed on the direct sound and audio corresponding to the left channel and the direct sound and audio corresponding to the right channel.

30. The method of claim 29, wherein: The sub-channels further include a left surround channel and a right surround channel. Before the second reverberation audio and the direct sound audio are fused to generate the optimized target audio, the method further includes: A crosstalk cancellation process is performed on the direct sound audio corresponding to the left surround sound channel and the direct sound audio corresponding to the right surround sound channel.

31. The method according to any one of claims 24 to 30, characterized in that: Before fusing the second reverberation audio and the direct sound audio to generate the optimized target audio, the method further includes: Downmixing is performed according to the direct sound audio corresponding to each of the sub-channels.

Images