Sound field expansion method, audio equipment and computer readable storage medium

Abstract

The invention discloses a sound field expansion method, an audio device and a computer readable storage medium, and relates to the technical field of audio processing, and the method comprises the steps: obtaining a target transfer function between a test audio device at an expansion target position and two ears of a user, and obtaining personalized acoustic transmission data of a target user, the personalized acoustic transmission data comprises at least one head related transfer function HRTF generated based on the personal auditory physiological data of the target user; rendering the original audio according to the personalized acoustic transmission data to obtain personalized audio; and performing crosstalk elimination processing on the personalized audio according to the target transfer function to obtain a target audio, and playing the target audio. While the sound field is effectively expanded, the crosstalk is reduced, and the sound quality is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio processing, in particular to a sound field expansion method, an audio device and a computer readable storage medium. BACKGROUND

[0002] Sound field expansion refers to an acoustic phenomenon that the sound field perceived by a listener is wider than the actual position of the loudspeaker. The sound field expansion is similar to a virtual loudspeaker, which can expand the sound source position to a wider position than the actual position of the loudspeaker, i.e. making the sound played by the sound source equivalent to the effect of the sound emitted from a virtual loudspeaker at a wider position.

[0003] In the field of audio processing, actual audio signals are mostly two-channel stereo signals. The sound field expansion technology is based on two-channel stereo sound, without increasing the sound channel and the sound box. By processing the signal, the listener feels that the sound comes from multiple directions, producing a simulated stereo sound field.

[0004] However, the current sound field expansion function (i.e. virtual surround sound function) is mainly realized by a head related transfer function (HRTF) algorithm. When a general HRTF is used for sound field expansion, the general HRTF may not match the individual head of a person, resulting in a phenomenon that the sound field effect does not meet the expectation. In addition, there may be a crosstalk problem. Crosstalk refers to the phenomenon that the sound signal of one sound channel is transmitted through the air and received by the other ear. In an audio device, the crosstalk problem is particularly prominent because the sound is directly transmitted in the air.

[0005] Therefore, how to effectively expand the sound field while reducing crosstalk and protecting the sound quality has become a technical problem to be solved. SUMMARY

[0006] The main purpose of the present application is to provide a sound field expansion method, an audio device and a computer readable storage medium, which aims to solve the technical problem of how to effectively expand the sound field while reducing crosstalk and protecting the sound quality.

[0007] To achieve the above purpose, the present application provides a sound field expansion method, which comprises the following steps: obtaining a target transfer function between a test audio device at an expansion target position and the ears of a user, and obtaining individual acoustic transmission data of a target user, wherein the individual acoustic transmission data comprises at least one head related transfer function (HRTF) generated based on individual auditory physiological data of the target user; rendering original audio according to the individual acoustic transmission data to obtain individual audio; According to the target transfer function, the crosstalk cancellation processing is performed on the personalized audio to obtain target audio, and the target audio is played.

[0008] In an embodiment, the step of obtaining the target transfer function between the test audio device at the target position and the user's binaural ears comprises: obtaining an artificial head transfer function and a free field transfer function; performing an inverse operation on the free field transfer function to obtain a free field inverse transfer function; multiplying the artificial head transfer function and the free field inverse transfer function to obtain the target transfer function between the test audio device at the target position and the user's binaural ears.

[0009] In an embodiment, the step of obtaining the artificial head transfer function and the free field transfer function comprises: When the test audio device is placed at the target position, and the test audio device outputs a sound signal, the artificial head transfer function is measured by a preset microphone in a preset artificial head ear canal; and When the preset artificial head is removed, and the test audio device outputs a sound signal, the free field transfer function is measured by a preset microphone placed at the left and right ear positions before the preset artificial head is removed.

[0010] In an embodiment, the step of performing crosstalk cancellation processing on the personalized audio according to the target transfer function to obtain target audio comprises: performing an inverse operation on the target transfer function to obtain a target inverse transfer function; multiplying the personalized audio and the target inverse transfer function to obtain target audio.

[0011] In an embodiment, the step of obtaining the personalized acoustic transmission data of the target user comprises: training a preset model to obtain a trained general model based on the obtained group auditory training data set, wherein the group auditory training data set comprises multiple human auditory physiological data and respective corresponding HRTF labels, and the general model comprises a general encoder and a general decoder; obtaining personal auditory physiological data of the target user; obtaining a general decoder from the general model, performing personalized training on the general decoder based on the personal auditory physiological data to obtain a personalized decoder, and generating personalized acoustic transmission data corresponding to the personal auditory physiological data based on the personalized decoder.

[0012] In an embodiment, before the step of rendering original audio to obtain personalized audio according to the personalized acoustic transmission data, the method further comprises: obtain an amplitude spectrum and a phase spectrum of the individual acoustic transmission data, and obtain a preset masking sound threshold; adjust the amplitude spectrum based on the preset masking sound threshold to obtain an adjusted amplitude spectrum, wherein the adjustment of the amplitude spectrum comprises reducing an amplitude value of a target frequency point, the target frequency point being a frequency point in the amplitude spectrum whose amplitude value is less than the preset masking sound threshold; combine the adjusted amplitude spectrum and the phase spectrum to obtain adjusted individual acoustic transmission data; based on the adjusted individual acoustic transmission data, perform the step of rendering original audio to obtain individual audio according to the individual acoustic transmission data.

[0013] In an embodiment, before the step of obtaining the preset masking sound threshold, the method further comprises: obtain an audio parameter of the original audio, wherein the audio parameter comprises a power density spectrum and a sound pressure level; identify at least one masking sound frequency point in the original audio based on the audio parameter, and obtain an individual sound pressure level of each masking sound frequency point; determine a minimum value of each individual sound pressure level and an absolute auditory threshold, and set the minimum value as the preset masking sound threshold.

[0014] In an embodiment, after the step of obtaining the target transfer function between the test audio device and the user's binaural ears at the target position, the method further comprises: obtain an environmental acoustic parameter of an environment in which a target user is currently located, wherein the environmental acoustic parameter comprises a reflected sound intensity, a reverberation time, and a dominant reflection direction; generate a corresponding environmental compensation coefficient based on the environmental acoustic parameter; multiply the environmental compensation coefficient and the target transfer function to obtain a compensated target transfer function; based on the compensated target transfer function, perform the step of performing crosstalk cancellation processing on the individual audio according to the target transfer function to obtain target audio.

[0015] In addition, to achieve the above-mentioned purposes, the present application also provides an audio device, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the sound field expansion method as described above.

[0016] In addition, to achieve the above object, the present application also provides a readable storage medium, which is a computer readable storage medium, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the steps of the sound field expansion method.

[0017] The present application also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the sound field expansion method as described above.

[0018] The one or more technical solutions proposed in the present application have at least the following technical effects: The present application solves the technical problems of poor sound field effect and existence of sound crosstalk caused by the mismatch between the general HRTF and the individual physiological characteristics of the target user by obtaining the target transfer function between the test audio equipment and the user's binaural ears at the expansion target position and combining the personalized acoustic transmission data of the target user. Specifically, the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the individual auditory physiological data of the target user. By using these personalized HRTFs to render the original audio, the generated personalized audio can better fit the user's auditory perception, thereby improving the naturalness and immersion of the sound field expansion effect. Further, the crosstalk elimination processing is performed on the personalized audio according to the target transfer function, which reduces the degree of sound signal reception of one sound channel by the other ear, thereby enhancing the positioning accuracy and spatial sense of the sound. Finally, the output target audio not only conforms to the individual physiological characteristics of the user in terms of sound field expansion effect, effectively ensuring the sound quality, but also eliminates crosstalk, achieving the goal of effectively expanding the sound field while reducing crosstalk and ensuring the sound quality. BRIEF DESCRIPTION OF DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0021] Figure 1 Flowchart of the first embodiment of the sound field expansion method of the present application; Figure 2 Schematic diagram of the application scenario related to an embodiment of the sound field expansion method of the present application; Figure 3A schematic diagram of a crosstalk elimination process involved in the first embodiment of the sound field expansion method of the present application; Figure 4 A schematic diagram of a scene of generating individual acoustic transmission data involved in an embodiment of the sound field expansion method of the present application; Figure 5 A schematic diagram of a process of obtaining individual auditory physiological data involved in an embodiment of the sound field expansion method of the present application; Figure 6 A schematic diagram of a device structure of a hardware running environment involved in the sound field expansion method device in the embodiment of the present application.

[0022] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0023] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0024] In digital signal processing, actual audio signals are mostly double-channel stereo signals. Stereo expansion technology is based on double-channel stereo sound, without increasing the sound channel and sound box, by processing the signal, the listener feels that the sound comes from multiple directions, and a simulated stereo sound field is produced. Stereo expansion technology has become an indispensable technology. It is mainly applied to far-field sound sources, such as scenes using sound boxes. In recent years, the use of near-ear open audio devices such as VR (Virtual Reality) and AR (Augmented Reality) has become more and more widespread, and the demand for sound field expansion functions of near-ear open audio devices has gradually increased. Currently, the stereo expansion function is mainly realized through the Head Related Transfer Function (HRTF) algorithm. Generally, a common HRTF is used to expand the sound field, which may cause the sound field effect to be less than expected due to the mismatch between the common HRTF and the individual head, and may cause crosstalk in the audio device, resulting in a narrow sound image. Therefore, it is particularly important to effectively expand the sound field while being suitable for open devices, reducing crosstalk and ensuring sound quality.

[0025] Based on this, the main solution of the present application is: obtaining a target transfer function between a test audio device at an extended target position and the user's two ears, and obtaining personalized acoustic transmission data of a target user, wherein the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the personal auditory physiological data of the target user; rendering the original audio according to the personalized acoustic transmission data to obtain personalized audio; and performing crosstalk cancellation processing on the personalized audio according to the target transfer function to obtain target audio and playing the target audio.

[0026] The present application solves the technical problems of poor sound field effect and sound crosstalk caused by the mismatch between the general HRTF and the personal physiological characteristics in the prior art by obtaining the target transfer function between the test audio device at the extended target position and the user's two ears, and combining the personalized acoustic transmission data of the target user. Specifically, the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the personal auditory physiological data of the target user. By using these personalized HRTFs to render the original audio, the generated personalized audio can better fit the user's auditory perception, thereby improving the naturalness and immersion of the sound field expansion effect. Further, the crosstalk cancellation processing is performed on the personalized audio according to the target transfer function, which reduces the degree of sound signal reception of one sound channel by the other ear, thereby enhancing the positioning accuracy and spatial sense of the sound. Finally, the output target audio not only better meets the personalized physiological characteristics of the user in terms of sound field expansion effect, effectively ensuring the sound quality, but also performs crosstalk cancellation, achieving the goal of effectively expanding the sound field while reducing crosstalk and ensuring the sound quality.

[0027] It should be noted that the execution subject of each embodiment of the sound field expansion method of the present application can be an audio device capable of realizing the above functions, such as an AR helmet, a VR helmet, smart audio glasses, a neck-hung audio box, an open earphone, a far ear open audio device such as a sound system and a television, and the like. The sound field expansion method of each embodiment of the present application does not make specific limitations on this. Exemplarily, the execution subject of each embodiment of the present application is a near-ear open audio device.

[0028] Based on this, the present application proposes a sound field expansion method of the first embodiment, as shown in Figure 1 The sound field expansion method includes the following steps S10-S30: Step S10, obtaining a target transfer function between a test audio device at an extended target position and the user's two ears, and obtaining personalized acoustic transmission data of a target user, wherein the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the personal auditory physiological data of the target user; The test audio device can be specifically a device with sound playing function, and further, the test audio device can be a near-ear open audio device as an execution subject. Exemplarily, the description and explanation of each embodiment of the present application is carried out by taking the test audio device as a loudspeaker. The extended target position is a virtual position where a user expects to feel a sound source, and is used to simulate the position of a sound source in a wider sound effect scene.

[0029] The near-ear open audio device is relatively close to the human ear in terms of the position of the loudspeaker or the loudspeaker, and the near-ear open device is generally an all-in-one machine, and the distance between the playing device and the human ear is almost not adjustable, that is, in general, the sound played by the near-ear open audio device and heard by the human ear is near-field sound effect. It can be understood in combination with Figure 2 Figure 2 is a schematic diagram of an application scenario provided by the present embodiment, assuming that the relative position between the position of the user's head and the loudspeaker of the device is as shown in Figure 2 When the user uses the near-ear open audio device, even if the position of the playing device can be adjusted slightly, the user cannot experience the listening experience that can be generated by the dashed loudspeaker position in the figure, and the user can only experience the near-field listening experience from the actual loudspeaker. When the user wants to experience a wider sound effect, for example, the user wants to hear the sound emitted from the dashed loudspeaker position in Figure 2 , the position can be set as the extended target position. The target transfer function between the loudspeaker of the extended target position and the user's ears represents the transfer function of the sound from the loudspeaker at the position to the human ear.

[0030] It should be noted that each transfer function mentioned in the present embodiment is an acoustic transfer function. The acoustic transfer function is the transfer function from a sound source to a reproduction area. The transfer function refers to the ratio of the Laplace transform (or z-transform) of the response (i.e., output) quantity of a linear system under zero initial condition to the Laplace transform of the excitation (i.e., input) quantity. In the present embodiment, the target transfer function between the test audio device and the user's ears is the transfer function from the output sound source (i.e., the loudspeaker or the loudspeaker) of the test audio device to the user's ears, which is used to reflect the changes of the output signal in the process of transferring the output signal of the test audio device to the user's ears.

[0031] ​The personalized acoustic transmission data can include at least one head-related transfer function (HRTF), which is a frequency domain filter function describing the reflection, diffraction and scattering effects of physiological structures such as the head, pinna and shoulders on sound waves during the propagation of sound from a free field to the ears. The head-related transfer function (HRTF) can be used to simulate the changes of sound waves from the sound source position to the human ear, and can make the virtual sound source have real spatial properties. For the same target user, there can be multiple head-related transfer functions (HRTFs), for example, the personalized acoustic transmission data can include a first head-related transfer function (HRTF) for the left ear and a second head-related transfer function (HRTF) for the right ear.

[0032] In step S20, the original audio is rendered according to the personalized acoustic transmission data to obtain personalized audio. The original audio can be, for example, audio input externally to the near-ear open audio device or local audio stored in the near-ear open audio device, which has not been subjected to any acoustic processing or personalized acoustic optimization.

[0033] Since the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the personal auditory physiological data of the target user, each head-related transfer function (HRTF) in the personalized acoustic transmission data can reflect the personal auditory physiological characteristics of the target user. Therefore, the original audio is rendered by the personalized acoustic transmission data in this embodiment, so that the rendered audio is more consistent with the auditory physiological characteristics of the target user, thereby improving the sound quality performance of the audio device.

[0034] In step S30, crosstalk cancellation processing is performed on the personalized audio according to the target transfer function to obtain target audio, and the target audio is played.

[0035] It can be understood that since the loudspeaker or speaker of the near-ear open audio device is not an ideal sound source, and the loudspeaker or speaker of the near-ear open audio device cannot be directly placed in the user's ear canal as a playback device, crosstalk problems often occur during the transmission of the target audio. In order to avoid this problem, crosstalk cancellation processing is performed on the personalized audio before the target audio is generated, i.e., the crosstalk problem that occurs during the transmission of the target audio after playback is offset. The target audio is obtained after the crosstalk cancellation processing of the personalized audio, which can offset the influence of the playback device itself and the influence of the user's head on the sound signal during the sound transmission, and retain the influence of the user's head on the sound transmission result. When the target audio enters the user's ears, the user can feel that the source of the sound signal is the extended target position located at a remote end.

[0036] It can be understood that, since the target audio has been subjected to crosstalk cancellation processing based on the crosstalk cancellation algorithm provided by the embodiment before being output via the loudspeaker, the influence of the near-ear open audio device itself and the human head on the sound transmission result is eliminated, so that when the target audio enters the user's two ears, it can achieve the same effect as the personalized audio. If the personalized audio is a left channel, the target audio is only transmitted to the left ear; if the personalized audio is a right channel, the target audio is only transmitted to the right ear; and if the personalized audio is a double channel, the target audio outputs the left and right channel sound signals to the left and right ears, respectively.

[0037] The embodiment obtains the target transfer function between the test audio device at the expanded target position and the user's two ears, and combines the personalized acoustic transmission data of the target user, to solve the technical problems of poor sound field effect and sound crosstalk caused by the mismatch between the general HRTF and the personal physiological characteristics in the prior art. Specifically, the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the personal auditory physiological data of the target user. By using these personalized HRTFs to render the original audio, the generated personalized audio can better fit the user's auditory perception, thereby improving the naturalness and immersion of the sound field expansion effect. Further, the personalized audio is subjected to crosstalk cancellation processing according to the target transfer function, reducing the degree to which the sound signal of one channel is received by the other ear, thereby enhancing the positioning accuracy and spatial sense of the sound. Finally, the output target audio not only better meets the user's personalized physiological characteristics in terms of sound field expansion effect, effectively ensuring the sound quality, but also is subjected to crosstalk cancellation, achieving the goal of effectively expanding the sound field while reducing crosstalk and ensuring the sound quality.

[0038] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described hereinafter. On this basis, the step of obtaining the target transfer function between the test audio device at the expanded target position and the user's two ears includes: Step A10, obtaining an artificial head transfer function and a free field transfer function; It should be noted that the target transfer function can be understood as the influence of the user's head profile on the sound signal transmission result. The embodiment obtains two different acoustic transfer functions based on two different sound transmission scenarios, and then obtains the target transfer function by calculation. Among them, the artificial head transfer function is obtained when the test audio device is placed at the expanded target position (i.e. Figure 2The artificial head transfer function is measured by the preset microphone in the ear canal of the preset artificial head when the test audio device is placed in the extended target position and the test audio device outputs a sound signal. The free field transfer function is measured by the preset microphone placed in the left and right ear positions before the preset artificial head is removed when the test audio device outputs a sound signal.

[0039] In a possible implementation, the step of obtaining the artificial head transfer function and the free field transfer function comprises: Step A201, obtaining the artificial head transfer function by the preset microphone in the ear canal of the preset artificial head when the test audio device is placed in the extended target position and the test audio device outputs a sound signal; and Step A202, obtaining the free field transfer function by the preset microphone placed in the left and right ear positions before the preset artificial head is removed when the test audio device outputs a sound signal.

[0040] It should be noted that, in the embodiment, the preset artificial head is an auxiliary device for assisting in measuring the acoustic transfer function, which is constructed by simulating the head of a user, and can simulate a scenario in which the user receives a sound signal emitted by a test audio device. The preset artificial head is provided with left and right ears and ear canals, and a microphone for receiving a sound signal can be placed in the ear canal in advance.

[0041] As an example, in combination with the application scenario shown in FIG. 1, the test audio device is placed in the virtual loudspeaker setting position, that is, the position of the dotted line loudspeaker, and the acoustic transfer function from the sound source to the left and right ears of the preset artificial head is measured by using the two preset microphones in the ear canals of the preset artificial head, and the acoustic transfer function is denoted as H1. Figure 2 Figure 2 As an example, in combination with the application scenario shown in FIG. 1, the test audio device is placed in the virtual loudspeaker setting position, that is, the position of the dotted line loudspeaker, and the acoustic transfer function from the sound source to the left and right ears of the preset artificial head is measured by using the two preset microphones in the ear canals of the preset artificial head, and the acoustic transfer function is denoted as H1.

[0042] As an example, in combination with the application scenario shown in FIG. 1, the test audio device is placed in the virtual loudspeaker setting position, that is, the position of the dotted line loudspeaker, and the acoustic transfer function from the sound source to the left and right ears of the preset artificial head is measured by using the two preset microphones in the ear canals of the preset artificial head, and the acoustic transfer function is denoted as H1. Figure 2 Figure 2 As an example, in combination with the application scenario shown in FIG. 1, the test audio device is placed in the virtual loudspeaker setting position, that is, the position of the dotted line loudspeaker, and the acoustic transfer function from the sound source to the left and right ears of the preset artificial head is measured by using the two preset microphones in the ear canals of the preset artificial head, and the acoustic transfer function is denoted as H1.

[0043] Step A20, performing an inverse operation on the free field transfer function to obtain a free field inverse transfer function; ​​Step A30, multiplying the artificial head transfer function with the free field inverse transfer function, to obtain a target transfer function between the test audio device and the user's binaural ears at the extended target position.

[0044] In this embodiment, the free field transfer function H2 containing the influence of the test audio device on the sound transmission result is first inversely operated to obtain a free field inverse transfer function, denoted as H2', and then the artificial head transfer function H1 containing the influence of the playback device and the preset artificial head on the sound transmission result is multiplied by H2' to obtain the target transfer function H. It should be noted that the H2' obtained after the inverse operation can eliminate the influence of the playback device on the sound transmission result, and after being multiplied by H1, the part of the influence of the test audio device on the sound transmission result in H1 can be eliminated, and the influence of the preset artificial head on the sound transmission result is retained as the target transfer function H.

[0045] In a possible implementation, the step of performing crosstalk cancellation processing on the individual audio according to the target transfer function to obtain target audio includes: Step B10, inversely operating the target transfer function to obtain a target inverse transfer function; Step B20, multiplying the individual audio with the target inverse transfer function to obtain the target audio.

[0046] As can be seen from the above steps, the target transfer function H represents the influence of the head contour on the sound transmission result, and it should be understood that the target inverse transfer function obtained by inversely operating H is equivalent to a unit matrix, which represents the elimination of the influence of the head contour on the sound transmission result. The target audio obtained by processing the individual audio with the target inverse transfer function can obviously offset the influence of the head contour on the sound transmission result when the sound signal is transmitted, so that the sound signal received by the user's binaural ears can be consistent with the target audio, and the target audio is from the wider far-field sound effect of the virtual loudspeaker (i.e. the loudspeaker at the extended target position), thereby obtaining the extended sound field after crosstalk cancellation.

[0047] As an example, in combination with Figure 2 As can be seen from the application scenario shown in FIG. 1, when the original audio X of the near-ear open audio device is given, the individual audio XM is obtained by processing the original audio X through the individual HRFT algorithm module and the binaural synthesis algorithm module, and the effect of the extended sound field is achieved. After the crosstalk cancellation algorithm module is processed, the target audio is output through the SPK (speaker, loudspeaker, loudspeaker), and the target audio is played into the human ear. Referring to FIG. 1, the basic process implemented by the crosstalk cancellation algorithm module is as follows. Figure 3 ​Step S11, obtaining the individual audio XM after rendering; Step S12, placing the test audio device at the target position of the virtual loudspeaker after sound field expansion (i.e., the expansion target position), and measuring the transfer function of the sound source to the human ear using the microphone in the preset artificial head ear, denoted as H1; Step S13, placing two microphones at the left and right ears of the preset artificial head, removing the preset artificial head, and measuring the transfer function when the device works in a free field, denoted as H2; Step S14, performing inverse operation on H2, denoted as H2'; Step S15, calculating the target transfer function H, specifically, H = H1H2', and performing inverse operation on H, denoted as B; Step S16, generating the target audio Y, specifically, Y = XMBH, and the target audio Y is the audio after removing the crosstalk.

[0048] Based on the first embodiment and / or the second embodiment of the present application, in the third embodiment of the present application, the same or similar contents as the above-mentioned embodiments one and two can be referred to the above description, and will not be described hereinafter. On this basis, the step of obtaining the individual acoustic transmission data of the target user comprises: Step C10, training the preset model to obtain a trained general model through the obtained group auditory training data set, wherein the group auditory training data set comprises a plurality of human auditory physiological data and respective corresponding HRTF labels, and the general model comprises a general encoder and a general decoder; It should be noted that the group auditory training data set comprises a plurality of human auditory physiological data, specifically, a deep learning model is trained using a large amount of existing HRTF data (existing HRTF data is open source HRTF data), that is, the group auditory training data set is obtained. The existing HRTF data can include a large amount of HRTF data of different people, covering a large amount of HRTF data of people of different ages, ear shapes, and head shapes. That is, the plurality of human auditory physiological data is derived from different human bodies, and the preset model can include a preset encoder and a preset decoder, and each human auditory physiological data corresponds to an HRTF label; the group auditory training data set can be predetermined, and each human auditory physiological data can reflect the auditory physiological characteristics of the human body, for example, the human auditory physiological data can include data of the ear canal resonance frequency, the angle between the tragus and the cymba concha, and the ear folding complexity, and the present embodiment does not make specific limitation thereon. The acquisition method of the human auditory physiological data can be the same as that of the individual auditory physiological data of the target user. The human auditory physiological data is data that can be processed by the preset encoder. The general model comprises a general decoder and a general encoder. After the preset model is trained, the general model comprising the general decoder and the general encoder can be obtained.

[0049] Exemplarily, a population auditory training data set is acquired, a preset model is trained through the population auditory training data set, and a general model composed of a general decoder and a general encoder is obtained. The general model can also be used to generate a general HRTF, and the general HRTF can also simulate the sound wave change of the sound source propagating to the human ear, but the general HRTF is difficult to adapt to the individualized differences of the user's hearing, so there is still a problem of poor sound quality performance of the audio device when the audio is rendered through the general HRTF. Therefore, after the general model is determined, the general decoder in the general model is fine-tuned to obtain a personalized decoder, so that the personalized acoustic transmission data of the target user can be generated through the personalized decoder subsequently.

[0050] In step C20, personal auditory physiological data of the target user is acquired. It should be noted that the personal auditory physiological data can reflect the personal hearing physiological characteristics of the target user. The personal auditory physiological data can be obtained by three-dimensional scanning of the head, can be obtained from the head measurement parameters of the target user, and can also be obtained by geometric modeling of the head of the user. This embodiment does not make specific limitation.

[0051] In step C30, a general decoder is acquired from the general model, the general decoder is personalized trained through the personal auditory physiological data, a personalized decoder is obtained, and personalized acoustic transmission data corresponding to the personal auditory physiological data is generated based on the personalized decoder.

[0052] It should be noted that the personal auditory physiological data can reflect the personal hearing physiological characteristics of the target user. The personal auditory physiological data can be obtained by three-dimensional scanning of the head, can be obtained from the head measurement parameters of the target user, and can also be obtained by geometric modeling of the head of the user. This embodiment does not make specific limitation.

[0053] Exemplarily, the general decoder is obtained from the general model, the general decoder is unsupervised trained through the personal auditory physiological data, and the personalized decoder is obtained. The personal auditory physiological data can be input into the general encoder or the preset encoder to encode the personal auditory physiological data through the general encoder or the preset encoder. The result of the personal auditory physiological data after encoding is input into the personalized decoder, and the personalized decoder outputs the personal auditory physiological data. The individual acoustic transmission data can include left ear acoustic transmission data and right ear acoustic transmission data, and the left ear acoustic transmission data and the right ear acoustic transmission data can each include a corresponding head-related transfer function HRTF. The left ear acoustic transmission data and the right ear acoustic transmission data can be symmetrical, or can be personalized and trained to be asymmetric according to the characteristics of the left and right ears of the target user.

[0054] Specifically, in other embodiments, the left ear acoustic transmission data and the right ear acoustic transmission data can also be asymmetric, for example, the left ear personal physiological data and the right ear personal physiological data can be obtained from the personal auditory physiological data. The left ear personal physiological data is input into the general encoder or the preset encoder to encode the left ear personal physiological data through the general encoder or the preset encoder. The result of the left ear personal physiological data after encoding is input into the personalized decoder, and the personalized decoder outputs the left ear acoustic transmission data of the left ear personal physiological data. The right ear personal physiological data is input into the general encoder or the preset encoder to encode the right ear personal physiological data through the general encoder or the preset encoder. The result of the right ear personal physiological data after encoding is input into the personalized decoder, and the personalized decoder outputs the right ear acoustic transmission data of the right ear personal physiological data. In this way, the accuracy of the individual acoustic transmission data can be improved, and the left and right ear differences of the same user can be adapted.

[0055] The embodiment trains the general decoder first, so as to facilitate the adjustment of the network parameters of the general decoder according to the personal auditory physiological data, obtain the personalized decoder, and then facilitate the generation of the individual acoustic transmission data in combination with the personalized decoder, so as to improve the accuracy of the audio rendering, make the rendered audio conform to the auditory physiological characteristics of the target user, and provide good sound quality experience for the user.

[0056] For better understanding of the embodiment, please refer to Figure 4 , Figure 4A scene schematic diagram of the personalization stage and the training stage of the general model in the embodiment is given. In the training stage of the general model, the population auditory training dataset 10 can be input to the preset encoder 20, and the output result of the preset encoder 20 is input to the preset decoder 30. The preset decoder 30 can output the HRTF training result 40. The preset encoder 20 and the preset decoder 30 can be trained until the training of the general decoder is completed. The general decoder can be personalized trained by using the personal auditory physiological data 500 to obtain the personalized decoder 60.

[0057] In the personalization stage, acoustic measurement can be performed to obtain the personal auditory physiological data 500. The general decoder can be personalized trained by using the personal auditory physiological data 500 to obtain the personalized decoder 60. The personal auditory physiological data 500 is input to the personalized decoder 60, and the personalized decoder can output the personalized acoustic transmission data 70. Figure 4 The process of personalizing the general decoder by using the personal auditory physiological data 500 is not shown in the embodiment.

[0058] In a feasible embodiment, the step C10 further includes steps C101-C102. The step C101 obtains target human auditory physiological data from the population auditory training dataset, inputs the target human auditory physiological data to the preset model, encodes the target human auditory physiological data by using the preset encoder in the preset model, inputs the encoded result to the preset decoder, and outputs the HRTF training result by using the preset decoder. The step C102 determines the training loss between the HRTF training result and the target HRTF label of the target human auditory physiological data. If the training loss is greater than a preset loss threshold, new target human auditory physiological data is obtained from the population auditory training dataset again, and the step of inputting the target human auditory physiological data to the preset model is returned until the training loss is less than or equal to the preset loss threshold, and the training of the general model is completed.

[0059] It should be noted that the preset encoder can encode the target human auditory physiological data, and can encode the high-dimensional target human auditory physiological data into a low-dimensional auditory-related feature vector to perform down-sampling encoding on the target human auditory physiological data. Specifically, the preset encoder can encode the features affecting the HRTF from the target human auditory physiological data, thereby reducing the model complexity and improving the training accuracy. The result of encoding the target human auditory physiological data by the preset encoder can be input to the preset decoder, and the result of encoding is decoded by the preset decoder, thereby generating an HRTF training result. The HRTF training result is the HRTF corresponding to the target human auditory physiological data output by the preset decoder in the preset model.

[0060] The training loss can represent the difference between the target HRTF label and the HRTF training result. The greater the training loss, the greater the difference between the target HRTF label and the HRTF training result. The preset loss threshold can be determined based on actual conditions, which is not limited in the embodiment. In the case where the training loss is greater than the preset loss threshold, it indicates that the difference between the target HRTF label and the HRTF training result is large, so new target human auditory physiological data needs to be obtained for training, until the training loss is less than or equal to the preset loss threshold, and a trained general model is obtained.

[0061] For example, the target human auditory physiological data is input to the preset model, the target human auditory physiological data is encoded by the preset encoder in the preset model, the encoded result is input to the preset decoder, and the HRTF training result is output by the preset decoder; the training loss between the HRTF training result and the target HRTF label of the target human auditory physiological data is calculated, and in the case where the training loss is greater than the preset loss threshold, new target human auditory physiological data is obtained from the group auditory training data set again, and the step of inputting the target human auditory physiological data to the preset model is returned, until the training loss is less than or equal to the preset loss threshold, and a trained general model is obtained.

[0062] The preset model is trained by the group auditory training data set to obtain a general model, and then the general decoder in the general model is used to determine the individual acoustic transmission data, so as to provide good sound quality experience for each user.

[0063] In a possible embodiment, step C20 further includes steps C201-C202: Step C201, obtaining head structure parameters of the target user, the head structure parameters including auricle height, auricle width, ear canal angle, head width and shoulder-neck parameters; Step C202, constructing a head geometry model of the target user through the head structure parameters, and extracting the personal hearing physiological data from the head geometry model.

[0064] It should be noted that the head width is the transverse distance between the left and right tragus, and the head width determines the binaural time difference (ITD, Interaural Time Difference). The shoulder-neck parameters can include shoulder width, neck height, etc., and the shoulder-neck parameters affect the reflection and scattering of low-frequency sound waves. The pinna height is the vertical distance from the tragus to the highest point of the helix, and the pinna width can be the maximum horizontal distance from the front edge of the helix to the back edge of the helix. The ear canal angle can be the angle between the ear canal axis and the head sagittal plane (left-right symmetry plane), which affects the sound wave incidence direction.

[0065] The head geometry model of the target user can be constructed through the head structure parameters. A 3D modeling software can be called to construct the head geometry model according to the head structure parameters. The features directly related to the hearing function can be directly extracted from the head geometry model to obtain the personal hearing physiological data. For example, the personal hearing physiological data can include the ear canal resonance frequency, the angle between the tragus and the concha, the ITD reference value, etc., which are not limited in the embodiment. In other embodiments, the head structure parameters can also include parameters such as mandibular width and ear canal length, which are not limited in the embodiment.

[0066] For example, the head structure parameters of the target user are obtained, which can be obtained by taking a photo. A modeling software is called to construct a head geometry model of the target user through the head structure parameters, and the personal hearing physiological data is obtained from the head geometry model.

[0067] The embodiment constructs the head geometry model of the target user through the head structure parameters, and then extracts the personal hearing physiological data from the head geometry model, and then determines the individual acoustic transmission data, so as to improve the sound quality performance of the audio equipment in the subsequent process.

[0068] In a feasible embodiment, step C20 further includes step C203 and / or step C204: Step C203, obtaining three-dimensional head scanning data of the user, and determining the personal hearing physiological data from the three-dimensional head scanning data; In step C204, the head measurement parameters of the user are obtained, the target sub-auditory physiological data corresponding to each sub-measurement parameter in the head measurement parameters are obtained from a preset audiological database, and personal audiological data composed of a plurality of target sub-auditory physiological data are obtained, wherein the plurality of sub-measurement parameters included in the head measurement parameters are respectively an external ear structure parameter, a head-neck connection parameter, and a head body parameter, and the preset audiological database includes preset sub-auditory physiological data corresponding to a plurality of preset sub-measurement parameters.

[0069] It should be noted that the three-dimensional head scanning data can be obtained by 3D structured light and / or laser scanning on the target user, and the personal audiological data related to hearing can be extracted from the three-dimensional head scanning data.

[0070] The head measurement parameters can be obtained by directly measuring the head and the shoulder-neck connected with the head of the target user, for example, by multi-angle photography. The head measurement parameters can include a plurality of sub-measurement parameters, and the plurality of sub-measurement parameters can be respectively an external ear structure parameter, a head-neck connection parameter, and a head body parameter. Each sub-measurement parameter can further include a plurality of sub-parameters. The plurality of sub-parameters included in the external ear structure parameter can be respectively an auricle contour, an auricle wrinkle, an auricle size, an ear canal length and diameter, an ear cavity depth, etc. The plurality of sub-parameters included in the head-neck connection parameter can be respectively a shoulder width and a neck length, and the plurality of sub-parameters included in the head body parameter can include a head diameter and a binaural distance.

[0071] The preset audiological database can be determined in advance based on actual conditions, and the preset audiological database can include preset sub-auditory physiological data corresponding to a plurality of preset sub-measurement parameters. The target sub-auditory physiological data of each sub-measurement parameter can be searched in the preset audiological database.

[0072] For example, the target user can be subjected to 3D structured light and / or laser scanning to obtain three-dimensional head scanning data, and the personal audiological data related to hearing can be obtained from the three-dimensional head scanning data.

[0073] The head measurement parameters of the user can also be obtained, the target sub-auditory physiological data corresponding to each sub-measurement parameter in the head measurement parameters can be obtained from a preset audiological database, and personal audiological data composed of a plurality of target sub-auditory physiological data are obtained. Specifically, each sub-measurement parameter further includes a plurality of sub-parameters, the preset sub-auditory physiological data corresponding to each sub-parameter can be searched in the preset audiological database, and personal audiological data composed of a plurality of preset sub-auditory physiological data are obtained. The preset sub-measurement parameter also includes a plurality of preset sub-parameters, and the preset sub-measurement parameter can correspond to a plurality of preset sub-auditory physiological data, which are not limited in the embodiment.

[0074] The embodiment can obtain personal auditory physiological data in multiple ways, thereby improving the flexibility of data acquisition. For example, reference can also be made to Figure 5 , Figure 5 The multiple ways of obtaining personal auditory physiological data are given, the personal auditory physiological data 500 can be extracted from the head geometric model 100, the personal auditory physiological data 500 can also be extracted from the three-dimensional head scan data 200, and the target sub-auditory physiological data corresponding to each sub-measurement parameter in the head measurement parameter 300 can be searched in the preset auditory physiological database 400 to obtain the personal auditory physiological data 500 composed of multiple target sub-auditory physiological data.

[0075] Based on the first embodiment, the second embodiment and / or the third embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as the above-mentioned embodiments one, embodiments two and embodiments three can be referred to the above introduction, and the subsequent will not be repeated. On this basis, before the step of rendering the original audio to obtain the personalized audio according to the personalized acoustic transmission data, the method further comprises: Step D10, obtaining the amplitude spectrum and the phase spectrum of the personalized acoustic transmission data, and obtaining a preset masking sound threshold; The personalized acoustic transmission data can be subjected to spectrum analysis to extract its amplitude spectrum and phase spectrum. The amplitude spectrum reflects the amplitude of different frequency components, and the phase spectrum reflects the phase information of different frequency components.

[0076] The preset masking sound threshold can be set or determined in advance based on the masking effect characteristics of the human ear, for example, it can be set as the lowest intensity level at which a specific sound (masked sound) can be perceived in the presence of masking sound. Masking sound refers to sound that can affect the perception of other sounds, which can be in the form of pure tone, complex tone or noise, etc. Masking sound will increase the hearing threshold of the masked sound, so that the sound that can be heard in a quiet environment becomes difficult to perceive in the presence of masking sound.

[0077] Step D20, adjusting the amplitude spectrum based on the preset masking sound threshold to obtain an adjusted amplitude spectrum, wherein the adjustment of the amplitude spectrum includes reducing the amplitude value of a target frequency point, and the target frequency point is a frequency point in the amplitude spectrum whose amplitude value is less than the preset masking sound threshold; For those target frequency points whose amplitude values are less than the preset masking sound threshold, the amplitude values are reduced. The purpose of this adjustment is to reduce the audio energy of these frequency points, so that their influence on the user's hearing in subsequent audio processing is minimized. For example, the amplitude values of these target frequency points can be reduced to a preset lower level, or adjusted according to a specific attenuation function, to ensure that the energy of the audio signal at these frequency points does not interfere with the user's auditory experience. In this way, the frequency response characteristics of the HRTF can be optimized to better conform to the laws of human auditory perception.

[0078] It should be noted that for frequency points greater than or equal to the masking sound threshold, the amplitude values can remain unchanged, or the corresponding optimization adjustment method can be set based on actual needs, and the present embodiment does not make specific limitations thereto.

[0079] Step D30, combining the adjusted amplitude spectrum and the phase spectrum to obtain the adjusted individual acoustic transmission data; Step D40, based on the adjusted individual acoustic transmission data, performing the step of rendering the original audio to obtain individual audio according to the individual acoustic transmission data.

[0080] The adjusted amplitude spectrum and the obtained phase spectrum are combined to obtain the adjusted individual acoustic transmission data. This combination can be achieved by combining the adjusted amplitude spectrum and the original phase spectrum in complex form, thereby generating a new individual acoustic transmission function. The adjusted individual acoustic transmission data can better adapt to the hearing needs of the target user, while reducing the influence of masking sound and improving the accuracy and effect of audio processing. In this way, the generated sound field is more in line with the laws of human auditory perception.

[0081] In one possible implementation, before the step of obtaining the preset masking sound threshold, the method further comprises: Step E10, obtaining the audio parameters of the original audio, wherein the audio parameters include power density spectrum and sound pressure level; The original audio is analyzed to extract its audio parameters. The audio parameters can include power density spectrum and sound pressure level. The power density spectrum reflects the energy distribution of the audio signal at different frequencies, while the sound pressure level indicates the intensity of the audio signal. By analyzing the power density spectrum of the original audio, the energy distribution of the audio signal at each frequency can be understood; by measuring the sound pressure level, the overall intensity of the audio signal can be determined.

[0082] Step E20, identifying at least one masking sound frequency point in the original audio based on the audio parameters, and obtaining the individual sound pressure level of each masking sound frequency point; The masking audio frequency point can be a frequency point meeting preset masking sound characteristics, such as a local peak point in a power density spectrum that maintains a maximum amplitude within a preset bandwidth (such as 0.5 Bark), a frequency point with a sound pressure level greater than an absolute hearing threshold of a human ear, and the like.

[0083] For each masking audio frequency point, an individual sound pressure level of the masking audio frequency point is further obtained, that is, a sound pressure level at the frequency point. The individual sound pressure level reflects the intensity of the audio signal at the frequency point.

[0084] Step E30, determining a minimum value between each of the individual sound pressure levels and the absolute hearing threshold, and setting the minimum value as the preset masking sound threshold.

[0085] For each identified masking audio frequency point, the individual sound pressure level of the masking audio frequency point is compared with the absolute hearing threshold. The absolute hearing threshold refers to the lowest sound intensity that can be perceived by a human ear, and is a fixed value, usually expressed in decibels (dB). By comparing the individual sound pressure level with the absolute hearing threshold, a minimum value between the two is determined. This minimum value will be set as the preset masking sound threshold.

[0086] The embodiment determines a minimum value between the individual sound pressure level of each masking audio frequency point and the absolute hearing threshold as the preset masking sound threshold. This threshold determination comprehensively considers the energy of the masking audio frequency point and the hearing sensitivity of the human ear, ensuring that the masking sound can have a masking effect on other sounds, and avoiding excessive processing and distortion due to the masking sound being too strong. At the same time, the masking sound threshold determined based on the original audio can more accurately reflect the actual masking ability of the audio, ensuring that the set threshold meets the actual masking characteristics of the current audio to be output, further improving the quality and effect of subsequent audio processing.

[0087] In a possible implementation, after the step of obtaining the target transfer function between the test audio device at the extended target position and the user's two ears, the method further includes: Step F10, obtaining an environmental acoustic parameter of an environment in which a target user is currently located, wherein the environmental acoustic parameter includes a reflected sound intensity, a reverberation time, and a dominant reflection direction; After obtaining the target transfer function between the test audio device at the extended target position and the user's two ears, the environmental acoustic parameter of the environment in which the target user is currently located is further obtained. These parameters include the reflected sound intensity, the reverberation time, and the dominant reflection direction. Specifically, a microphone array can be placed in the environment in which the target user is currently located, and a set of test signals can be played through the microphone array, and then the room impulse response (RIR) is collected in real time, and the environmental acoustic parameter is analyzed and calculated based on the collected room impulse response.

[0088] The reflected sound intensity refers to the energy of the reflected sound in the environment within a specific time window, such as the reflected sound energy within 5-50 ms after the test signal is emitted; the reverberation time refers to the time required for the sound to decay to a certain level in the environment, such as the time required for the test signal to decay by 60 dB; and the dominant reflection direction refers to the main propagation direction of the reflected sound, such as the direction with the strongest reflected sound.

[0089] Step F20, generating a corresponding environment compensation coefficient based on the environment acoustic parameters; After obtaining the environment acoustic parameters, a corresponding environment compensation coefficient is generated based on these parameters. The generation of the environment compensation coefficient aims to adjust the target transfer function according to the environmental acoustic characteristics, in order to compensate for the influence of the environment on audio propagation and perception.

[0090] Specifically, the corresponding relationship between different environment acoustic parameters and environment compensation coefficients can be measured by experiment or preset according to experience, so as to obtain the environment compensation coefficient corresponding to the environment acoustic parameters based on the corresponding relationship. A prediction model with environment acoustic parameters as input and environment compensation coefficients as output can also be pre-trained, so that after obtaining the environment acoustic parameters, the environment acoustic parameters are input into the prediction model, and the corresponding environment compensation coefficient is output.

[0091] It should be noted that the transfer function can be essentially represented as a complex matrix, and thus the environment compensation coefficient can be a coefficient matrix.

[0092] Step F30, multiplying the environment compensation coefficient with the target transfer function to obtain the compensated target transfer function; Step F40, performing the crosstalk cancellation processing of the individual audio based on the compensated target transfer function to obtain the target audio.

[0093] Based on the compensated target transfer function, the step of performing crosstalk cancellation processing of the individual audio based on the target transfer function is performed to obtain the target audio. The purpose of crosstalk cancellation processing is to eliminate the crosstalk effect of audio signals in the propagation process due to the mutual interference between loudspeakers, thereby improving the positioning accuracy and spatial sense of audio. Based on the compensated target transfer function, the influence of environmental acoustic characteristics on audio propagation can be more accurately considered, thereby ensuring that the target audio played subsequently can meet the acoustic transmission characteristics of the current space environment, so that better audio effects can be achieved in different environments. For example, in an environment with strong reflected sound, crosstalk cancellation processing based on the compensated target transfer function can reduce the interference of reflected sound on audio positioning, so that users can more accurately perceive the source direction of the audio, improving the immersion and realism of the audio.

[0094] Further, an audio device is also provided in the embodiments of the present application, and the audio device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the computer program is configured to implement the steps of the sound field extension method as described above.

[0095] Reference Figure 6 which shows a structural schematic diagram of an audio device suitable for implementing the embodiments of the present application. The audio device in the embodiments of the present application can also include, but is not limited to, mobile terminals such as AR headgear, VR headgear, earphones, mobile phones, servers, notebook computers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. Figure 6 The audio device shown is only an example and should not bring any limitation to the functions and use range of the embodiments of the present application.

[0096] As Figure 6 shown, the audio device can include a processing apparatus 1001 (for example, a central processor, a graphics processor, and the like) which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the audio device are also stored. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input apparatus 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, and the like; an output apparatus 1008 including, for example, a liquid crystal display (LCD), a test audio device, a vibrator, and the like; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, and the like; and a communication apparatus 1009. The communication apparatus 1009 can allow the audio device to communicate with other devices wirelessly or by wire to exchange data. Although the audio device with various systems is shown in the figure, it should be understood that all the systems shown are not required to be implemented or possessed. More or fewer systems can be alternatively implemented or possessed.

[0097] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program codes for executing the method shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network through a communication device, or installed from the storage device 1003, or installed from the ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiments disclosed in the present application are executed.

[0098] The audio device provided by the embodiments of the present application adopts the sound field expansion method in the above embodiments, which can solve the technical problem of how to effectively expand the sound field while reducing crosstalk and protecting the sound quality. Compared with the prior art, the audio device provided by the present application has the same beneficial effects as the sound field expansion method provided by the above embodiments, and other technical features in the audio device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0099] It should be understood that parts of the present application can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0100] The above is merely specific implementation of the present application, but the protection scope of the present application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0101] In addition, in order to achieve the above-mentioned purpose, the embodiments of the present application also provide a readable storage medium having computer readable program instructions (i.e. computer program) stored thereon, the computer readable program instructions being used to execute the sound field expansion method in the above embodiments.

[0102] The computer readable storage medium provided by the embodiments of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer readable storage medium may include, but are not limited to, an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments, the computer readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer readable storage medium can be transmitted by any suitable medium, including but not limited to an electric wire, an optical cable, an RF (Radio Frequency), and the like, or any suitable combination thereof.

[0103] The computer readable storage medium described above may be contained in an audio device, or may exist separately without being assembled into an audio device.

[0104] The computer readable storage medium described above carries one or more programs, which, when executed by the audio device, cause the audio device to: obtain a target transfer function between a test audio device and the user's ears at an extended target position, and obtain individual acoustic transmission data of a target user, wherein the individual acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on personal auditory physiological data of the target user; render original audio according to the individual acoustic transmission data to obtain individual audio; perform crosstalk cancellation processing on the individual audio according to the target transfer function to obtain target audio, and play the target audio.

[0105] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0106] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0107] The modules involved in the embodiments of the present application can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0108] The readable storage medium provided by the present application is a computer readable storage medium, which stores computer readable program instructions (i.e. computer program) for executing the above-mentioned sound field expansion method, and can solve the technical problem of how to effectively expand the sound field while reducing crosstalk and protecting the sound quality. Compared with the prior art, the computer readable storage medium provided by the present application has the same beneficial effects as the sound field expansion method provided by the above-mentioned embodiments, which will not be described here.

[0109] In addition, the embodiment of the present application further provides a computer program product comprising a computer program, which, when executed by a processor, implements the steps of the sound field extension method as described above.

[0110] The computer program product embodiment of the present application is basically the same as the above-mentioned sound field extension method embodiments, and will not be described here.

[0111] It should be noted that in this paper, the term "comprise", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or system. Without more limitations, the element defined by the sentence "comprises a" does not exclude the presence of other identical elements in the process, method, article or system including the element.

[0112] The above-mentioned serial numbers of the embodiments of the present application are only for description, not representing the advantages and disadvantages of the embodiments.

[0113] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment method can be realized by software plus the necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is the better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of software sensors, which are stored in a storage medium (such as ROM / RAM, magnetic disc, optical disc) as described above, including a number of instructions to make an audio device (which can be a mobile phone, computer, server or network device, etc.) execute the method described in each embodiment of the present application.

[0114] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied to other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A sound field expansion method, characterized in that: The sound field expansion method comprises the following steps: Obtaining a target transfer function between the test audio device at the expanded target location and the user's ears, and obtaining personalized acoustic transmission data of the target user, wherein the personalized acoustic transmission data includes at least one head-related transfer function (HRTF) generated based on the target user's personal auditory physiological data; Rendering the original audio according to the personalized acoustic transmission data to obtain personalized audio; The personalized audio is subjected to crosstalk cancellation processing according to the target transfer function to obtain target audio, and the target audio is played.

2. The sound field expansion method according to claim 1, wherein: The step of obtaining a target transfer function between the test audio device at the expanded target position and the user's ears includes: Obtain artificial head transfer function and free field transfer function; performing an inverse operation on the free-field transfer function to obtain an inverse free-field transfer function; The artificial head transfer function is multiplied by the free-field inverse transfer function to obtain a target transfer function between the test audio device at the extended target position and the user's ears.

3. The sound field expansion method according to claim 2, wherein: The step of obtaining the artificial head transfer function and the free field transfer function comprises: When the test audio device is placed at the expansion target position and the test audio device outputs a sound signal, the artificial head transfer function is measured by a preset microphone in the ear canal of the preset artificial head; and When the preset artificial head is removed and the test audio device outputs a sound signal, a free-field transfer function is measured by using preset microphones placed at the left and right ear positions before the preset artificial head is removed.

4. The sound field expansion method according to claim 2, wherein: The step of performing crosstalk cancellation processing on the personalized audio according to the target transfer function to obtain the target audio includes: performing an inverse operation on the target transfer function to obtain a target inverse transfer function; The personalized audio is multiplied by the target inverse transfer function to obtain the target audio.

5. The sound field expansion method according to claim 1, wherein: The step of obtaining the target user's personalized acoustic transmission data includes: A pre-set model is trained using the acquired group auditory training dataset to obtain a trained general model, wherein the group auditory training dataset includes a plurality of human auditory physiological data and respective corresponding HRTF labels, and the general model includes a general encoder and a general decoder; Acquiring personal auditory physiological data of the target user; A universal decoder is obtained from the universal model, and personalized training is performed on the universal decoder using the personal auditory physiological data to obtain a personalized decoder. Personalized acoustic transmission data corresponding to the personal auditory physiological data is generated based on the personalized decoder.

6. The sound field expansion method according to any one of claims 1 to 5, characterized in that: Before the step of rendering the original audio according to the personalized acoustic transmission data to obtain the personalized audio, the method further includes: Obtaining the amplitude spectrum and phase spectrum of the individual acoustic transmission data, and obtaining a preset covert sound threshold; Adjusting the amplitude spectrum based on the preset covert sound threshold to obtain the adjusted amplitude spectrum, wherein the adjustment of the amplitude spectrum includes reducing the amplitude value of a target frequency point, the target frequency point being a frequency point in the amplitude spectrum having an amplitude value less than the preset covert sound threshold; combining the adjusted amplitude spectrum and the phase spectrum to obtain the adjusted individual acoustic transmission data; Based on the adjusted personalized acoustic transmission data, the step of rendering the original audio according to the personalized acoustic transmission data to obtain personalized audio is performed.

7. The sound field expansion method according to claim 6, wherein: Before the step of obtaining the preset covert sound threshold, the method further includes: Acquiring audio parameters of the original audio, wherein the audio parameters include power density spectrum and sound pressure level; identifying at least one masking audio frequency point in the original audio based on the audio parameters, and obtaining individual sound pressure levels of each masking audio frequency point; The minimum value between each of the individual sound pressure levels and the absolute hearing threshold is determined, and the minimum value is set as the preset covert sound threshold.

8. The sound field expansion method according to any one of claims 1 to 5, characterized in that: After the step of obtaining the target transfer function between the test audio device at the expanded target location and the user's ears, the method further includes: Acquiring the ambient acoustic parameters of the target user's current environment, wherein the ambient acoustic parameters include reflected sound intensity, reverberation time, and dominant reflection direction; generating a corresponding environmental compensation coefficient based on the environmental acoustic parameter; multiplying the target transfer function by the environmental compensation coefficient to obtain the compensated target transfer function; Based on the compensated target transfer function, the step of performing crosstalk cancellation processing on the personalized audio according to the target transfer function to obtain the target audio is performed.

9. An audio device, characterized in that include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the sound field expansion method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a sound field expansion program, which, when executed by a processor, implements the steps of the sound field expansion method according to any one of claims 1 to 8.

Images