Audio data processing method, audio data processing device, medium and electronic equipment

Abstract

The invention provides an audio data processing method, an audio data processing device, a computer readable storage medium and electronic equipment, and relates to the technical field of audio processing. The audio data processing method comprises the following steps: acquiring to-be-processed audio data collected in an original sound field, wherein the to-be-processed audio data comprises to-be-processed audio data corresponding to various equipment attitude data; acquiring a head-related transformation function corresponding to each piece of equipment attitude data; determining conversion parameters between the original sound field and a target sound field according to a loudspeaker array of the target sound field; and determining the audio data of the target sound field based on the conversion parameters, the head-related transformation function and the audio data to be processed under the various equipment attitude data. According to the invention, the to-be-processed audio data in the original sound field can be accurately and conveniently processed to obtain the audio data in the target sound field.

Description

Background Art

[0002] Spatial audio is a stereo and surround sound audio technology that can bring users a more immersive and spatial audio listening experience. In the prior art, audio data with specific spatial audio effects usually relies on professional mixing producers to produce original audio data through specific hardware equipment or technology, or is collected through microphones in a professional sound field. The hardware and labor costs are high, and the production process is complicated. Therefore, how to simply and effectively process audio data to obtain audio data in the expected sound field is a technical problem that needs to be solved in the prior art. Summary of the invention

[0003] The present disclosure provides an audio data processing method, an audio data processing device, a computer-readable storage medium and an electronic device, thereby at least to a certain extent solving the problem that it is difficult to conveniently and effectively obtain audio data in an expected sound field in the prior art.

[0004] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0005] According to a first aspect of the present disclosure, there is provided an audio data processing method, comprising: obtaining audio data to be processed collected in an original sound field, the audio data to be processed comprising audio data to be processed corresponding to a plurality of device posture data; obtaining a head-related transformation function corresponding to each of the device posture data; determining, according to a speaker array of a target sound field, a transformation parameter between the original sound field and the target sound field; and determining the audio data of the target sound field based on the transformation parameter, the head-related transformation function and the audio data to be processed under a plurality of device posture data.

[0006] According to a second aspect of the present disclosure, there is provided an audio data processing apparatus, comprising: a data acquisition module, for acquiring audio data to be processed collected in an original sound field, the audio data to be processed comprising audio data to be processed corresponding to a plurality of device posture data; a function acquisition module, for acquiring a head-related transformation function corresponding to each of the device posture data; a parameter determination module, for determining, according to a speaker array of the target sound field, a transformation parameter between the original sound field and the target sound field; and an audio determination module, for determining the audio data of the target sound field based on the transformation parameter, the head-related transformation function and the audio data to be processed under a plurality of device posture data.

[0007] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the audio data processing method of the first aspect and possible implementation methods thereof are implemented.

[0008] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor, wherein the processor is configured to execute the audio data processing method of the first aspect and possible implementation thereof by executing the executable instructions.

[0009] The technical solution disclosed in this disclosure has the following beneficial effects:

[0010] The audio data to be processed collected in the original sound field is obtained, and the audio data to be processed includes the audio data to be processed corresponding to a plurality of device posture data; the head-related transformation function corresponding to each device posture data is obtained; according to the speaker array of the target sound field, the conversion parameters between the original sound field and the target sound field are determined; based on the conversion parameters, the head-related transformation function and the audio data to be processed under the plurality of device posture data, the audio data of the target sound field is determined. On the one hand, this exemplary embodiment is based on the audio data to be processed and the device posture data collected in the original sound field, combined with the conversion parameters between the original sound field and the target sound field, and the head-related transformation function, that is, the audio data under the target sound field can be accurately and effectively calculated, and the audio processing process is highly convenient and has a wide range of applications; on the other hand, compared with the prior art that relies on the mixing production technology of professional mixing producers or the audio collection technology of professional sound fields, this exemplary embodiment, combined with the actual requirements for the target sound field, can obtain the audio data under the target sound field by processing the audio data to be processed in the original sound field, which greatly saves the hardware cost and manpower cost of spatial audio data processing and has a relatively wide range of applications.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0013] Figure 1 A flowchart showing a method for processing audio data in this exemplary embodiment is shown;

[0014] Figure 2 A schematic diagram showing the layout of a speaker array in this exemplary embodiment is shown;

[0015] Figure 3 A spatial schematic diagram showing another loudspeaker array in this exemplary embodiment;

[0016] Figure 4 A sub-flow chart showing a method for processing audio data in this exemplary embodiment;

[0017] Figure 5 A schematic diagram of spherical coordinates in this exemplary embodiment is shown;

[0018] Figure 6 A schematic diagram showing a process of binaural signal audio acquisition in this exemplary embodiment is shown;

[0019] Figure 7 A schematic diagram showing signal transmission of sound field reconstruction in this exemplary embodiment is shown;

[0020] Figure 8 A schematic diagram of a process for determining spatial audio data in this exemplary embodiment is shown;

[0021] Fig. 9 A structural block diagram of an audio data device in this exemplary embodiment is shown;

[0022] Fig.10 A structural diagram of an electronic device in this exemplary embodiment is shown. DETAILED DESCRIPTION

[0023] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0024] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0025] Combine the following Figure 1 This section describes the flow of the audio data processing method. Figure 1 As shown, the audio data processing method may include the following steps S110 to S140:

[0026] Step S110, obtaining the audio data to be processed collected in the original sound field, wherein the audio data to be processed includes the audio data to be processed corresponding to a plurality of device posture data.

[0027] Among them, the audio data to be processed refers to the audio data that needs to be processed in this exemplary embodiment, which can be different types of sound data, such as voice audio data, music audio data, notification audio data, etc. The audio data to be processed can be monophonic audio data or dual-channel audio data, etc. This exemplary embodiment can obtain the audio data to be processed from different sound sources, and can also obtain the audio data to be processed in a variety of different ways, for example, it can obtain the audio data to be processed from audio, video and other applications, and can also obtain the audio data to be processed by recording audio on the spot in real time by real-time sound collection. When the audio data to be processed is generated, the corresponding recording sound field is the original sound field. For example, when the music is obtained from the audio application, the recording environment in which it is located when the music is recorded is the original sound field; or when the voice audio is recorded on the spot by real-time sound collection, the recording environment on the spot is the original sound field, and so on.

[0028] The device corresponding to the audio data to be processed refers to a sound receiving device used to collect the audio data to be processed. The position of the device in the original sound field can be used to simulate the position of a person receiving the audio data. The device can receive sounds from different sound sources in the original sound field. For example, in the original sound field, there is a speaker on each side of the device, and the device can receive audio data emitted by the speakers on both sides. The device posture data corresponding to the audio data to be processed refers to data used to reflect the position or state of the device in the original sound field, such as the position coordinates, orientation, elevation angle, motion trajectory, motion direction, or the distance or angle from each speaker of the device. The recorded audio data to be processed can change according to the posture change of the recording device. For example, when the recording device simulates the rotation of the human ear, the volume or spatial sense of the audio data received on both sides will change. When acquiring the audio data to be processed, this exemplary embodiment can acquire the audio data to be processed corresponding to a plurality of device posture data, wherein the plurality of device posture data may include a plurality of postures of the device rotating in the horizontal direction or a pitch posture in the vertical direction, or a posture of moving and changing on a plane, etc. The device posture data of the recording device corresponding to the audio data to be processed can be obtained by configuring corresponding sensors in the device. For example, a sensor device such as an accelerometer or a gyroscope can be configured in the device to obtain acceleration sensor data and posture data, and then determine the device posture data based on the sensor data.

[0029] Step S120, obtaining a head-related transformation function corresponding to each device posture data.

[0030] Among them, the head-related transformation function refers to the description of the transmission process of sound waves from the sound source to the ears, and contains information about the location of the sound source. Usually, the sound waves emitted by the sound source reach the ears after being scattered by the head, auricle, trunk, etc. The physical process can be regarded as a linear time-invariant sound filtering system, and its characteristics can be fully described by the frequency domain transfer function of the system, which is the head-related transformation function. In this exemplary embodiment, different device posture data can correspond to different head-related transformation functions. The head-related transformation function can be determined based on pre-configured data information, based on the device posture data, or calculated based on the device posture data, etc., and the present disclosure does not make specific limitations on this.

[0031] Step S130 : determining conversion parameters between the original sound field and the target sound field according to the speaker array of the target sound field.

[0032] Among them, the target sound field refers to the reconstructed sound field in which the audio data is expected to be played after audio processing. Generally, when the sound source is located at different positions in the sound field space, different sound effects can be presented. The target sound field in this exemplary embodiment can be an expected sound field or a virtual sound field. By processing the audio data to be processed, it can be converted into a sound effect of the virtual sound sources at multiple specific positions in the target sound field, so that the user feels that the sound seems to be emitted from the location of the virtual sound source in the target sound field. For example, the original sound field is a sound field including two speakers, or a binaural recording scene, and the target sound field is a sound field including 6 speakers, which has a certain spatial audio effect, and the sound effects are different. The speaker array can reflect information such as the arrangement of the sound sources in the sound field environment, the layout position or the number of sound sources. For example Figure 2 The 5.1 speaker array shown includes three front speakers 210, 220, 230, which are placed in front of the audience A, namely, the left front, the center front, and the right front, two rear speakers 240, 250, which are placed behind the audience, namely, the left rear and the right rear, and a subwoofer sound source 260, which can be placed at a random position or a preset position. The position or layout of the specific speakers can be pre-configured or custom-set, for example, the target sound field can be the sound field of a 5.1 speaker array, or the sound field of a 7.1 speaker array, etc. Figure 3 A schematic diagram of a virtual loudspeaker array in space is shown. Figure 3 In the spatial coordinates shown, there are four speakers, namely a first speaker 310, a second speaker 320, a third speaker 330 and a fourth speaker. The relative angles corresponding to the speakers are respectively the first speaker 210 (0°, 0°), the second speaker 220 (120°, 109°), the third speaker (0°, 109°), and the fourth speaker (-120°, 109°). Figure 3 This is only described as an example of a speaker array, and speaker information such as the number and layout of the specific speaker array can be customized according to actual needs.

[0033] In this exemplary embodiment, after determining the target sound field, the speaker array therein can be determined. For example, when the target sound field is a 5.1 speaker array, the speaker array of 6 speakers included therein can be determined. Further, the conversion parameters between the original sound field and the target sound field can be determined according to the speaker array of the target sound field. The conversion parameters can reflect the mapping relationship between the target sound field and the original sound field, and the original sound field can be transformed to the target sound field through the conversion parameters. In this exemplary embodiment, the conversion parameters can be a conversion matrix determined according to the speaker array, expressed as invMtr or invert Matrix.

[0034] This exemplary embodiment can utilize a speaker array to determine conversion parameters between the original sound field and the target sound field, and then process the audio data to be processed to present the sound field distribution of the spatial audio corresponding to the target sound field in a specific target sound field, so that the audience can feel as if they have the listening experience of the target sound field and experience the real sound effect and sound quality after the sound field reconstruction.

[0035] Step S140 , determining audio data of the target sound field based on the conversion parameters, the head-related transfer function, and the audio data to be processed under various device posture data.

[0036] The audio data to be processed under multiple device posture data refers to the audio data to be processed obtained when the audio receiving device is in different postures. For example, when there is a speaker on each side of the original sound field, the audio data to be processed received will be different when the angle or direction of the receiving device is rotated. The audio data to be processed under multiple device posture data can be the audio data to be processed collected under multiple pre-set device postures.

[0037] This exemplary embodiment determines the audio data of the target sound field based on the conversion parameters, the head-related transformation function, and the audio data to be processed under the various device posture data. The audio data may include the sound field information of the target sound field, the position of each sound source, and the volume of the audio emitted by each sound source.

[0038] In summary, in this exemplary embodiment, the audio data to be processed collected in the original sound field is obtained, and the audio data to be processed includes the audio data to be processed corresponding to a plurality of device posture data; the head-related transformation function corresponding to each device posture data is obtained; according to the speaker array of the target sound field, the conversion parameters between the original sound field and the target sound field are determined; based on the conversion parameters, the head-related transformation function and the audio data to be processed under the plurality of device posture data, the audio data of the target sound field is determined. On the one hand, this exemplary embodiment is based on the audio data to be processed and the device posture data collected in the original sound field, combined with the conversion parameters between the original sound field and the target sound field, and the head-related transformation function, that is, the audio data under the target sound field can be accurately and effectively calculated, and the audio processing process is highly convenient and has a wide range of applications; on the other hand, compared with the prior art that relies on the mixing production technology of professional mixing producers or the audio collection technology of professional sound fields, this exemplary embodiment, combined with the actual requirements for the target sound field, can obtain the audio data under the target sound field by processing the audio data to be processed in the original sound field, which greatly saves the hardware cost and manpower cost of spatial audio data processing, and has a relatively wide range of applications.

[0039] In an exemplary embodiment, if Figure 4 As shown, the above step S130 may include:

[0040] Step S410, establishing an audio rendering equation from the target sound field to the audio data to be processed based on the spherical harmonics of the target sound field;

[0041] Step S420: Process the conversion parameters, the head-related transformation function, and the to-be-processed audio data under various device posture data through the audio rendering equation to determine the audio data of the target sound field.

[0042] Among them, the spherical harmonic function is the angular part of the solution of the Laplace equation in the spherical coordinate system. After the variables are separated from the expression of the Laplace equation in the spherical coordinate system, the partial differential equation of the angular part can become a spherical harmonic function. That is, the present exemplary embodiment can obtain the spherical harmonic function of the target sound field by deducing the angular part of the solution of the Laplace equation in the spherical coordinate system. Furthermore, the audio data of the target sound field can be expressed by the spherical harmonic function, that is, based on the spherical harmonic function of the target sound field, an audio rendering equation from the target sound field to the audio data to be processed can be established.

[0043] Specifically, spherical coordinates can be expressed as:

[0044]

[0045] The Laplace equation in three-dimensional space can be expressed as:

[0046]

[0047] Substituting the above spherical coordinates into the Laplace equation, we can get the following equation:

[0048]

[0049] in, represents the Laplace operator, represents the azimuth angle, θ represents the zenith angle, such as Figure 5 shown.

[0050] Solving the audio data of the target sound field through the audio rendering equation may refer to the process of performing audio rendering processing on the audio to be processed to obtain audio data with an audio space effect in the target sound field.

[0051] like Figure 6As shown, in the original sound field, assuming that the binaural signal audio 640 collected by the listener or the sound receiving device is obtained by means of "three-dimensional continuous sound field 610" + "virtual array rendering 620" + "personalized head-related transformation function 630", then it can satisfy the preset audio rendering equation, and by substituting the conversion parameters, head-related transformation function and the audio data to be processed under various device posture data into the audio rendering equation, the audio data of the target sound field can be solved.

[0052] In an exemplary embodiment, the above-mentioned audio data to be processed includes left channel data and right channel data, that is, the audio data to be processed is binaural signal audio; the audio rendering equation can be expressed as:

[0053] hoaSig·invMtr·HL=sigL

[0054] hoaSig·invMtr·HR=sigR

[0055] Among them, hoaSig represents the audio data of the target sound field, invMtr represents the conversion parameter, HL represents the head-related transformation function of the left ear, HR represents the head-related transformation function of the right ear, sigL represents the left channel data in the audio data to be processed, and sigR represents the right channel data in the audio data to be processed. Among them, HL and HR are a set of corresponding head-related transformation functions. When the device posture changes, the corresponding head-related transformation function will also change, and it will correspond to different combinations of the head-related transformation function HL of the left ear and the head-related transformation function HR of the right ear.

[0056] In this exemplary embodiment, hoaSig is a kind of sound field information, which can be expressed by spherical harmonics. The more coefficients of spherical harmonics, the more complete the expression of the sound field; the head-related transformation function HL of the left ear and the head-related transformation function HR of the right ear can be directly obtained by looking up the table; according to the given speaker array, the conversion parameter invMtr can be determined; the left channel data sigL and the right channel data sigR are recorded left and right channel signals, which can be determined according to the recorded audio data to be processed. Therefore, substituting the conversion parameters, the head-related transformation function, and the audio data to be processed under various device posture data into the audio rendering equation to solve the audio data of the target sound field can be transformed into a solution process for the unknown variable hoaSig.

[0057] In this exemplary embodiment, after determining the speaker array of the target sound field, the conversion parameters corresponding to the speaker array, such as the conversion matrix, can be determined. The processing of audio data is actually the reconstruction of the virtual speaker sound field in the target sound field. This exemplary embodiment can use the conversion parameters corresponding to the speaker array of the target sound field to reconstruct the target sound field. Figure 7As shown, given the ideal reconstruction signal P, for a given space and speaker distribution, if the acoustic transfer function G from the speaker to the reconstruction area is obtained, then a linear equation system GS = P can be established to solve the speaker input signal S in the sense of least squares, so that the reconstructed target sound field can be well matched with the original sound field, thus achieving the purpose of truly reproducing the sound field. Among them, the reconstruction signal P refers to the signal of the virtual speaker obtained using the transfer function and the conversion parameters; G refers to the fixed transfer function determined according to the size of the speaker array to be reconstructed; E represents the system energy.

[0058] In an exemplary embodiment, the above step S420 may include:

[0059] Performing Fourier transform on the conversion parameters, the head-related transformation function, and the audio data to be processed under the posture data of each device;

[0060] Determine intermediate audio data according to the conversion parameters after Fourier transformation, the head-related transformation function, and the audio data to be processed;

[0061] Perform an inverse Fourier transform on the intermediate audio data to obtain the audio data of the target sound field.

[0062] In this exemplary embodiment, an audio rendering equation is taken as: h·i·l=s for example, wherein h represents the audio data of the target sound field, i represents the conversion parameter, l represents the head-related transformation function, and s represents the channel data in the audio data to be processed. After substituting the conversion parameter, the head-related transformation function, and the audio data to be processed under various device posture data into the audio rendering equation, the conversion parameter, the head-related transformation function, and the audio data to be processed under each device posture data can be first Fourier transformed, which can be expressed as: H×I×L=S. It can be seen that the audio data of the target sound field required to be solved in this exemplary embodiment can be obtained based on H, and according to the conversion parameter, the head-related transformation function, and the audio data to be processed after the Fourier transformation, the intermediate audio data can be calculated, for example, the intermediate audio data can be expressed as: H=S / (I×L), and finally the intermediate audio data is inversely Fourier transformed, for example, H is inversely Fourier transformed, and the convolution signal h can be determined, that is, the audio data of the target sound field can be obtained.

[0063] In an exemplary embodiment, the above step S420 may include the following steps:

[0064] By means of audio rendering, the conversion parameters, the head-related transformation function corresponding to each device posture data, and the audio data to be processed under each device posture data are processed to determine the audio data relationship corresponding to each device posture data;

[0065] The audio data of the target sound field is determined by combining the audio data relationship formulas corresponding to the posture data of various devices.

[0066] In practical applications, for example, when the audio data to be processed is binaural recording, including left channel data and right channel data, if the order of the audio data hoaSig of the target sound field is required to be sufficient, that is, the coefficients of the spherical harmonics are sufficient, it is necessary to apply the device posture data, such as IMU (inertial measurement unit) data, and the recorded audio data to be processed, such as a combination of binaural audio data, within a period of time, that is, a slight rotation of the human head or the recording device during the recording process can optimize the coefficients of the spherical harmonics.

[0067] Therefore, this exemplary embodiment can utilize the slight rotation of the recording device during the recording process to optimize the spherical harmonic coefficients, and the rotation of the recording device can determine different device postures, and then correspond to different device posture data. This exemplary embodiment can determine the audio rendering equations of different postures under each device posture: hoaSig·invMtr·HL=sigL, wherein sigL will vary with the rotation of the device. Then, the conversion parameters, the head-related transformation function corresponding to each device posture data, and the audio data to be processed under each device posture data are substituted into the audio rendering equation, and the audio data relationship corresponding to each device posture data can be obtained. If multiple audio data relationship equations need to be satisfied at the same time, the coefficients of hoaSig need to be increased, so as to achieve the effect of optimizing the spherical harmonic coefficients and then solve the audio data of the target sound field.

[0068] In an exemplary embodiment, after determining the audio data of the target sound field, the audio data processing method further includes:

[0069] The audio data of the target sound field is encoded in a preset multi-channel format to obtain spatial audio data.

[0070] Among them, spatial audio data refers to audio data with spatial audio effects when played, which can restore the spatial sense of sound emitted by each speaker in the target sound field, and provide listeners with a real and rich listening experience. After determining the audio data of the target sound field, this exemplary embodiment can encode its audio data in a preset multi-channel format, wherein the preset multi-channel format can refer to the channel encoding format corresponding to the speaker array of the target sound field. For example, when the target sound field is a 5.1 speaker array, the number of channels is 6, and the audio data of the target sound field can be encoded according to the preset multi-channel format, so that the encoded spatial audio data can present the spatial audio effect of the 5.1 speaker array through 6 channels.

[0071] Figure 8A schematic diagram of a process for determining spatial audio data in this exemplary embodiment is shown, which may specifically include: obtaining recorded audio data to be processed, the audio data to be processed including binaural recording signals 810; processing the audio data to be processed by the above-mentioned audio data processing algorithm 820 to determine the audio data of the target sound field, the audio data of the target sound field including 5.1 multi-channel audio data 830; performing multi-channel encoding processing 840 on the 5.1 multi-channel audio data by a preset multi-channel format to determine spatial audio data 850.

[0072] This exemplary embodiment, on the one hand, can convert binaurally recorded audio data to be processed into multi-channel spatial audio data, so that a multi-perspective spatial audio playback experience can be presented based on the spatial audio data, ensuring that the sound source object recorded in the recording maintains the correct spatial orientation during spatial audio playback, thereby enhancing the user's sense of immersion; on the other hand, compared with the existing need for professional mixing processing to provide spatial audio data with spatial audio effects, this exemplary embodiment produces spatial audio data through binaural recording, without the need for professional production by a mixer or the use of professional sound field acquisition equipment, greatly reducing the hardware cost of producing spatial audio data and allowing more people to conveniently produce spatial audio data; on another hand, this exemplary embodiment encodes the audio data of the target sound field through a preset multi-channel format, and can save the spatial audio data in a universal multi-channel format, thereby ensuring the universality of the generated spatial audio data content, and can achieve playback on devices that support the preset multi-channel spatial audio, meeting the needs of users to share audio across devices.

[0073] In an exemplary embodiment, the number of channels of the preset multi-channel format is greater than the number of channels of the audio data to be processed.

[0074] In particular, considering that this exemplary embodiment can be applied to the sound field reconstruction scenario of spatial audio, such as processing binaural audio signals into 5.1 multi-channel audio data, the number of channels of the preset multi-channel format can be set to be greater than the number of channels of the audio data to be processed, so as to ensure that the processed spatial audio data has rich and diverse spatial audio effects.

[0075] In an exemplary embodiment, the audio data of the target sound field is sound source distribution data based on an n-order; the order can be customized according to actual scene requirements, or determined according to requirements for solving the sound distribution of the audio data of the target sound field. Specifically, after step S110, the audio data processing method further includes:

[0076] The value of n is determined according to the number of device postures represented by the device posture data.

[0077] Assuming that the listener is located at the center of a sphere, and the ears receive sounds from all directions on the sphere, if the recorded spatial sound field can be transmitted to the ears in this way, rather than just the two speakers in front, it will provide a more credible and immersive experience. Such a system is an advanced technology in sound field reconstruction technology.

[0078] The audio data of the target sound field in this exemplary embodiment can be sound source distribution data based on order n. The audio data hoaSig of the target sound field can correspond to multiple unknowns according to the set order. For example, when the order is set to 4, 9 or 16, 4, 9 or 16 hoaSig unknowns can be estimated. The more orders are set, the more coefficients of hoaSig can be solved, thereby realizing the expression of sound distribution at different points in the spherical space by audio data hoaSig of different orders.

[0079] In this exemplary embodiment, each device posture data can provide an audio data relationship formula, and the audio data of the target sound field can be determined by solving multiple audio data relationship formulas. The more device posture data there are, the more audio data relationship formulas there are, and the larger the corresponding order n is. Therefore, this exemplary embodiment can determine the order n according to the number of device postures represented by the device posture data.

[0080] The exemplary embodiment of the present disclosure also provides an audio data processing device. Fig. 9 As shown, the audio data processing device 900 may include: a data acquisition module 910, used to acquire the audio data to be processed collected in the original sound field, the audio data to be processed includes the audio data to be processed corresponding to a plurality of device posture data; a function acquisition module 920, used to acquire the head-related transformation function corresponding to each device posture data; a parameter determination module 930, used to determine the conversion parameters between the original sound field and the target sound field according to the speaker array of the target sound field; an audio determination module 940, used to determine the audio data of the target sound field based on the conversion parameters, the head-related transformation function and the audio data to be processed under a plurality of device posture data.

[0081] In an exemplary embodiment, the audio determination module includes: an equation establishing unit, which is used to establish an audio rendering equation from the target sound field to the audio data to be processed based on the spherical harmonics of the target sound field; a data solving unit, which is used to process the conversion parameters, head-related transformation functions and the audio data to be processed under various device posture data through the audio rendering equation to determine the audio data of the target sound field.

[0082] In an exemplary embodiment, the data solution unit includes: a Fourier transform subunit, which is used to perform Fourier transform on the conversion parameters, head-related transform functions and audio data to be processed under each device posture data; an intermediate data calculation subunit, which is used to determine the intermediate audio data based on the conversion parameters, head-related transform functions and audio data to be processed after Fourier transform; and an inverse Fourier transform subunit, which is used to perform inverse Fourier transform on the intermediate audio data to obtain audio data of the target sound field.

[0083] In an exemplary embodiment, a data solving unit includes: a relationship obtaining subunit, which is used to process conversion parameters, head-related transformation functions corresponding to each type of device posture data, and audio data to be processed under each type of device posture data through an audio rendering method, and determine the audio data relationship corresponding to each type of device posture data; an audio data solving subunit, which is used to determine the audio data of the target sound field by combining the audio data relationship corresponding to multiple types of device posture data.

[0084] In an exemplary embodiment, the audio data to be processed includes left channel data and right channel data; the audio rendering equation is:

[0085] hoaSig·invMtr·HL=sigL

[0086] hoaSig·invMtr·HR=sigR

[0087] Among them, hoaSig represents the audio data of the target sound field, invMtr represents the conversion parameter, HL represents the head-related transformation function of the left ear, HR represents the head-related transformation function of the right ear, sigL represents the left channel data in the audio data to be processed, and sigR represents the right channel data in the audio data to be processed.

[0088] In an exemplary embodiment, the audio data processing apparatus further includes: an encoding unit for encoding the audio data of the target sound field in a preset multi-channel format after determining the audio data of the target sound field to obtain spatial audio data.

[0089] In an exemplary embodiment, the number of channels of the preset multi-channel format is greater than the number of channels of the audio data to be processed.

[0090] In an exemplary embodiment, the audio data of the target sound field is sound source distribution data based on order n; the audio data processing device also includes: an n value determination unit, which is used to determine the value of n according to the number of device postures represented by the device posture data after obtaining the audio data to be processed collected in the original sound field and the device posture data corresponding to the audio data to be processed.

[0091] The specific details of each part of the above device have been described in detail in the implementation method part, so they will not be repeated here.

[0092] The exemplary embodiments of the present disclosure further provide a computer-readable storage medium, which can be implemented in the form of a program product, including program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification, for example, to execute Figure 1 or Figure 4 The program product may be in the form of a portable compact disk read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0093] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory, a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0094] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0095] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0096] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0097] The exemplary embodiments of the present disclosure further provide an electronic device. The electronic device may include a processor and a memory, wherein the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned audio data processing method by executing the executable instructions.

[0098] Below Fig.10 The mobile terminal 1000 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.10 The construction in can also be applied to fixed type equipment.

[0099] like Fig.10 As shown, the mobile terminal 1000 may specifically include: a processor 1001, a memory 1002, a bus 1003, a mobile communication module 1004, an antenna 1, a wireless communication module 1005, an antenna 2, a display screen 1006, a camera module 1007, an audio module 1008, a power module 1009 and a sensor module 1010.

[0100] Processor 1001 may include one or more processing units, for example: processor 1001 may include AP (Application Processor), modem processor, GPU (Graphics Processing Unit), ISP (Image Signal Processor), controller, encoder, decoder, DSP (Digital Signal Processor), baseband processor and / or NPU (Neural-Network Processing Unit), etc.

[0101] The encoder can encode (i.e. compress) an image or video to reduce the data size for easy storage or transmission. The decoder can decode (i.e. decompress) the encoded data of an image or video to restore the image or video data. The mobile terminal 1000 can support one or more encoders and decoders, such as: image formats such as JPEG (Joint Photographic Experts Group), PNG (Portable Network Graphics), BMP (Bitmap), and video formats such as MPEG (Moving Picture Experts Group) 1, MPEG10, H.1063, H.1064, and HEVC (High Efficiency Video Coding).

[0102] The processor 1001 may be connected to the memory 1002 or other components via a bus 1003 .

[0103] The memory 1002 may be used to store computer executable program codes, which may include instructions. The processor 1001 executes various functional applications and data processing of the mobile terminal 1000 by running the instructions stored in the memory 1002. The memory 1002 may also store application data, such as images, videos, and other files.

[0104] The communication function of the mobile terminal 1000 can be implemented by the mobile communication module 1004, antenna 1, wireless communication module 1005, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1004 can provide 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1000. The wireless communication module 1005 can provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. applied to the mobile terminal 1000.

[0105] The display screen 1006 is used to implement display functions, such as displaying user interfaces, images, videos, etc. The camera module 1007 is used to implement shooting functions, such as shooting images, videos, etc. The audio module 1008 is used to implement audio functions, such as playing audio, collecting voice, etc. The power module 1009 is used to implement power management functions, such as charging the battery, powering the device, monitoring the battery status, etc. The sensor module 1010 may include one or more sensors for implementing corresponding sensing detection functions. For example, the sensor module 1010 may include an inertial sensor, which is used to detect the motion posture of the mobile terminal 1000 and output inertial sensing data.

[0106] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or implementations combining hardware and software aspects, which may be collectively referred to herein as "circuit", "module" or "system". Those skilled in the art will readily think of other implementations of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptive changes of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and implementation are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0107] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. An audio data processing method, characterized in that: include: Acquire audio data to be processed collected in the original sound field, wherein the audio data to be processed includes audio data to be processed corresponding to a plurality of device posture data; Obtaining a head-related transformation function corresponding to each of the device posture data; Determining, according to the speaker array of the target sound field, a conversion parameter between the original sound field and the target sound field; Based on the conversion parameters, the head-related transformation function, and the to-be-processed audio data under a plurality of device posture data, the audio data of the target sound field is determined.

2. The method according to claim 1, characterized in that The step of determining the audio data of the target sound field based on the conversion parameters, the head-related transformation function, and the audio data to be processed under the various device posture data includes: Based on the spherical harmonics of the target sound field, establishing an audio rendering equation from the target sound field to the audio data to be processed; The conversion parameters, the head-related transformation function and the to-be-processed audio data under various device posture data are processed through the audio rendering equation to determine the audio data of the target sound field.

3. The method according to claim 2, characterized in that The step of processing the conversion parameter, the head-related transformation function, and the to-be-processed audio data under a plurality of device posture data by using the audio rendering equation to determine the audio data of the target sound field includes: Performing Fourier transform on the conversion parameter, the head-related transformation function and the audio data to be processed under each of the device posture data; Determine intermediate audio data according to the conversion parameters after Fourier transformation, the head-related transformation function, and the audio data to be processed; Performing an inverse Fourier transform on the intermediate audio data to obtain audio data of the target sound field.

4. The method according to claim 2, characterized in that: The step of processing the conversion parameter, the head-related transformation function, and the to-be-processed audio data under a plurality of device posture data by using the audio rendering equation to determine the audio data of the target sound field includes: By means of the audio rendering method, the conversion parameters, the head-related transformation function corresponding to each type of device posture data, and the to-be-processed audio data under each type of device posture data are processed to determine an audio data relationship equation corresponding to each type of device posture data; The audio data of the target sound field is determined by combining the audio data relationship expressions corresponding to the multiple device posture data.

5. The method according to claim 2, characterized in that: The audio data to be processed includes left channel data and right channel data; the audio rendering equation is: hoaSig·invMtr·HL=sigL hoaSig·invMtr·HR=sigR Among them, hoaSig represents the audio data of the target sound field, invMtr represents the conversion parameter, HL represents the head-related transformation function of the left ear, HR represents the head-related transformation function of the right ear, sigL represents the left channel data in the audio data to be processed, and sigR represents the right channel data in the audio data to be processed.

6. The method according to any one of claims 1 to 5, characterized in that: After determining the audio data of the target sound field, the method further includes: The audio data of the target sound field is encoded in a preset multi-channel format to obtain spatial audio data.

7. The method according to claim 1, characterized in that The audio data of the target sound field is sound source distribution data based on order n; After acquiring the audio data to be processed collected in the original sound field, the method further includes: The value of n is determined according to the number of device postures represented by the device posture data.

8. An audio data processing device, characterized in that: include: A data acquisition module, used to acquire the audio data to be processed collected in the original sound field, wherein the audio data to be processed includes the audio data to be processed corresponding to a plurality of device posture data; A function acquisition module, used for acquiring a head-related transformation function corresponding to each of the device posture data; A parameter determination module, used to determine the conversion parameters between the original sound field and the target sound field according to the speaker array of the target sound field; The audio determination module is used to determine the audio data of the target sound field based on the conversion parameters, the head-related transformation function and the audio data to be processed under multiple device posture data.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 7 by executing the executable instructions.

Images