Audio processing method and electronic equipment

Abstract

The embodiment of the invention provides an audio processing method and electronic equipment. The method comprises the steps that firstly, a source audio signal, a corresponding sound source position of the source audio signal in a virtual scene and a corresponding first position of a user in the virtual scene are acquired, and the virtual scene is obtained by modeling a real scene; according to the sound source position and the first position, obtaining a first acoustic feature, the first acoustic feature being obtained by adjusting a second acoustic feature according to the conversion information, and the second acoustic feature being obtained by performing acoustic feature extraction according to the first position, the virtual scene and the sound source position, the conversion information is used for describing a conversion relationship between acoustic features of the same position in the virtual scene and the real scene; a spatial audio signal is then generated from the first acoustic feature and the source audio signal. In this way, the user can experience the spatial audio signal with high quality and strong sense of immediacy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing, and in particular to an audio processing method and electronic device. Background Art

[0002] At present, many electronic devices (such as mobile phones, augmented reality (AR) devices, virtual reality (VR) devices, etc.) have spatial audio functions, which can render source audio signals into spatial audio signals, so that users can perceive the position, distance and spatial sense of sound and image in the audio, bringing users an immersive listening experience.

[0003] Generally, the method of rendering the source audio signal into a spatial audio signal is to render the source audio signal using the acoustic response of the target space. However, the acoustic response generated by existing computing technology is often different from the acoustic response in the real scene, which makes users experience low quality of spatial audio signal and weak sense of presence. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides an audio processing method and an electronic device, which can enable users to experience spatial audio signals with higher quality and stronger sense of presence.

[0005] It should be noted that the application scenarios of the present application may include: scenarios in which users use AR / VR devices to experience AR / VR projects (such as AR / VR science lectures, AR / VR cinemas, AR / VR concerts, etc.), and scenarios in which users use mobile phones, tablets, laptops, personal computers, smart watches and other terminal devices + headphones to experience listening projects (such as crosstalk, movies, concerts, music concerts, etc.).

[0006] In a first aspect, an embodiment of the present application provides an audio processing method, which includes: first, obtaining a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene; according to the sound source position and the first position, obtaining a first acoustic feature, the first acoustic feature is obtained by adjusting the second acoustic feature according to conversion information, the second acoustic feature is obtained by extracting acoustic features according to the first position, the virtual scene, and the sound source position, and the conversion information is used to describe the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene; then, generating a spatial audio signal according to the first acoustic feature and the source audio signal.

[0007] That is to say, the present application adjusts the acoustic features of the first position corresponding to the user's current position in the virtual scene (i.e., the second acoustic features) according to the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene, so that the acoustic features of the first position corresponding to the user in the virtual scene (i.e., the first acoustic features) are closer to the acoustic features of the user's position in the real scene corresponding to the virtual scene; in this way, the spatial audio signal subsequently generated according to the adjusted acoustic features (i.e., the first acoustic features) is closer to the spatial audio signal heard by the user at the position in the real scene corresponding to the virtual scene, so that users can experience a spatial audio signal with higher quality and stronger sense of presence.

[0008] It should be noted that the conversion information can be used to adjust the second acoustic feature of any first position in the virtual environment to obtain the corresponding first acoustic feature.

[0009] Exemplarily, the virtual scene where the user is located may refer to the virtual scene corresponding to the listening project or AR / VR project selected (or experienced) by the user. For example, if the listening project is a concert, the corresponding virtual scene is a virtual gymnasium or virtual studio; if the listening project is a movie, the corresponding virtual scene is a virtual cinema; if the AR / VR project is a concert, the corresponding virtual scene is a virtual concert hall; if the AR / VR project is a lecture, the corresponding virtual scene is a virtual lecture hall, and so on. In other words, the virtual scene where the user is located may refer to the virtual scene corresponding to the audio signal of the user experience space.

[0010] Exemplarily, the virtual scene is obtained by modeling the real scene, and there may be many ways to generate the virtual scene, which is not limited in this application. A real scene refers to an objectively existing scene, such as a movie theater, a concert hall, a lecture hall, etc.; a virtual scene is a scene constructed using virtual technology, not an objectively existing scene. The virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user is obtained by modeling the real scene. For example, a virtual concert hall is obtained by modeling a real concert hall; for another example, a virtual cinema is obtained by modeling a real cinema; for another example, a virtual lecture hall is obtained by modeling a real lecture hall; and so on.

[0011] It should be noted that the real scene currently in which the user is located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may be the same scene or different scenes, and this application does not limit this. In other words, the real scene currently in which the user is located and the real scene used to model the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may be the same scene or different scenes, and this application does not limit this.

[0012] Exemplarily, when the real scene in which the user is currently located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user, the user's position in the real scene is the same as the corresponding position of the user in the virtual scene, that is, the user's position in the real scene is the first position.

[0013] Exemplarily, when the real scene in which the user is currently located is not the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user: In one possible manner, the default position can be determined as the first position corresponding to the user in the virtual scene; wherein the default position can be set as required, for example, it can be the position with the best audio experience in the virtual scene, and this application does not limit this. In one possible manner, the terminal device can provide position options in the virtual scene, so that the user can select the position option corresponding to the desired experience position in the virtual scene; then, the terminal device can use the position corresponding to the position option selected by the user as the first position corresponding to the user in the virtual scene.

[0014] Exemplarily, the modeling methods may include multiple methods, such as manual modeling, modeling based on visual information, modeling based on auditory information, etc., which are not limited in the present application.

[0015] Exemplarily, spatial audio processing (such as rendering) may be performed according to the first acoustic feature and the source audio signal to generate a spatial audio signal (such as a binaural rendering signal).

[0016] Exemplarily, the source audio signal may include an audio file (such as a music file, a crosstalk file, etc.), or an audio file included in a multimedia file (such as an audio file included in a movie file, etc.).

[0017] Exemplarily, the first acoustic feature may be generated in advance and stored in a database, or may be generated in real time, and this application does not impose any limitation on this.

[0018] Exemplarily, the second acoustic feature may be generated in advance and stored in a database, or may be generated in real time, and this application does not impose any limitation on this.

[0019] Exemplarily, the conversion information may be generated in advance and stored in a database, or may be generated in real time, and this application does not impose any limitation on this.

[0020] It should be noted that the first acoustic feature, the second acoustic feature and the conversion information can be stored in the same database or in different databases.

[0021] Exemplarily, the method of extracting acoustic features may include, but is not limited to: acoustic simulation, machine learning, numerical calculation, etc., which is not limited in this application.

[0022] According to the first aspect, the conversion information is determined by analyzing an acoustic feature group, the acoustic feature group includes a third acoustic feature and a fourth acoustic feature, the fourth acoustic feature is an acoustic feature of the second position in the real scene, and the third acoustic feature is an acoustic feature of the second position in the virtual scene. In this way, by obtaining and analyzing the acoustic features of the same position in the real scene and the virtual scene, the accurate conversion relationship between the acoustic features of the same position in the real scene and the virtual scene can be determined.

[0023] According to the first aspect, or any implementation of the first aspect above, there are one or more acoustic feature groups, there are one or more second positions, and the second positions corresponding to the third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group are the same.

[0024] For example, when the real scene is a uniform room (for example, the room is square, the wall materials are uniform, etc., where the room can be understood as an indoor scene), and the difference between the real acoustic features at different positions is relatively small, the first position can be one, thereby reducing the workload. For another example, when the real scene is an uneven room (for example, the room is asymmetrical, the wall materials are diverse, etc.), and the difference between the real acoustic features at different positions is relatively large, the second position can be multiple, thereby increasing the accuracy of the generated conversion information. It should be understood that the number of second positions can be determined according to demand, and this application does not limit this.

[0025] Exemplarily, when the sound source device is deployed at different positions (hereinafter referred to as preset sound source positions) in the real scene, the fourth acoustic feature of the same position is different; correspondingly, when the virtual sound source is deployed at different preset sound source positions in the virtual scene, the third acoustic feature of the same position is different; that is, in the real scene, each second position corresponds to one or more fourth acoustic features; in the virtual scene, each second position corresponds to one or more third acoustic features. Therefore, a second position can correspond to one or more acoustic feature groups, and the third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group correspond to the same second position, and the corresponding preset sound source positions are the same.

[0026] It should be noted that the present application only needs to analyze the conversion relationship between the acoustic features of one or more positions (i.e., one or more second positions) in the virtual scene and the real scene to obtain conversion information; and the conversion information is universally applicable to any position in the same room (i.e., the same virtual scene), rather than being able to be used only to adjust the acoustic features of a first position close to a second position.

[0027] According to the first aspect, or any implementation of the first aspect above, the conversion information includes a conversion function, and the conversion function is obtained by analyzing the signal processing results of the third acoustic feature and the fourth acoustic feature.

[0028] Exemplarily, the third acoustic feature and the fourth acoustic feature in an acoustic feature group may be subjected to signal processing respectively to obtain a signal processing result; then, the signal processing result of the third acoustic feature and the signal processing result of the fourth acoustic feature may be analyzed to obtain a conversion function; in this case, the conversion function may be used as conversion information. For example, for a third acoustic response such as a room impulse response (RIR) included in the third acoustic feature and a fourth acoustic response such as RIR included in the fourth acoustic feature, wherein the third acoustic feature is represented by RIR 1 and the fourth acoustic feature is represented by RIR 2; RIR 1 and RIR 2 may be subjected to frequency domain conversion respectively to obtain a frequency domain response of RIR 1 (referred to as frequency response 1) and a frequency domain response of RIR 2 (referred to as frequency response 2). Then, in a possible manner, a frequency response conversion function may be calculated using frequency response 1 and frequency response 2, and the frequency response conversion function may be used as a conversion function; in a possible manner, signal analysis may be performed on frequency response 1 and frequency response 2 respectively, and then a conversion function may be determined based on the signal analysis results.

[0029] According to the first aspect, or any implementation of the first aspect above, the conversion information includes a feature change rate, where the feature change rate is a change rate of the third acoustic feature relative to the fourth acoustic feature.

[0030] Exemplarily, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be numerically analyzed respectively to obtain the feature change rate; in this case, the feature change rate can be used as conversion information. Specifically, the change rate of the third acoustic feature in the acoustic feature group relative to the fourth acoustic feature in the acoustic feature group can be calculated as the feature change rate. For example, for the third acoustic parameter such as the direct mixing ratio contained in the third acoustic feature, and the fourth acoustic parameter such as the direct mixing ratio contained in the fourth acoustic feature, wherein the direct mixing ratio in the third acoustic parameter is represented by DRR 1, and the direct mixing ratio in the fourth acoustic parameter is represented by DRR 2; the corresponding feature change rate can be (DRR 1-DRR 2) / DRR2, or, DRR1 / DRR2. It should be understood that the conversion information may include multiple feature change rates (multiple feature change rates correspond one-to-one to multiple acoustic parameters), and the present application does not limit this.

[0031] According to the first aspect, or any implementation of the first aspect above, the conversion information includes model output information obtained by inputting the third acoustic feature and the fourth acoustic feature into the model.

[0032] Exemplarily, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be processed by machine learning to obtain conversion information. Specifically, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be input into an AI model (or a machine learning model, hereinafter referred to as a second model), and the second model processes the third acoustic feature and the fourth acoustic feature in an acoustic feature group and outputs conversion information; that is, the model output information of the second model is the conversion information.

[0033] Exemplarily, the model output information may include a conversion function, a characteristic change rate, and other forms of information, which are not limited in the present application.

[0034] It should be understood that other methods may be used to generate conversion information, and this application does not limit this.

[0035] According to the first aspect, or any implementation of the first aspect, the first acoustic feature includes at least one of the following: acoustic response, energy information, acoustic parameter or acoustic feature.

[0036] The acoustic response may include but is not limited to: room impulse response, binaural room impulse response (Binaural Room Impulse Response, BRIR), higher order ambisonics (Higher Order Ambisonics, HOA), etc.

[0037] The energy information may include multiple frequency bands corresponding to the acoustic response, energy distribution in multiple directions, energy attenuation, etc.

[0038] Acoustic parameters may include environmental acoustic parameters and binaural acoustic parameters. Among them, environmental acoustic parameters include but are not limited to: direct-to-Reverberation energy Ratio (DRR), reverberation time (Reverberation Time) (such as T20 (time for energy to decay by 20 dB), T30, T60, etc.), clarity (Clarity) (such as C50 (sound energy ratio before and after 50 ms), C80, etc.) and the like. Binaural acoustic parameters may include but are not limited to: interaural time difference (ITD), interaural level difference (ILD), interaural cross correlation coefficient (IACC), etc.).

[0039] The acoustic features may include numerical analysis features obtained by performing numerical analysis on acoustic parameters (for example, principal component analysis (PCA) analysis); and may also include output information of the first model obtained by inputting at least one of the acoustic response, energy information or acoustic parameters into an AI model (hereinafter referred to as the first model).

[0040] According to the first aspect, or any implementation of the first aspect above, the fourth acoustic feature is obtained by processing a test audio signal received at a second position in the real scene.

[0041] Exemplarily, a sound source device can be deployed in a real scene. After the user carries the receiving device to the second position, the sound source device can be controlled to play a test audio signal. Correspondingly, the receiving device at the second position can receive the test audio signal; then, the receiving device can process the received test audio signal to obtain a fourth acoustic feature; it should be understood that the fourth acoustic feature is a real acoustic feature.

[0042] For example, the sound source device plays a test audio signal such as a pulse signal, a noise signal, or a sweep signal, and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in a format such as RIR, BRIR, or HOA.

[0043] For another example, the sound source device plays sound signals such as pulse signals, noise signals, and sweep frequency signals, and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, and acoustic characteristics.

[0044] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in the format of RIR, BRIR, HOA, etc.

[0045] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, acoustic characteristics, etc.

[0046] Among them, the sound source equipment includes but is not limited to: speaker equipment (such as home audio), terminal equipment with external speaker function (such as tablets, large screens, etc.), professional acoustic measurement equipment, etc.

[0047] The receiving device includes but is not limited to: terminal devices including microphones (such as tablets, mobile phones, AR / VR devices, etc.), professional acoustic measurement equipment, etc.

[0048] According to the first aspect, or any implementation of the first aspect above, the fourth acoustic feature is user-defined input.

[0049] Since the acoustic features actually measured in some real scenes may not be the most accurate match to the user's real hearing experience, the acoustic features that match the user's real hearing experience in some real scenes are directly input, so that the adjusted acoustic features are closer to the acoustic features that match the user's real hearing experience in some real scenes. In this way, the spatial audio signal subsequently generated based on the adjusted acoustic features is closer to the spatial audio signal heard by the user at the position in the real scene corresponding to some virtual scene.

[0050] Exemplarily, “certain real scenes” may include: real scenes with unavoidable interfering noise, real scenes where the visual presentation of surface materials does not match their actual acoustic effects, and real scenes that serve as multi-functional spaces (for example, real scenes that serve as lecture halls, concert halls, and film and television appreciation halls).

[0051] According to the first aspect, or any implementation method of the first aspect above, a first acoustic feature is obtained according to a sound source position and a first position, including: determining one or more acoustic feature groups, one acoustic feature group including a third acoustic feature and a fourth acoustic feature, the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene; analyzing the third acoustic feature and the fourth acoustic feature in the one or more acoustic feature groups to obtain conversion information; extracting acoustic features according to the first position, the virtual scene and the sound source position to obtain a second acoustic feature; adjusting the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0052] In this case, the real scene in which the user is currently located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user, and conversion information, second acoustic features and first acoustic features can be generated in real time.

[0053] According to the first aspect, or any implementation method of the first aspect above, a first acoustic feature is obtained according to a sound source position and a first position, including: selecting a second acoustic feature set corresponding to a virtual scene from a plurality of second acoustic feature sets stored in a database; wherein the plurality of second acoustic feature sets correspond one-to-one to a plurality of preset virtual scenes, a second acoustic feature set includes a plurality of second preset acoustic features, a second acoustic feature set is obtained by extracting acoustic features according to a preset virtual scene, a plurality of fourth positions and a plurality of preset sound source positions, a fourth position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set; selecting a second acoustic feature from a second acoustic feature set corresponding to a virtual scene according to the sound source position and the first position; selecting conversion information from a plurality of preset conversion information stored in a database according to the virtual scene, the plurality of preset conversion information corresponding one-to-one to a plurality of preset virtual scenes; adjusting the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0054] In this case, the real scene in which the user is currently located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may or may not be the same scene. In this way, the terminal device does not need to generate conversion information and the second acoustic feature in real time, which can shorten the time to generate spatial audio signals and improve user experience. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient way.

[0055] According to the first aspect, or any implementation method of the first aspect above, a first acoustic feature is obtained according to a sound source position and a first position, including: selecting a first acoustic feature set corresponding to a virtual scene from multiple first acoustic feature sets stored in a database; wherein multiple first acoustic feature sets correspond one-to-one to multiple preset virtual scenes, multiple first acoustic feature sets correspond one-to-one to multiple second acoustic feature sets, a second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple third positions and multiple preset sound source positions, a third position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set, and a first preset acoustic feature in a first acoustic feature set is obtained by adjusting a second preset acoustic feature in the corresponding second acoustic feature set according to conversion information; according to the sound source position and the first position, a first acoustic feature is selected from the first acoustic feature set corresponding to the virtual scene.

[0056] In this case, the real scene in which the user is currently located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may be the same scene or different scenes. In this way, the terminal device does not need to generate conversion information, the second acoustic feature, and the first acoustic feature in real time, which can further shorten the time for generating spatial audio signals and improve user experience. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient way.

[0057] In a second aspect, an embodiment of the present application provides an audio processing device, the device comprising:

[0058] A first acquisition module is used to acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene;

[0059] a second acquisition module, configured to acquire a first acoustic feature according to a sound source position and a first position, wherein the first acoustic feature is obtained by adjusting a second acoustic feature according to conversion information, and the second acoustic feature is obtained by extracting acoustic features according to the first position, a virtual scene, and a sound source position, and the conversion information is used to describe a conversion relationship between acoustic features at the same position in a virtual scene and a real scene;

[0060] The audio signal generating module is used to generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0061] It should be understood that the audio processing device of the second aspect can execute any implementation of the first aspect, which will not be described in detail herein.

[0062] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0063] In a third aspect, an embodiment of the present application provides an electronic device, comprising: a memory and a processor, wherein the memory is coupled to the processor; the memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the audio processing method in the first aspect or any possible implementation of the first aspect.

[0064] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0065] In a fourth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the audio processing method in the first aspect or any possible implementation of the first aspect are executed.

[0066] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0067] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the audio processing method in the first aspect or any possible implementation of the first aspect.

[0068] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0069] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the audio processing method in the first aspect or any possible implementation of the first aspect.

[0070] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here.

[0071] In a seventh aspect, an embodiment of the present application provides an augmented reality (AR) device, the AR device comprising: a display module, an image acquisition module, a headset, and a processor, wherein:

[0072] The processor is used to execute the audio processing method in the first aspect or any possible implementation manner of the first aspect;

[0073] The earphone is used to play the spatial audio signal generated by the audio processing method in the first aspect or any possible implementation manner of the first aspect.

[0074] Exemplarily, the display module can be used to display images; the image acquisition module is used to acquire images.

[0075] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can refer to the technical effects corresponding to the above-mentioned first aspect and any implementation of the first aspect, which will not be repeated here.

[0076] In an eighth aspect, an embodiment of the present application provides an augmented reality VR device, the VR device comprising: a display module, an image acquisition module, a headset, and a processor, wherein:

[0077] The processor is used to execute the audio processing method in the first aspect or any possible implementation manner of the first aspect;

[0078] The earphone is used to play the spatial audio signal generated by the audio processing method in the first aspect or any possible implementation manner of the first aspect.

[0079] Exemplarily, the display module can be used to display images; the image acquisition module is used to acquire images.

[0080] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can refer to the technical effects corresponding to the first aspect and any implementation of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1A is a schematic diagram of an application scenario shown as an example;

[0082] Figure 1B is a schematic diagram of an application scenario shown as an example;

[0083] Figure 1C is a schematic diagram of an application scenario shown as an example;

[0084] Figure 1D is a schematic diagram of an application scenario shown as an example;

[0085] Figure 2is a schematic diagram of an audio processing process shown as an example;

[0086] Figure 3 is a schematic diagram of an audio processing process shown as an example;

[0087] Figure 4 is a schematic diagram of an audio processing process shown as an example;

[0088] Figure 5 is a schematic diagram of an audio processing process shown as an example;

[0089] Fig. 6A is a schematic diagram of a frequency response curve of an acoustic characteristic shown as an example;

[0090] Figure 6B is a schematic diagram of a frequency response curve of an acoustic characteristic shown as an example;

[0091] Figure 7 is a schematic diagram of an audio processing device shown as an example;

[0092] Figure 8 Schematic diagram of the structure of the device shown as an example. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0094] The term "and / or" in this article is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0095] The terms "first" and "second" in the description and claims of the embodiments of the present application are used to distinguish different objects rather than to describe a specific order of objects. For example, a first target object and a second target object are used to distinguish different target objects rather than to describe a specific order of target objects.

[0096] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0097] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "multiple" refers to two or more than two. For example, multiple processing units refer to two or more processing units; multiple systems refer to two or more systems.

[0098] Figure 1A FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1A In FIG. 1 , an application scenario of a user using an AR device to experience an AR concert is shown. It should be understood that the user can also use the AR device to experience other AR projects such as an AR science lecture hall, an AR cinema, etc., and this application does not limit this.

[0099] in, Figure 1A The real scene where the user is located and the virtual scene corresponding to the AR project that the user experiences are the same scene, that is, the real scene where the user is located is the concert hall, and the virtual scene corresponding to the AR project that the user experiences is the virtual concert hall.

[0100] Reference Figure 1A For example, when a user wants to experience an AR concert in an empty concert hall, he or she can wear an AR device; then, the user can select the AR concert hall as the AR project to be experienced in the AR device, and select the music file to be experienced and perform a playback operation. Subsequently, the AR device can process the music file selected by the user, such as rendering, to obtain a spatial audio signal and play the spatial audio signal through headphones; and the AR device can also display the performance screen through the display screen, so that the user can see the performer playing at the corresponding position on the concert hall stage through the display screen. In this way, through the combination of vision and hearing, users can experience the concert immersively.

[0101] It should be understood that Figure 1A There may be one or more users using AR devices to experience an AR concert in a concert hall, and this application does not impose any restrictions on this.

[0102] Figure 1B FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1B In the figure, the application scenario of a user using a VR device to experience a VR cinema is shown. It should be understood that the user can also use the VR device to experience other VR projects such as a VR concert hall, a VR science lecture hall, etc., and this application does not limit this.

[0103] in, Figure 1B The real scene where the user is located and the virtual scene corresponding to the VR project that the user experiences are not the same scene, that is, the real scene where the user is located is the living room, and the virtual scene corresponding to the VR project that the user experiences is the virtual cinema.

[0104] Reference Figure 1BFor example, when a user wants to experience a VR cinema in the living room, he or she can wear a VR device; then, the user can select a VR cinema as the VR project to be experienced in the VR device, and select the movie file to be experienced and perform a playback operation. Subsequently, the VR device can process the audio file contained in the movie file, such as rendering, to obtain a spatial audio signal and play the spatial audio signal through headphones; and the VR device can also display the movie screen on the display screen; in this way, through the combination of vision and hearing, the user can experience the movie immersively.

[0105] Figure 1C FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1C In the figure, an application scenario is shown in which a user uses (mobile phone + headphones) to experience a concert.

[0106] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as crosstalk, movies, concerts, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application. This application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to experience listening.

[0107] in, Figure 1C The real scene where the user is located and the virtual scene corresponding to the listening project experienced by the user are the same scene, that is, the real scene where the user is located is the concert hall, and the virtual scene corresponding to the listening project experienced by the user is the virtual concert hall.

[0108] Reference Figure 1C For example, when a user wants to experience a concert in an empty concert hall, he or she can wear headphones and connect the headphones to a mobile phone; then, the user can select the concert as the listening project he or she wants to experience in the mobile phone, and select the music file he or she wants to experience and perform a play operation. Subsequently, the mobile phone can process the music file selected by the user, such as rendering, to obtain a spatial audio signal; then, the spatial audio signal can be sent to the headphones for playback; and the mobile phone can also play the video of the concert performance. In this way, the user can use the mobile phone to immersively experience the concert in the concert hall.

[0109] It should be understood that Figure 1C There may be one or more users using mobile phones to experience a concert in a concert hall, and this application does not impose any restrictions on this.

[0110] Figure 1D FIG. 1 is a schematic diagram of an exemplary application scenario. Figure 1D In the figure, an application scenario in which a user uses (mobile phone + headphones) to experience crosstalk is shown.

[0111] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as concerts, movies, concerts, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application, and this application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to listen to music.

[0112] in, Figure 1D The real scene where the user is located and the virtual scene corresponding to the listening project experienced by the user are not the same scene, that is, the real scene where the user is located is the bedroom, and the virtual scene corresponding to the listening project experienced by the user is the virtual studio.

[0113] Reference Figure 1D For example, when a user wants to experience crosstalk in the bedroom, he can wear headphones and connect the headphones to the mobile phone; then, the user can select crosstalk as the listening item he wants to experience in the mobile phone, and select the crosstalk file he wants to experience and perform the play operation. Subsequently, the mobile phone can process the crosstalk file selected by the user, such as rendering, to obtain a spatial audio signal; then, the spatial audio signal can be sent to the headphones for playback; and the mobile phone can also play the crosstalk video screen. In this way, users can use their mobile phones in the bedroom to immersively experience crosstalk.

[0114] It should be understood that the present application can also be applied to other listening scenarios or other virtual scenarios, and the present application is not limited to this.

[0115] The process of generating a spatial audio signal is described below.

[0116] Figure 2 FIG. 1 is a schematic diagram of an exemplary audio processing process. Figure 2 The steps in the embodiments can be executed by terminal devices such as AR devices, VR devices, mobile phones, tablet computers, laptops, personal computers, smart watches, etc.

[0117] S201, obtaining a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene.

[0118] For example, when a user needs to listen to music, he can wear headphones and connect a mobile phone to the headphones; then, he can select a listening project and an audio file that he wants to experience in a terminal device such as a mobile phone, tablet computer, laptop computer, personal computer, smart watch, etc.; then, he can perform a play operation. When a user needs to experience an AR / VR project, he can wear an AR / VR device, then select an AR / VR project that he wants to experience and a multimedia file that he wants to experience; then, he can perform a play operation.

[0119] Exemplarily, after the user performs a play operation, the terminal device can respond to the user operation to obtain the audio file selected by the user (or the audio file contained in the multimedia file selected by the user), that is, obtain the source audio signal; and the terminal device can also obtain the first position corresponding to the user in the virtual scene and the sound source position corresponding to the source audio signal in the virtual scene, so as to facilitate the subsequent generation of a spatial audio signal matching the first position and the sound source position.

[0120] Exemplarily, the virtual scene where the user is located in S201 may refer to the virtual scene corresponding to the listening project or AR / VR project selected (or experienced) by the user. For example, if the listening project is a concert, the corresponding virtual scene is a virtual gymnasium or virtual studio; if the listening project is a movie, the corresponding virtual scene is a virtual cinema; if the AR / VR project is a concert, the corresponding virtual scene is a virtual concert hall; if the AR / VR project is a lecture, the corresponding virtual scene is a virtual lecture hall, and so on. In other words, the virtual scene where the user is located in S201 may refer to the virtual scene corresponding to the audio signal of the user experience space.

[0121] Exemplarily, the virtual scene is obtained by modeling the real scene; there may be many ways to generate a virtual scene, which is not limited in this application and will be described in detail later. A real scene refers to an objectively existing scene, such as a movie theater, a concert hall, a lecture hall, etc.; a virtual scene is a scene constructed using virtual technology, and is not an objectively existing scene. The virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user is obtained by modeling the real scene. For example, a virtual concert hall is obtained by modeling a real concert hall; for another example, a virtual cinema is obtained by modeling a real cinema; for another example, a virtual lecture hall is obtained by modeling a real lecture hall; and so on.

[0122] It should be noted that the real scene currently in which the user is located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user can be the same scene. Figure 1A and Figure 1C As shown in the embodiment, it may not be the same scene as above Figure 1B and Figure 1D In other words, the real scene currently in which the user is located and the real scene used to model the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user can be the same scene. Figure 1A and Figure 1C As shown in the embodiment, it may not be the same scene as above Figure 1B and Figure 1D As shown in the embodiments, this application is not limited to this.

[0123] Exemplarily, when the real scene in which the user is currently located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user, the user's position in the real scene is the same as the corresponding position of the user in the virtual scene, that is, the user's position in the real scene is the first position.

[0124] Exemplarily, when the real scene in which the user is currently located is not the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user: In one possible manner, the default position can be determined as the first position corresponding to the user in the virtual scene; wherein the default position can be set as required, for example, it can be the position with the best audio experience in the virtual scene, and this application does not limit this. In one possible manner, the terminal device can provide position options in the virtual scene, so that the user can select the position option corresponding to the desired experience position in the virtual scene before performing the playback operation; then, the terminal device can use the position corresponding to the position option selected by the user as the first position corresponding to the user in the virtual scene.

[0125] S202, obtaining a first acoustic feature according to the sound source position and the first position, wherein the first acoustic feature is obtained by adjusting the second acoustic feature according to the conversion information, and the second acoustic feature is obtained by extracting acoustic features according to the first position, the virtual scene and the sound source position, and the conversion information is used to describe the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene.

[0126] Next, a first acoustic feature may be obtained, and the first acoustic feature may be used for spatial audio processing to generate a spatial audio signal.

[0127] Exemplarily, the first acoustic feature includes, but is not limited to: acoustic response, energy information, acoustic parameters or acoustic features, etc., which are not limited in the present application.

[0128] The acoustic response may include but is not limited to: Room Impulse Response (RIR), Binaural Room Impulse Response (BRIR), Higher Order Ambisonics (HOA), and the like.

[0129] The energy information may include multiple frequency bands corresponding to the acoustic response, energy distribution in multiple directions, energy attenuation, etc.

[0130] Acoustic parameters may include environmental acoustic parameters and binaural acoustic parameters. Among them, environmental acoustic parameters include but are not limited to: direct-to-Reverberation energy Ratio (DRR), reverberation time (Reverberation Time) (such as T20 (time for energy to decay by 20 dB), T30, T60, etc.), clarity (Clarity) (such as C50 (sound energy ratio before and after 50 ms), C80, etc.) and the like. Binaural acoustic parameters may include but are not limited to: interaural time difference (ITD), interaural level difference (ILD), interaural cross correlation coefficient (IACC), etc.).

[0131] The acoustic features may include numerical analysis features obtained by performing numerical analysis on acoustic parameters (for example, principal component analysis (PCA)); and may also include output information of the first model obtained by inputting at least one of the acoustic response, energy information or acoustic parameters into an AI model (hereinafter referred to as the first model).

[0132] Among them, for the convenience of subsequent explanation, the acoustic response contained in the first acoustic feature can be called the first acoustic response; the energy information contained in the first acoustic feature can be called the first energy information; the acoustic parameter contained in the first acoustic feature can be called the first acoustic parameter; and the acoustic feature contained in the first acoustic feature can be called the first acoustic feature.

[0133] In S202, a possible method may be to first obtain conversion information, wherein the conversion information may be used to describe the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene. It should be noted that the conversion information may be pre-generated and stored in the database, or may be generated in real time, and this application does not limit this; wherein the generation method of the conversion information is described later. Next, a second acoustic feature is obtained; wherein the second acoustic feature is the acoustic feature of the first position in the virtual scene, and may be obtained by extracting the acoustic feature according to the first position, the virtual scene and the sound source position. wherein the method of extracting the acoustic feature may include but is not limited to: acoustic simulation, machine learning, numerical calculation, etc., and this application does not limit this. It should be noted that the second acoustic feature may be pre-generated and stored in the database, or may be generated in real time, and this application does not limit this. thereafter, the second acoustic feature may be adjusted according to the conversion information to obtain the first acoustic feature. in this way, the acoustic feature of the first position in the virtual scene may be adjusted to be closer to the acoustic feature of the first position in the real scene, thereby improving the quality and presence of the spatial audio signal, thereby improving the user experience. Among them, the acoustic features of any position in the virtual scene can be called virtual acoustic features, and the acoustic features of any position in the real scene can be called real acoustic features; that is, the first acoustic feature and the second acoustic feature are virtual acoustic features.

[0134] In S202, a possible method may be to adjust the second preset acoustic features of each position in various preset virtual scenes (there are one or more second preset acoustic features for each position, and the multiple second preset acoustic features for each position correspond to multiple virtual sound source positions) in advance according to various preset conversion information (one preset conversion information corresponds to a preset virtual scene), so as to obtain the first preset acoustic features of each position in the various virtual scenes and store them in a database; in this way, the first acoustic feature can be directly obtained from the database according to the virtual scene, the sound source position and the first position.

[0135] It should be understood that the second acoustic characteristic may include, but is not limited to: a second acoustic response, second energy information, a second acoustic parameter or a second acoustic characteristic.

[0136] It should be noted that the conversion information, the second acoustic feature and the first acoustic feature can be stored in the same database or in different databases, and this application does not impose any restrictions on this.

[0137] S203: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0138] Afterwards, spatial audio processing (such as rendering) may be performed according to the first acoustic feature and the source audio signal to obtain a spatial audio signal (such as a binaural rendering signal).

[0139] Then, the AR / VR device can play the spatial audio signal through its built-in headphones; terminal devices such as mobile phones, tablets, laptops, smart watches, etc. can play the spatial audio signal through headphones connected to them; in this way, users can hear the spatial audio signal through headphones.

[0140] The following describes a process of generating the conversion information and a process of adjusting the second acoustic feature to obtain the first acoustic feature.

[0141] Figure 3 FIG. 1 is a schematic diagram of an audio processing process shown as an example. Figure 3 In the embodiment of the present invention, the real scene currently in which the user is located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user (that is, the application scene can be the above-mentioned Figure 1A and Figure 1C ); In this case, the conversion information, the second acoustic feature, and the first acoustic feature can all be generated in real time.

[0142] S301, constructing a virtual scene based on the real scene.

[0143] In one possible approach, manual modeling can be performed. For example, the size, position, surface material and other information of each object in the real scene can be input into the modeling software, and the modeling software performs 3D modeling based on the user's input information to obtain a virtual scene.

[0144] In one possible approach, modeling can be performed based on visual information. For example, a user wears an AR / VR device and captures an image sequence that covers the real scene by moving or rotating the AR / VR device; the AR / VR device then performs 3D modeling based on the image sequence to obtain a virtual scene.

[0145] It should be understood that other methods may be used to construct a virtual scene, and this application does not limit this.

[0146] It should be noted that S301 can be a pre-executed step or a real-time executed step, and this application does not limit this.

[0147] S302: Acquire fourth acoustic features of one or more second positions in the real scene.

[0148] In a possible manner, the fourth acoustic feature may be obtained by processing an audio signal received at a second position in the real scene.

[0149] Exemplarily, a sound source device can be deployed in a real scene. After the user carries the receiving device to the second position, the sound source device can be controlled to play a test audio signal. Correspondingly, the receiving device at the second position can receive the test audio signal; then, the receiving device can process the received test audio signal to obtain a fourth acoustic feature; it should be understood that the fourth acoustic feature is a real acoustic feature.

[0150] For example, the sound source device plays a test audio signal such as a pulse signal, a noise signal, or a sweep signal, and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in a format such as RIR, BRIR, or HOA.

[0151] For another example, the sound source device plays sound signals such as pulse signals, noise signals, and sweep frequency signals, and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, and acoustic characteristics.

[0152] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in the format of RIR, BRIR, HOA, etc.

[0153] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, acoustic characteristics, etc.

[0154] Among them, the sound source equipment includes but is not limited to: speaker equipment (such as home audio), terminal equipment with external speaker function (such as tablets, large screens, etc.), professional acoustic measurement equipment, etc.

[0155] The receiving device includes but is not limited to: terminal devices including microphones (such as tablets, mobile phones, AR / VR devices, etc.), professional acoustic measurement equipment, etc.

[0156] Among them, the second position is one or more. For example, when the real scene is a uniform room (for example, the room is square, the wall material is uniform, etc., where the room can be understood as an indoor scene) and the difference between the real acoustic characteristics of different positions is relatively small, the first position can be one, thereby reducing the workload. For another example, when the real scene is an uneven room (for example, the room is asymmetrical, the wall materials are diverse, etc.) and the difference between the real acoustic characteristics of different positions is relatively large, the second position can be multiple, thereby increasing the accuracy of the generated conversion information. It should be understood that the number of second positions can be determined according to demand, and this application does not limit this.

[0157] In one possible manner, when the terminal device is an AR / VR device, the AR / VR device may display a measurement instruction on a display screen, wherein the measurement instruction may include a second position. In this way, the user may wear the AR / VR device and move according to the second position displayed on the display screen of the AR / VR device. During the user's movement, the AR / VR device may locate and obtain the position information in real time, and provide a moving prompt based on the obtained position information; when the AR / VR device determines that the user has moved to the second position based on the obtained position information, the user may be prompted to stop moving. After the AR / VR device receives the test audio signal at a second position and processes and obtains the fourth acoustic feature, the test success message may be played; and when there are multiple second positions, the AR / VR device may prompt the user to move to the next second position until the AR / VR device obtains the fourth acoustic features at all second positions in the real scene. When the AR / VR device does not receive the test audio signal or obtain the fourth acoustic feature at the second position, the test failure message may be played, and the user may be prompted to test again, that is, the user may be prompted to stay at the current second position to receive the test audio signal. In this case, the AR / VR device can record the position information of one or more second positions to facilitate the subsequent determination of the third acoustic characteristics of one or more second positions in the virtual scene.

[0158] In one possible manner, when the terminal device is an AR / VR device, the AR / VR device may display a measurement instruction on a display screen, wherein the measurement instruction may include the number of second positions. In this way, the user may first select a corresponding number of second positions in the AR / VR device according to the number of second positions displayed on the display screen; and then wear the AR / VR device to move to the second position selected by the user. During the user's movement, the AR / VR device may locate and obtain position information in real time, and provide a moving prompt according to the obtained position information; when the AR / VR device determines that the user has moved to the second position according to the obtained position information, the user may be prompted to stop moving. After the AR / VR device receives a test audio signal at a second position and processes and obtains a fourth acoustic feature, a test success message may be played; and when there are multiple second positions, the AR / VR device may prompt the user to move to the next second position until the AR / VR device obtains the fourth acoustic features at all second positions in the real scene. When the AR / VR device does not receive a test audio signal or obtains the fourth acoustic feature at the second position, a test failure message may be played, and the user may be prompted to test again, that is, the user may be prompted to stay at the current second position to receive the test audio signal. In this case, the AR / VR device can record the position information of one or more second positions to facilitate the subsequent determination of the third acoustic characteristics of one or more second positions in the virtual scene.

[0159] In one possible way, when the terminal device is a terminal device with low positioning accuracy such as a tablet, a laptop computer, and a mobile phone, a virtual scene can be displayed in the terminal device, and the second position can be marked in the virtual scene; in this way, the user carries the terminal device, moves to the second position marked by the terminal device in the virtual scene, and then stops; then performs a recording operation in the terminal device, and the terminal device can receive the test audio signal at the second position. After the terminal device receives the test audio signal at the second position and processes to obtain the fourth acoustic feature, the test success information can be played; and when there are multiple second positions, the terminal device can prompt the user to move to the next second position until the terminal device obtains the fourth acoustic features at all second positions in the real scene. When the terminal device does not receive the test audio signal at the second position or does not obtain the fourth acoustic feature, the test failure information can be played, and the user can be prompted to test again, that is, the user is prompted to stay at the current second position to receive the test audio signal. In this case, the user can enter the position information of one or more second positions (for example, the distance from the wall, the direction of an object relative to the user, the height of the user, etc.) in the terminal device, so as to facilitate the subsequent determination of the third acoustic features of one or more second positions in the virtual scene.

[0160] It should be understood that the fourth acoustic characteristic may include, but is not limited to: a fourth acoustic response, fourth energy information, a fourth acoustic parameter or a fourth acoustic characteristic.

[0161] In addition, the position information of the sound source device in the real scene may also be recorded to facilitate the subsequent determination of the third acoustic features of one or more second positions in the virtual scene.

[0162] In a possible manner, the fourth acoustic feature is user-defined input.

[0163] Since the acoustic features actually measured in some real scenes may not be the most appropriate to the acoustic features that match the user's actual hearing perception, the acoustic features that match the user's actual hearing perception in some real scenes are directly input, so that the adjusted acoustic features are closer to the acoustic features that match the user's actual hearing perception in some real scenes. In this way, the spatial audio signal subsequently generated based on the adjusted acoustic features is closer to the spatial audio signal heard by the user in the real scene corresponding to some virtual scene.

[0164] Exemplary, “certain real-life scenarios” may include:

[0165] 1. Real scenes with inevitable interference noise. For example, there is inevitable interference noise in the real scene or nearby (such as construction in the next room). The fourth acoustic feature to be input by the user can be determined through the design parameters of the real scene, past related data, etc.

[0166] 2. A real scene where the visual presentation of the surface material does not match its actual acoustic effect. For this real scene, if the fourth acoustic feature is obtained by processing the test audio signal received at the second position in the real scene, then when the spatial audio signal is subsequently generated and played, the user's hearing experience may not match the visual perception.

[0167] 3. Real scenes as multi-functional spaces, that is, real scenes are used as multi-functional spaces; for example, a real scene is used as a lecture hall, a concert hall, and a film and television appreciation hall. Since users have different requirements for the auditory sense of spatial audio signals in different virtual scenes, the virtual acoustic characteristics of different virtual scenes are different; if the real acoustic characteristics are measured in the same real scene for different virtual scenes, then when the spatial audio signal is subsequently generated and played, it may cause the user's auditory sense to be inconsistent with the visual perception.

[0168] It should be noted that, in one possible manner, a sound source device may be deployed at one position (hereinafter referred to as a preset sound source position); in this way, one second position may correspond to one fourth acoustic feature. In one possible manner, a sound source device may be deployed at multiple preset sound source positions, in this way, one second position may correspond to multiple fourth acoustic features.

[0169] S303: Acquire third acoustic features of one or more second positions in the virtual scene.

[0170] Accordingly, the acoustic feature extraction can be performed based on the position information of the sound source device recorded in S302, the position information of the second position recorded in S302, the test audio signal played by the sound source device in S302, and the virtual scene to determine the third acoustic feature of one or more second positions. Similarly, one second position can correspond to one or more third acoustic features. It should be understood that the third acoustic feature is a virtual acoustic feature.

[0171] It should be understood that the third acoustic characteristic may include, but is not limited to: a third acoustic response, third energy information, a third acoustic parameter or a third acoustic characteristic.

[0172] In this way, after executing S302 and S303, one or more acoustic feature groups can be obtained for a second position. An acoustic feature group includes a third acoustic feature and a fourth acoustic feature. The third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group have the same corresponding second position and the same corresponding preset sound source position.

[0173] S304: Analyze the third acoustic feature and the fourth acoustic feature in one or more acoustic feature groups to obtain conversion information.

[0174] Exemplarily, the present application may use signal processing, machine learning, numerical analysis, etc. to analyze and compare the third acoustic feature and the fourth acoustic feature in one or more acoustic feature groups to obtain conversion information.

[0175] The following uses an acoustic feature group as an example to illustrate the method of generating conversion information.

[0176] In a possible manner, the third acoustic feature and the fourth acoustic feature in an acoustic feature group may be subjected to signal processing respectively to obtain a signal processing result; then, the signal processing result of the third acoustic feature and the signal processing result of the fourth acoustic feature may be analyzed to obtain a conversion function; in this case, the conversion function may be used as conversion information. For example, for a third acoustic response such as RIR included in the third acoustic feature and a fourth acoustic response such as RIR included in the fourth acoustic feature, wherein the RIR in the third acoustic response is represented by RIR1 and the RIR in the fourth acoustic response is represented by RIR2; RIR1 and RIR2 may be subjected to frequency domain conversion respectively to obtain a frequency domain response of RIR1 (referred to as frequency response 1) and a frequency domain response of RIR2 (referred to as frequency response 2). Then, in a possible manner, a frequency response conversion function may be calculated using frequency response 1 and frequency response 2, and the frequency response conversion function may be used as a conversion function; in a possible manner, signal analysis may be performed on frequency response 1 and frequency response 2 respectively, and then a conversion function may be determined based on the signal analysis results.

[0177] In one possible manner, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be numerically analyzed respectively to obtain the feature change rate; in this case, the feature change rate can be used as conversion information. Specifically, the change rate of the third acoustic feature in the acoustic feature group relative to the fourth acoustic feature in the acoustic feature group can be calculated as the feature change rate. For example, for the third acoustic parameter such as the direct mixing ratio contained in the third acoustic feature, and the fourth acoustic parameter such as the direct mixing ratio contained in the fourth acoustic feature, wherein the direct mixing ratio in the third acoustic parameter is represented by DRR 1, and the direct mixing ratio in the fourth acoustic parameter is represented by DRR 2; the corresponding feature change rate can be (DRR 1-DRR 2) / DRR 2, or, DRR1 / DRR2. It should be understood that the conversion information may include multiple feature change rates (multiple feature change rates correspond one-to-one to multiple acoustic parameters), and the present application does not limit this.

[0178] In one possible method, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be processed by machine learning to obtain conversion information. Specifically, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be input into an AI model (or a machine learning model, hereinafter referred to as the second model), and the second model processes the third acoustic feature and the fourth acoustic feature in an acoustic feature group and outputs conversion information; that is, the model output information of the second model is the conversion information.

[0179] It should be understood that other methods may be used to generate conversion information, and this application does not limit this.

[0180] S305: Acquire a source audio signal, a sound source position corresponding to the source audio signal in the virtual scene, and a first position corresponding to the user in the virtual scene.

[0181] Exemplarily, S305 may refer to the description of S201 above, which will not be repeated here.

[0182] S306: Extract acoustic features according to the first position, the sound source position of the source audio signal, and the virtual scene to obtain a second acoustic feature.

[0183] Exemplarily, the second acoustic feature may be understood as a virtual acoustic feature at a first position when a virtual sound source located at a sound source position in a virtual scene plays a source audio signal.

[0184] S307: Adjust the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0185] Exemplarily, the second acoustic feature can be adjusted according to the conversion information to obtain the first acoustic feature; in this way, when the virtual sound source located at the sound source position in the virtual scene plays the source audio signal, the virtual acoustic feature at the first position can be adjusted to be close to or consistent with the real acoustic feature at the first position when the real sound source located at the sound source position in the real scene plays the source audio signal.

[0186] Exemplarily, different adjustment methods may be used for different types of second acoustic features. For example, when the second acoustic feature is a second acoustic response, a frequency response curve adjustment algorithm based on signal processing may be used. Specifically, the second acoustic response may be converted into a frequency domain to obtain a frequency domain response of the second acoustic response (hereinafter referred to as frequency response 3); then, the frequency response 3 may be adjusted according to the conversion function determined above to obtain an adjusted frequency response 3; thereafter, the adjusted frequency response 3 may be transformed into a first acoustic response, that is, a first acoustic feature.

[0187] For another example, when the second acoustic feature includes a second acoustic response and a second acoustic parameter, a reverberation adjustment algorithm based on signal processing can be used. Specifically, for the second DRR in the second acoustic parameter, the second DRR can be adjusted according to the change rate of the direct mixing ratio calculated above to obtain the first DRR (that is, the first acoustic parameter included in the first acoustic feature); then, the second acoustic response can be adjusted according to the first DRR to obtain the first acoustic response. It should be understood that for other second acoustic parameters, adjustments can also be made according to the corresponding feature change rates, and the present application does not limit this.

[0188] For another example, the adjustment algorithm is an adjustment algorithm based on an AI model. Specifically, the second acoustic feature and the conversion information can be input into an AI model (hereinafter referred to as a third model), and the third model adjusts the second acoustic feature according to the conversion information and outputs the first acoustic feature.

[0189] It should be understood that other adjustment algorithms may also be used to adjust the second acoustic feature to obtain the first acoustic feature, and this application does not limit this.

[0190] It should be noted that the conversion information obtained in S304 can be used to adjust the second acoustic features of any first position in the virtual environment to obtain the first acoustic features. In other words, S304 only needs to analyze the third acoustic features and the fourth acoustic features of one or more second positions, and the conversion information obtained has universal applicability to any position in the same room (i.e., virtual scene), rather than being used only to adjust the acoustic features of the first position close to the second position.

[0191] S308: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0192] Exemplarily, when the first acoustic feature is a first acoustic response, the first acoustic response may be convolved with the source audio signal to obtain a spatial audio signal.

[0193] Exemplarily, when the first acoustic feature includes a first acoustic response and first energy information, the energy of the spatial audio signal may be adjusted according to the first energy information; then, the adjusted first acoustic response is convolved with the source audio to obtain the spatial audio signal.

[0194] Exemplarily, when the first acoustic feature includes a first acoustic parameter and a first acoustic feature, a first acoustic response may be generated according to the first acoustic parameter and the first acoustic feature; then, the first acoustic response may be convolved with the source audio to obtain a spatial audio signal.

[0195] Exemplarily, when the first acoustic feature includes a first acoustic response, a first acoustic parameter and a first acoustic feature, the first acoustic response can be first optimized according to the first acoustic parameter and the first acoustic feature, and then the optimized first acoustic response is convolved with the source audio to obtain a spatial audio signal.

[0196] It should be understood that there are many ways to generate a spatial audio signal according to the first acoustic feature, and the present application does not limit this.

[0197] In one possible method, preset conversion information corresponding to multiple preset virtual scenes can be pre-generated and stored in a database, and second preset acoustic features of multiple positions in multiple preset virtual scenes can be pre-generated and stored in a database; in this way, during the application process, the conversion information corresponding to the current virtual scene can be read from the database, and the first position in the current virtual scene can be judged in real time during the application process, so as to read the second acoustic feature matching the first position and the sound source position in the current virtual scene from the database, and adjust the second acoustic feature in real time to obtain the first acoustic feature matching the first position and the sound source position in the current virtual scene. Using the database, the time for generating spatial audio signals can be shortened and the user experience can be improved. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient way.

[0198] Figure 4 FIG. 1 is a schematic diagram of an audio processing process shown as an example. Figure 4 In the embodiment of the present invention, it is not limited whether the real scene currently in which the user is located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user. In other words, Figure 4 The application scenario of the embodiment can be the above Figure 1A to Figure 1D .

[0199] S401, obtaining a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene.

[0200] S402: Select a second acoustic feature set corresponding to the virtual scene from a plurality of second acoustic feature sets stored in a database.

[0201] S403: Select a second acoustic feature from a second acoustic feature set corresponding to the virtual scene according to the sound source position of the source audio signal and the first position.

[0202] Exemplarily, a plurality of second acoustic feature sets may be generated in advance for a plurality of preset virtual scenes; wherein the plurality of second acoustic feature sets correspond one-to-one to the plurality of preset virtual scenes.

[0203] Specifically, for a preset virtual scene, multiple positions (hereinafter referred to as the fourth position) can be identified in advance in the preset virtual scene; for example, the preset virtual scene is divided into a 10*10 grid, and the intersection of the grid lines or the center point of the grid can be used as the fourth position. Then, for each fourth position, acoustic feature extraction can be performed based on the fourth position, the preset sound source position in the preset virtual scene and the preset virtual scene to obtain the second preset acoustic feature of the fourth position. Among them, there can be multiple preset sound source positions in the preset virtual scene, so that for one fourth position, multiple second preset acoustic features can be obtained. In this way, the second preset acoustic features of multiple fourth positions can be obtained; the second preset acoustic features of these multiple fourth positions can constitute a second acoustic feature set corresponding to the preset virtual scene. Afterwards, the second acoustic feature set can be stored in a database, and a corresponding relationship between the second acoustic feature set and the preset virtual scene is established.

[0204] According to the above method, a plurality of second acoustic feature sets may be generated; and the plurality of second acoustic feature sets may be stored in a database, and a correspondence between the plurality of second acoustic feature sets and corresponding preset virtual scenes may be established.

[0205] Exemplarily, a second acoustic feature set corresponding to the current virtual scene can be first filtered out from the database; then, based on the sound source position and the fourth position of the source audio signal, a second preset acoustic feature whose preset sound source position is the same as or closest to the sound source position, and whose fourth position is the same as or closest to the first position, is selected from the filtered second acoustic feature set as the second acoustic feature.

[0206] S404, selecting conversion information from a plurality of preset conversion information stored in a database according to the virtual scene; wherein the plurality of preset conversion information corresponds one-to-one to a plurality of preset virtual scenes.

[0207] Exemplarily, for one of the multiple preset virtual scenes, preset conversion information can be generated in advance according to the description in S302 to S304 above, and the preset conversion information can be stored in a database, and a corresponding relationship between the preset conversion information and the preset virtual scene can be established. In the process of executing S404, preset conversion information corresponding to the preset virtual scene and the same as the current virtual scene can be selected from the database as the conversion information corresponding to the current virtual scene.

[0208] S405: Adjust the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0209] S406: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0210] S405 to S406 may refer to the description of S307 to S308 above, which will not be repeated here.

[0211] In one possible approach, the first acoustic features of multiple positions in multiple preset virtual scenes can be generated in advance and stored in a database; thus, during the application process, the first acoustic features that match the first position and the sound source position in the current virtual scene can be read from the database, which can further shorten the time for generating spatial audio signals and improve the user experience. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient way.

[0212] Figure 5 FIG. 1 is a schematic diagram of an audio processing process shown as an example. Figure 5 In the embodiment of the present invention, it is not limited whether the real scene currently in which the user is located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user. In other words, Figure 5 The application scenario of the embodiment can be the above Figure 1A to Figure 1D .

[0213] S501, obtaining a source audio signal, a sound source position of the source audio signal in a virtual scene, and a first position corresponding to a user in the virtual scene.

[0214] S502: Select a first acoustic feature set corresponding to the virtual scene from a plurality of first acoustic feature sets stored in a database.

[0215] S503: Select a first acoustic feature from a first acoustic feature set corresponding to the virtual scene according to the sound source position and the first position of the source audio signal.

[0216] Exemplarily, for one of the multiple preset virtual scenes, according to the above Figure 3 The embodiment generates corresponding conversion information, and according to Figure 4 The embodiment generates a corresponding second acoustic feature set; then, according to the conversion information corresponding to the preset virtual scene, the second preset acoustic features of multiple positions (hereinafter referred to as third positions) in the second acoustic feature set corresponding to the preset virtual scene can be adjusted to obtain the first preset acoustic features of multiple third positions; the first preset acoustic features of the multiple third positions can form a first acoustic feature set, that is, a first acoustic feature set corresponding to the preset virtual scene. Then, the first acoustic feature set corresponding to the preset virtual scene can be stored in a database, and a corresponding relationship between the first acoustic feature set and the preset virtual scene can be established.

[0217] It should be understood that there may be multiple preset sound source positions in the preset virtual scene, so that for one third position, multiple second preset acoustic features can be obtained. Correspondingly, for one third position, multiple first preset acoustic features can be obtained, and each first preset acoustic feature corresponds to one preset acoustic position.

[0218] Exemplarily, a first acoustic feature set corresponding to the current virtual scene can be first filtered out from the database; then, from the filtered first acoustic feature set, a first preset acoustic feature whose preset sound source position is the same as or closest to the sound source position, and whose third position is the same as or closest to the first position, is selected as the first acoustic feature.

[0219] S504: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0220] S504 may refer to the description of S308 above, which will not be repeated here.

[0221] Fig. 6A and Figure 6B Schematic diagram of the frequency response curve of the acoustic characteristics shown exemplarily. Fig. 6A and Figure 6B In the figure are the frequency response curves of the real acoustic feature (such as curve 1), the frequency response curves of the adjusted virtual acoustic feature (i.e., the first acoustic feature) (i.e., curve 2), and the frequency response curves of the unadjusted virtual acoustic feature (i.e., the second acoustic feature) (curve 3) when the user is at different first positions in the virtual scene. By comparing curve 1, curve 2, and curve 3, it can be seen that the frequency response curve of the adjusted virtual acoustic feature is closer to the frequency response curve of the real acoustic feature; that is, by adopting the method provided by the present application, the virtual acoustic feature can be made closer to the real acoustic feature, thereby improving the quality of the spatial audio signal.

[0222] Figure 7 is a schematic diagram of an illustrative audio processing device.

[0223] A first acquisition module 701 is used to acquire a source audio signal, a sound source position of the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene;

[0224] A second acquisition module 702 is used to acquire a first acoustic feature according to the sound source position and the first position, where the first acoustic feature is obtained by adjusting the second acoustic feature according to the conversion information, and the second acoustic feature is obtained by extracting acoustic features according to the first position, the virtual scene and the sound source position, and the conversion information is used to describe the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene;

[0225] The audio signal generating module 703 is configured to generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0226] Exemplarily, the conversion information is determined by analyzing an acoustic feature group, where the acoustic feature group includes a third acoustic feature and a fourth acoustic feature, where the fourth acoustic feature is an acoustic feature of a second position in a real scene, and the third acoustic feature is an acoustic feature of the second position in a virtual scene.

[0227] Exemplarily, there are one or more acoustic feature groups, there are one or more second positions, and the second positions corresponding to the third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group are the same.

[0228] Exemplarily, the conversion information includes a conversion function, and the conversion function is obtained by analyzing the signal processing results of the third acoustic feature and the fourth acoustic feature.

[0229] Exemplarily, the conversion information includes a feature change rate, where the feature change rate is a change rate of the third acoustic feature relative to the fourth acoustic feature.

[0230] Exemplarily, the conversion information includes model output information obtained by inputting the third acoustic feature and the fourth acoustic feature into the model.

[0231] Exemplarily, the first acoustic characteristic includes at least one of the following: acoustic response, energy information, acoustic parameter or acoustic characteristic.

[0232] Exemplarily, the fourth acoustic feature is obtained by processing a test audio signal received at a second position in the real scene.

[0233] Exemplarily, the fourth acoustic feature is user-defined input.

[0234] Exemplarily, the second acquisition module 702 is specifically used to determine one or more acoustic feature groups, one acoustic feature group includes a third acoustic feature and a fourth acoustic feature, the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene; analyze the third acoustic feature and the fourth acoustic feature in the one or more acoustic feature groups to obtain conversion information; extract acoustic features according to the first position, the virtual scene and the sound source position to obtain the second acoustic feature; adjust the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0235] Exemplarily, the second acquisition module 702 is specifically used to select a second acoustic feature set corresponding to the virtual scene from multiple second acoustic feature sets stored in the database; wherein the multiple second acoustic feature sets correspond one-to-one to the multiple preset virtual scenes, a second acoustic feature set includes multiple second preset acoustic features, a second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple fourth positions and multiple preset sound source positions, a fourth position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set; according to the sound source position and the first position, select the second acoustic feature from the second acoustic feature set corresponding to the virtual scene; according to the virtual scene, select conversion information from multiple preset conversion information stored in the database, and the multiple preset conversion information correspond one-to-one to the multiple preset virtual scenes; adjust the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0236] Exemplarily, the second acquisition module 702 is specifically used to acquire the first acoustic feature, including:

[0237] A first acoustic feature set corresponding to a virtual scene is selected from multiple first acoustic feature sets stored in a database; wherein the multiple first acoustic feature sets correspond one-to-one to the multiple preset virtual scenes, the multiple first acoustic feature sets correspond one-to-one to the multiple second acoustic feature sets, a second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple third positions and multiple preset sound source positions, a fourth position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set, and a first preset acoustic feature in a first acoustic feature set is obtained by adjusting a second preset acoustic feature in the corresponding second acoustic feature set according to conversion information; according to the sound source position and the first position, a first acoustic feature is selected from the first acoustic feature set corresponding to the virtual scene.

[0238] An embodiment of the present application also provides an AR device, which includes a display module, an image acquisition module, headphones and a processor, wherein: the processor is used to execute the audio processing method in the above embodiment; the headphones are used to play the spatial audio signal generated by the audio processing method in the above embodiment.

[0239] An embodiment of the present application also provides a VR device, which includes a display module, an image acquisition module, headphones and a processor, wherein: the processor is used to execute the audio processing method in the above embodiment; the headphones are used to play the spatial audio signal generated by the audio processing method in the above embodiment.

[0240] In one example, Figure 8A schematic block diagram of a device 800 according to an embodiment of the present application is shown. The device 800 may include: a processor 801 and a transceiver / transceiver pin 802 , and optionally, a memory 803 .

[0241] The components of the device 800 are coupled together via a bus 804, wherein the bus 804 includes a power bus, a control bus, and a status signal bus in addition to a data bus. However, for the sake of clarity, all buses are referred to as bus 804 in the figure.

[0242] Optionally, the memory 803 may be used to store instructions in the aforementioned method embodiment. The processor 801 may be used to execute instructions in the memory 803, and control the receiving pin to receive a signal, and control the sending pin to send a signal.

[0243] The apparatus 800 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0244] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0245] The embodiment of the present application also provides a chip, including one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the above-mentioned related method steps implement the steps of the method in the above-mentioned embodiment. The interface circuit is a transceiver / transceiver pin 802.

[0246] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0247] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0248] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory so that the chip executes the methods in the above-mentioned method embodiments.

[0249] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above and will not be repeated here.

[0250] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0251] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0252] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0253] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0254] Any content of each embodiment of the present application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of the present application.

[0255] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0256] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (ReadOnly Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disk (CD-ROM) or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0257] Those skilled in the art should be aware that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented with hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein the communication media include any media that facilitates the transmission of a computer program from one place to another. The storage medium can be any available medium that a general or special-purpose computer can access.

[0258] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. An audio processing method, characterized in that: The method comprises: Acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene; Acquiring a first acoustic feature according to the sound source position and the first position, wherein the first acoustic feature is obtained by adjusting a second acoustic feature according to conversion information, and the second acoustic feature is obtained by extracting acoustic features according to the first position, the virtual scene, and the sound source position, and the conversion information is used to describe a conversion relationship between acoustic features at the same position in the virtual scene and the real scene; A spatial audio signal is generated according to the first acoustic feature and the source audio signal.

2. The method according to claim 1, characterized in that The conversion information is determined by analyzing an acoustic feature group, wherein the acoustic feature group includes a third acoustic feature and a fourth acoustic feature, wherein the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene.

3. The method according to claim 2, characterized in that There are one or more acoustic feature groups, there are one or more second positions, and the second positions corresponding to the third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group are the same.

4. The method according to claim 2 or 3, characterized in that: The conversion information includes a conversion function, and the conversion function is obtained by analyzing signal processing results of the third acoustic feature and the fourth acoustic feature.

5. The method according to any one of claims 2 to 4, characterized in that: The conversion information includes a feature change rate, where the feature change rate is a change rate of the third acoustic feature relative to the fourth acoustic feature.

6. The method according to any one of claims 2 to 5, characterized in that: The conversion information includes model output information obtained by inputting the third acoustic feature and the fourth acoustic feature into a model.

7. The method according to any one of claims 1 to 6, characterized in that: The first acoustic feature includes at least one of the following: acoustic response, energy information, acoustic parameter or acoustic feature.

8. The method according to any one of claims 2 to 7, characterized in that: The fourth acoustic feature is obtained by processing a test audio signal received at the second position in the real scene.

9. The method according to any one of claims 2 to 7, characterized in that: The fourth acoustic feature is input by a user.

10. The method according to any one of claims 1 to 9, characterized in that: Acquiring a first acoustic feature according to the sound source position and the first position includes: Determine one or more acoustic feature groups, one acoustic feature group includes a third acoustic feature and a fourth acoustic feature, the fourth acoustic feature is an acoustic feature of a second position in the real scene, and the third acoustic feature is an acoustic feature of the second position in the virtual scene; analyzing the third acoustic feature and the fourth acoustic feature in the one or more acoustic feature groups to obtain the conversion information; Extracting acoustic features according to the first position, the virtual scene, and the sound source position to obtain the second acoustic features; The second acoustic feature is adjusted according to the conversion information to obtain the first acoustic feature.

11. The method according to any one of claims 1 to 9, characterized in that: Acquiring a first acoustic feature according to the sound source position and the first position includes: Selecting a first acoustic feature set corresponding to the virtual scene from a plurality of first acoustic feature sets stored in a database; wherein the plurality of first acoustic feature sets correspond one-to-one to a plurality of preset virtual scenes, the plurality of first acoustic feature sets correspond one-to-one to a plurality of second acoustic feature sets, a second acoustic feature set is obtained by extracting acoustic features according to a preset virtual scene, a plurality of third positions, and a plurality of preset sound source positions, a third position and a preset sound source position are used to determine a second preset acoustic feature in the second acoustic feature set, and a first preset acoustic feature in a first acoustic feature set is obtained by adjusting a second preset acoustic feature in the corresponding second acoustic feature set according to the conversion information; The first acoustic feature is selected from a first acoustic feature set corresponding to the virtual scene according to the sound source position and the first position.

12. The method according to any one of claims 1 to 9, characterized in that: Acquiring a first acoustic feature according to the sound source position and the first position includes: Selecting a second acoustic feature set corresponding to the virtual scene from a plurality of second acoustic feature sets stored in a database; wherein the plurality of second acoustic feature sets correspond one-to-one to a plurality of preset virtual scenes, a second acoustic feature set includes a plurality of second preset acoustic features, the second acoustic feature set is obtained by extracting acoustic features according to a preset virtual scene, a plurality of fourth positions and a plurality of preset sound source positions, and a fourth position and a preset sound source position are used to determine a second preset acoustic feature in the second acoustic feature set; Selecting the second acoustic feature from a second acoustic feature set corresponding to the virtual scene according to the sound source position and the first position; According to the virtual scene, selecting the conversion information from a plurality of preset conversion information stored in the database, wherein the plurality of preset conversion information corresponds one-to-one to a plurality of preset virtual scenes; The second acoustic feature is adjusted according to the conversion information to obtain the first acoustic feature.

13. An audio processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene; a second acquisition module, configured to acquire a first acoustic feature according to the sound source position and the first position, wherein the first acoustic feature is obtained by adjusting a second acoustic feature according to conversion information, and the second acoustic feature is obtained by extracting acoustic features according to the first position, the virtual scene, and the sound source position, and the conversion information is used to describe a conversion relationship between acoustic features at the same position in the virtual scene and the real scene; The audio signal generating module is configured to generate a spatial audio signal according to the first acoustic feature and the source audio signal.

14. An augmented reality (AR) device, characterized in that: The AR device includes a display module, an image acquisition module, a headset and a processor, wherein: The processor is used to execute the audio processing method as described in any one of claims 1 to 12 above; The earphone is used to play the spatial audio signal generated by the audio processing method according to any one of claims 1 to 12.

15. A virtual reality VR device, characterized in that: The VR device includes a display module, an image acquisition module, headphones and a processor, wherein: The processor is used to execute the audio processing method as described in any one of claims 1 to 12 above; The earphone is used to play the spatial audio signal generated by the audio processing method according to any one of claims 1 to 12.

16. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the audio processing method according to any one of claims 1 to 12.

17. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as claimed in any one of claims 1 to 12 are executed.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the audio processing method according to any one of claims 1 to 12.

19. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 12 are performed.

Images