Audio processing method and electronic device

Abstract

Provided in the embodiments of the present application are an audio processing method and an electronic device. The method comprises: first, acquiring a source audio signal, a corresponding sound source position of the source audio signal in a virtual scene, and a corresponding first position of a user in the virtual scene, wherein the virtual scene is obtained by constructing a real scene; acquiring a first acoustic feature, wherein the first acoustic feature is obtained by converting, when the source audio signal is located at the sound source position, an acoustic feature of the first position in the virtual scene on the basis of an acoustic feature of a second position in the real scene and an acoustic feature of a second position in the virtual scene; and then generating a spatial audio signal on the basis of the first acoustic feature and the source audio signal. In this way, the user can experience spatial audio signals with relatively high quality and relatively strong sense of presence.

Description

Audio processing method and electronic device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on August 15, 2023, with application number 202311035340.3 and application name “Audio Processing Method and Electronic Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of audio processing, and in particular to an audio processing method and electronic device. Background Art

[0003] Currently, many electronic devices (such as mobile phones, augmented reality (AR) devices, virtual reality (VR) devices, etc.) have spatial audio functions, which can render the source audio signal into spatial audio signals, so that users can perceive the position, distance and spatial sense of sound and image in the audio, bringing users an immersive listening experience.

[0004] Typically, source audio signals are rendered into spatial audio signals using the acoustic response of the target space. However, the acoustic responses generated by existing computational techniques often differ from those in real-world scenes, resulting in lower-quality spatial audio signals and a weaker sense of presence.

[0005] Summary of the Invention

[0006] In order to solve the above technical problems, the present application provides an audio processing method and an electronic device, which can enable users to experience spatial audio signals with high quality and strong sense of presence.

[0007] It should be noted that the application scenarios of this application may include: scenarios in which users use AR / VR devices to experience AR / VR projects (such as AR / VR science lectures, AR / VR cinemas, AR / VR concerts, etc.), and scenarios in which users use mobile phones, tablets, laptops, personal computers, smart watches and other terminal devices + headphones to experience listening projects (such as crosstalk, movies, concerts, music concerts, etc.).

[0008] In a first aspect, an embodiment of the present application provides an audio processing method, which includes: first, obtaining a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene; according to the sound source position and the first position, obtaining a first acoustic feature, the first acoustic feature is obtained by adjusting the second acoustic feature according to conversion information, the second acoustic feature is obtained by extracting acoustic features based on the first position, the virtual scene, and the sound source position, and the conversion information is used to describe the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene; then, generating a spatial audio signal based on the first acoustic feature and the source audio signal.

[0009] That is to say, the present application adjusts the acoustic features of the first position corresponding to the user's current position in the virtual scene (i.e., the second acoustic features) based on the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene, so that the acoustic features of the first position corresponding to the user in the virtual scene (i.e., the first acoustic features) are closer to the acoustic features of the user's position in the real scene corresponding to the virtual scene; in this way, the spatial audio signal subsequently generated based on the adjusted acoustic features (i.e., the first acoustic features) is closer to the spatial audio signal heard by the user at the position in the real scene corresponding to the virtual scene, so that the user can experience a spatial audio signal with higher quality and stronger sense of presence.

[0010] It should be noted that the conversion information can be used to adjust the second acoustic feature of any first position in the virtual environment to obtain the corresponding first acoustic feature.

[0011] For example, the virtual scene in which the user is located can refer to the virtual scene corresponding to the listening project or AR / VR project selected (or experienced) by the user. For example, if the listening project is a concert, the corresponding virtual scene is a virtual stadium or virtual studio; if the listening project is a movie, the corresponding virtual scene is a virtual cinema; if the AR / VR project is a concert, the corresponding virtual scene is a virtual concert hall; if the AR / VR project is a lecture, the corresponding virtual scene is a virtual lecture hall, and so on. In other words, the virtual scene in which the user is located can refer to the virtual scene corresponding to the spatial audio signal experienced by the user.

[0012] Exemplarily, the virtual scene is obtained by modeling the real scene, and there may be multiple ways to generate the virtual scene, which is not limited by this application. The real scene refers to an objectively existing scene, such as a cinema, a concert hall, a lecture hall, etc.; the virtual scene is a scene constructed using virtual technology, and is not an objectively existing scene. The virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user is obtained by modeling the real scene. For example, a virtual concert hall is obtained by modeling a real concert hall; for another example, a virtual cinema is obtained by modeling a real cinema; for another example, a virtual lecture hall is obtained by modeling a real lecture hall; and so on.

[0013] It should be noted that the real scene currently in which the user is located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may be the same scene or different scenes, and this application does not impose any restrictions on this. In other words, the real scene currently in which the user is located and the real scene used to model the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may be the same scene or different scenes, and this application does not impose any restrictions on this.

[0014] Exemplarily, when the real scene in which the user is currently located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user, the user's position in the real scene is the same as the corresponding position of the user in the virtual scene, that is, the user's position in the real scene is the first position.

[0015] For example, when the real scene in which the user is currently located is not the same as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user: in one possible way, the default position can be determined as the first position corresponding to the user in the virtual scene; wherein, the default position can be set as required, for example, it can be the position with the best audio experience in the virtual scene, and this application does not limit this. In one possible way, the terminal device can provide position options in the virtual scene, so that the user can select the position option corresponding to the desired experience position in the virtual scene; then, the terminal device can use the position corresponding to the position option selected by the user as the first position corresponding to the user in the virtual scene.

[0016] Exemplarily, there may be multiple modeling methods, such as manual modeling, modeling based on visual information, modeling based on auditory information, etc., which are not limited in this application.

[0017] Exemplarily, spatial audio processing (such as rendering) can be performed according to the first acoustic feature and the source audio signal to generate a spatial audio signal (such as a binaural rendering signal).

[0018] Exemplarily, the source audio signal may include an audio file (such as a music file, a crosstalk file, etc.), or an audio file included in a multimedia file (such as an audio file included in a movie file, etc.).

[0019] Illustratively, the first acoustic feature may be generated in advance and stored in a database, or may be generated in real time, and this application does not impose any limitation thereto.

[0020] Exemplarily, the second acoustic feature may be generated in advance and stored in a database, or may be generated in real time, and this application does not impose any limitation thereto.

[0021] Exemplarily, the conversion information may be generated in advance and stored in a database, or may be generated in real time, and this application does not impose any limitation on this.

[0022] It should be noted that the first acoustic feature, the second acoustic feature and the conversion information can be stored in the same database or different databases.

[0023] Exemplarily, the methods of extracting acoustic features may include, but are not limited to: acoustic simulation, machine learning, numerical calculation, etc., which are not limited in this application.

[0024] According to the first aspect, the conversion information is determined by analyzing an acoustic feature set, the acoustic feature set including a third acoustic feature and a fourth acoustic feature, where the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene. Thus, by obtaining and analyzing the acoustic features of the same position in both the real and virtual scenes, an accurate conversion relationship between the acoustic features of the same position in the real and virtual scenes can be determined.

[0025] According to the first aspect, or any implementation of the first aspect above, there are one or more acoustic feature groups, there are one or more second positions, and the second positions corresponding to the third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group are the same.

[0026] For example, when the real scene is a uniform room (for example, the room is square, the wall materials are uniform, etc., where the room can be understood as an indoor scene), and the difference between the real acoustic characteristics of different positions is relatively small, the second position can be one, thereby reducing the workload. For another example, when the real scene is an uneven room (for example, the room is asymmetrical, the wall materials are diverse, etc.), and the difference between the real acoustic characteristics of different positions is relatively large, the second position can be multiple, thereby increasing the accuracy of the generated conversion information. It should be understood that the number of second positions can be determined according to demand, and this application does not impose any restrictions on this.

[0027] For example, when sound source devices are deployed at different locations (hereinafter referred to as preset sound source locations) in a real scene, the fourth acoustic features of the same location are different; correspondingly, when virtual sound sources are deployed at different preset sound source locations in a virtual scene, the third acoustic features of the same location are different; that is, in a real scene, each second location corresponds to one or more fourth acoustic features; in a virtual scene, each second location corresponds to one or more third acoustic features. Therefore, a second location can correspond to one or more acoustic feature groups, and the third acoustic features and fourth acoustic features belonging to the same acoustic feature group correspond to the same second location, and the corresponding preset sound source locations are the same.

[0028] It should be noted that this application only needs to analyze the conversion relationship between the acoustic features of one or more positions (i.e., one or more second positions) in the virtual scene and the real scene to obtain conversion information; and the conversion information is universally applicable to any position in the same room (i.e., the same virtual scene), rather than being able to be used only to adjust the acoustic features of the first position close to the second position.

[0029] According to the first aspect, or any implementation of the first aspect above, the conversion information includes a conversion function, and the conversion function is obtained by analyzing signal processing results of the third acoustic feature and the fourth acoustic feature.

[0030] For example, the third and fourth acoustic features in an acoustic feature group can be signal processed separately to obtain signal processing results. The signal processing results of the third and fourth acoustic features can then be analyzed to obtain a transfer function. In this case, the transfer function can be used as the conversion information. For example, the third acoustic feature includes a third acoustic response, such as a room impulse response (RIR), and the fourth acoustic feature includes a fourth acoustic response, such as an RIR, where the third acoustic feature is represented by RIR 1 and the fourth acoustic feature is represented by RIR 2. Frequency domain conversion can be performed on RIR 1 and RIR 2 to obtain the frequency domain response of RIR 1 (referred to as frequency response 1) and the frequency domain response of RIR 2 (referred to as frequency response 2). In one possible approach, a frequency response transfer function can be calculated using frequency response 1 and frequency response 2, and the frequency response transfer function can be used as the transfer function. In another possible approach, signal analysis can be performed on frequency response 1 and frequency response 2, and the transfer function can be determined based on the signal analysis results.

[0031] According to the first aspect, or any implementation of the first aspect above, the conversion information includes a feature change rate, where the feature change rate is a change rate of the third acoustic feature relative to the fourth acoustic feature.

[0032] Exemplarily, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be numerically analyzed respectively to obtain a feature change rate; in this case, the feature change rate can be used as conversion information. Specifically, the change rate of the third acoustic feature in the acoustic feature group relative to the fourth acoustic feature in the acoustic feature group can be calculated as the feature change rate. For example, for the third acoustic parameter such as the direct mixing ratio contained in the third acoustic feature, and the fourth acoustic parameter such as the direct mixing ratio contained in the fourth acoustic feature, wherein the direct mixing ratio in the third acoustic parameter is represented by DRR 1, and the direct mixing ratio in the fourth acoustic parameter is represented by DRR 2; the corresponding feature change rate can be (DRR 1-DRR 2) / DRR 2, or, DRR1 / DRR2. It should be understood that the conversion information can include multiple feature change rates (multiple feature change rates correspond one-to-one to multiple acoustic parameters), and this application does not limit this.

[0033] According to the first aspect, or any implementation of the first aspect above, the conversion information includes model output information obtained by inputting the third acoustic feature and the fourth acoustic feature into the model.

[0034] For example, a machine learning method can be used to process the third and fourth acoustic features in an acoustic feature group to obtain conversion information. Specifically, the third and fourth acoustic features in an acoustic feature group can be input into an AI model (or a machine learning model, hereinafter referred to as a second model), and the second model processes the third and fourth acoustic features in an acoustic feature group and outputs conversion information; that is, the model output information of the second model is the conversion information.

[0035] For example, the model output information may include a conversion function, a characteristic change rate, and other forms of information, which are not limited in this application.

[0036] It should be understood that other methods may be used to generate conversion information, and this application does not limit this.

[0037] According to the first aspect, or any implementation of the first aspect, the first acoustic feature includes at least one of the following: acoustic response, energy information, acoustic parameter, or acoustic feature.

[0038] The acoustic response may include but is not limited to: room impulse response, binaural room impulse response (BRIR), higher order ambisonics (HOA), etc.

[0039] The energy information may include multiple frequency bands corresponding to the acoustic response, energy distribution in multiple directions, energy attenuation, etc.

[0040] Acoustic parameters may include ambient acoustic parameters and binaural acoustic parameters. Ambient acoustic parameters include, but are not limited to, direct-to-reverberation energy ratio (DRR), reverberation time (such as T20 (the time it takes for energy to decay by 20 dB), T30, T60, etc.), clarity (such as C50 (the ratio of sound energy before and after 50 ms), C80, etc.). Binaural acoustic parameters may include, but are not limited to, interaural time difference (ITD), interaural level difference (ILD), interaural cross correlation coefficient (IACC), etc.

[0041] Acoustic features may include numerical analysis features obtained by performing numerical analysis (for example, principal component analysis (PCA)) on acoustic parameters; and may also include output information of the first model obtained by inputting at least one of acoustic response, energy information or acoustic parameters into an AI model (hereinafter referred to as the first model).

[0042] According to the first aspect, or any implementation of the first aspect above, the fourth acoustic feature is obtained by processing a test audio signal received at a second position in a real scene.

[0043] Exemplarily, a sound source device can be deployed in a real scene. After the user carries the receiving device to the second position, the sound source device can be controlled to play a test audio signal. Correspondingly, the receiving device at the second position can receive the test audio signal; then, the receiving device can process the received test audio signal to obtain a fourth acoustic feature; it should be understood that the fourth acoustic feature is a real acoustic feature.

[0044] For example, the sound source device plays a test audio signal such as a pulse signal, a noise signal, or a sweep signal, and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in a format such as RIR, BRIR, or HOA.

[0045] For another example, the sound source device plays sound signals such as pulse signals, noise signals, and sweep frequency signals, and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, and acoustic characteristics.

[0046] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in the format of RIR, BRIR, HOA, etc.

[0047] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, acoustic characteristics, etc.

[0048] Among them, sound source equipment includes but is not limited to: speaker equipment (such as home audio), terminal equipment with external speaker function (such as tablets, large screens, etc.), professional acoustic measurement equipment, etc.

[0049] The receiving device includes but is not limited to: terminal devices including microphones (such as tablets, mobile phones, AR / VR devices, etc.), professional acoustic measurement equipment, etc.

[0050] According to the first aspect, or any implementation of the first aspect above, the fourth acoustic feature is user-defined input.

[0051] Since the acoustic features obtained through actual measurements in certain real scenes may not be the most accurate match for the user's actual hearing experience, directly inputting the acoustic features that match the user's actual hearing experience in certain real scenes makes the adjusted acoustic features closer to the acoustic features that match the user's actual hearing experience in certain real scenes. In this way, the spatial audio signal subsequently generated based on the adjusted acoustic features is closer to the spatial audio signal heard by the user at the position in the real scene corresponding to the certain virtual scene.

[0052] Exemplarily, "certain real scenes" may include: real scenes with unavoidable interfering noise, real scenes where the visual presentation of surface materials is inconsistent with their actual acoustic effects, and real scenes that serve as multi-functional spaces (for example, real scenes that serve as both a lecture hall, a concert hall, and a film and television appreciation hall).

[0053] According to the first aspect, or any implementation method of the first aspect above, a first acoustic feature is obtained according to the sound source position and the first position, including: determining one or more acoustic feature groups, one acoustic feature group including a third acoustic feature and a fourth acoustic feature, the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene; analyzing the third acoustic feature and the fourth acoustic feature in the one or more acoustic feature groups to obtain conversion information; extracting acoustic features according to the first position, the virtual scene and the sound source position to obtain a second acoustic feature; adjusting the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0054] In this case, the real scene in which the user is currently located is the same as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user, and conversion information, second acoustic features and first acoustic features can be generated in real time.

[0055] According to the first aspect, or any implementation method of the first aspect above, a first acoustic feature is obtained according to the sound source position and the first position, including: selecting a second acoustic feature set corresponding to the virtual scene from multiple second acoustic feature sets stored in a database; wherein the multiple second acoustic feature sets correspond one-to-one to multiple preset virtual scenes, a second acoustic feature set includes multiple second preset acoustic features, a second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple fourth positions and multiple preset sound source positions, and a fourth position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set; according to the sound source position and the first position, a second acoustic feature is selected from the second acoustic feature set corresponding to the virtual scene; according to the virtual scene, conversion information is selected from multiple preset conversion information stored in the database, and the multiple preset conversion information corresponds one-to-one to the multiple preset virtual scenes; the second acoustic feature is adjusted according to the conversion information to obtain the first acoustic feature.

[0056] In this case, the real scene in which the user is currently located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may or may not be the same scene. In this way, the terminal device does not need to generate conversion information and second acoustic features in real time, which can shorten the time to generate spatial audio signals and improve user experience. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient way.

[0057] According to the first aspect, or any implementation method of the first aspect above, a first acoustic feature is obtained according to the sound source position and the first position, including: selecting a first acoustic feature set corresponding to the virtual scene from multiple first acoustic feature sets stored in a database; wherein, multiple first acoustic feature sets correspond one-to-one to multiple preset virtual scenes, multiple first acoustic feature sets correspond one-to-one to multiple second acoustic feature sets, a second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple third positions and multiple preset sound source positions, a third position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set, a first preset acoustic feature in a first acoustic feature set is obtained by adjusting a second preset acoustic feature in the corresponding second acoustic feature set according to conversion information; according to the sound source position and the first position, a first acoustic feature is selected from the first acoustic feature set corresponding to the virtual scene.

[0058] In this case, the real scene in which the user is currently located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user may be the same scene or different scenes. In this way, the terminal device does not need to generate conversion information, the second acoustic feature, and the first acoustic feature in real time, which can further shorten the time to generate the spatial audio signal and improve the user experience. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient way.

[0059] In a second aspect, an embodiment of the present application provides an audio processing device, the device comprising:

[0060] a first acquisition module, configured to acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene;

[0061] a second acquisition module, configured to acquire a first acoustic feature based on the sound source position and the first position, wherein the first acoustic feature is obtained by adjusting the second acoustic feature based on the conversion information, and the second acoustic feature is obtained by extracting acoustic features based on the first position, the virtual scene, and the sound source position, wherein the conversion information is used to describe a conversion relationship between acoustic features at the same position in the virtual scene and the real scene;

[0062] The audio signal generating module is configured to generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0063] It should be understood that the audio processing device of the second aspect can execute any implementation of the first aspect, which will not be described in detail here.

[0064] The second aspect and any implementation of the second aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the second aspect and any implementation of the second aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0065] In a third aspect, an embodiment of the present application provides an electronic device comprising: a memory and a processor, the memory being coupled to the processor; the memory storing program instructions, which, when executed by the processor, enables the electronic device to execute the audio processing method in the first aspect or any possible implementation of the first aspect.

[0066] The third aspect and any implementation of the third aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the third aspect and any implementation of the third aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0067] In a fourth aspect, an embodiment of the present application provides a chip comprising one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the audio processing method in the first aspect or any possible implementation of the first aspect are executed.

[0068] The fourth aspect and any implementation of the fourth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fourth aspect and any implementation of the fourth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0069] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a computer or a processor, the computer or the processor executes the audio processing method in the first aspect or any possible implementation of the first aspect.

[0070] The fifth aspect and any implementation of the fifth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the fifth aspect and any implementation of the fifth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0071] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer or the processor executes the audio processing method in the first aspect or any possible implementation of the first aspect.

[0072] The sixth aspect and any implementation of the sixth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the sixth aspect and any implementation of the sixth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0073] In a seventh aspect, an embodiment of the present application provides an augmented reality (AR) device, comprising: a display module, an image acquisition module, headphones, and a processor, wherein:

[0074] The processor is configured to execute the audio processing method in the first aspect or any possible implementation manner of the first aspect;

[0075] The earphones are used to play the spatial audio signal generated by the audio processing method in the first aspect or any possible implementation manner of the first aspect.

[0076] Exemplarily, the display module can be used to display images; the image acquisition module is used to acquire images.

[0077] The seventh aspect and any implementation of the seventh aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the seventh aspect and any implementation of the seventh aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0078] In an eighth aspect, an embodiment of the present application provides an augmented reality (VR) device, comprising: a display module, an image acquisition module, headphones, and a processor, wherein:

[0079] The processor is configured to execute the audio processing method in the first aspect or any possible implementation manner of the first aspect;

[0080] The earphones are used to play the spatial audio signal generated by the audio processing method in the first aspect or any possible implementation manner of the first aspect.

[0081] Exemplarily, the display module can be used to display images; the image acquisition module is used to acquire images.

[0082] The eighth aspect and any implementation of the eighth aspect correspond to the first aspect and any implementation of the first aspect, respectively. The technical effects corresponding to the eighth aspect and any implementation of the eighth aspect can be referred to the technical effects corresponding to the first aspect and any implementation of the first aspect, and will not be repeated here.

[0083] In a ninth aspect, the present application provides an audio processing method, which includes: first, obtaining a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by constructing a real scene; then, obtaining a first acoustic feature, which is obtained by converting the acoustic feature of the first position in the virtual scene when the source audio signal is at the sound source position based on the acoustic feature of the second position in the real scene and the acoustic feature of the second position in the virtual scene; thereafter, generating a spatial audio signal based on the first acoustic feature and the source audio signal.

[0084] In one possible approach, the virtual scene is obtained by modeling a real scene.

[0085] Illustratively, the acoustic feature of the second position in the real scene may refer to the fourth acoustic feature in the following specific implementation.

[0086] Exemplarily, the acoustic feature of the first position in the virtual scene when the source audio signal is at the sound source position may refer to the second acoustic feature in the following specific implementation manner.

[0087] Illustratively, the acoustic feature of the second position in the virtual scene may refer to the third acoustic feature in the following specific implementation.

[0088] In one possible approach, obtaining the first acoustic feature may include converting the acoustic feature at the first position in the virtual scene based on the acoustic feature at the second position in the real scene and the acoustic feature at the second position in the virtual scene to obtain the first acoustic feature. In other words, the acoustic feature at the first position in the virtual scene is converted in real time during application to obtain the first acoustic feature.

[0089] In one possible approach, obtaining the first acoustic feature may include: selecting a first acoustic feature set corresponding to a virtual scene from multiple first acoustic feature sets corresponding to multiple preset virtual scenes stored in a database; and selecting a first acoustic feature from the first acoustic feature set corresponding to the virtual scene based on the sound source position and the first position. In other words, first acoustic features for multiple positions are predetermined, and during application, the database is searched for a first acoustic feature that matches the current situation (i.e., the current sound source position and the first position).

[0090] According to a ninth aspect, the first acoustic feature includes at least one of the following: an acoustic response, energy information, an acoustic parameter, or an acoustic feature.

[0091] According to the ninth aspect, or any implementation of the ninth aspect, the second position is one or more.

[0092] According to the ninth aspect, or any implementation of the ninth aspect, the first positions corresponding to the user in the virtual scene at different times are the same or different.

[0093] According to the ninth aspect, or any implementation of the ninth aspect, the second position is the same as or different from the first position.

[0094] According to the ninth aspect, or any implementation of the ninth aspect above, the spatial audio signal generated based on the first acoustic feature and the source audio signal is different from the spatial audio signal generated based on the acoustic feature of the first position in the virtual scene and the source audio signal.

[0095] According to the ninth aspect, or any implementation of the ninth aspect above, the method further includes: reading the acoustic features of the second position in the real scene from a database.

[0096] According to the ninth aspect, or any implementation of the ninth aspect above, the acoustic characteristics of the second position in the real scene are obtained by collecting environmental acoustic characteristics using professional acoustic measurement equipment.

[0097] According to the ninth aspect, or any implementation of the ninth aspect above, the method further includes: obtaining user-defined acoustic features of a second position in the real scene.

[0098] According to the ninth aspect, or any implementation of the ninth aspect above, before obtaining the source audio signal, the sound source position corresponding to the source audio signal in the virtual scene, and the first position corresponding to the user in the virtual scene, the method also includes: obtaining the user operation performed by the user in the user interface; based on the user operation, generating the acoustic characteristics of the second position in the real scene.

[0099] According to the ninth aspect, or any implementation of the ninth aspect above, before obtaining the source audio signal, the sound source position corresponding to the source audio signal in the virtual scene, and the first position corresponding to the user in the virtual scene, the method also includes: obtaining the user's position information; when determining that the user moves to the second position based on the position information, generating the acoustic characteristics of the second position in the real scene.

[0100] According to the ninth aspect, or any implementation of the ninth aspect above, the method further includes: acquiring the acoustic characteristics of the second position in the real scene based on the sound source position, the first position and the second position.

[0101] The technical effects corresponding to the ninth aspect and any implementation method of the ninth aspect can be referred to the technical effects corresponding to the above-mentioned first aspect and any implementation method of the first aspect, and will not be repeated here.

[0102] In addition, the present application also provides an audio processing device, an augmented reality AR device, a virtual reality VR device, an electronic device, a chip, a computer-readable storage medium, and a computer program product, etc. for executing the method in the above-mentioned first aspect and any one of the implementation methods of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] FIG1A is a schematic diagram illustrating an exemplary application scenario;

[0104] FIG1B is a schematic diagram illustrating an exemplary application scenario;

[0105] FIG1C is a schematic diagram illustrating an exemplary application scenario;

[0106] FIG1D is a schematic diagram illustrating an exemplary application scenario;

[0107] FIG2 is a schematic diagram illustrating an exemplary audio processing process;

[0108] FIG3 is a schematic diagram illustrating an exemplary audio processing process;

[0109] FIG4 is a schematic diagram illustrating an exemplary audio processing process;

[0110] FIG5 is a schematic diagram illustrating an exemplary audio processing process;

[0111] FIG6A is a schematic diagram of a frequency response curve of an acoustic characteristic;

[0112] FIG6B is a schematic diagram of a frequency response curve of an exemplary acoustic feature;

[0113] FIG7 is a schematic diagram illustrating an exemplary audio processing device;

[0114] FIG8 is a schematic structural diagram of an exemplary device. DETAILED DESCRIPTION

[0115] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0116] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0117] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0118] In the embodiments of this application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplarily" or "for example" is intended to present the relevant concepts in a concrete manner.

[0119] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0120] Figure 1A is a schematic diagram illustrating an exemplary application scenario. Figure 1A illustrates an application scenario in which a user uses an AR device to experience an AR concert. It should be understood that users can also use AR devices to experience other AR projects, such as AR science lectures, AR cinemas, and so on, and this application does not limit this.

[0121] Among them, the real scene where the user is in Figure 1A and the virtual scene corresponding to the AR project experienced by the user are the same scene, that is, the real scene where the user is in is a concert hall, and the virtual scene corresponding to the AR project experienced by the user is a virtual concert hall.

[0122] 1A , for example, when a user wishes to experience an AR concert in an unoccupied concert hall, the user can wear an AR device. The user can then select the AR concert hall as the AR project they wish to experience, and select the music file they wish to experience and play it. The AR device can then process the music file selected by the user, such as rendering it, to obtain a spatial audio signal and play the spatial audio signal through headphones. The AR device can also display the performance image on a display screen, allowing the user to see the performer playing at the corresponding position on the concert hall stage. In this way, through the combination of vision and hearing, the user can experience the concert immersively.

[0123] It should be understood that there may be one or more users using AR devices to experience the AR concert in the concert hall in FIG1A , and this application does not impose any restrictions on this.

[0124] Figure 1B is a schematic diagram illustrating an exemplary application scenario. Figure 1B illustrates an application scenario where a user uses a VR device to experience a VR cinema. It should be understood that users can also use VR devices to experience other VR projects, such as a VR concert hall, a VR science lecture hall, and so on, and this application does not limit this.

[0125] Among them, the real scene where the user is in Figure 1B is not the same as the virtual scene corresponding to the VR project that the user experiences, that is, the real scene where the user is in is the living room, and the virtual scene corresponding to the VR project that the user experiences is a virtual cinema.

[0126] Referring to Figure 1B , for example, when a user wishes to experience a VR cinema in their living room, they can wear a VR device. Next, the user can select VR Cinema as the VR project they wish to experience, select a movie file, and play it. The VR device can then process, such as rendering, the audio file contained in the movie file to generate a spatial audio signal, which is then played through headphones. The VR device can also display the movie footage on a display screen. This allows the user to immerse themselves in the movie through a combination of visual and auditory perception.

[0127] Figure 1C is a schematic diagram of an exemplary application scenario. In Figure 1C, an application scenario is shown in which a user uses (mobile phone + headphones) to experience a concert.

[0128] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as crosstalk, movies, concerts, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application. This application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to experience listening.

[0129] Among them, the real scene where the user is in Figure 1C and the virtual scene corresponding to the listening project experienced by the user are the same scene, that is, the real scene where the user is in is a concert hall, and the virtual scene corresponding to the listening project experienced by the user is a virtual concert hall.

[0130] Referring to Figure 1C , for example, when a user wishes to experience a concert in an unoccupied concert hall, they can put on headphones and connect them to their phone. The user can then select the concert as the desired listening experience on their phone, select the desired music file, and play it. The phone can then process the selected music file, such as rendering it, to generate a spatial audio signal. This spatial audio signal can then be sent to the headphones for playback. The phone can also play a video of the concert performance. This allows the user to immerse themselves in the concert hall using their phone.

[0131] It should be understood that there may be one or more users using mobile phones to experience the concert in the concert hall in FIG1C , and this application does not impose any limitation on this.

[0132] Figure 1D is a schematic diagram of an exemplary application scenario. In Figure 1D, it shows an application scenario in which a user uses (mobile phone + earphones) to experience crosstalk.

[0133] It should be understood that users can also use (mobile phone + headphones) to experience other listening projects such as concerts, movies, concerts, etc., and this application does not limit this. In addition, the mobile phone is only an example of this application. This application can also use terminal devices such as tablets, laptops, personal computers, smart watches, etc. + headphones to listen to music.

[0134] Among them, the real scene where the user is in Figure 1D is not the same as the virtual scene corresponding to the listening project experienced by the user, that is, the real scene where the user is in is the bedroom, and the virtual scene corresponding to the listening project experienced by the user is the virtual studio.

[0135] Referring to Figure 1D , for example, when a user wishes to experience crosstalk in their bedroom, they can wear headphones and connect them to their phone. The user can then select crosstalk as the desired listening item on their phone, select the desired crosstalk file, and play it. The phone can then process the selected crosstalk file, such as rendering it, to generate a spatial audio signal. This spatial audio signal can then be sent to the headphones for playback. The phone can also play the crosstalk video. This allows users to immersively experience crosstalk in their bedroom using their phone.

[0136] It should be understood that the present application can also be applied to other listening scenarios or other virtual scenarios, and the present application does not limit this.

[0137] The following describes the process of generating a spatial audio signal.

[0138] Figure 2 is a schematic diagram illustrating an exemplary audio processing process. The steps in the embodiment of Figure 2 can be executed by terminal devices such as AR devices, VR devices, mobile phones, tablet computers, laptops, personal computers, smart watches, etc.

[0139] S201 , obtaining a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene.

[0140] For example, when a user wants to listen to music, they can wear headphones and connect their phone to the headphones. Then, they can select a music item and the desired audio file on a terminal device such as a phone, tablet, laptop, personal computer, or smartwatch, and then play it. When a user wants to experience an AR / VR project, they can wear an AR / VR device, select the desired AR / VR project and the desired multimedia file, and then play it.

[0141] Exemplarily, after the user performs a play operation, the terminal device can respond to the user operation and obtain the audio file selected by the user (or the audio file contained in the multimedia file selected by the user), that is, obtain the source audio signal; and the terminal device can also obtain the first position corresponding to the user in the virtual scene and the sound source position corresponding to the source audio signal in the virtual scene, so as to subsequently generate a spatial audio signal that matches the first position and the sound source position.

[0142] For example, the virtual scene in which the user is located in S201 may refer to the virtual scene corresponding to the listening project or AR / VR project selected (or experienced) by the user. For example, if the listening project is a concert, the corresponding virtual scene is a virtual stadium or virtual studio; if the listening project is a movie, the corresponding virtual scene is a virtual cinema; if the AR / VR project is a concert, the corresponding virtual scene is a virtual concert hall; if the AR / VR project is a lecture, the corresponding virtual scene is a virtual lecture hall, and so on. In other words, the virtual scene in which the user is located in S201 may refer to the virtual scene corresponding to the spatial audio signal experienced by the user.

[0143] Exemplarily, the virtual scene is obtained by modeling the real scene; there are many ways to generate a virtual scene, which is not limited in this application and will be described in detail later. A real scene refers to an objectively existing scene, such as a cinema, a concert hall, a lecture hall, etc.; a virtual scene is a scene constructed using virtual technology, and is not an objectively existing scene. The virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user is obtained by modeling the real scene. For example, a virtual concert hall is obtained by modeling a real concert hall; for another example, a virtual cinema is obtained by modeling a real cinema; for another example, a virtual lecture hall is obtained by modeling a real lecture hall; and so on.

[0144] It should be noted that the real scene in which the user is currently located and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user can be the same scene as shown in the embodiments of Figures 1A and 1C above, or they can be different scenes as shown in the embodiments of Figures 1B and 1D above, and this application does not impose any restrictions on this. In other words, the real scene in which the user is currently located and the real scene used to model the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user can be the same scene as shown in the embodiments of Figures 1A and 1C above, or they can be different scenes as shown in the embodiments of Figures 1B and 1D above, and this application does not impose any restrictions on this.

[0145] Exemplarily, when the real scene in which the user is currently located is the same scene as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user, the user's position in the real scene is the same as the corresponding position of the user in the virtual scene, that is, the user's position in the real scene is the first position.

[0146] For example, when the real scene in which the user is currently located is not the same as the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user: in one possible way, the default position can be determined as the first position corresponding to the user in the virtual scene; wherein, the default position can be set as required, for example, it can be the position with the best audio experience in the virtual scene, and this application does not limit this. In one possible way, the terminal device can provide a position option in the virtual scene, so that the user can select the position option corresponding to the desired experience position in the virtual scene before performing the playback operation; then, the terminal device can use the position corresponding to the position option selected by the user as the first position corresponding to the user in the virtual scene.

[0147] S202, obtaining a first acoustic feature, where the first acoustic feature is obtained by converting the acoustic feature of the first position in the virtual scene when the source audio signal is at the sound source position based on the acoustic feature of the second position in the real scene and the acoustic feature of the second position in the virtual scene.

[0148] Next, a first acoustic feature may be obtained, and the first acoustic feature may be used for spatial audio processing to generate a spatial audio signal.

[0149] Exemplarily, the first acoustic feature may be obtained by converting the acoustic feature of the first position in the virtual scene when the source audio signal is at the sound source position in the virtual scene, based on the acoustic feature of the second position in the real scene and the acoustic feature of the second position in the virtual scene.

[0150] Illustratively, the first acoustic feature includes but is not limited to: acoustic response, energy information, acoustic parameters or acoustic features, etc., which are not limited in this application.

[0151] The acoustic response may include, but is not limited to, room impulse response (RIR), binaural room impulse response (BRIR), higher order ambisonics (HOA), and the like.

[0152] The energy information may include multiple frequency bands corresponding to the acoustic response, energy distribution in multiple directions, energy attenuation, etc.

[0153] Acoustic parameters may include ambient acoustic parameters and binaural acoustic parameters. Ambient acoustic parameters include, but are not limited to, direct-to-reverberation energy ratio (DRR), reverberation time (such as T20 (the time it takes for energy to decay by 20 dB), T30, T60, etc.), clarity (such as C50 (the ratio of sound energy before and after 50 ms), C80, etc.). Binaural acoustic parameters may include, but are not limited to, interaural time difference (ITD), interaural level difference (ILD), interaural cross correlation coefficient (IACC), etc.

[0154] Acoustic features may include numerical analysis features obtained by performing numerical analysis (for example, principal component analysis (PCA)) on acoustic parameters; and may also include output information of the first model obtained by inputting at least one of acoustic response, energy information or acoustic parameters into an AI model (hereinafter referred to as the first model).

[0155] In order to facilitate subsequent explanation, the acoustic response contained in the first acoustic feature can be called the first acoustic response; the energy information contained in the first acoustic feature can be called the first energy information; the acoustic parameters contained in the first acoustic feature can be called the first acoustic parameters; and the acoustic features contained in the first acoustic feature can be called the first acoustic features.

[0156] In one possible scenario, the first acoustic feature can be obtained by adjusting the second acoustic feature based on the conversion information, the second acoustic feature is obtained by extracting the acoustic features based on the first position, the virtual scene and the sound source position, and the conversion information is used to describe the conversion relationship between the acoustic features of the same position in the virtual scene and the real scene; wherein the conversion information can be generated based on the acoustic features of the second position in the real scene and the acoustic features of the second position in the virtual scene.

[0157] In S202, one possible approach may be to first obtain conversion information, where the conversion information can be used to describe the conversion relationship between the acoustic features at the same location in the virtual scene and the real scene. It should be noted that the conversion information can be pre-generated and stored in a database, or generated in real time, and this application does not limit this. The method for generating the conversion information will be described later. Next, a second acoustic feature is obtained. The second acoustic feature is the acoustic feature of the first location in the virtual scene, and can be obtained by extracting the acoustic features based on the first location, the virtual scene, and the location of the sound source. The method for extracting the acoustic features may include, but is not limited to, acoustic simulation, machine learning, numerical calculation, etc., and this application does not limit this. It should be noted that the second acoustic feature can be pre-generated and stored in a database, or generated in real time, and this application does not limit this. The second acoustic feature can then be adjusted based on the conversion information to obtain the first acoustic feature. In this way, the acoustic features of the first location in the virtual scene can be adjusted to more closely match the acoustic features of the first location in the real scene, thereby improving the quality and sense of presence of the spatial audio signal, thereby enhancing the user experience. Among them, the acoustic features of any position in the virtual scene can be called virtual acoustic features, and the acoustic features of any position in the real scene can be called real acoustic features; that is, the first acoustic feature and the second acoustic feature are virtual acoustic features.

[0158] In S202, a possible approach may be to adjust the second preset acoustic features of each position in various preset virtual scenes (each position has one or more second preset acoustic features, and the multiple second preset acoustic features of each position correspond to multiple virtual sound source positions) in advance according to various preset conversion information (one preset conversion information corresponds to a preset virtual scene), obtain the first preset acoustic features of each position in various virtual scenes and store them in a database; in this way, the first acoustic features can be directly obtained from the database based on the virtual scene, the sound source position and the first position.

[0159] It should be understood that the second acoustic characteristic may include, but is not limited to: a second acoustic response, second energy information, a second acoustic parameter, or a second acoustic characteristic.

[0160] It should be noted that the conversion information, the second acoustic feature and the first acoustic feature can be stored in the same database or in different databases, and this application does not impose any restrictions on this.

[0161] S203: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0162] Afterwards, spatial audio processing (such as rendering) may be performed based on the first acoustic feature and the source audio signal to obtain a spatial audio signal (such as a binaural rendering signal).

[0163] Then, the AR / VR device can play the spatial audio signal through its built-in headphones; terminal devices such as mobile phones, tablets, laptops, smart watches, etc. can play the spatial audio signal through the headphones connected to them; in this way, users can hear the spatial audio signal through the headphones.

[0164] The following describes a process of generating the conversion information and a process of adjusting the second acoustic feature to obtain the first acoustic feature.

[0165] Figure 3 is a schematic diagram illustrating an exemplary audio processing process. In the embodiment of Figure 3, the user's current real scene and the virtual scene corresponding to the listening project (or AR / VR project) selected (or experienced) by the user are the same scene (i.e., the application scene can be Figures 1A and 1C above); in this case, the conversion information, the second acoustic feature, and the first acoustic feature can all be generated in real time.

[0166] S301, constructing a virtual scene based on the real scene.

[0167] In one possible approach, manual modeling can be used. For example, the size, position, surface material, and other information of each object in the real scene can be input into the modeling software, and the modeling software performs 3D modeling based on the user's input information to obtain a virtual scene.

[0168] One possible approach is to model the scene based on visual information. For example, a user wearing an AR / VR device can capture a sequence of images that overlap the real scene by moving or rotating the device. The AR / VR device then performs 3D modeling based on the image sequence to create a virtual scene.

[0169] It should be understood that other methods can also be used to construct virtual scenes, and this application does not limit this.

[0170] It should be noted that S301 can be a step executed in advance or a step executed in real time, and this application does not limit this.

[0171] S302: Acquire fourth acoustic features of one or more second positions in the real scene.

[0172] In one possible manner, the fourth acoustic feature may be obtained by processing an audio signal received at a second position in a real scene.

[0173] Exemplarily, a sound source device can be deployed in a real scene. After the user carries the receiving device to the second position, the sound source device can be controlled to play a test audio signal. Correspondingly, the receiving device at the second position can receive the test audio signal; then, the receiving device can process the received test audio signal to obtain a fourth acoustic feature; it should be understood that the fourth acoustic feature is a real acoustic feature.

[0174] For example, the sound source device plays a test audio signal such as a pulse signal, a noise signal, or a sweep signal, and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in a format such as RIR, BRIR, or HOA.

[0175] For another example, the sound source device plays sound signals such as pulse signals, noise signals, and sweep frequency signals, and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, and acoustic characteristics.

[0176] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain an acoustic response in the format of RIR, BRIR, HOA, etc.

[0177] For another example, the sound source device plays a natural audio signal (such as voice, song, pure music, video, etc.), and the receiving device at the second position receives and processes the test audio signal to obtain energy information, acoustic parameters, acoustic characteristics, etc.

[0178] Among them, sound source equipment includes but is not limited to: speaker equipment (such as home audio), terminal equipment with external speaker function (such as tablets, large screens, etc.), professional acoustic measurement equipment, etc.

[0179] The receiving device includes but is not limited to: terminal devices including microphones (such as tablets, mobile phones, AR / VR devices, etc.), professional acoustic measurement equipment, etc.

[0180] Among them, the second position is one or more. For example, when the real scene is a uniform room (for example, the room is square, the wall materials are uniform, etc., wherein the room can be understood as an indoor scene) and the difference between the real acoustic characteristics of different positions is relatively small, the second position can be one, thereby reducing the workload. For another example, when the real scene is an uneven room (for example, the room is asymmetrical, the wall materials are diverse, etc.) and the difference between the real acoustic characteristics of different positions is relatively large, the second position can be multiple, thereby increasing the accuracy of the generated conversion information. It should be understood that the number of second positions can be determined according to demand, and this application does not impose any restrictions on this.

[0181] In one possible approach, when the terminal device is an AR / VR device, the AR / VR device may display measurement instructions on its display screen, where the measurement instructions may include a second location. This allows the user to wear the AR / VR device and move according to the second location displayed on the AR / VR device's display screen. As the user moves, the AR / VR device may locate the user in real time to obtain location information and provide movement prompts based on the obtained location information. When the AR / VR device determines, based on the obtained location information, that the user has moved to the second location, it may prompt the user to stop moving. After the AR / VR device receives a test audio signal at a second location and processes it to obtain a fourth acoustic feature, it may play a test success message. Furthermore, if there are multiple second locations, the AR / VR device may prompt the user to move to the next second location until the fourth acoustic features are obtained for all second locations in the real scene. If the AR / VR device does not receive the test audio signal or obtain the fourth acoustic feature at a second location, it may play a test failure message and prompt the user to retest, i.e., prompt the user to remain at the current second location to receive the test audio signal. In this case, the AR / VR device can record the position information of one or more second locations to facilitate subsequent determination of the third acoustic characteristics of the one or more second locations in the virtual scene.

[0182] In one possible approach, when the terminal device is an AR / VR device, the AR / VR device may display measurement instructions on its display screen, where the measurement instructions may include the number of second locations. This allows the user to first select a corresponding number of second locations on the AR / VR device based on the number of second locations displayed on the display screen. The user can then wear the AR / VR device and move to the selected second locations. As the user moves, the AR / VR device can locate the user in real time and obtain location information, providing movement prompts based on the obtained location information. When the AR / VR device determines, based on the obtained location information, that the user has moved to a second location, it can prompt the user to stop moving. After the AR / VR device receives a test audio signal at a second location and processes it to obtain a fourth acoustic feature, it can play a test success message. If there are multiple second locations, the AR / VR device can prompt the user to move to the next second location until the AR / VR device obtains the fourth acoustic features at all second locations in the real scene. If the AR / VR device does not receive a test audio signal or obtain the fourth acoustic feature at a second location, it can play a test failure message and prompt the user to retest, i.e., to remain at the current second location and receive the test audio signal. In this case, the AR / VR device can record the position information of one or more second locations to facilitate subsequent determination of the third acoustic characteristics of the one or more second locations in the virtual scene.

[0183] In one possible approach, when the terminal device is a tablet, laptop, or mobile phone with low positioning accuracy, a virtual scene can be displayed on the terminal device, and a second location can be identified in the virtual scene. In this way, the user carries the terminal device, moves to the second location identified by the terminal device in the virtual scene, and then stops. A recording operation is then performed on the terminal device, at which point the terminal device can receive a test audio signal at the second location. After the terminal device receives the test audio signal at the second location and processes it to obtain the fourth acoustic characteristic, a test success message can be played. Furthermore, if there are multiple second locations, the terminal device can prompt the user to move to the next second location until the terminal device obtains the fourth acoustic characteristics at all second locations in the real scene. If the terminal device does not receive the test audio signal or obtain the fourth acoustic characteristic at the second location, a test failure message can be played and the user can be prompted to test again, i.e., the user can remain at the current second location to receive the test audio signal. In this case, the user can enter location information of one or more second locations (e.g., distance from a wall, direction of an object relative to the user, user height, etc.) into the terminal device to facilitate subsequent determination of the third acoustic characteristics of one or more second locations in the virtual scene.

[0184] It should be understood that the fourth acoustic characteristic may include, but is not limited to: a fourth acoustic response, fourth energy information, a fourth acoustic parameter or a fourth acoustic characteristic.

[0185] In addition, the position information of the sound source device in the real scene can also be recorded to facilitate the subsequent determination of the third acoustic characteristics of one or more second positions in the virtual scene.

[0186] In one possible approach, the fourth acoustic feature is user-defined input.

[0187] Since the acoustic features obtained through actual measurements in certain real scenarios may not be the most accurate match for the user's actual hearing experience, directly inputting the acoustic features that match the user's actual hearing experience in certain real scenarios makes the adjusted acoustic features closer to the acoustic features that match the user's actual hearing experience in certain real scenarios. In this way, the spatial audio signal subsequently generated based on the adjusted acoustic features is closer to the spatial audio signal heard by the user in the real scene corresponding to the certain virtual scene.

[0188] For example, “certain real-world scenarios” may include:

[0189] 1. Real-world scenarios with unavoidable interfering noise. For example, there may be unavoidable interfering noise in the real scene or nearby (e.g., construction in the next room). The fourth acoustic feature to be customized by the user can be determined based on the design parameters of the real scene, historical relevant data, etc.

[0190] 2. Real-world scenarios where the visual presentation of a surface material does not match its actual acoustic effect. In such scenarios, if the fourth acoustic feature is derived from the test audio signal received at the second location in the real scene, the subsequent generation and playback of the spatial audio signal may result in a discrepancy between the user's auditory experience and their visual perception.

[0191] 3. A real scene used as a multifunctional space: that is, a real scene used as a multifunctional space; for example, a real scene can serve as a lecture hall, a concert hall, and a film and television viewing room. Because users have different listening requirements for spatial audio signals in different virtual scenes, the virtual acoustic characteristics of different virtual scenes vary. If the real acoustic characteristics are measured in the same real scene for different virtual scenes, the subsequent generation and playback of spatial audio signals may result in a discrepancy between the user's auditory and visual perception.

[0192] It should be noted that in one possible approach, a sound source device can be deployed at one location (hereinafter referred to as a preset sound source location); in this way, one fourth acoustic feature can be obtained corresponding to one second location. In another possible approach, a sound source device can be deployed at multiple preset sound source locations; in this way, multiple fourth acoustic features can be obtained corresponding to one second location.

[0193] S303: Acquire third acoustic features of one or more second positions in the virtual scene.

[0194] Accordingly, acoustic feature extraction can be performed based on the location information of the sound source device recorded in S302, the location information of the second location recorded in S302, the test audio signal played by the sound source device in S302, and the virtual scene to determine one or more third acoustic features for the second location. Similarly, one or more third acoustic features can be obtained for each second location. It should be understood that the third acoustic feature is a virtual acoustic feature.

[0195] It should be understood that the third acoustic feature may include, but is not limited to: a third acoustic response, third energy information, a third acoustic parameter or a third acoustic feature.

[0196] In this way, after executing S302 and S303, one or more acoustic feature groups can be obtained for a second position. An acoustic feature group includes a third acoustic feature and a fourth acoustic feature. The third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group have the same corresponding second position and the same corresponding preset sound source position.

[0197] S304: Analyze the third acoustic feature and the fourth acoustic feature in the one or more acoustic feature groups to obtain conversion information.

[0198] Illustratively, the present application may use signal processing, machine learning, numerical analysis, etc. to analyze and compare the third acoustic feature and the fourth acoustic feature in one or more acoustic feature groups to obtain conversion information.

[0199] The following describes a method for generating conversion information using an acoustic feature group as an example.

[0200] In one possible approach, the third and fourth acoustic features in an acoustic feature group can be signal processed separately to obtain signal processing results. Subsequently, the signal processing results of the third and fourth acoustic features can be analyzed to obtain a transfer function. In this case, the transfer function can be used as the conversion information. For example, for a third acoustic response (e.g., RIR) included in the third acoustic feature and a fourth acoustic response (e.g., RIR) included in the fourth acoustic feature, where the RIR in the third acoustic response is represented by RIR1 and the RIR in the fourth acoustic response is represented by RIR2, frequency domain conversion can be performed on RIR1 and RIR2, respectively, to obtain the frequency domain response of RIR1 (referred to as frequency response 1) and the frequency domain response of RIR2 (referred to as frequency response 2). Subsequently, in one possible approach, a frequency response transfer function can be calculated using frequency response 1 and frequency response 2, and the frequency response transfer function can be used as the transfer function. In another possible approach, signal analysis can be performed on frequency response 1 and frequency response 2, respectively, and the transfer function can be determined based on the signal analysis results.

[0201] In one possible way, the third acoustic feature and the fourth acoustic feature in an acoustic feature group can be numerically analyzed respectively to obtain the feature change rate; in this case, the feature change rate can be used as conversion information. Specifically, the change rate of the third acoustic feature in the acoustic feature group relative to the fourth acoustic feature in the acoustic feature group can be calculated as the feature change rate. For example, for the third acoustic parameter such as the direct mixing ratio contained in the third acoustic feature, and the fourth acoustic parameter such as the direct mixing ratio contained in the fourth acoustic feature, wherein the direct mixing ratio in the third acoustic parameter is represented by DRR 1, and the direct mixing ratio in the fourth acoustic parameter is represented by DRR 2; the corresponding feature change rate can be (DRR 1-DRR 2) / DRR 2, or, DRR1 / DRR2. It should be understood that the conversion information may include multiple feature change rates (multiple feature change rates correspond one-to-one to multiple acoustic parameters), and this application does not limit this.

[0202] In one possible approach, a machine learning approach can be used to process the third and fourth acoustic features in an acoustic feature group to obtain conversion information. Specifically, the third and fourth acoustic features in an acoustic feature group can be input into an AI model (or a machine learning model, hereinafter referred to as a second model), and the second model processes the third and fourth acoustic features in an acoustic feature group and outputs conversion information; that is, the model output information of the second model is the conversion information.

[0203] It should be understood that other methods may be used to generate conversion information, and this application does not limit this.

[0204] S305: Obtain a source audio signal, a sound source position corresponding to the source audio signal in the virtual scene, and a first position corresponding to the user in the virtual scene.

[0205] For example, S305 may refer to the description of S201 above, which will not be repeated here.

[0206] S306: Extract acoustic features based on the first position, the sound source position of the source audio signal, and the virtual scene to obtain a second acoustic feature.

[0207] Exemplarily, the second acoustic feature may be understood as a virtual acoustic feature at a first position when a virtual sound source located at a sound source position in a virtual scene plays a source audio signal.

[0208] S307: Adjust the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0209] Exemplarily, the second acoustic feature can be adjusted according to the conversion information to obtain the first acoustic feature; in this way, when the virtual sound source located at the sound source position in the virtual scene plays the source audio signal, the virtual acoustic feature at the first position can be adjusted to be close to or consistent with the real acoustic feature at the first position when the real sound source located at the sound source position in the real scene plays the source audio signal.

[0210] For example, different adjustment methods can be used for different types of second acoustic features. For example, when the second acoustic feature is a second acoustic response, a frequency response curve adjustment algorithm based on signal processing can be used. Specifically, the second acoustic response can be transformed into the frequency domain to obtain a frequency domain response of the second acoustic response (hereinafter referred to as frequency response 3). Then, frequency response 3 can be adjusted according to the above-determined conversion function to obtain an adjusted frequency response 3. Thereafter, the adjusted frequency response 3 is transformed into the first acoustic response, i.e., the first acoustic feature.

[0211] For another example, when the second acoustic feature includes a second acoustic response and a second acoustic parameter, a reverberation adjustment algorithm based on signal processing can be used. Specifically, for the second DRR in the second acoustic parameter, the second DRR can be adjusted according to the change rate of the direct mixing ratio calculated above to obtain the first DRR (that is, the first acoustic parameter included in the first acoustic feature); thereafter, the second acoustic response can be adjusted according to the first DRR to obtain the first acoustic response. It should be understood that for other second acoustic parameters, adjustments can also be made according to the corresponding characteristic change rates, and this application does not impose any restrictions on this.

[0212] For another example, the adjustment algorithm is an adjustment algorithm based on an AI model. Specifically, the second acoustic feature and the conversion information can be input into the AI ​​model (hereinafter referred to as the third model), and the third model adjusts the second acoustic feature according to the conversion information and outputs the first acoustic feature.

[0213] It should be understood that other adjustment algorithms may also be used to adjust the second acoustic feature to obtain the first acoustic feature, and this application does not impose any limitation on this.

[0214] It should be noted that the conversion information obtained in S304 can be used to adjust the second acoustic characteristics of any first location in the virtual environment to obtain the first acoustic characteristics. In other words, S304 only needs to analyze the third and fourth acoustic characteristics of one or more second locations. The resulting conversion information is universally applicable to any location in the same room (i.e., virtual scene), and is not limited to adjusting the acoustic characteristics of first locations close to the second location.

[0215] S308: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0216] Exemplarily, when the first acoustic feature is a first acoustic response, the first acoustic response may be convolved with the source audio signal to obtain a spatial audio signal.

[0217] Exemplarily, when the first acoustic feature includes a first acoustic response and first energy information, the energy of the spatial audio signal may be adjusted according to the first energy information; then, the adjusted first acoustic response is convolved with the source audio to obtain the spatial audio signal.

[0218] Exemplarily, when the first acoustic feature includes a first acoustic parameter and a first acoustic feature, a first acoustic response may be generated according to the first acoustic parameter and the first acoustic feature; then, the first acoustic response may be convolved with the source audio to obtain a spatial audio signal.

[0219] Exemplarily, when the first acoustic feature includes a first acoustic response, a first acoustic parameter and a first acoustic feature, the first acoustic response can be first optimized according to the first acoustic parameter and the first acoustic feature, and then the optimized first acoustic response can be convolved with the source audio to obtain a spatial audio signal.

[0220] It should be understood that there are many ways to generate a spatial audio signal according to the first acoustic feature, and this application does not limit this.

[0221] In one possible approach, preset conversion information corresponding to a plurality of preset virtual scenes can be pre-generated and stored in a database, as well as second preset acoustic features for a plurality of positions in a plurality of preset virtual scenes can be pre-generated and stored in a database; in this way, during the application process, the conversion information corresponding to the current virtual scene can be read from the database, and the first position in the current virtual scene can be judged in real time during the application process, thereby reading the second acoustic feature that matches the first position and the sound source position in the current virtual scene from the database, and adjusting the second acoustic feature in real time to obtain the first acoustic feature that matches the first position and the sound source position in the current virtual scene. By using the database, the time for generating spatial audio signals can be shortened, and the user experience can be improved. In addition, virtual scenes that may be used multiple times at different times, by different users, and in different applications can also be called in a more efficient manner.

[0222] Figure 4 is a schematic diagram illustrating an exemplary audio processing process. In the embodiment of Figure 4 , the user's current real scene is not restricted to whether the virtual scene corresponding to the audio project (or AR / VR project) selected (or experienced) by the user is the same scene. In other words, the application scenario of the embodiment of Figure 4 can be the aforementioned Figures 1A to 1D.

[0223] S401: Acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to a user in the virtual scene.

[0224] S402: Select a second acoustic feature set corresponding to the virtual scene from a plurality of second acoustic feature sets stored in a database.

[0225] S403 : Select a second acoustic feature from a second acoustic feature set corresponding to the virtual scene according to the sound source position of the source audio signal and the first position.

[0226] Exemplarily, a plurality of second acoustic feature sets may be generated in advance for a plurality of preset virtual scenes; wherein the plurality of second acoustic feature sets correspond one-to-one to the plurality of preset virtual scenes.

[0227] Specifically, for a preset virtual scene, multiple positions (hereinafter referred to as fourth positions) can be identified in advance in the preset virtual scene; for example, the preset virtual scene is divided into a 10*10 grid, and the intersection of the grid lines or the center point of the grid can be used as the fourth position. Then, for each fourth position, acoustic feature extraction can be performed based on the fourth position, the preset sound source position in the preset virtual scene and the preset virtual scene to obtain the second preset acoustic feature of the fourth position. Among them, there can be multiple preset sound source positions in the preset virtual scene. In this way, multiple second preset acoustic features can be obtained for one fourth position. In this way, the second preset acoustic features of multiple fourth positions can be obtained; the second preset acoustic features of these multiple fourth positions can constitute a second acoustic feature set corresponding to the preset virtual scene. Afterwards, the second acoustic feature set can be stored in a database, and a correspondence between the second acoustic feature set and the preset virtual scene can be established.

[0228] According to the above method, multiple second acoustic feature sets can be generated; and the multiple second acoustic feature sets can be stored in a database, and a correspondence between the multiple second acoustic feature sets and the corresponding preset virtual scenes can be established.

[0229] Exemplarily, a second acoustic feature set corresponding to the current virtual scene can be first filtered out from the database; then, based on the sound source position and the fourth position of the source audio signal, a second preset acoustic feature whose preset sound source position is the same as or closest to the sound source position, and whose fourth position is the same as or closest to the first position, is selected from the filtered second acoustic feature set as the second acoustic feature.

[0230] S404 , selecting conversion information from a plurality of preset conversion information stored in a database according to the virtual scene; wherein the plurality of preset conversion information corresponds one-to-one to a plurality of preset virtual scenes.

[0231] For example, for one of the multiple preset virtual scenes, preset conversion information can be generated and stored in a database in advance according to the descriptions in S302 to S304 above, and a correspondence between the preset conversion information and the preset virtual scene can be established. During S404, preset conversion information corresponding to the preset virtual scene that is identical to the current virtual scene can be selected from the database as the conversion information corresponding to the current virtual scene.

[0232] S405: Adjust the second acoustic feature according to the conversion information to obtain the first acoustic feature.

[0233] S406: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0234] S405 to S406 may refer to the description of S307 to S308 above, and will not be repeated here.

[0235] One possible approach is to pre-generate first acoustic features for multiple locations in various preset virtual scenes and store them in a database. This allows the user to retrieve the first acoustic features that match the first location and sound source position in the current virtual scene from the database during application, further reducing the time required to generate spatial audio signals and improving the user experience. Furthermore, this allows for a more efficient recall of virtual scenes that may be used multiple times at different times, by different users, and in different applications.

[0236] Figure 5 is a schematic diagram illustrating an exemplary audio processing process. In the embodiment of Figure 5 , the user's current real scene is not restricted to whether the virtual scene corresponding to the audio project (or AR / VR project) selected (or experienced) by the user is the same scene. In other words, the application scenario of the embodiment of Figure 5 can be the aforementioned Figures 1A to 1D.

[0237] S501: Obtain a source audio signal, a sound source position of the source audio signal in a virtual scene, and a first position corresponding to a user in the virtual scene.

[0238] S502: Select a first acoustic feature set corresponding to a virtual scene from a plurality of first acoustic feature sets stored in a database.

[0239] S503 : Select a first acoustic feature from a first acoustic feature set corresponding to the virtual scene according to the sound source position and the first position of the source audio signal.

[0240] Exemplarily, for one of the multiple preset virtual scenes, corresponding conversion information is generated in advance according to the embodiment of FIG. 3 , and a corresponding second acoustic feature set is generated according to the embodiment of FIG. 4 ; thereafter, the second preset acoustic features of multiple positions (hereinafter referred to as third positions) in the second acoustic feature set corresponding to the preset virtual scene can be adjusted according to the conversion information corresponding to the preset virtual scene to obtain the first preset acoustic features of multiple third positions; the first preset acoustic features of these multiple third positions can form a first acoustic feature set, that is, a first acoustic feature set corresponding to the preset virtual scene. Then, the first acoustic feature set corresponding to the preset virtual scene can be stored in a database, and a corresponding relationship between the first acoustic feature set and the preset virtual scene can be established.

[0241] It should be understood that there may be multiple preset sound source positions in the preset virtual scene. Thus, for one third position, multiple second preset acoustic features may be obtained. Correspondingly, for one third position, multiple first preset acoustic features may be obtained, each first preset acoustic feature corresponding to one preset acoustic position.

[0242] Exemplarily, a first acoustic feature set corresponding to the current virtual scene can be first filtered out from the database; then, from the filtered out first acoustic feature set, a first preset acoustic feature whose preset sound source position is the same as or closest to the sound source position, and whose third position is the same as or closest to the first position, is selected as the first acoustic feature.

[0243] S504: Generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0244] S504 can refer to the description of S308 above, which will not be repeated here.

[0245] Figures 6A and 6B are schematic diagrams of frequency response curves of acoustic features as examples. Figures 6A and 6B show the frequency response curves of the real acoustic feature (such as curve 1), the frequency response curves of the adjusted virtual acoustic feature (i.e., the first acoustic feature) (i.e., curve 2), and the frequency response curves of the unadjusted virtual acoustic feature (i.e., the second acoustic feature) (i.e., curve 3) when the user is at different first positions in the virtual scene. By comparing curves 1, 2, and 3, it can be seen that the frequency response curve of the adjusted virtual acoustic feature is closer to the frequency response curve of the real acoustic feature; that is, by adopting the method provided in this application, the virtual acoustic feature can be made closer to the real acoustic feature, thereby improving the quality of the spatial audio signal.

[0246] FIG. 7 is a schematic diagram illustrating an exemplary audio processing device.

[0247] A first acquisition module 701 is configured to acquire a source audio signal, a sound source position of the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene;

[0248] A second acquisition module 702 is configured to acquire a first acoustic feature based on the sound source position and the first position, where the first acoustic feature is obtained by adjusting the second acoustic feature based on the conversion information, and the second acoustic feature is obtained by extracting acoustic features based on the first position, the virtual scene, and the sound source position, where the conversion information is used to describe a conversion relationship between acoustic features at the same position in the virtual scene and the real scene;

[0249] The audio signal generating module 703 is configured to generate a spatial audio signal according to the first acoustic feature and the source audio signal.

[0250] Exemplarily, the conversion information is determined by analyzing an acoustic feature group, where the acoustic feature group includes a third acoustic feature and a fourth acoustic feature, where the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene.

[0251] Exemplarily, there are one or more acoustic feature groups, there are one or more second positions, and the third acoustic feature and the fourth acoustic feature belonging to the same acoustic feature group correspond to the same second position.

[0252] Exemplarily, the conversion information includes a conversion function, which is obtained by analyzing signal processing results of the third acoustic feature and the fourth acoustic feature.

[0253] Exemplarily, the conversion information includes a feature change rate, where the feature change rate is a change rate of the third acoustic feature relative to the fourth acoustic feature.

[0254] Exemplarily, the conversion information includes model output information obtained by inputting the third acoustic feature and the fourth acoustic feature into the model.

[0255] Exemplarily, the first acoustic feature includes at least one of the following: acoustic response, energy information, acoustic parameter or acoustic feature.

[0256] Exemplarily, the fourth acoustic feature is obtained by processing a test audio signal received at a second position in the real scene.

[0257] Exemplarily, the fourth acoustic feature is user-defined input.

[0258] Exemplarily, the second acquisition module 702 is specifically used to determine one or more acoustic feature groups, where one acoustic feature group includes a third acoustic feature and a fourth acoustic feature, where the fourth acoustic feature is the acoustic feature of the second position in the real scene, and the third acoustic feature is the acoustic feature of the second position in the virtual scene; analyze the third acoustic feature and the fourth acoustic feature in the one or more acoustic feature groups to obtain conversion information; extract acoustic features according to the first position, the virtual scene, and the sound source position to obtain a second acoustic feature; and adjust the second acoustic feature according to the conversion information to obtain a first acoustic feature.

[0259] Exemplarily, the second acquisition module 702 is specifically used to select a second acoustic feature set corresponding to the virtual scene from multiple second acoustic feature sets stored in the database; wherein, the multiple second acoustic feature sets correspond one-to-one to multiple preset virtual scenes, a second acoustic feature set includes multiple second preset acoustic features, a second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple fourth positions and multiple preset sound source positions, a fourth position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set; according to the sound source position and the first position, the second acoustic feature is selected from the second acoustic feature set corresponding to the virtual scene; according to the virtual scene, conversion information is selected from multiple preset conversion information stored in the database, and the multiple preset conversion information corresponds one-to-one to multiple preset virtual scenes; the second acoustic feature is adjusted according to the conversion information to obtain the first acoustic feature.

[0260] Exemplarily, the second acquisition module 702 is specifically configured to acquire the first acoustic feature, including:

[0261] A first acoustic feature set corresponding to a virtual scene is selected from multiple first acoustic feature sets stored in a database; wherein, multiple first acoustic feature sets correspond one-to-one to multiple preset virtual scenes, and multiple first acoustic feature sets correspond one-to-one to multiple second acoustic feature sets. A second acoustic feature set is obtained by extracting acoustic features based on a preset virtual scene, multiple third positions, and multiple preset sound source positions. A third position and a preset sound source position are used to determine a second preset acoustic feature in a second acoustic feature set. A first preset acoustic feature in a first acoustic feature set is obtained by adjusting a second preset acoustic feature in the corresponding second acoustic feature set according to conversion information. A first acoustic feature is selected from the first acoustic feature set corresponding to the virtual scene based on the sound source position and the first position.

[0262] An embodiment of the present application also provides an AR device, which includes a display module, an image acquisition module, headphones and a processor, wherein: the processor is used to execute the audio processing method as in the above embodiment; the headphones are used to play the spatial audio signal generated by the audio processing method in the above embodiment.

[0263] An embodiment of the present application also provides a VR device, which includes a display module, an image acquisition module, headphones and a processor, wherein: the processor is used to execute the audio processing method in the above embodiment; and the headphones are used to play the spatial audio signal generated by the audio processing method in the above embodiment.

[0264] In an example, FIG8 shows a schematic block diagram of an apparatus 800 according to an embodiment of the present application. The apparatus 800 may include: a processor 801 and a transceiver / transceiver pin 802 , and optionally, a memory 803 .

[0265] The various components of the device 800 are coupled together via a bus 804, wherein the bus 804 includes, in addition to a data bus, a power bus, a control bus, and a status signal bus. However, for the sake of clarity, all buses are referred to as bus 804 in the figure.

[0266] Optionally, the memory 803 may be used to store instructions in the aforementioned method embodiment. The processor 801 may be used to execute the instructions in the memory 803 and control the receiving pin to receive a signal and control the transmitting pin to send a signal.

[0267] The apparatus 800 may be the electronic device or a chip of the electronic device in the above method embodiment.

[0268] Among them, all relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0269] The present application also provides a chip including one or more interface circuits and one or more processors. The one or more processors receive or send data via the one or more interface circuits. When the one or more processors execute computer instructions, the steps of the above-mentioned related methods are implemented. The interface circuit is a transceiver / transceiver pin 802.

[0270] This embodiment further provides a computer-readable storage medium, in which computer instructions are stored. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the method in the above-mentioned embodiment.

[0271] This embodiment further provides a computer program product, which includes computer instructions. When the computer instructions are executed by a computer or a processor, the computer executes the above-mentioned related steps to implement the method in the above-mentioned embodiment.

[0272] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the chip to execute the methods in the above-mentioned method embodiments.

[0273] Among them, the electronic device, computer-readable storage medium, computer program product or chip provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0274] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0275] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of modules or units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0276] Units described as separate components may or may not be physically separate, and components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0277] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0278] Any content of each embodiment of this application, as well as any content of the same embodiment, can be freely combined. Any combination of the above content is within the scope of this application.

[0279] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0280] The steps of the method or algorithm described in conjunction with the disclosure of the embodiments of the present application can be implemented in a hardware manner, or can be implemented by a processor executing a software instruction. The software instruction can be composed of corresponding software modules, and the software module can be stored in a random access memory (Random Access Memory, RAM), a flash memory, a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (Erasable Programmable ROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM), a register, a hard disk, a mobile hard disk, a read-only compact disc (CD-ROM) or any other form of storage medium well known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and can write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0281] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer-readable storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0282] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. An audio processing method, characterized in that: The method comprises: Acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by constructing a real scene; Acquiring a first acoustic feature, where the first acoustic feature is obtained by converting an acoustic feature of a first position in the virtual scene when the source audio signal is at the sound source position according to an acoustic feature of a second position in the real scene and an acoustic feature of a second position in the virtual scene; A spatial audio signal is generated according to the first acoustic feature and the source audio signal.

2. The method according to claim 1, characterized in that: The first acoustic feature includes at least one of the following: acoustic response, energy information, acoustic parameter or acoustic feature.

3. The method according to claim 1 or 2, characterized in that: The second position is one or more.

4. The method according to any one of claims 1 to 3, characterized in that: The first positions corresponding to the users in the virtual scene at different times are the same or different.

5. The method according to any one of claims 1 to 4, characterized in that: The second position is the same as or different from the first position.

6. The method according to any one of claims 1 to 5, characterized in that: There is a difference between a spatial audio signal generated according to the first acoustic feature and the source audio signal and a spatial audio signal generated according to an acoustic feature of a first position in the virtual scene and the source audio signal.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquire a user-defined acoustic feature of a second position in the real scene.

8. The method according to any one of claims 1 to 6, characterized in that: Before acquiring the source audio signal, the sound source position corresponding to the source audio signal in the virtual scene, and the first position corresponding to the user in the virtual scene, the method further includes: Get the user operations performed by the user in the user interface; Based on the user operation, an acoustic feature of a second position in the real scene is generated.

9. The method according to any one of claims 1 to 6, characterized in that: Before acquiring the source audio signal, the sound source position corresponding to the source audio signal in the virtual scene, and the first position corresponding to the user in the virtual scene, the method further includes: Get the user's location information; When it is determined according to the position information that the user moves to a second position, an acoustic feature of the second position in the real scene is generated.

10. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Acquiring an acoustic feature of a second position in the real scene according to the sound source position, the first position, and the second position.

11. An audio processing method, characterized in that: The method comprises: Acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by modeling a real scene; Acquiring a first acoustic feature according to the sound source position and the first position, wherein the first acoustic feature is obtained by adjusting a second acoustic feature according to conversion information, and the second acoustic feature is obtained by extracting acoustic features according to the first position, the virtual scene, and the sound source position, and the conversion information is used to describe a conversion relationship between acoustic features at the same position in the virtual scene and the real scene; A spatial audio signal is generated according to the first acoustic feature and the source audio signal.

12. An audio processing device, characterized in that: The device comprises: A first acquisition module, used to acquire a source audio signal, a sound source position corresponding to the source audio signal in a virtual scene, and a first position corresponding to the user in the virtual scene, wherein the virtual scene is obtained by constructing a real scene; a second acquisition module, configured to acquire a first acoustic feature, wherein the first acoustic feature is obtained by converting the acoustic feature of the first position in the virtual scene when the source audio signal is at the sound source position according to the acoustic feature of the second position in the real scene and the acoustic feature of the second position in the virtual scene; The audio signal generating module is configured to generate a spatial audio signal according to the first acoustic feature and the source audio signal.

13. An augmented reality (AR) device, characterized in that: The AR device includes a display module, an image acquisition module, a headset and a processor, wherein: The processor is used to execute the audio processing method as described in any one of claims 1 to 11 above; The earphone is used to play the spatial audio signal generated by the audio processing method according to any one of claims 1 to 11.

14. A virtual reality VR device, characterized in that: The VR device includes a display module, an image acquisition module, headphones and a processor, wherein: The processor is used to execute the audio processing method as described in any one of claims 1 to 11 above; The earphone is used to play the spatial audio signal generated by the audio processing method according to any one of claims 1 to 11.

15. An electronic device, characterized in that: include: a memory and a processor, the memory being coupled to the processor; The memory stores program instructions, and when the program instructions are executed by the processor, the electronic device executes the audio processing method according to any one of claims 1 to 11.

16. A chip, characterized in that: It comprises one or more interface circuits and one or more processors; the one or more processors receive or send data through the one or more interface circuits, and when the one or more processors execute computer instructions, the steps of the method as claimed in any one of claims 1 to 11 are executed.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program runs on a computer or a processor, the computer or the processor executes the audio processing method according to any one of claims 1 to 11.

18. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a computer or a processor, the steps of the method according to any one of claims 1 to 11 are performed.

Images