Spatial audio synthesis method, electronic device and computer-readable storage medium

Abstract

The embodiment of the present application discloses a spatial audio synthesis method, an electronic device and a computer-readable storage medium, wherein the method includes: determining a first relative position set based on a first position of a target object in a target space and an absolute position of each microphone in the target space, the first relative position set including the relative position of each microphone relative to the first position; determining first head movement information based on the head movement information of the target object in the first position; synthesizing spatial audio of the first position based on the first relative position set, the first head movement information and the first audio set; the first audio set including the ambient audio collected by each microphone when the target object is in the first position. Based on the method described in the present application, the efficiency of synthesizing spatial audio can be effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio processing technology, and in particular to a spatial audio synthesis method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Spatial audio is a type of audio that goes far beyond stereo and surround sound, and can enhance the user's sense of immersion and presence. Currently, there are too few spatial audios. When replaying existing non-spatial audio, spatial audio can be obtained by spatially expanding the non-spatial audio through spatial audio expansion technology. This spatial audio can replicate the spatial sound field and simulate the user's sound experience in the spatial sound field. However, the efficiency of obtaining spatial audio through spatial audio expansion technology is low. Summary of the invention

[0003] The present application provides a spatial audio synthesis method, an electronic device, and a computer-readable storage medium, which can effectively improve the efficiency of synthesized spatial audio.

[0004] In a first aspect, an embodiment of the present application provides a spatial audio synthesis method, the method comprising: determining a first relative position set based on a first position of a target object in a target space and the absolute positions of each microphone in the target space, the first relative position set including the relative positions of each microphone relative to the first position; determining first head movement information based on head movement information of the target object in the first position; synthesizing spatial audio at the first position based on the first relative position set, the first head movement information and a first audio set; the first audio set including ambient audio collected by each microphone when the target object is in the first position.

[0005] In the above embodiment, since the ambient audio collected by each microphone can reflect the spatial sound field at each microphone position, the spatial audio at the first position obtained based on the ambient audio collected by each microphone, the first relative position set and the first head movement information can reproduce the spatial sound field that the target object can feel when it is at the first position. This method of obtaining spatial audio using the collected ambient audio can effectively improve the efficiency of synthesizing spatial audio.

[0006] In a possible implementation, the method further includes: receiving a position setting request for the target object; and determining the first position in response to the position setting request.

[0007] Since the position setting request can be set by the user according to his own needs, the first position determined based on the position setting request can be flexibly adapted to the user's own needs, and the spatial audio of the first position obtained can provide the user with a more immersive and realistic sound field experience.

[0008] In a possible implementation, the method further includes: acquiring first sensor information of the target object; the first sensor information includes movement information of the target object; and determining the first position based on the first sensor information.

[0009] Since the first sensor information includes movement information, and the movement information is truly perceived movement information, a more accurate first position can be determined based on the first sensor information, and the obtained spatial audio of the first position can replicate a more accurate spatial sound field of the first position.

[0010] In a possible implementation, the method further includes: receiving a head movement setting request for the target object; and determining head movement information of the target object in the first position in response to the head movement setting request.

[0011] Since the head movement setting request can be set by the user according to his or her own needs, the head movement information of the first position determined based on the head movement setting request can be flexibly adapted to the user's own needs, and the spatial audio of the first position obtained can provide the user with a more immersive and realistic sound field experience.

[0012] In a possible implementation, the method further includes: acquiring second sensor information of the target object; the second sensor information includes head movement information of the target object in the first position.

[0013] Since the second sensor information includes the head movement information at the first position, and the head movement information is the real perceived head movement information, more accurate head movement information of the first position can be obtained, and the obtained spatial audio of the first position can reproduce the spatial sound field of the first position more accurately.

[0014] In a possible implementation, the spatial audio of the first position is synthesized based on the first relative position set, the first head movement information and the first audio set, specifically including: obtaining the head-related transfer function corresponding to each microphone from a database based on the first head movement information and the first relative position set; obtaining the mapped ambient audio of each microphone based on the head-related transfer function of each microphone and the ambient audio collected by each microphone; and obtaining the spatial audio of the first position based on the mapped ambient audio of each microphone.

[0015] In a possible implementation, the method further includes at least one of the following: transmitting the spatial audio at the first position; playing the spatial audio at the first position; or saving the spatial audio at the first position.

[0016] By transmitting / playing / saving the spatial audio of the first position, a post-playback or real-time playback of the spatial sound field of the first position can be achieved.

[0017] In a possible embodiment, the method also includes: in response to the target object moving from a first position to a second position in the target space, determining a second relative position set based on the second position and the absolute positions of each microphone, the second relative position set including the relative positions of each microphone relative to the second position; determining second head movement information based on the head movement information of the target object at the second position; synthesizing spatial audio at the second position based on the second relative position set, the second head movement information and the second audio set; the second audio set including the ambient audio collected by each microphone when the target object is at the second position.

[0018] The spatial audio of the second position is synthesized in response to the position movement of the target object, and the spatial audio of the second position can replicate the spatial sound field of the second position, so that the obtained spatial sound field can be updated according to the moved position.

[0019] In a second aspect, an embodiment of the present application provides an electronic device, comprising a memory and one or more processors; the memory is coupled to the one or more processors and is used to store a computer program, the computer program comprising program instructions; the one or more processors call the program instructions so that the electronic device executes: based on a first position of a target object in a target space and an absolute position of each microphone in the target space, determining a first relative position set, the first relative position set including the relative positions of each microphone relative to the first position; based on head movement information of the target object at the first position, determining first head movement information; based on the first relative position set, the first head movement information and the first audio set, synthesizing spatial audio at the first position; the first audio set including ambient audio collected by each microphone when the target object is at the first position.

[0020] In a possible implementation, the one or more processors further call the program instructions to cause the electronic device to execute: receiving a location setting request for a target object; and determining a first location in response to the location setting request.

[0021] In a possible implementation, the one or more processors also call the program instructions to cause the electronic device to execute: obtaining first sensor information of the target object; the first sensor information includes movement information of the target object; and determining a first position based on the first sensor information.

[0022] In one possible implementation, the one or more processors also call the program instructions so that the electronic device executes: receiving a head movement setting request for a target object; and determining head movement information of the target object in a first position in response to the head movement setting request.

[0023] In a possible implementation, the one or more processors further call the program instructions to cause the electronic device to execute: acquiring second sensor information of the target object; the second sensor information includes head movement information of the target object in the first position.

[0024] In a possible implementation, when the one or more processors call the program instructions so that the electronic device executes synthesis of the spatial audio of the first position based on the first relative position set, the first head movement information and the first audio set, it specifically includes: obtaining the head-related transfer function corresponding to each microphone from a database based on the first head movement information and the first relative position set; obtaining the mapped ambient audio of each microphone based on the head-related transfer function of each microphone and the ambient audio collected by each microphone; and obtaining the spatial audio of the first position based on the mapped ambient audio of each microphone.

[0025] In a possible implementation, the one or more processors further call the program instructions so that the electronic device performs at least one of the following: transmitting the spatial audio at the first position; playing the spatial audio at the first position; or saving the spatial audio at the first position.

[0026] In a possible implementation, the one or more processors also call the program instructions so that the electronic device executes: in response to the target object moving from a first position to a second position in the target space, determining a second relative position set based on the second position and the absolute positions of each microphone, the second relative position set including the relative positions of each microphone relative to the second position; determining second head movement information based on the head movement information of the target object at the second position; synthesizing the spatial audio of the second position based on the second relative position set, the second head movement information and the second audio set; the second audio set including the ambient audio collected by each microphone when the target object is at the second position.

[0027] In a third aspect, an embodiment of the present application provides a chip system, which is applied to an electronic device, and the chip system includes one or more processors, which are used to call program instructions so that the electronic device executes the method described in the first aspect or any possible implementation method of the first aspect.

[0028] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and the computer program includes program instructions. When the program instructions are run on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation manner of the first aspect.

[0029] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, including a computer program, which includes program instructions. When the program instructions are run on an electronic device, the electronic device executes the method described in the first aspect or any possible implementation method of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a schematic diagram of a 6DOF provided in an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0032] Figure 3A is a schematic diagram of a home environment provided in an embodiment of the present application;

[0033] Figure 3B is a schematic diagram of an office environment provided in an embodiment of the present application;

[0034] Figure 4 is a flowchart of a spatial audio synthesis method provided in an embodiment of the present application;

[0035] Figure 5 It is a two-dimensional coordinate schematic diagram provided in an embodiment of the present application;

[0036] Figure 6 It is a schematic diagram of an interface of an application provided in an embodiment of the present application;

[0037] Figure 7 is a schematic diagram of an interface of another application provided in an embodiment of the present application;

[0038] Figure 8 is a schematic diagram of an interface of another application provided in an embodiment of the present application;

[0039] Fig. 9 is a flowchart of another spatial audio synthesis method provided in an embodiment of the present application;

[0040] Fig.10 It is a schematic diagram of the software structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the embodiments of the present application will be described clearly and in detail below in conjunction with the accompanying drawings. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in the text is only a description of the association relationship of associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0042] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0043] The term "user interface (UI)" in the following embodiments of the present application refers to a medium interface for interaction and information exchange between an application or operating system and a user, which realizes the conversion between the internal form of information and the form acceptable to the user. The user interface is a source code written in a specific computer language such as Java and extensible markup language (XML). The interface source code is parsed and rendered on an electronic device and finally presented as content that the user can recognize. The commonly used form of user interface is a graphical user interface (GUI), which refers to a user interface related to computer operations that is displayed in a graphical manner. It can be a visual interface element such as text, icons, buttons, menus, tabs, text boxes, dialog boxes, status bars, navigation bars, widgets, etc. displayed on the display screen of an electronic device.

[0044] The concepts and terms involved in this application are first explained below to facilitate understanding by those skilled in the art.

[0045] 1. Spatial Audio

[0046] For a sound source located in a certain space, the sound emitted by the sound source can be heard by the listener after propagation in the space and binaural reflection. The listener can feel the distance between the sound source and the listener and the direction of the sound source based on the characteristics of the sound heard, so as to judge the location of the sound source. In other words, the listener can perceive rich spatial information (i.e. spatial sound field) through the sound heard. This kind of sound containing spatial information is spatial audio.

[0047] Currently, spatial audio can be obtained by spatially expanding non-spatial audio through spatial audio expansion technology. For example, multiple speakers at different positions are used to play the same audio at the same time to simulate sounds from different directions. Alternatively, the sound emitted by the sound source is processed based on the position information of the sound source and the listener to obtain spatial audio. However, the efficiency of obtaining spatial audio based on spatial audio expansion technology is low.

[0048] 2. Factors affecting spatial audio

[0049] Spatial audio can be differentiated by the influence of ambient effect and binaural effect.

[0050] Among them, the environmental effect refers to the phenomenon that the sound will attenuate and distort as the distance it propagates from the sound source to the listener, and will experience reflection when encountering obstacles in the space, thus changing the characteristics of the sound. The binaural effect refers to the fact that the shapes and contours of the ears of different listeners are different, and the sound emitted by the sound source has different reflection paths when it is reflected by the ears, so the characteristics of the sound heard by different listeners are different.

[0051] 3. 6DOF six degrees of freedom (6DOF)

[0052] 6DOF may include head movement 3DOF and walking 3DOF.

[0053] See also Figure 1 , Figure 1 It is a schematic diagram of a 6DOF provided in an embodiment of the present application. Among them, taking the object as a person, the center point of the human body's trunk is point O, and three-dimensional coordinate axes are established with point O as the coordinate origin: x-axis, y-axis, and z-axis. Walking 3DOF refers to translational movement in the directions of these three coordinate axes. Head movement 3DOF refers to rotational movement around the three coordinate axes. The rotational movement around these three coordinate axes can be represented by the yaw angle Raw, pitch angle Pitch, and roll angle Roll in the figure. Among them, the yaw angle Raw represents the angle of rotation around the y-axis, the pitch angle Pitch represents the angle of rotation around the x-axis, and the roll angle Roll represents the angle of rotation around the z-axis.

[0054] Walking 3DOF may cause the absolute position of the object to change, and head 3DOF may cause the head movement information of the object to change.

[0055] In order to improve the efficiency of synthesizing spatial audio, the present application proposes a spatial audio synthesis method, an electronic device and a computer-readable storage medium. The electronic device may be a terminal device or server with data processing function, for example, the terminal device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0056] The hardware structures of electronic devices are described below with examples.

[0057] See also Figure 2 , Figure 2 1 is a schematic diagram of the hardware structure of an electronic device proposed in the present application. For example, the electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0058] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0059] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0060] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0061] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the provided memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0062] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0063] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0064] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicrOLED, Micro-OLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0065] The electronic device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0066] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the electronic device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0067] Video codecs are used to compress or decompress digital videos. The electronic device 100 may support one or more video codecs. Thus, the electronic device 100 may play or record videos in a variety of coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0068] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the electronic device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0069] The electronic device 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor, etc. For example, in the present application, the electronic device 100 can play spatial audio.

[0070] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0071] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The electronic device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0072] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the human ear.

[0073] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can make a sound by approaching the microphone 170C with his mouth to input the sound signal into the microphone 170C. The electronic device 100 can be provided with at least one microphone 170C. In other embodiments, the electronic device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device 100 can also be provided with three, four or more microphones 170C to realize the collection of sound signals, noise reduction, identification of sound sources, and directional recording functions, etc.

[0074] The earphone jack 170D is used to connect a wired earphone.

[0075] The earphone jack 170D is used to connect a wired earphone.

[0076] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be disposed on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc.

[0077] The gyro sensor 180B may be used to determine the motion posture of the electronic device 100. In some embodiments, the angular velocity of the electronic device 100 around three axes (ie, x, y, and z axes) may be determined by the gyro sensor 180B.

[0078] The air pressure sensor 180C is used to measure air pressure.

[0079] The magnetic sensor 180D includes a Hall sensor.

[0080] The acceleration sensor 180E can detect the magnitude of the acceleration of the electronic device 100 in various directions (generally three axes). When the electronic device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the electronic device 100.

[0081] The distance sensor 180F is used to measure the distance. The electronic device 100 can measure the distance by infrared or laser.

[0082] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0083] The ambient light sensor 180L is used to sense the brightness of the ambient light. The electronic device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed brightness of the ambient light.

[0084] The fingerprint sensor 180H is used to collect fingerprints.

[0085] The temperature sensor 180J is used to detect temperature. In some embodiments, the electronic device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy.

[0086] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0087] The bone conduction sensor 180M can obtain a vibration signal. In some embodiments, the bone conduction sensor 180M can obtain a vibration signal of a vibrating bone block of a human vocal part.

[0088] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0089] The following describes the application scenarios of the spatial audio synthesis method provided by this application:

[0090] The spatial audio synthesis method provided in the present application can be applied to replay spatial audio in various spatial environments, which may be a home environment, an office environment, a karaoke environment, etc.

[0091] Take the home environment as an example. Figure 3A A home environment is shown. When the user is not in the home environment, the spatial audio at any position in the home environment can be obtained by the method proposed in the present application, and the sound experience at any position in the home environment can be obtained by listening to the spatial audio. Among them, the home environment includes a living room area and a bedroom area, and multiple smart devices located in the home environment are shown: the living room area includes a smart speaker, a smart TV and a smart monitor, and the bedroom area includes a smart computer and a smart phone. For example, the user can assume that he is on the sofa in the living room area of ​​the home environment, and assume that his head movement information is as follows: the head is facing the direction of the smart TV, and the rotation angle, pitch angle, and roll angle of the head are all zero. Then, through the relative position of the microphones carried in the above-mentioned smart devices relative to the sofa, and the user's head movement information, the home environment audio collected by these microphones is mapped to the sofa to obtain the spatial audio at the sofa. By listening to the spatial audio at the sofa, the user can get the sound experience of being at the sofa.

[0092] Take the office environment as an example. Figure 3BAn office environment is shown. When the user is not in the office environment, the spatial audio at any position in the office environment can be obtained by the method proposed in the present application, and the sound experience at any position in the office environment can be obtained by listening to the spatial audio. Among them, multiple smart devices located in the office environment are shown: a smart printer, a smart projector, a smart speaker, a smart computer and a smart phone. For example, if the user is assumed to be at a desk in the office environment, and assumes that his head movement information is as follows: the head is facing the direction of the smart computer, and the rotation angle, pitch angle and roll angle of the head are all zero. Then, through the relative position of the microphone carried in the above-mentioned smart device relative to the desk, and the user's head movement information, the office environment audio collected by these microphones is mapped to the desk to obtain the spatial audio at the desk. By listening to the spatial audio at the desk, the user can get the sound experience of being at the desk.

[0093] Taking the KTV environment as an example, after recording the singing audio, the user can add different spatial audio to the singing audio in the audio-repairing APP to simulate the singing effect of the user in different spaces. For example, the user can add spatial audio of KTV, spatial audio of bathroom, spatial audio of concert, etc., and these spatial audios can be obtained by the method proposed in this application. For example, the spatial audio of KTV can be the spatial audio of the center position of the KTV environment (the center position is preset or can be selected by the user), the KTV environment audio collected by the microphones of each smart device in the real KTV environment, and then based on the relative position of each microphone relative to the center position and the assumed head movement information (the head movement information can be preset or can be selected by the user), the KTV environment audio collected by each microphone is mapped to the center position to obtain the spatial audio of the center position. By adding the spatial audio of the center position, the user's singing audio can have the spatial information at the center position.

[0094] Optionally, when replaying the spatial audio of each of the above-mentioned spatial environments, a real-time playback method or a post-playback method may be adopted.

[0095] Optionally, when replaying spatial audio, it can be used in combination with video playback devices, sensor devices, etc. to enhance the user's sense of reality. For example, in a VR / AR scenario, the user's position information and head movement information are obtained through a sensor device, and then the ambient audio collected by each microphone in the spatial environment is mapped according to the position information and the head movement information to obtain the user's spatial audio. Finally, by playing the spatial audio and synchronously playing the corresponding spatial video, the user's visual and auditory experience in the spatial environment can be simulated.

[0096] The spatial audio synthesis method provided in the embodiment of the present application can also be applied to more scenarios, which are not limited here.

[0097] Below through Figure 4 The spatial audio synthesis method provided in the embodiment of the present application is specifically described:

[0098] See also Figure 4 , Figure 4 4 is a flowchart of a spatial audio synthesis method provided in an embodiment of the present application, the method comprising steps 401 to 403. Figure 4 The execution subject of the method shown may be an electronic device, or a chip in an electronic device. The following description is made by taking an electronic device as an execution subject of the method as an example, and the electronic device may be the electronic device 100 introduced in the above content. Among them:

[0099] Step 401: Determine a first relative position set based on a first position of a target object in a target space and absolute positions of each microphone in the target space.

[0100] The target space is a closed space environment, for example, the target space can be the home environment, office environment, karaoke environment, etc. listed above.

[0101] The target object is located in the target space, and the target object can be a virtual object (such as a virtual person) or a real object (such as a robot) to simulate a real user in the target space. For example, when the real user is not in the target space, the virtual object (virtual person) can be used as the target object; or, if there is a robot or an organism in the target space, and the robot or organism can move in the target space, then when the real user is not in the target space, the robot or organism can be used as the target object.

[0102] The first position of the target object refers to the absolute position of the target object in the target space. Figure 3A For example, the first location of the target object may be the sofa in the living room area of ​​the home environment. Figure 3B Taking the office environment in the example, the first position of the target object may be a desk in the office environment.

[0103] Each microphone is a microphone carried by each smart device in the target space. Figure 3A Taking the home environment in the video as an example, various smart devices with microphones can include: smart computers, smart speakers, smart monitoring, smart phones, smart TVs, etc. Figure 3BTaking the office environment in an office as an example, a variety of smart devices with microphones may include: smart printers, smart projectors, smart speakers, smart computers and smart phones.

[0104] Optionally, these smart devices carrying microphones may be distributed at different locations in the target space and connected to the same communication network.

[0105] The absolute position of each microphone refers to the absolute position of each microphone in the target space when the smart devices where the microphones are located remain fixed.

[0106] The first relative position set refers to the relative position of each microphone relative to the first position. Specifically, the absolute position of each microphone can be determined first, and then the relative position of each microphone relative to the first position can be determined based on the first position and the absolute position of each microphone.

[0107] Optionally, the absolute position of each microphone, the first position, and the relative position of each microphone with respect to the first position mentioned above may be expressed in the form of two-dimensional coordinates or in the form of three-dimensional coordinates, which is not limited in the present application.

[0108] For example, Figure 5 As shown, taking the form of two-dimensional coordinates as an example, a two-dimensional coordinate system can be first established with the center of the head of the target object as the origin, and then the coordinates of microphones 1 to 5 in the two-dimensional coordinate system can be obtained according to the absolute positions of microphones 1 to 5 and the first position: ( )、( )、( )、( )、( ), these five coordinates are also the first relative position set.

[0109] The absolute position of each microphone and the method for determining the first position are described below:

[0110] 1. How to determine the absolute position of each microphone

[0111] Optionally, the absolute position of each microphone in the target space may be used as the absolute position of each microphone; or, the absolute position of an electronic device associated with each microphone in the target space may be used as the absolute position of each microphone.

[0112] The absolute position of each microphone in the target space, or the absolute position of each associated electronic device in the target space, can be determined and updated with the help of an existing positioning algorithm, which includes, but is not limited to, an indoor WIFI positioning algorithm, an ultrasonic positioning algorithm, a visual positioning algorithm, and the like.

[0113] 2. How to determine the first position

[0114] In a possible implementation, the electronic device may receive a location setting request for a target object; and determine a first location in response to the location setting request.

[0115] For example, an application (APP) for listening to spatial audio is installed in the electronic device, and the user can trigger the electronic device to receive a position setting request by operating the APP, and the electronic device further determines the first position.

[0116] For example, the app can support users to customize the location for listening to spatial audio, such as Figure 6 As shown, Figure 6 is a schematic diagram of an interface of the application provided by the present application, wherein the interface displays a schematic diagram of a target space (with the target space being Figure 3A For example, the user can select a position as the first position in the target space schematic diagram to simulate the scene where the target object (virtual object) is at the first position, and trigger the electronic device to receive a position setting request. For example, the user selects point A in the target space as the first position of the target object through operation 601, and triggers the electronic device to receive a position setting request through operation 602 on the "OK" control, and the position setting request includes the position of point A.

[0117] Alternatively, the app can support users to select the location where they want to listen to spatial audio, such as Figure 7 As shown, Figure 7 is another interface diagram of the application provided by the present application, which displays a schematic diagram of the target space and the position points to be selected. The position points to be selected are located in the target space, including point B next to the sofa in the living room, point C on the bed, and point D next to the desk. The user can select any position point from the various position points to be selected to simulate the scenario where the target object is at the selected position point, and trigger the electronic device to receive a position setting request. For example, the user selects point B as the first position of the target object through operation 701, and triggers the electronic device to receive a position setting request through operation 702 for the "OK" control, and the position setting request includes the position of point B.

[0118] It is understandable that the above method of triggering the electronic device to receive the location setting request through the APP is only an example. In specific implementation, other methods can also be used to trigger the electronic device to receive the location setting request, which is not limited here.

[0119] Based on this approach, the user can trigger the electronic device to receive a location setting request according to his or her own needs, so that the determined first location can be flexibly adapted to the user's own needs.

[0120] In another possible implementation, the electronic device may acquire first sensor information of the target object; the first sensor information includes movement information of the target object; and the first position is determined based on the first sensor information.

[0121] Optionally, the first sensor information may be acquired in real time, or acquired in a historical period.

[0122] For example, in a real-time playback scenario, the first sensor information may include real-time movement information of the user (such as movement information collected in real time when the user wears an AR / VR device).

[0123] For another example, in a post-playback scenario, the first sensor information may include the historical movement information of a robot or any organism at a specific moment in a historical period (e.g., at a party, the party scene is recorded by the robot's movement, and the spatial audio of the specific moment can be generated later). The specific moment may be selected by the user or may be automatically selected. For example, at a party, the robot moves at the party scene for 3 minutes, wherein the robot is located at the center of the party scene at the 1st minute, so the user may select the historical movement information corresponding to the 1st minute as the first sensor information. Alternatively, for the historical movement information covering 3 minutes, the historical movement information corresponding to the 30th second may be used as the first sensor information at intervals of 30 seconds to generate the spatial audio at the 30th second; then the historical movement information corresponding to the 1st minute may be used as the first sensor information to generate the spatial audio at the 1st minute, and so on.

[0124] Optionally, the first sensor information may include but is not limited to: image information, posture information, communication information, etc.

[0125] For example, the image information includes image information collected by a camera when the target object moves. The electronic device can extract environmental features from the collected image information and match the environmental features with a stored environmental feature library to obtain the first position of the target object.

[0126] For another example, the position information includes accelerations in three directions collected by the inertial navigation sensor, and the electronic device can deduce the first position of the target object based on the accelerations in three directions and the initial position of the target object. The initial position of the target object can be set by the user or obtained based on the sensor information of the initial position (such as identifying the initial position of the target object based on the initial image information, etc.).

[0127] For another example, the communication information includes a wireless fidelity (Wi-Fi) signal, and the electronic device can match the first location of the target object according to the Wi-Fi information obtained from the target object and a Wi-Fi fingerprint library.

[0128] Based on this approach, since the movement information included in the first sensor information is real-time / historical movement information that is actually perceived, a more accurate first position can be determined based on the first sensor information.

[0129] Step 402: Determine first head movement information based on the head movement information of the target object at the first position.

[0130] The head information of the target object includes, but is not limited to, the orientation of the target object, the rotation angles of the target object on the x-axis, y-axis, and z-axis (ie, the yaw angle Raw, the pitch angle Pitch, and the roll angle Roll), etc.

[0131] In a possible implementation, the electronic device may receive a head motion setting request for a target object; and determine head motion information of the target object in a first position in response to the head motion setting request.

[0132] For example, for an APP that listens to spatial audio, the electronic device displays the APP Figure 6 or Figure 7 After the interface shown, you can display Figure 8 The interface shown in the figure. Figure 8 As shown, the interface displays input boxes 801 to 803, and an "OK" control 804. The user can first input the head pitch angle Pitch (rotation angle around the x-axis) in the input box 801, the head yaw angle Raw (rotation angle around the y-axis) in the input box 802, and the head roll angle Roll (rotation angle around the z-axis) in the input box 803, and then click the "OK" control 804, thereby triggering the electronic device to receive a head movement setting request, which includes the Pitch, Raw and Roll input by the user.

[0133] Alternatively, the electronic device may display a head image of the target object, and the user may slide the screen in a preset direction to simulate the head movement of the target object, thereby triggering the electronic device to receive a head movement setting request. For example, the user may slide the head image displayed on the screen up and down to simulate the rotation angle of the target object's head around the y-axis.

[0134] This application does not limit the triggering conditions for receiving a head movement setting request.

[0135] Based on this approach, the user can trigger the electronic device to receive a head movement setting request according to his or her own needs, so that the determined head movement information at the first position can be flexibly adapted to the user's own needs.

[0136] In another possible implementation, the electronic device may acquire second sensor information of the target object; the second sensor information includes head movement information of the target object in the first position.

[0137] Optionally, the second sensor information may be acquired in real time, or acquired in a historical period.

[0138] For example, in a real-time playback scenario, the second sensor information may include real-time head movement information of the user, such as real-time movement information collected when the user wears an AR / VR device.

[0139] For another example, in a post-playback scenario, the second sensor information may include historical head movement information of a robot or any organism at a specific moment in a historical period, wherein the specific moment may be determined according to the above description and will not be described in detail here.

[0140] Optionally, the second sensor information may include, but is not limited to: posture information, image information, etc. For example, the posture information includes rotation angles in three directions collected by the inertial navigation sensor. For another example, the image information includes image information collected by a camera when the target object is walking. The electronic device can extract environmental features from the collected image information and match the environmental features with a stored environmental feature library to obtain the head movement information of the target object in the first position.

[0141] Based on this approach, since the head movement information included in the second sensor information is real-time / historical head movement information that is actually perceived, more accurate head movement information of the first position can be determined based on the second sensor information.

[0142] Step 403: synthesize the spatial audio of the first position based on the first relative position set, the first head movement information and the first audio set.

[0143] The first audio set includes environmental audio collected by each microphone when the target object is at the first position.

[0144] Optionally, the first audio set may be triggered by the spatial audio of the first position synthesized this time. For example, after the electronic device receives the above position setting request or head movement setting request, it may notify each microphone to start collecting ambient audio, thereby obtaining the first audio set.

[0145] Optionally, the first audio set may be triggered to be collected when each microphone moves. For example, when each microphone moves from other positions to the absolute position of each microphone, the ambient audio may be collected to obtain the first audio set.

[0146] Optionally, the first audio set may be triggered by the last change in the state of the target space. For example, the state of the target space may be caused by factors such as weather changes, changes in the layout of the target space, and changes in the external environment of the target space. When the state of the target space changes, the propagation of sound in the target space will also change. In this case, it is necessary to trigger each microphone to collect the changed ambient audio, thereby obtaining the first audio set.

[0147] In the present application, the method of triggering each microphone to collect the first audio set is not limited to the methods listed above.

[0148] In a possible implementation, the electronic device may obtain the head-related transfer function corresponding to each microphone from a database based on the first head movement information and the first relative position set; obtain the mapped ambient audio of each microphone based on the head-related transfer function of each microphone and the ambient audio collected by each microphone; and obtain the spatial audio of the first position based on the mapped ambient audio of each microphone.

[0149] The head-related transfer function corresponding to each microphone is used to indicate the influence of the environmental effect and the binaural effect on the characteristics of the spatial audio (i.e., the change of the spatial sound field) when the sound collected by each microphone is transmitted from each microphone to the ears of the target object. The database stores the corresponding relationship between the first head movement information, the first relative position set, and the head-related transfer function corresponding to each microphone.

[0150] Optionally, the head-related transfer functions corresponding to the microphones include a left-ear head-related transfer function and a right-ear head-related transfer function, and the mapped ambient audio of the microphones includes a left-ear mapped ambient audio and a right-ear mapped ambient audio of the microphones. The spatial audio at the first position includes the spatial audio corresponding to the left ear and the spatial audio corresponding to the right ear when the target object is at the first position.

[0151] For example, the spatial audio corresponding to the left ear when the target object is at the first position can be obtained based on the following formula 1: , based on the following formula 2, the spatial audio corresponding to the right ear when the target object is in the first position is obtained :

[0152] (Formula 1)

[0153] (Formula 2)

[0154] in, Indicates the ambient audio collected by microphone 1. Indicates the ambient audio collected by microphone 2; and represents the left ear head related transfer function and the right ear head related transfer function of microphone 1; and y represents the left ear head related transfer function and the right ear head related transfer function of microphone 2. represents the convolution process, Represents the left ear mapping ambient audio of microphone 1, Indicates the right ear mapping ambient audio of microphone 1, Represents the left ear mapping ambient audio of microphone 2, Represents the right ear mapped ambient audio from microphone 2.

[0155] In a possible implementation, after obtaining the spatial audio at the first position, the electronic device may further perform one or more of the following: transmitting the spatial audio at the first position; playing the spatial audio at the first position; or saving the spatial audio at the first position. Based on this approach, post-playback and real-time playback of the spatial audio at the first position may be achieved.

[0156] In the above-described embodiment, since the ambient audio collected by each microphone can reflect the spatial sound field at each microphone position, and the head-related transfer function corresponding to each microphone can indicate: when the sound collected by each microphone is transmitted from each microphone to the ears of the target object, the influence of the ambient effect and binaural effect on the spatial sound field. Therefore, the spatial audio of the first position obtained based on the ambient audio collected by each microphone, the first relative position set and the first head movement information can reproduce the spatial sound field that the target object can feel when it is in the first position. This method of obtaining spatial audio using the collected ambient audio can effectively improve the efficiency of synthesizing spatial audio. Moreover, since this method can recycle the collected first audio set, it can effectively simplify the processing flow of synthesizing spatial audio and reduce the processing complexity.

[0157] pass Figure 4 In the embodiment shown, the spatial audio of the first position can be obtained. Fig. 9 The process shown introduces a method for obtaining spatial audio at multiple locations, which includes steps 901 to 905:

[0158] Step 901: Determine a relative position set based on the current position of the target object in the target space and the absolute position of each microphone in the target space.

[0159] Step 902: Determine head movement information based on the head movement information of the target object at the current position.

[0160] Step 903: synthesize spatial audio of the current position of the target object based on the relative position set, the head movement information, and the audio set.

[0161] Step 904: Determine whether the current location of the target object has been updated.

[0162] If the electronic device determines that the current location of the target object has been updated, step 905 is executed.

[0163] Step 905: Update the current location of the target object.

[0164] After the electronic device executes step 905 , it may jump to execute step 901 .

[0165] For example, when the current position of the target object is the first position, the electronic device can determine the spatial audio of the target object at the first position by executing steps 901 to 904. Then, the electronic device can update the target object from the first position to the second position by executing step 905, that is, determine that the current position of the target object is the second position. Then, the electronic device executes steps 901 to 904 again to determine the spatial audio of the target object at the second position. The specific implementation method of the electronic device executing steps 901 to 904 to determine the spatial audio of the target object at the first position can be referred to. Figure 4 The embodiments shown are not described in detail here.

[0166] The following describes a specific implementation method of the electronic device executing step 901 to step 904 again to determine that the target object is located at the second position of the spatial audio:

[0167] In one possible implementation, in response to a target object moving from a first position to a second position in the target space, a second relative position set is determined based on the second position and the absolute positions of each microphone; second head movement information is determined based on the head movement information of the target object at the second position; and spatial audio at the second position is synthesized based on the second relative position set, the second head movement information and the second audio set.

[0168] The second position may be determined by referring to the first position. The second relative position set may be determined by referring to the first relative position set. The second head movement information may be determined by referring to the first head movement information.

[0169] The second audio set includes the ambient audio collected by each microphone when the target object is at the second position. The second audio set is the same as or different from the first audio set.

[0170] For example, if the absolute position of each microphone does not change and the state of the target space does not change after the first audio set is collected, the first audio set and the second audio set are the same. Conversely, if the absolute position of each microphone does not change and / or the state of the target space changes after the first audio set is collected, each microphone can be re-controlled to collect ambient audio, and when the spatial audio of the second position needs to be determined, the ambient audio re-collected by each microphone is used as the second audio set.

[0171] For a specific implementation manner of synthesizing the spatial audio of the second position based on the second relative position set, the second head movement information, and the second audio set, reference may be made to the implementation manner of step 403 above.

[0172] based on Fig. 9 In the described manner, if the current position of the target object keeps changing, spatial audio at multiple different positions can be obtained according to the change of the current position of the target object.

[0173] In a possible implementation, when the electronic device obtains spatial audio at multiple different positions, it can play the multiple spatial audios in a preset order, thereby improving the user's immersion. The preset order may be the time order corresponding to the multiple different positions. For example, the user first selects the first position and then the second position through the position setting operation, then the time order of the first position precedes the time order of the second position. Alternatively, in the sensor information of the target object that the electronic device can receive for a total of 3 minutes, the first position is the sensor information of 3 minutes, the position determined according to the sensor information of the 1st minute, and the second position is the position determined according to the sensor information of the 1st minute and 30 seconds, then the time order of the first position precedes the time order of the second position. Therefore, the electronic device can play the spatial audio at the first position first, and then play the spatial audio at the second position.

[0174] The software structure of the electronic device of this application is introduced below:

[0175] See also Fig.10 , Fig.10 This is a schematic diagram of the software structure of an electronic device proposed in this application. The layered architecture divides the software into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the hardware abstraction layer (HAL) layer, and the kernel layer.

[0176] The application layer can include a series of application packages. Fig.10 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications (also referred to as applications).

[0177] The application framework layer provides application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Fig. 9 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0178] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0179] Content providers are used to store and retrieve data and make it accessible to applications. This data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0180] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0181] The phone manager is used to provide communication functions of the electronic device 100, such as management of call status (including connecting, hanging up, etc.).

[0182] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0183] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications from applications running in the background, or a notification that appears on the screen in the form of a dialogue interface. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0184] The hardware abstraction layer may include multiple functional modules, such as a position processing module, a head information processing module, and an audio processing module.

[0185] The position processing module is used to determine a first relative position set based on a first position of a target object in the target space and an absolute position of each microphone in the target space. The first relative position set includes relative positions of each microphone relative to the first position.

[0186] The head information processing module is used to determine first head movement information based on the head movement information of the target object at the first position.

[0187] An audio processing module is used to synthesize spatial audio at a first position based on a first relative position set, first head movement information and a first audio set; the first audio set includes ambient audio collected by each microphone when the target object is at the first position.

[0188] The functions of the position processing module, the head information processing module and the audio processing module can be specifically implemented according to the method described in the above method embodiment.

[0189] The kernel layer is the layer between hardware and software. The kernel layer contains at least display driver, camera driver, audio driver, and sensor driver.

[0190] It should be understood that each step in the above method embodiment provided by the present application can be completed by an integrated logic circuit of hardware in a processor or by instructions in the form of software. The method steps disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or by a combination of hardware and software modules in a processor.

[0191] The present application also provides an electronic device, which may include: a memory and a processor. The memory may be used to store a computer program, which includes program instructions; the processor may be used to call the program instructions in the memory, so that the electronic device executes the method in any one of the above embodiments.

[0192] The present application also provides a chip system, which includes at least one processor for implementing the functions involved in the method executed by the electronic device in any of the above embodiments.

[0193] In one possible design, the chip system also includes a memory, which is used to store program instructions and data, and the memory is located inside or outside the processor.

[0194] The chip system may be composed of the chip, or may include the chip and other discrete devices.

[0195] Optionally, the processor in the chip system may be one or more. The processor may be implemented by hardware or by software. When implemented by hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented by software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0196] Optionally, the memory in the chip system may also be one or more. The memory may be integrated with the processor or may be separately arranged with the processor, which is not limited in the embodiments of the present application. Exemplarily, the memory may be a non-transient processor, such as a read-only memory ROM, which may be integrated with the processor on the same chip or may be arranged on different chips respectively. The embodiments of the present application do not specifically limit the type of memory and the arrangement of the memory and the processor.

[0197] Exemplarily, the chip system may be a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on chip (SoC), a central processor unit (CPU), a network processor (NP), a digital signal processor (DSP), a microcontroller unit (MCU), a programmable logic device (PLD), or other integrated chips.

[0198] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed, the computer executes the method executed by the electronic device in any of the above embodiments.

[0199] The various implementation modes of the present application can be combined arbitrarily to achieve different technical effects.

[0200] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs, and the computer program includes program instructions. When the program instructions are loaded and executed on the computer, the process or function described in this application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by the computer or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0201] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

[0202] In short, the above description is only an embodiment of the technical solution of the present invention, and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made according to the disclosure of the present invention shall be included in the protection scope of the present invention.

Claims

1. A spatial audio synthesis method, characterized in that: The method is applied to an electronic device, and the method comprises: Outputting a spatial display interface of an application for listening to spatial audio, the spatial display interface displays a target space and a first determination control; the target space includes a plurality of candidate position points of a target object; the target object is a virtual object or a real object in the target space; the user of the electronic device is not in the target space; receiving, in the target space, a selection operation input by the user for the plurality of candidate position points, and determining a first position point from the plurality of candidate position points in response to the selection operation; receiving a first click operation input by the user on the first determination control, and determining the first position point as a first position of the target object in response to the first click operation; Determine a first relative position set based on the first position and the absolute positions of the microphones in the target space, the first relative position set including the relative positions of the microphones relative to the first position; Outputting a head movement information selection interface of the application, wherein the head movement information selection interface displays a head angle input box and a second confirmation control; receiving head angle information input by the user in the head angle input box, wherein the head angle information includes a head pitch angle, a head yaw angle, and a head roll angle; receiving a second click operation input by the user on the second determination control, and in response to the second click operation, determining that the head angle information is head movement information of the target object at the first position; Determining first head movement information based on head movement information of the target object at the first position; Based on the first head movement information and the first relative position set, a head-related transfer function corresponding to each microphone is obtained from a database; the database stores a correspondence between the first head movement information, the first relative position set and the head-related transfer function corresponding to each microphone; based on the head-related transfer function of each microphone, a convolution operation is performed on the ambient audio collected by each microphone in the first audio set to obtain the mapped ambient audio of each microphone; based on the mapped ambient audio of each microphone, the spatial audio of the first position is obtained; the first audio set includes the ambient audio collected by each microphone when the target object is at the first position; the first audio set is the ambient audio collected by each microphone notified by the electronic device after receiving the first click operation or the second click operation; The spatial audio at the first position is played.

2. The method according to claim 1, characterized in that The method further comprises: In response to the target object moving from the first position to a second position in the target space, determining a second relative position set based on the second position and the absolute positions of the respective microphones, the second relative position set including relative positions of the respective microphones relative to the second position; determining second head movement information based on the head movement information of the target object at the second position; Based on the second relative position set, the second head movement information and the second audio set, the spatial audio of the second position is synthesized; the second audio set includes the ambient audio collected by the various microphones when the target object is in the second position.

3. An electronic device, characterized in that: It comprises a memory and one or more processors; the memory is coupled to the one or more processors and is used to store a computer program, the computer program comprises program instructions; the one or more processors call the program instructions so that the electronic device executes the method as claimed in claim 1 or 2.

4. A computer-readable storage medium, characterized in that: The computer program comprises a computer program, wherein the computer program comprises program instructions, and when the program instructions are executed on an electronic device, the electronic device executes the method as claimed in claim 1 or 2.

Images