Stereo generation method and device, computer equipment and storage medium

Abstract

The invention relates to a stereophonic sound generation method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring respective sound source positions and sound signals of a plurality of sound sources in an environment space where a target object is located; performing sound source category analysis on each sound source, and determining a foreground sound source and at least two background sound sources in each sound source; based on the position of the target object in the environment space and the respective sound source position of each background sound source, carrying out sound mixing processing on the sound signal of each background sound source to obtain a sound mixing signal; stereophonic reconstruction is carried out on the sound mixing signal to obtain background stereophonic sound; and superposing foreground stereophonic sound obtained by performing stereophonic reconstruction on the sound signal of the foreground sound source and background stereophonic sound to obtain stereophonic sound matched with the position of the target object. By adopting the method, on the premise that the perception experience of a user is not influenced, the calculation overhead in the stereo generation process is effectively reduced, and calculation resources are saved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a stereo generation method, apparatus, computer equipment, storage medium and computer program product. Background Art

[0002] With the rapid development of computer technology, people have higher and higher requirements for the stereoscopic sense and spatial layering of the sound played by the terminal. Taking the metaverse as an example, by reconstructing the stereo sound signals of each sound source in the reconstructed virtual space, a virtual stereo sound simulating the real scene can be obtained.

[0003] In traditional technology, it is necessary to perform stereo reconstruction on the audio signals of each sound source in the ambient space to obtain the required stereo sound. However, the computational overhead of stereo reconstruction is relatively large, resulting in a very high computational overhead for stereo generation. Even if the computing module is transferred to the server for implementation, it still requires a lot of computing resources. Summary of the invention

[0004] Based on this, it is necessary to provide a stereo generation method, apparatus, computer device, computer-readable storage medium and computer program product that can save computing resources in order to solve the above technical problems.

[0005] In a first aspect, the present application provides a stereo generation method. The method comprises:

[0006] Acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0007] Performing sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source;

[0008] Based on the position of the target object in the environment space and the sound source positions of the background sound sources, mixing the sound signals of the background sound sources to obtain a mixed signal;

[0009] Perform stereo reconstruction on the mixed signal to obtain background stereo;

[0010] The foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo sound to obtain a stereo sound that matches the position of the target object.

[0011] In a second aspect, the present application further provides a stereo sound generation device. The device comprises:

[0012] An acquisition module, used to acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0013] A sound source category analysis module, used to perform sound source category analysis on each of the sound sources to determine a foreground sound source and at least two background sound sources in each of the sound sources;

[0014] A mixing module, configured to perform mixing processing on the sound signals of each background sound source based on the position of the target object in the environment space and the sound source position of each background sound source to obtain a mixed signal;

[0015] A stereo reconstruction module, used for performing stereo reconstruction on the mixed signal to obtain background stereo;

[0016] The stereo sound generation module is used to perform superposition processing on the foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source and the background stereo sound to obtain the stereo sound matching the position of the target object.

[0017] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0018] Acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0019] Performing sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source;

[0020] Based on the position of the target object in the environment space and the sound source positions of the background sound sources, mixing the sound signals of the background sound sources to obtain a mixed signal;

[0021] Perform stereo reconstruction on the mixed signal to obtain background stereo;

[0022] The foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo sound to obtain a stereo sound that matches the position of the target object.

[0023] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0024] Acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0025] Performing sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source;

[0026] Based on the position of the target object in the environment space and the sound source positions of the background sound sources, mixing the sound signals of the background sound sources to obtain a mixed signal;

[0027] Perform stereo reconstruction on the mixed signal to obtain background stereo;

[0028] The foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo sound to obtain a stereo sound that matches the position of the target object.

[0029] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0030] Acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0031] Performing sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source;

[0032] Based on the position of the target object in the environment space and the sound source positions of the background sound sources, mixing the sound signals of the background sound sources to obtain a mixed signal;

[0033] Perform stereo reconstruction on the mixed signal to obtain background stereo;

[0034] The foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo sound to obtain a stereo sound that matches the position of the target object.

[0035] The stereo sound generation method, device, computer equipment, storage medium and computer program product obtain the sound source positions and sound signals of each of the multiple sound sources in the environment space where the target object is located; perform sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source; perform mixing processing on the sound signals of each background sound source based on the location of the target object in the environment space and the sound source positions of each background sound source to obtain a mixed signal; perform stereo reconstruction on the mixed signal to obtain background stereo; and perform superposition processing on the foreground stereo obtained by stereo reconstruction of the sound signal of the foreground sound source and the background stereo to obtain a stereo sound matching the location of the target object. In the process of stereo generation, each sound source is first divided into foreground sound source and background sound source through sound source category analysis, and the foreground stereo obtained by stereo reconstruction of the sound signal of the foreground sound source can ensure the clarity of the foreground stereo; for the background sound source, a group stereo reconstruction process is performed, specifically, the sound signals of multiple background sound sources are first mixed to obtain a mixed signal, and then the mixed signal is stereo reconstructed to obtain the background stereo, which can realize the integrated reconstruction of multiple background sound sources, thereby reducing the number of stereo reconstruction processes and reducing the computational overhead in the stereo generation process. The above method can effectively reduce the computational overhead in the stereo generation process and save computing resources without affecting the user's perception experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A diagram showing an application environment of a stereo sound generation method in some embodiments;

[0037] Figure 2 A diagram showing an application environment of a stereo sound generation method in some other embodiments;

[0038] Figure 3 is a schematic diagram of a stereo sound generation process in some embodiments;

[0039] Figure 4 is a schematic flow chart of a stereo sound generation method in some embodiments;

[0040] Figure 5 is a schematic diagram of a virtual scene in some embodiments;

[0041] Figure 6 A schematic diagram of an environmental space in some embodiments;

[0042] Figure 7 A schematic diagram of multiple intervals of a stereo reconstruction area in some embodiments;

[0043] Figure 8 Schematic diagram of the flow of stereo generation method in some other embodiments;

[0044] Fig. 9 is a schematic diagram of a stereo sound generation process in some embodiments;

[0045] Fig.10 is a schematic diagram of HRTF stereo reconstruction processing in some embodiments;

[0046] Fig.11 is a structural block diagram of a stereo sound generating device in some embodiments;

[0047] Fig.12 is an internal structure diagram of a computer device in some embodiments;

[0048] Fig.13 1 is a diagram of the internal structure of a computer device in some other embodiments. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] First, the terms involved in the embodiments of the present application are briefly explained:

[0051] Virtual stereo: It uses dual-channel processing technology to convert sound signals from different directions of sound sources in a real sound environment into dual-channel sound signals, and finally plays them through two channels or two speakers to achieve sound field reproduction. Virtual stereo generates stereo effects by simulating the sound delay and sound intensity difference from the sound source to the left and right ears through algorithms. Virtual stereo compresses the original multi-channel information in the real environment into dual channels, effectively reducing the cost of transmission and playback equipment. It is the most cost-effective way to obtain stereo effects. The input of the virtual stereo algorithm is usually the mono sound signal of the sound source and the azimuth coordinate information of the sound source, and its output is the left and right dual-channel sound signals.

[0052] Metaverse: It is a virtual world that is linked and created by scientific and technological means, mapped and interacted with the real world, and has a digital living space with a new social system. It integrates a variety of new technologies to produce new Internet applications and social forms that blend the virtual and the real. It provides an immersive experience based on extended reality technology, generates a mirror image of the real world based on digital twin technology, and builds an economic system based on blockchain technology. It closely integrates the virtual world with the real world in the economic system, social system, and identity system, and allows each user to produce content and edit the world.

[0053] Spatial sound effect: It is a sound that has been processed with certain audio technology to allow users to hear more three-dimensional and spatially layered sounds. Through headphones or a combination of two or more speakers, the actual auditory scene is played back, allowing listeners to clearly identify the direction, distance and movement trajectory of different acoustic objects. It also allows listeners to feel surrounded by the sound in all directions, giving listeners an immersive auditory experience as if they were in the actual environment.

[0054] The stereo sound generation method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the application scenario may include a terminal 102 and a server 104. The terminal 102 communicates with the server 104 via a network. The communication network may be a wired network or a wireless network. Therefore, the terminal 102 and the server 104 may be directly or indirectly connected via a wired or wireless communication. For example, the terminal 102 may be indirectly connected to the server 104 via a wireless access point, or the terminal 102 may be directly connected to the server 104 via the Internet, and this application does not limit this.

[0055] Among them, the terminal 102 can be, but is not limited to, various smart phones, tablet computers, Internet of Things devices and portable wearable devices. The Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The embodiments of the present application can be applied to stereo generation scenarios in real environments or virtual environments, such as virtual reality applications, three-dimensional maps, event simulations, and game applications. Further, the terminal 102 may include a sound-emitting terminal corresponding to the sound source, and a playback terminal corresponding to the listener. The sound-emitting terminal can be installed with a client related to sound signal generation, and the playback terminal can be installed with a client related to audio playback. The above-mentioned client can be software, a web page or a small program, etc. In some embodiments, the terminal 102 can be used as a sound-emitting terminal and a playback terminal at the same time. The server 104 is a background server corresponding to the software, web page or small program, etc., or a server specifically used for stereo generation. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The data storage system can store data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers.

[0056] It should be noted that the stereo sound generation method in the embodiment of the present application can be executed by the terminal 102 or the server 104 alone, or can be executed by the terminal 102 and the server 104 together. Taking the case where the server 104 executes alone as an example, Figure 2 As shown in FIG. 1 , the environment space may include N sound sources, each of which emits a sound signal through a sound-emitting terminal, and the listener can play the stereo sound in the environment space through the playback terminal. The environment space may include multiple listeners. Since the positions of the listeners in the environment space may be different, it is necessary to match the corresponding stereo sound for each listener. The matched listener of the currently generated stereo sound is taken as the target object, as shown in FIG. Figure 3 As shown, in the process of generating stereo sound matching the position of the target object, the server 104: obtains the sound source position and sound signal of each sound source from the sound source terminals corresponding to each of the multiple sound sources in the environment space where the target object is located; performs sound source category analysis on each sound source to determine the foreground sound source and at least two background sound sources in each sound source; performs mixing processing on the sound signals of each background sound source based on the position of the target object in the environment space and the sound source positions of each background sound source to obtain a mixed signal; performs stereo reconstruction on the mixed signal to obtain background stereo sound; and performs superposition processing on the foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source and the background stereo sound to obtain a stereo sound matching the position of the target object. Then, the server 104 sends the generated stereo sound to the playback terminal corresponding to the target object, so that the target object can play the stereo sound in the environment space through the playback terminal.

[0057] It can be understood that when the data processing capability of the playback terminal meets the data processing requirements of the stereo generation process, the server 104 may not be involved, and the playback terminal corresponding to the target object can directly obtain the sound source position and sound signal from the sound-emitting terminal corresponding to each sound source, and obtain stereo that matches the position of the target object based on the sound source position and sound signal.

[0058] In some embodiments, Figure 4 As shown, a stereo generation method is provided, which can be executed by a computer device, and the computer device can be Figure 1 The terminal or server shown in this embodiment is applied to Figure 1 The server in the example is used to illustrate the following steps:

[0059] Step S402: Acquire the sound source positions and sound signals of each of the multiple sound sources in the environment space where the target object is located.

[0060] The environmental space refers to the space in the real scene or virtual scene. The real scene refers to the scene existing in the objective world, and the virtual scene refers to the scene created by computer technology that is different from the real scene. The virtual scene can be a simulation of the real scene, a semi-simulated and semi-fictional three-dimensional world, or a purely fictional three-dimensional world. Figure 5 As shown, the virtual scene can be, for example, a scene such as a somatosensory game, a multiplayer electronic game, or a metaverse world. The environment space can contain a variety of objects such as people, animals, virtual characters, etc., and the target object refers to the listener in the environment space who receives the sound signal generated by the sound source. The target object can include listeners in the real scene or listeners constructed in the virtual scene. In other words, the stereo generated in this application is a stereo that matches a target object in the environment space.

[0061] Sound is a kind of mechanical wave, and a sound source refers to an object that can generate mechanical waves. In physics, an object that is making a sound is called a sound source. For example, vocal cords, tuning forks, drums, etc. are all sound sources. The sound sources in the embodiments of the present application may include multiple sound sources of different types. The sound sources may be, for example, sound sources in a real environment, such as tuning forks and drums, or they may be virtually constructed sound sources. Exemplarily, there may be ambient sounds such as rain and wind in the environment space, and there may also be multiple objects that can make sounds, such as multiple characters and scenery, which are used as different sound sources. For example Figure 6 The sound signals contained in the environmental space shown may include the voices, laughter, and footsteps of different people in different directions around, the engine sound of a car approaching from far to near, the sound of a pedestrian path, and the sound of wind and rain. Furthermore, each sound source carries its own sound signal and information such as the location of the sound source. The location of the sound source can be characterized by position information such as coordinates, distance, and azimuth, for example. It can be understood that the sound source in the embodiment of the present application includes at least three sound sources, and the sound signal carried by the sound source includes at least one single-channel audio signal. Exemplarily, the sound signal of the sound source may include a left-channel audio signal and a right-channel audio signal.

[0062] Specifically, the server can determine multiple sound sources contained in the environment space where the target object is located, and obtain the sound source position and sound signal of each sound source. For example, the sound signal of the sound source collected by the user's recording device can also be the sound signal of the sound source constructed by the virtual space sound effect system, that is, the server.

[0063] Furthermore, the sound signal of the sound source may be a sound signal captured in real time or recorded in advance, which is not limited here. Moreover, the acquired sound signal may be associated with time. For the current moment, if the sound signal is captured in real time, the acquired sound signal is the sound signal captured in real time at the current moment. If the sound source of the virtual space is pre-set by the virtual reality application, the acquired sound signal is the sound signal output when the pre-set audio is played according to the time sequence to the current moment.

[0064] Step S404: performing sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source.

[0065] Among them, the sound signals emitted by each sound source include foreground sound and background sound. Foreground sound refers to the sound as the main content. For example, the dialogue or narration in a TV series, movie, animation or game; for example, the conversation sound in life scenes. In actual scenes, foreground sound usually includes human voice. The human voice can be the sound emitted by a natural person or the sound synthesized by a machine. Background sound refers to the sound used as background in the audio. For example, the music used to adjust the atmosphere in a TV series, movie, animation or game, and for example, the ambient sound such as rain, wind, dog barking in life scenes. In actual scenes, background sound can also include human voice. For example, the cheering sound used to create the atmosphere in the game; for example, the sound of students discussing quietly in the classroom. Correspondingly, the foreground sound source refers to the sound source whose sound signal emitted is the foreground sound, and the background sound source refers to the sound source whose sound signal emitted is the background sound.

[0066] In the real acoustic world, although our ears can receive many sound signals from different sound sources, there may be only a limited number of them that really attract our attention, such as the foreground sound sources mentioned above. For the foreground sound sources, the human ear can clearly identify the information content and can more accurately identify the direction of the sound source; on the contrary, the background sound sources in the real acoustic world are a kind of foil to the human ear, that is, the human ear can perceive their existence but does not pay much attention to the information content of their sound. Based on this, the server can perform sound source category analysis on each sound source and determine the foreground sound source and at least two background sound sources in each sound source.

[0067] In one embodiment, the server may perform a sound source category analysis based on at least one of the type and energy of the sound signal to determine a foreground sound source and at least two background sound sources among the sound sources. For example, the sound source of a human voice may be determined as a foreground sound source, and the sound source of an ambient sound may be determined as a background sound source. For another example, the sound source of a sound signal whose energy meets an energy condition may be determined as a foreground sound source, and the remaining sound sources may be determined as background sound sources. For another example, the sound source of a human voice whose energy meets an energy condition may be determined as a foreground sound source, and the remaining sound sources may be determined as background sound sources. Among them, energy meeting an energy condition may mean that the energy is greater than an energy threshold, or that the energy is greater than or equal to the energy threshold.

[0068] In one embodiment, the target object can initiate an interactive operation with the server to determine the foreground sound source of interest to itself, and the server responds to the interactive operation to determine the foreground sound source and the background sound source among the sound sources. The triggering mode and type of the interactive operation can be associated with the application scenario, the type of interactive device held by the target object, and other factors. Exemplarily, if the application scenario is that the user plays a somatosensory game through a somatosensory device, the type of interactive operation can be the user's limb movements. For another example, if the application scenario is that the user plays an immersive game through VR (Virtual Reality) glasses, the type of interactive operation can be the user's head movement, such as turning the head. Alternatively, the VR glasses do not capture the user's head movements, but capture the user's eye movements by acquiring the eye position, then the type of interactive operation can be the user's eye movements.

[0069] In one embodiment, the server can determine the target object's perception ability of each of the sound sources based on the position of each sound source and the position of the target object in the environment space; perform sound source category analysis on each sound source according to the perception ability corresponding to each sound source, and determine the foreground sound source and at least two background sound sources in each sound source. Among them, the target object's perception ability of the sound source refers to the target object's perception ability of the sound signal currently emitted by the sound source. It can be understood that the target object's perception ability of the sound signal of the sound source is inversely correlated with the distance between the target object and the sound source, and positively correlated with the energy of the sound signal of the sound source. That is, the closer the distance between the target object and the sound source, the greater the energy of the sound signal of the sound source, and the stronger the target object's perception ability of the sound source. The server can determine the corresponding sound source with relatively strong perception ability as the foreground sound source, and determine the corresponding sound source with relatively weak perception ability as the background sound source.

[0070] It should be noted that, when the sound signal is associated with time, for the same target object, the sound source categories corresponding to the same sound source at different times may be different. Take the case where the sound source is a movable sound source and the sound source category is determined based on the target object's perception of the sound source as an example. If the sound source distance between a certain sound source and the target object decreases during movement, even if the energy of the sound signal emitted by the sound source remains unchanged, the target object's perception of the sound source will increase as the distance decreases, then the sound source can be switched from a background sound source to a foreground sound source. In other words, using the method of the present application, stereo sound associated with time can be generated to ensure the temporal continuity of the generated stereo sound.

[0071] Step S406 , based on the position of the target object in the environment space and the sound source positions of the background sound sources, the sound signals of the background sound sources are mixed to obtain a mixed signal.

[0072] Among them, the mixing process is the process of integrating sound signals from multiple sources into a mixed signal. The mixed signal is a new sound signal obtained after the mixing process. For example, the sound signal of the new sound source I is obtained by mixing the sound signals of sound source 1 and sound source 2, and the sound signal of the new sound source I is the mixed signal. It can be understood that multiple background sound sources can be merged into a new sound source through the mixing process. The specific algorithm for mixing multiple sound signals may include direct addition method, averaging method, clamping method, normalization method, adaptive mixing weighting, automatic alignment algorithm, etc.

[0073] Specifically, for each background sound source, the server can perform distance attenuation processing on the sound signal of the background sound source based on the sound source position of the background sound source and the position of the target object, and determine the sound perception signal of the target object to the background sound source. Then, each sound perception signal is mixed to obtain the mixed signal corresponding to each background sound source. The server can also group each sound perception signal, and perform mixing processing on each sound perception signal of each group respectively to obtain the mixed signal corresponding to each group. Exemplarily, the server can divide the sound perception signals of the background sound sources in the same interval into one group according to the position of each background sound source in the environmental space; the server can also divide the sound perception signals of the background sound sources corresponding to the same type of sound-emitting objects into one group according to the sound-emitting object type of each background sound source.

[0074] It should be noted that, in the process of mixing, the sound signals of the same channel are mixed separately to obtain the mixed signals of the corresponding channels. Take the case where both the background sound source 1 and the background sound source 2 include the left channel sound signal and the right channel sound signal as an example. In the process of mixing the sound signals of the background sound source 1 and the background sound source 2, the server first performs distance attenuation processing on each sound signal to obtain the corresponding sound perception signal. The sound perception signal includes the left channel perception signal and the right channel perception signal. Then, the server performs mixing processing on the left channel perception signals of the background sound source 1 and the background sound source 2 to obtain the left channel mixed signal, and performs mixing processing on the right channel perception signals of the background sound source 1 and the background sound source 2 to obtain the right channel mixed signal.

[0075] Taking the average method as an example, the left and right channel signals output by each background sound source are superimposed and summed, and then the average value is taken to obtain the mixing signal of the corresponding channel. The specific formula is:

[0076]

[0077]

[0078] Here, lout and rout are the left channel mixing signal and the right channel mixing signal in the stereo mixing signal.

[0079] The specific process of distance attenuation processing is introduced below. The server can determine the sound source distance between each background sound source and the target object based on the sound source position of the background sound source and the position of the target object. It can be understood that the distance values ​​between different sound sources and the target object can be the same or different. For example, the distance values ​​between sound source 1, sound source 2 and target object A are both 2 meters, and the distance between sound source 3 and target object A is 5 meters. Then, for each background sound source, the server can amplify or attenuate the sound signal of the background sound source based on the sound source distance corresponding to the background sound source to obtain the sound perception signal of the target object to the background sound source.

[0080] Step S408: Perform stereo reconstruction on the mixed signal to obtain background stereo.

[0081] Among them, stereo reconstruction is the process of restoring a given single-channel audio signal into a multi-channel stereo audio to make the audio have a three-dimensional sense. It can be understood that the reconstructed stereo includes at least two channels of sound signals, and the sound signals of each channel are played through different channels or speakers respectively, so as to achieve the reproduction of the sound field and produce a stereo effect.

[0082] Specifically, the server can perform stereo reconstruction on the mixed signal to obtain background stereo. The specific method of stereo reconstruction is not unique. For example, the server can use the collected stereo samples to train a neural network model, and then input the mixed signal to be reconstructed into the neural network model to reconstruct the background stereo. For another example, the server can use virtual stereo reconstruction technology to perform stereo reconstruction on the mixed signal to obtain background stereo. The virtual stereo reconstruction technology can be, for example, a virtual stereo reconstruction technology based on HRTF (Head Related Transfer Function).

[0083] Furthermore, when the sound signals of the background sound sources are grouped for mixing, the server can perform stereo reconstruction on the mixed signals corresponding to each group to obtain the reconstructed stereo corresponding to each group, and then superimpose the reconstructed stereo to obtain the background stereo. Similar to the mixing process, in the process of superimposing multiple reconstructed stereos to obtain the background stereo, the sound signals of the same channel in the reconstructed stereo are superimposed to obtain the superimposed signals of the corresponding channels, and the superimposed signals of each channel together constitute the background stereo. The specific method of the superposition process can be linear superposition or nonlinear superposition, which is not limited here.

[0084] Step S410: performing stereo reconstruction on the sound signal of the foreground sound source and superimposing the foreground stereo sound with the background stereo sound to obtain a stereo sound that matches the position of the target object.

[0085] Among them, the specific definition of the foreground sound source is as mentioned above and will not be repeated here. Specifically, the server can perform stereo reconstruction on the sound signal of the foreground sound source to obtain the foreground stereo corresponding to the foreground sound source. Then, the foreground stereo and the background stereo are superimposed to obtain a stereo matching the position of the target object. The specific algorithm of the superposition processing can be, for example, linear superposition or nonlinear superposition, etc., which is not limited here.

[0086] Further, the specific method for stereo reconstruction of the sound signal of the foreground sound source can refer to the method for stereo reconstruction of the mixed signal in the previous text. Take the virtual stereo reconstruction technology based on HRTF as an example. The server can first perform distance attenuation processing on the sound signal of the foreground sound source based on the sound source position of the foreground sound source and the position of the target object, and determine the sound perception signal of the target object to the foreground sound source. The sound perception signal may include perception signals corresponding to each of the multiple channels. Then, the server processes each single-channel perception signal of the foreground sound source through HRTF to obtain foreground stereo. Among them, the input of the HRTF function is a single-channel perception signal and azimuth information, and the output is a stereo signal. The azimuth information refers to the azimuth information of the output sound source of the single-channel perception signal relative to the listener (i.e., the target object). The stereo reconstruction based on HRTF is to convolve the original single-channel perception signal u(n) with the HRIR (Head Related Impulse Response, head-related impulse response) excitation data h(n) of the corresponding azimuth, and the HRIR excitation data here can be found from the relevant data set of HRTF. The data set may be, for example, an artificial head data set from MIT or a real person data set from CIPIC. Exemplarily, the formula for calculating the two-channel stereo signal y(n) by HRTF is as follows:

[0087]

[0088] Among them, h(n) includes the HRIR data of the left channel and the right channel, so the generated y(n) also includes the processing results of the left channel and the right channel.

[0089] In a specific embodiment, the number of foreground sound sources is at least two. In the case of this embodiment, the stereo generation method further includes: performing stereo reconstruction on the sound signal of each foreground sound source to obtain foreground stereo sounds corresponding to each foreground sound source; step S410 includes: superimposing each foreground stereo sound and background stereo sound to obtain stereo sounds matching the position of the target object.

[0090] Specifically, when the number of foreground sound sources is at least two, stereo reconstruction is performed on the sound signal of each foreground sound source to obtain the foreground stereo corresponding to each foreground sound source, and then each foreground stereo and background stereo are superimposed to obtain a stereo matching the position of the target object. Further, the left channel audio signal and the right channel audio signal of all stereo audio signals (including foreground stereo and background stereo) can be mixed by averaging method, that is, each channel audio signal and the right channel audio signal are superimposed and summed, and then the average value is taken to obtain the left channel audio signal and the right channel audio signal of the stereo audio signal finally output. In this way, it is equivalent to independent stereo reconstruction of each foreground sound source, which can ensure the clarity of the foreground stereo corresponding to each foreground sound source, meet the user's perception experience, and improve the stereo sound effect.

[0091] In the above stereo generation method, each sound source is first divided into a foreground sound source and a background sound source through sound source category analysis, and the foreground stereo obtained by stereo reconstruction of the sound signal of the foreground sound source can ensure the clarity of the foreground stereo; for the background sound source, a group stereo reconstruction process is performed, specifically, the sound signals of multiple background sound sources are first mixed to obtain a mixed signal, and then the mixed signal is stereo reconstructed to obtain the background stereo, which can realize the integrated reconstruction of multiple background sound sources, thereby reducing the number of stereo reconstruction processes and reducing the computational overhead in the stereo generation process. The above method can effectively reduce the computational overhead in the stereo generation process and save computing resources without affecting the user's perception experience.

[0092] In one embodiment, step S404 includes: determining the target object's perception ability of each sound source based on the position of each sound source and the position of the target object in the environment space; performing sound source category analysis on each sound source according to the perception ability corresponding to each sound source, and determining a foreground sound source and at least two background sound sources in each sound source.

[0093] As mentioned above, the target object's perception of the sound signal of the sound source is inversely correlated with the distance between the target object and the sound source, and positively correlated with the energy of the sound signal of the sound source. The positive correlation of two variables means that: when other conditions remain unchanged, the two variables change in the same direction, and when one variable changes from large to small, the other variable also changes from large to small. It can be understood that the positive correlation here means that the direction of change is consistent, but it does not require that when one variable changes a little, the other variable must also change. For example, when variable a is 10 to 20, variable b can be set to 100, and when variable a is 20 to 30, variable b is 120. In this way, the direction of change of a and b is that when a increases, b also increases. But when a is in the range of 10 to 20, b can be unchanged. Correspondingly, the negative correlation of two variables means that when other conditions remain unchanged, the two variables change in opposite directions, and when one variable changes from large to small, the other variable changes from small to large. Similarly, the negative correlation here means that the direction of change is opposite, but it does not require that when one variable changes a little, the other variable must also change.

[0094] Specifically, the server may determine the target object's perception capability of each sound source based on the position of each sound source and the position of the target object in the environment space, and then perform sound source category analysis on each sound source according to the perception capability corresponding to each sound source, and determine the foreground sound source and at least two background sound sources in each sound source.

[0095] In the above embodiment, performing sound source category analysis based on perception capability can ensure that the analysis result matches the acoustic perception in a real scene, thereby improving the stereo sound effect reconstructed based on the sound source category analysis result.

[0096] Furthermore, the specific method of analyzing the sound source category of each sound source based on the perception ability is not unique. For example, the server may determine the sound source whose perception ability meets the perception condition as the foreground sound source, and determine the other sound sources in the environment space except the foreground sound source as the background sound source. The perception condition may be, for example, that the perception ability is greater than the perception ability threshold, or that the perception ability is greater than or equal to the perception ability threshold.

[0097] In one embodiment, sound source category analysis is performed on each sound source according to the perception capabilities corresponding to each sound source to determine a foreground sound source and at least two background sound sources in each sound source, including: sorting the sound sources by perception capabilities according to the perception capabilities corresponding to each sound source; and determining, in each sound source, a foreground sound source whose sorting position meets a position condition and at least two background sound sources whose sorting positions do not meet the position condition.

[0098] Among them, the position condition can be represented by position order or percentage. Exemplarily, for 10 sound sources in the environmental space, it can be determined that the sorting positions of the two sound sources with greater perception capabilities meet the position condition; it can also be determined that the sorting positions of the top 10% of the sound sources with greater perception capabilities meet the position condition. Specifically, the server can sort the perception capabilities of each sound source according to the perception capabilities corresponding to each sound source, and obtain an ascending sequence of perception capabilities from small to large, or a descending sequence of perception capabilities from large to small. Then, determine the foreground sound sources whose sorting positions meet the position condition, and at least two background sound sources whose sorting positions do not meet the position condition.

[0099] For example, for the N-1 sound sources in the environment space where the target object is located, the server can sort the perception capabilities corresponding to each sound source in order from large to small, determine the top M sound sources as foreground sound sources, and determine the remaining sound sources as background sound sources.

[0100] Furthermore, based on the object category of the target object, the sound conditions that the target object can perceive can be determined, and the effective sound sources that the target object can perceive can be determined from various sound sources, and then the foreground sound sources and the background sound sources can be further determined from the effective sound sources. The sound conditions can include at least one of the frequency conditions and the energy conditions. For example, the human ear hearing frequency range is 20Hz-20000Hz, and the minimum perceptible sound energy is 20μPa; the cat's hearing frequency range is 60Hz-65000Hz. In this way, the generated stereo can be matched with the object category of the target object, which is conducive to further improving the sound effect of the stereo.

[0101] In the above embodiment, the sound sources are ranked according to their perception capabilities, and the foreground sound sources and the background sound sources are determined based on the ranked positions. The algorithm is simple, which is conducive to further reducing the computational overhead in the stereo generation process and saving computational resources.

[0102] In one embodiment, based on the positions of each sound source and the position of the target object in the environmental space, the target object's perception ability of each sound source is determined, including: for each sound source, determining the sound source distance between the sound source position of the sound source and the position of the target object in the environmental space; based on the sound source distance corresponding to the sound source, performing signal conditioning on the sound signal of the sound source to determine the target object's perception ability of the sound source.

[0103] Among them, the perception ability can be characterized by the relative energy of the sound source. The relative energy of the sound source refers to the relative energy of the sound signal emitted by the sound source transmitted to the location of the target object. It can be understood that the transmission process of sound in the medium is accompanied by energy loss. Therefore, when the sound signal emitted by the sound source remains unchanged, the longer the transmission path, the greater the energy loss. Based on this, the server can determine the sound source distance between the sound source position of the sound source and the location of the target object in the ambient space for each sound source. Then, based on the sound source distance corresponding to the sound source, the sound signal of the sound source is conditioned to determine the relative energy of the sound signal of the sound source transmitted to the location of the target object, and then determine the perception ability of the target object represented by the relative energy of the sound source to the sound source.

[0104] In the above embodiment, the sound signal is conditioned based on the distance of the sound source to determine the target object's perception ability of the sound source, so that the perception ability is associated with the position, which is closer to the actual application scenario, thereby improving the accuracy of the foreground sound source and the background sound source determined based on the perception ability and enhancing the stereo sound effect.

[0105] In one embodiment, the server may establish a correspondence between the signal conditioning parameter and the sound source distance. Then, for each sound source, based on the sound source distance corresponding to the sound source and the correspondence between the sound source distance and the signal conditioning parameter, the signal conditioning parameter corresponding to the sound source is determined, and the sound signal of the sound source is signal conditioned based on the signal conditioning parameter to obtain the sound source relative energy of the sound source.

[0106] In one embodiment, based on the sound source distance corresponding to the sound source, signal conditioning is performed on the sound signal of the sound source to determine the perception ability of the target object to the sound source, including: determining a plurality of audio frames contained in the sound signal of the sound source; for each audio frame, based on the sound source sub-distance corresponding to the audio frame, signal conditioning is performed on the frame signal of the audio frame to determine the initial perception ability of the target object to the sound source in the audio frame; from each audio frame, a current audio frame and at least one historical audio frame that meets an interval condition with the current audio frame are determined; and statistical processing is performed on the initial perception abilities corresponding to the current audio frame and the historical audio frames to determine the perception ability of the target object to the sound source in the current audio frame.

[0107] Among them, an audio frame refers to a frame of audio obtained according to the encoding format of the sound signal. For different encoding formats, the length of the audio frame is different. For example, for the AMR (Adaptive Multi-Rate) format, every 20ms of audio is an audio frame. For the mp3 (Moving Picture Experts Group Audio Layer III, dynamic image expert compression standard audio layer 3) format, the number of audio frames is determined by the file size and frame length, and the length of each frame may be fixed or not. In practical applications, continuous audio frames can characterize the correlation between the sound signal and time. Correspondingly, when the sound signal is associated with time, the sound source distance corresponding to the sound source is also associated with time, and the sound source distance can include the sound source sub-distance corresponding to each audio frame. The sound source sub-distance refers to the distance between the sound source and the target object at the time node corresponding to the audio frame. The time node can be determined based on at least one of the start time node, the middle time node or the end time node of the audio frame. It can be understood that if the sound source and the target object remain relatively still, the sound source sub-distance corresponding to each audio frame is the same.

[0108] Specifically, the server can determine the multiple audio frames contained in the sound signal of the sound source, and determine the sound source sub-distance corresponding to each audio frame based on the time node corresponding to each audio frame and the sound source distance corresponding to each time node. Then, for each audio frame, based on the sound source sub-distance corresponding to the audio frame, the frame signal of the audio frame is signal conditioned to determine the initial perception ability of the target object to the sound source under the audio frame. Next, the server determines the current audio frame and at least one historical audio frame that meets the interval condition with the current audio frame from each audio frame, performs statistical processing on the initial perception ability corresponding to the current audio frame and the historical audio frame, and determines the perception ability of the target object to the sound source under the current audio frame. Among them, the interval condition can refer to the interval being less than the frame number threshold, or the interval being less than or equal to the frame number threshold. The frame number threshold can be, for example, 2 frames, 3 frames, etc. The specific algorithm for statistical processing can include at least one of addition, subtraction or multiplication.

[0109] In a specific application, the perception ability of the target object to the sound source in the current audio frame can be determined by weighted smoothing the initial perception abilities of the current audio frame and the historical audio frames. For example, the calculation formula for the perception ability Esm′(i, j) corresponding to the sound source i in the jth frame can be:

[0110] Esm′(i,j)=a*Esm′(i,j-1)+(1-a)*E′(i,j)

[0111] Among them, Esm′(i, j-1) is the perception ability corresponding to the sound source i in the j-1th frame, E′(i, j) is the initial perception ability corresponding to the sound source i in the jth frame, and a is a constant that can be set according to actual needs, for example, a=0.93.

[0112] In the above embodiment, the perceptual capability corresponding to the current audio frame is determined in combination with the perceptual capability corresponding to the historical audio frames, so that the determined perceptual capability is more in line with the actual situation, which is conducive to improving the accuracy of the determined perceptual capability.

[0113] In one embodiment, based on the sound source sub-distance corresponding to the audio frame, signal conditioning is performed on the frame signal of the audio frame to determine the initial perception ability of the target object to the sound source in the audio frame, including: determining the gain value corresponding to the audio frame based on the sound source sub-distance corresponding to the audio frame; and signal conditioning is performed on the frame signal of the audio frame based on the gain value to determine the initial perception ability of the target object to the sound source in the audio frame.

[0114] Among them, gain refers to the degree of increase in current, voltage or power of a component, circuit, device or system. The gain value refers to the amplification factor, that is, the gain value indicates the degree of amplification of the signal. The larger the gain value, the stronger the amplification effect on the signal. For example, the gain value can be set to 2 to the power of 0, 2 to the power of 2 / 3, 2 to the power of 4 / 3, and 2 to the power of 2, etc. The gain value in the embodiment of the present application can be a pre-set fixed gain value, or it can be a calculation method for pre-setting the gain value. The server automatically generates the corresponding gain value according to the calculation method. The calculation method and generation method of the gain value are not limited here.

[0115] Specifically, the server determines the gain value corresponding to the audio frame based on the sound source sub-distance corresponding to the audio frame, and then attenuates or amplifies the frame signal of the audio frame based on the gain value to determine the initial perception ability of the target object to the sound source under the audio frame. There is a mapping relationship between the sound source distance and the gain value between the sound source and the target object, for example, the preset gain value function includes the corresponding relationship between the sound source distance and the gain value.

[0116] Taking a two-dimensional environment space as an example, the distance value between each sound source and the target object can be obtained using the following formula:

[0117]

[0118] Among them, (x(0), y(0)) is the location of the target object, and (x(i), y(i)) is the location of the sound source i.

[0119] Correspondingly, the sound source sub-distance D(i,j) between the sound source i and the target object in the jth frame can be expressed as:

[0120]

[0121] Among them, (x(0,j), y(0,j)) is the position of the target object in the jth frame, and (x(i,j), y(i,j)) is the position of the sound source i in the jth frame.

[0122] After the server calculates the sound source sub-distance based on the above formula, it can input the sound source sub-distance D(i,j) into the preset gain value function to determine the gain value G(i,j) corresponding to the sound source i in the jth frame. Based on the gain value, according to the principle that the sound volume attenuates with distance, the sound signal emitted by the sound source i in the jth frame is attenuated or amplified to obtain the initial perception capability corresponding to the sound source i in the jth frame.

[0123] In a specific implementation, signal conditioning is performed on the frame signal of the audio frame based on the gain value, including: when the sound source sub-distance is greater than the distance threshold, the audio frame is amplified; when the sound source sub-distance is less than the distance threshold, the audio frame is attenuated. For example, assuming that the instantaneous energy of the frame signal of the sound source i in the jth frame is E(i, j), then the initial perception ability corresponding to the sound source i in the jth frame can be obtained by the relative energy E of the sound source in the frame. ′ (i,j) representation, the calculation formula is:

[0124]

[0125] Among them, d 0 is a distance threshold, such as 1 meter, and the relative energy of each sound source is referenced to d 0 Calculated.

[0126] Furthermore, the instantaneous energy can be obtained by filtering and statistically calculating the sound signal. For example, the sound signal of the sound source is processed by high-pass filtering (such as filtering the low frequency 250hz), and the instantaneous energy E(i) of the current frame (for example, 20ms is one frame) is calculated based on the filtered signal:

[0127]

[0128] Here s(i,k) is the sample value of the kth sample point of the sound source i in the jth frame after high-pass filtering.

[0129] In the above embodiment, a gain value is determined based on the sound source sub-distance, and signal conditioning is performed according to the gain value, so that a more accurate initial perception capability can be obtained.

[0130] In one embodiment, step S406 includes: determining the interval of each background sound source in the stereo reconstruction area of ​​the environment space according to the sound source position of each background sound source; performing mixing processing on the sound signals of each background sound source in the same interval to obtain an interval mixed signal corresponding to each interval. Step S408 includes: performing stereo reconstruction on each interval mixed signal to obtain an interval stereo for each interval; superimposing the interval stereo to obtain the background stereo corresponding to each background sound source.

[0131] The center of the stereo reconstruction area is the location of the target object in the environment space. The stereo reconstruction area can be a two-dimensional plane reconstruction area or a three-dimensional stereo reconstruction area, which is not limited here. The shape of the stereo reconstruction area is also not unique. For example, a plane area can be a square area or a three-dimensional area. Figure 7 The circular area shown. The stereo reconstruction area includes multiple intervals, and the space occupied by each interval can be the same or different.

[0132] Specifically, the server can determine the interval where each background sound source is located in the stereo reconstruction area of ​​the ambient space according to the sound source position of each background sound source, and then perform mixing processing on the sound signals of each background sound source in the same interval to obtain an interval mixing signal corresponding to each interval. Then, stereo reconstruction is performed on each interval mixing signal to obtain the interval stereo of each interval. Finally, the background stereo corresponding to each background sound source is obtained by superimposing the interval stereo. For the specific limitations of mixing processing, stereo reconstruction and superimposition processing, please refer to the above, which will not be repeated here.

[0133] In this embodiment, stereo reconstruction of the background sound source is performed between partitions, which can reduce resource overhead while ensuring the sound effect of the reconstructed background stereo sound.

[0134] The specific manner of dividing the stereo reconstruction area into a plurality of intervals is not unique.

[0135] In a specific embodiment, the server may determine a stereo reconstruction area in the environment space with the location of the target object as the center of the area, and divide the stereo reconstruction area into multiple intervals of the same size. Figure 7 As shown, the 360° plane reconstruction area can be equally divided into 8 intervals.

[0136] In a specific embodiment, the server can determine a stereo reconstruction area in the ambient space with the location of the target object as the center of the area, and divide the stereo reconstruction area into multiple intervals according to the distribution of each sound source in the stereo reconstruction area, so that the difference between the number of sound sources contained in each interval is less than a difference threshold.

[0137] In a specific embodiment, the stereo generation method also includes: obtaining the position of the target object in the ambient space and the object posture of the target object; determining a stereo reconstruction area in the ambient space with the position of the target object as the center of the area; and dividing the stereo reconstruction area into multiple intervals based on the object posture.

[0138] The object posture of the target object includes information such as the orientation and posture of the target object. The object posture can usually reflect the target object's perception needs for sound. Based on this, the server can obtain the location of the target object in the environment space and the object posture of the target object; determine the stereo reconstruction area in the environment space with the location of the target object as the center of the area; and divide the stereo reconstruction area into non-uniform intervals based on the object posture to obtain multiple intervals.

[0139] Take the case where the target object is a person as an example. For example, in a sleeping position, the need for sound acquisition is relatively weak. At this time, the stereo reconstruction area can be divided into relatively sparse intervals to minimize the number of intervals, thereby reducing the number of stereo reconstructions. When the head is tilted in a certain direction, it means that the target object has a strong need to perceive the sound in that direction. At this time, the stereo reconstruction area in that direction can be divided into relatively dense intervals to improve the stereo reconstruction effect. For another example, the sound source that the target object is facing is usually the sound source that the target object is relatively more concerned about. The area in front of the target object can be divided into K1 intervals, and the area behind the target object can be divided into K2 intervals, where K1 is greater than K2.

[0140] In this embodiment, the interval division is performed based on the object posture, so that the divided intervals can match the actual needs of the target object, thereby reducing resource overhead while ensuring the stereo reconstruction effect.

[0141] In one embodiment, stereo reconstruction is performed on each interval mixing signal to obtain interval stereo for each interval, including: for each interval, determining the characteristic position of the interval; based on the excitation signal corresponding to the characteristic position, stereo reconstruction is performed on the interval mixing signal of the interval to obtain interval stereo for the interval.

[0142] The characteristic position refers to the position of the orientation information in the stereo reconstruction process. The characteristic position may be, for example, the geometric center position of the interval, or the positions of multiple equal points of the interval, or the acoustic center calculated by weighted summation based on the sound source position and sound source weight of the background sound source in the interval.

[0143] Specifically, for each interval, the server can determine the characteristic position of the interval and determine the excitation signal corresponding to the characteristic position. Then, based on the excitation signal corresponding to the characteristic position, the server performs stereo reconstruction on the interval mixing signal of the interval to obtain the interval stereo of the interval. Take the virtual stereo reconstruction technology based on HRTF as an example. The excitation signal can specifically be HRIR excitation data, which can be found from the relevant data set of HRTF. In the process of stereo reconstruction of the interval mixing signal, the interval stereo of the interval can be obtained by convolving the interval mixing signal with the HRIR excitation data corresponding to the characteristic position. The interval mixing signal includes the mixing signals corresponding to each channel, and correspondingly, the obtained interval stereo includes the audio signals corresponding to each channel.

[0144] In the above embodiment, stereo reconstruction is performed based on the excitation signal corresponding to the feature position, which can reduce the number of stereo reconstruction processing times while ensuring the quality of the reconstructed interval stereo.

[0145] In one embodiment, the number of characteristic positions is at least two. In the case of this embodiment, based on the excitation signal corresponding to the characteristic position, the interval mixed signal of the interval is stereo-reconstructed to obtain the interval stereo of the interval, including: superimposing the excitation signal corresponding to each characteristic position to obtain the interval excitation signal of the interval; based on the interval excitation signal, the interval mixed signal of the interval is stereo-reconstructed to obtain the interval stereo of the interval.

[0146] The specific method of superimposing the excitation signals corresponding to each characteristic position can be linear superposition or nonlinear superposition, which is not limited here. Specifically, after determining the excitation signals corresponding to each characteristic position in the interval, the server can superimpose each excitation signal to obtain a new excitation signal, and use the new excitation signal as the interval excitation signal of the interval. Then, based on the interval excitation signal, the interval mixing signal of the interval is stereo reconstructed to obtain the interval stereo of the interval.

[0147] Exemplarily, multiple equal-dividing point positions within the interval can be used as feature positions, and the HRIR excitation data corresponding to each of these feature positions can be linearly superimposed to obtain interval HRIR excitation data, and HRTF convolution processing can be performed on the interval HRIR excitation data and the interval mixing signal to obtain interval stereo of the interval.

[0148] In the above embodiment, the interval excitation signal is determined by referring to the excitation signals at multiple characteristic positions, which can ensure the accuracy of the determined interval excitation signal and thus improve the sound effect of interval stereo.

[0149] In one embodiment, the excitation signal includes a left channel excitation signal and a right channel excitation signal; the interval mixing signal includes a left channel interval signal and a right channel interval signal. In the case of this embodiment, based on the excitation signal corresponding to the characteristic position, the interval mixing signal of the interval is stereo reconstructed to obtain the interval stereo of the interval, including: based on the left channel excitation signal corresponding to the characteristic position, the left channel interval signal of the interval is convoluted to obtain the left channel signal of the interval; based on the right channel excitation signal corresponding to the characteristic position, the right channel interval signal of the interval is convoluted to obtain the right channel signal of the interval; and the interval stereo including the left channel signal and the right channel signal is determined.

[0150] Specifically, after determining the characteristic position of the interval, the server can search for the excitation signal corresponding to the characteristic position, and the excitation signal includes a left channel excitation signal and a right channel excitation signal. Then, the server performs convolution processing on the left channel interval signal and the right channel interval signal of the interval respectively to obtain the left channel signal and the right channel signal of the interval, and the left channel signal and the right channel signal together constitute the stereo audio signal of the interval stereo.

[0151] In the above embodiment, stereo reconstruction processing is performed between partitions, which can effectively reduce the calculation overhead in the stereo generation process and realize real-time processing.

[0152] In one embodiment, Figure 8 As shown, a stereo reconstruction method is provided, which can be executed by a computer device, and the computer device can be Figure 1 The terminal or server shown, taking the computer device as a server as an example, in this embodiment, the method includes the following steps:

[0153] Step S801, obtaining the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0154] Step S802, for each sound source, determining a sound source distance between a sound source position of the sound source and a position of the target object, and determining a plurality of audio frames included in a sound signal of the sound source;

[0155] Step S803, for each audio frame, based on the sound source sub-distance corresponding to the audio frame, signal conditioning is performed on the frame signal of the audio frame to determine the initial perception ability of the target object to the sound source in the audio frame;

[0156] The sound source distance corresponding to the sound source includes the sound source sub-distances corresponding to each audio frame;

[0157] Step S804, determining, from each audio frame, a current audio frame and at least one historical audio frame that satisfies an interval condition with the current audio frame;

[0158] Step S805, statistically processing the initial perception capabilities corresponding to the current audio frame and the historical audio frames, to determine the perception capability of the target object to the sound source in the current audio frame;

[0159] Step S806, sorting the perception capabilities of the sound sources according to their corresponding perception capabilities;

[0160] Step S807, determining the foreground sound sources whose sorting positions satisfy the position condition and at least two background sound sources whose sorting positions do not satisfy the position condition among the sound sources;

[0161] Step S808, determining the interval where each background sound source is located in the stereo reconstruction area of ​​the environment space according to the sound source position of each background sound source;

[0162] The center of the stereo reconstruction area is the location of the target object in the environment space;

[0163] Step S809, mixing the sound signals of the background sound sources in the same interval respectively to obtain an interval mixed signal corresponding to each interval;

[0164] Step S810, for each interval, determining multiple characteristic positions of the interval, superimposing the excitation signals corresponding to the characteristic positions, and obtaining the interval excitation signal of the interval;

[0165] Step S811, based on the interval excitation signal, performing stereo reconstruction on the interval mixed signal of the interval to obtain the interval stereo of the interval;

[0166] Step S812, superimposing the stereo sounds of each interval to obtain the background stereo sounds corresponding to each background sound source;

[0167] Step S813, performing stereo reconstruction on the sound signal of each foreground sound source to obtain foreground stereo corresponding to each foreground sound source;

[0168] Step S814: performing superposition processing on the foreground stereo sounds and the background stereo sounds to obtain a stereo sound that matches the position of the target object.

[0169] In the above stereo generation method, each sound source is first divided into a foreground sound source and a background sound source through sound source category analysis, and the foreground stereo obtained by stereo reconstruction of the sound signal of the foreground sound source can ensure the clarity of the foreground stereo; for the background sound source, a group stereo reconstruction process is performed, specifically, the sound signals of multiple background sound sources are first mixed to obtain a mixed signal, and then the mixed signal is stereo reconstructed to obtain the background stereo, which can realize the integrated reconstruction of multiple background sound sources, thereby reducing the number of stereo reconstruction processes and reducing the computational overhead in the stereo generation process. The above method can effectively reduce the computational overhead in the stereo generation process and save computing resources without affecting the user's perception experience.

[0170] The present application also provides an application scenario, and the application scenario applies the above-mentioned stereo generation method. Specifically, the application of the stereo generation method in the application scenario is as follows:

[0171] The concept of the metaverse originated from the description of the metaverse by the famous American science fiction master Neal Stephenson in his novel Snow Crash in 1992: "Put on headphones and eyepieces, find the connection terminal, and you can enter the virtual space simulated by the computer and parallel to the real world in the form of a virtual avatar." The realization of the metaverse requires the integration of artificial intelligence virtual perception technologies such as audio, video, and perception to construct a computer virtual space that is close to the perception of the real world. Experiencers only need to use some hardware equipment, such as headphones, glasses, and somatosensory equipment, to experience the same sensory experience as the real world. Among them, virtual space sound effects are a very important part of similar metaverse applications. Through virtual space sound effects, binaural sound signals in a real environment are restored, and experiencers can perceive the stereo sound effect experience in a real environment by wearing headphones.

[0172] Spatial sound effects refer to the sound that has been processed by certain audio technologies to make the user hear a more three-dimensional and spatially layered sound. Through headphones or a combination of two or more speakers, the actual on-site auditory scene is restored, allowing the listener to clearly identify the position, distance and movement trajectory of different acoustic objects, and also allowing the listener to feel surrounded by the sound in all directions, giving the listener an immersive auditory experience as if they were in the actual environment. For example, Figure 6 The spatial sound effects shown include the voices, laughter, footsteps of different people in different directions around, the engine sound of a car approaching from far away, the sound of a pedestrian crossing, and the sound of wind and rain.

[0173] When a user uses an application similar to the metaverse or needs to experience the virtual space sound effect, the above-mentioned stereo generation method can be used, that is, the binaural sound signal in the real environment is restored through the virtual space sound effect, and the experiencer, that is, the user, can perceive the stereo sound effect experience in the real environment by wearing headphones. Specifically, the server obtains the sound source position and sound signal of each of the multiple sound sources in the environment space where the experiencer is located; the sound source category analysis is performed on each sound source to determine the foreground sound source and at least two background sound sources in each sound source; based on the experiencer's position in the environment space and the sound source position of each background sound source, the sound signal of each background sound source is mixed to obtain a mixed signal; the mixed signal is stereo reconstructed to obtain background stereo; the foreground stereo obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo to obtain a stereo matching the position of the experiencer. Then, the server sends the audio signal corresponding to the stereo to the terminal held by the experiencer for decoding and playback, so that the terminal basically does not have too much bandwidth and computing pressure.

[0174] In the process of stereo generation, each sound source is first divided into foreground sound source and background sound source through sound source category analysis, and the foreground stereo obtained by stereo reconstruction of the sound signal of the foreground sound source can ensure the clarity of the foreground stereo; for the background sound source, a group stereo reconstruction process is performed, specifically, the sound signals of multiple background sound sources are first mixed to obtain a mixed signal, and then the mixed signal is stereo reconstructed to obtain the background stereo, which can realize the integrated reconstruction of multiple background sound sources, thereby reducing the number of stereo reconstruction processes and reducing the computational overhead in the stereo generation process. The above method can effectively reduce the computational overhead in the stereo generation process and save computing resources without affecting the user's perception experience.

[0175] Traditional stereo generation methods have obvious technical bottlenecks. When the number of users increases, the computing overhead and network bandwidth consumption are very huge. For example, if there are N participants, the server has N*(N-1) signals to forward, and each user client will receive N-1 signals and perform stereo reconstruction processing and N-1 stereo mixing on the client with 2*(N-1) HRIR convolution calculations per frame. Virtual space applications simulate the audio experience of real scenes. The N value in actual applications is relatively large, which will lead to a large amount of network bandwidth consumption and huge client computing overhead. These will cause data packet loss problems, network delay jitter problems, and non-real-time computing processing problems, which will lead to terminal sound jamming and poor experience. When the number of users grows to a certain number, the existing mixing solution performs linear superposition processing on all sound source signals, which is very likely to cause sound distortion problems. If there are many sound sources, the sound will be noisy, causing discomfort to the listener. This series of problems cannot be well solved by existing solutions.

[0176] The traditional monophonic multi-person voice conference routing and mixing solution is only applicable to monophonic applications, and the sound source has no coordinate information. However, in virtual space multi-stereo applications, the sound source has clear coordinate position information, and the listener's auditory perception is related to the relative distance of the sound source. The closer the sound source is, the stronger the auditory perception is. In addition, the current application is a stereo application, and its sound masking effect is the result of binaural masking. Therefore, it is different from the monophonic application that only considers the sound source sound intensity for judgment, so the monophonic multi-person voice conference routing and mixing solution cannot be applied. In addition, the disadvantage of the routing scheme is that the number of channels selected each time is limited, and the sound signals of the unselected channels cannot be transmitted to the user terminal, that is, they cannot be perceived by the user. With the switching of each routing channel, part of the selected channel signals in the previous selection cycle will be replaced by the new channel, which will inevitably cause sound discontinuity. This problem may not be clearly perceived in mono multi-person conference applications. However, in the virtual stereo world that pursues realism, the stereo changes brought about by this switching are clearly perceived, so the routing scheme cannot be applied in virtual stereo mixing application scenarios that pursue close to real immersive hearing.

[0177] The stereo generation method proposed in the present application divides the multiple sound sources into foreground sound sources and background sound sources according to the perception ability. The foreground sound sources represent the sound sources that the target object pays more attention to, while the background sound sources are the sound sources that the target object is relatively less focused on. Then, the foreground sound sources are processed independently in stereo, and the background sound sources are processed in group stereo. Finally, the front and back sound sources that have been processed in stereo are mixed in stereo to obtain a stereo that matches the position of the target object. The solution of the present application can effectively solve the computational overhead and bandwidth consumption problems of the existing stereo mixing solution. At the same time, the solution of the present invention is in line with the human ear's auditory perception and understanding, and is a low-cost solution that allows users to hear naturally and comfortably and has controllable costs.

[0178] The following takes the case where the environment space is a virtual space as an example. Fig. 9 The stereo generation method of the present application is described in detail.

[0179] In one embodiment, the stereo generation process includes the following steps:

[0180] Step S901, receiving a sound signal and coordinate information of a sound source;

[0181] Step S902, calculating the distance from the sound source to the listener;

[0182] Step S903, calculating the relative energy of the sound source based on the distance;

[0183] Step S904, calculating the historical weighted relative energy of the sound source;

[0184] Step S905, dividing the foreground sound source and the background sound source;

[0185] Step S906, performing independent stereo processing on the foreground sound source;

[0186] Step S907, performing group stereo processing on the background sound source;

[0187] Step S908: stereo mixing.

[0188] Specifically, assume that there are N participants in the current virtual space, that is, N sound sources, each of which carries a corresponding sound signal and sound source position. Among them, the current user (target object) and N-1 other users have different relative distances in the virtual space, and this relative distance can be calculated by a geometric formula:

[0189]

[0190] Here, x0 and y0 are the plane rectangular coordinates of the user, and x(i) and y(i) are the plane rectangular coordinates of other users, and i is the serial number of other users, i=1~N-1.

[0191] The sound source sub-distance D(i,j) between the sound source i and the target object in the jth frame can be expressed as:

[0192]

[0193] Among them, (x(0,j), y(0,j)) is the position of the target object in the jth frame, and (x(i,j), y(i,j)) is the position of the sound source i in the jth frame.

[0194] The sound signals of each sound source are processed by high-pass filtering (such as filtering low frequency 250hz) to obtain a filtered signal, which can be used to calculate the instantaneous energy E(i) of the current frame (for example, 20ms is a frame):

[0195]

[0196] Here, s(i, k) is the sample value of the kth sample point of the sound source i in the current frame after high-pass filtering.

[0197] According to the inverse square law, the sound intensity is inversely proportional to the square of the distance, and the relative energy value of each sound source is calculated by the above distance results. For example, assuming that the instantaneous energy of the frame signal of the sound source i in the jth frame is k(i,j), then the initial perception ability corresponding to the sound source i in the jth frame can be calculated by the relative energy E of the sound source in the frame. ′ (i,j) representation, the calculation formula is:

[0198]

[0199] Among them, d 0 is a distance threshold, such as 1 meter, and the relative energy of each sound source is referenced to d 0 Calculated.

[0200] As calculated above, the current frame relative energy values ​​of the N-1 sound sources heard by the target object are obtained. The energy information of the historical frames is needed to measure the perceived strength of the sound. Therefore, the historical weighted relative energy value is calculated through weighted smoothing. The formula is as follows:

[0201] Esm′(i,j)=a*Esm′(i,j-1)+(1-a)*E′(i,j)

[0202] Among them, Esm ′ (i,j-1) represents the perception ability of sound source i in the j-1th frame, E ′ (i, j) represents the initial perception ability corresponding to the sound source i in the jth frame, and a is a constant that can be set according to actual needs, for example, a=0.93.

[0203] After calculating the historical weighted relative energy values ​​corresponding to each of the N-1 sound sources, the historical weighted relative energies are sorted from large to small, and the top M sound sources will be classified as foreground sound sources, while the remaining sound sources will be classified as background sound sources.

[0204] In the real acoustic world, although our ears can receive many sound signals from different sound sources, there may be only a limited number of them that really attract our attention, such as the foreground sound sources mentioned here. For the foreground sound sources, the human ear can clearly identify the information content and can more accurately identify the direction of the sound source; on the contrary, the background sound sources in the real acoustic world are a kind of foil to the human ear, that is, the human ear can perceive their existence but does not pay much attention to the information content of their sound until the background sound sources switch to the foreground sound sources.

[0205] Based on this, the present application adopts a relatively accurate independent virtual stereo processing method for the foreground sound source, and a group stereo processing method for the background sound source.

[0206] The independent virtual stereo processing method is to process the single sound source mono signal through the HRTF (Head Related Transfer Functions) head-related transfer function. Its input is the mono signal and azimuth information, and the output is a stereo signal. HRTF-based stereo generation is to convolve the original mono input signal u(n) with the HRIR data h(n) of the corresponding azimuth. The HRIR data here can be obtained from the MIT artificial head dataset or the CIPIC real person dataset. The formula for HRTF to calculate the two-channel stereo signal y(n) is as follows:

[0207]

[0208] like Fig.10 As shown, since h(n) includes the HRIR data of the left channel and the right channel, the generated y(n) also corresponds to the left channel and the right channel signal results.

[0209] The group virtual stereo processing method is to classify the background sound sources into a limited number of regional spaces according to their actual positions, such as Figure 7 As shown, it is a representation of a plane interval, dividing the 360-degree plane into 8 intervals. It can be understood that in practical applications, it can also be a three-dimensional space application, and the interval where the sound source is located can be determined according to the interval projected to the horizontal plane. And each background sound source that is finally divided into 8 intervals will be HRTF processed in each interval respectively. Here, the azimuth information used for HRTF processing in a certain interval can be the central azimuth information of the interval, and the corresponding HRIR excitation data is found based on the central azimuth information. It can also be that several azimuth points (usually the average points in the interval) are fixed from the interval, and the HRIR excitation data corresponding to these azimuth points are linearly superimposed as new HRIR data, as interval HRIR excitation data. Then, the HRIR data of the interval and the linear mixing result of the sound source signal divided into the interval are HRTF convolved to obtain the interval stereo corresponding to each interval. Finally, the stereo of each interval is linearly superimposed (i.e., stereo mixing), and the virtual stereo processing result of each background sound source is obtained, i.e., background stereo.

[0210] The stereo mixing mentioned above is obtained by mixing the left channel signal and the right channel signal separately. There are many mixing methods, such as direct addition, averaging, clamping, normalization, adaptive mixing weighting, and automatic alignment algorithm. The following takes the averaging method as an example, which is to add and average the left and right channel signals.

[0211]

[0212]

[0213] Here lout and rout are the left and right channels of the stereo output.

[0214] After the foreground sound source and the background sound source are processed by HRTF respectively, they are linearly superimposed to obtain the final multi-source stereo mixing output.

[0215] In terms of solution deployment, it can be deployed locally, that is, the local client receives the sound signals of all sound sources (usually obtained after decoding) and their respective position information, and then obtains the virtual stereo mixing result through the above processing steps and plays it locally. However, this method will pose a great challenge to the client's packet receiving bandwidth when used on a large scale. Therefore, for large-scale applications, it is recommended to deploy it in the cloud, that is, all users (including virtual sound sources) will converge on the server, and the server will obtain the final mixing result based on the sound source signals that each user should hear, and then send the final stream to the client for decoding and playback after stereo encoding. In this way, the client will basically not have too much bandwidth and computing pressure.

[0216] Although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0217] Based on the same inventive concept, the embodiment of the present application also provides a stereo sound generation device for implementing the above-mentioned stereo sound generation method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more stereo sound generation device embodiments provided below can refer to the above-mentioned limitations on the stereo sound generation method, and will not be repeated here.

[0218] In one embodiment, Fig.11 As shown, a stereo sound generation device 1100 is provided, comprising: an acquisition module 1101, a sound source category analysis module 1102, a mixing module 1103 and a stereo sound reconstruction module 1104, wherein:

[0219] An acquisition module 1101 is used to acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located;

[0220] A sound source category analysis module 1102 is used to perform sound source category analysis on each sound source to determine a foreground sound source and at least two background sound sources in each sound source;

[0221] A mixing module 1103 is used to mix the sound signals of the background sound sources based on the position of the target object in the environment space and the sound source positions of the background sound sources to obtain a mixed signal;

[0222] A stereo reconstruction module 1104 is used to perform stereo reconstruction on the mixed signal to obtain background stereo;

[0223] The stereo sound generation module 1105 is used to perform superposition processing on the foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source and the background stereo sound to obtain a stereo sound matching the position of the target object.

[0224] In one embodiment, the sound source category analysis module 1102 includes: a perception ability determination submodule, which is used to determine the perception ability of the target object to each sound source based on the position of each sound source and the position of the target object in the environmental space; a sound source category determination submodule, which is used to perform sound source category analysis on each sound source according to the perception ability corresponding to each sound source, and determine the foreground sound source and at least two background sound sources in each sound source.

[0225] In one embodiment, the sound source category determination submodule is specifically used to: sort the perceptual capabilities of each sound source according to the perceptual capabilities corresponding to each sound source; determine the foreground sound source whose sorting position meets the position condition, and at least two background sound sources whose sorting positions do not meet the position condition.

[0226] In one embodiment, the perception ability determination submodule includes: a sound source distance determination unit, which is used to determine, for each sound source, the sound source distance between the sound source position of the sound source and the position of the target object in the environmental space; and a perception ability determination unit, which is used to perform signal conditioning on the sound signal of the sound source based on the sound source distance corresponding to the sound source, and determine the perception ability of the target object to the sound source.

[0227] In one embodiment, the perception capability determination unit includes: an audio frame determination component for determining a plurality of audio frames included in the sound signal of the sound source; an initial perception capability determination component for performing signal conditioning on the frame signal of the audio frame based on the sound source sub-distance corresponding to the audio frame for each audio frame, and determining the initial perception capability of the target object to the sound source under the audio frame; a historical audio frame determination component for determining, from each audio frame, the current audio frame and at least one historical audio frame that satisfies the interval condition with the current audio frame; and a statistical component for performing statistical processing on the initial perception capabilities corresponding to the current audio frame and the historical audio frames, and determining the perception capability of the target object to the sound source under the current audio frame. The sound source distance corresponding to the sound source includes the sound source sub-distance corresponding to each audio frame.

[0228] In one embodiment, the initial perception capability determination component is specifically used to: determine the gain value corresponding to the audio frame based on the sound source sub-distance corresponding to the audio frame; perform signal conditioning on the frame signal of the audio frame based on the gain value to determine the initial perception capability of the target object to the sound source in the audio frame.

[0229] In one embodiment, the mixing module 1103 includes: an interval determination unit, which is used to determine the interval of each background sound source in the stereo reconstruction area of ​​the environmental space according to the sound source position of each background sound source; an interval mixing unit, which is used to mix the sound signals of each background sound source in the same interval to obtain an interval mixing signal corresponding to each interval. The stereo reconstruction module 1104 includes: an interval stereo reconstruction unit, which is used to perform stereo reconstruction on each interval mixing signal to obtain the interval stereo of each interval; and a background stereo generation unit, which is used to superimpose the interval stereo to obtain the background stereo corresponding to each background sound source. The center of the stereo reconstruction area is the position of the target object in the environmental space.

[0230] In one embodiment, the stereo generation device 1100 also includes an interval division module, which is used to: obtain the position of the target object in the ambient space and the object posture of the target object; determine the stereo reconstruction area in the ambient space with the position of the target object as the center of the area; and divide the stereo reconstruction area into multiple intervals based on the object posture.

[0231] In one embodiment, the interval stereo reconstruction unit includes: a feature position determination component, used to determine the feature position of each interval; an interval stereo reconstruction component, used to perform stereo reconstruction on the interval mixing signal of the interval based on the excitation signal corresponding to the feature position to obtain the interval stereo of the interval.

[0232] In one embodiment, the number of characteristic positions is at least two. In the case of this embodiment, the interval stereo reconstruction component is specifically used to: superimpose the excitation signals corresponding to each characteristic position to obtain the interval excitation signal of the interval; based on the interval excitation signal, perform stereo reconstruction on the interval mixing signal of the interval to obtain the interval stereo of the interval.

[0233] In one embodiment, the excitation signal includes a left channel excitation signal and a right channel excitation signal; the interval mixing signal includes a left channel interval signal and a right channel interval signal. In the case of this embodiment, the interval stereo reconstruction component is specifically used to: based on the left channel excitation signal corresponding to the feature position, convolve the left channel interval signal of the interval to obtain the left channel signal of the interval; based on the right channel excitation signal corresponding to the feature position, convolve the right channel interval signal of the interval to obtain the right channel signal of the interval; determine the interval stereo including the left channel signal and the right channel signal.

[0234] In one embodiment, the number of foreground sound sources is at least two. In this embodiment, the stereo sound generating device 1100 further includes: a foreground stereo sound reconstruction module, which is used to perform stereo sound reconstruction on the sound signal of each foreground sound source to obtain foreground stereo sounds corresponding to each foreground sound source. The stereo sound generating module is specifically used to: perform superposition processing on each foreground stereo sound and background stereo sound to obtain stereo sound matching the position of the target object.

[0235] Each module in the above stereo sound generation device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0236] In one embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig.12 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a stereo generation method is implemented.

[0237] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.13As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a stereo generation method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.

[0238] Those skilled in the art will understand that Fig.13 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0239] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0240] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0241] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0242] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions. In addition, the subject can choose not to authorize the subject information and related data, or refuse or conveniently refuse to push information, etc.

[0243] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0244] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0245] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A stereo sound generation method, characterized in that: The method comprises: Acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located; Performing sound source category analysis on each of the sound sources to determine a foreground sound source and at least two background sound sources in each of the sound sources; Based on the position of the target object in the environment space and the sound source position of each background sound source, mixing the sound signals of each background sound source to obtain a mixed signal; Performing stereo reconstruction on the mixed signal to obtain background stereo; The foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo sound to obtain a stereo sound matching the position of the target object.

2. The method according to claim 1, characterized in that: The performing sound source category analysis on each of the sound sources to determine a foreground sound source and at least two background sound sources in each of the sound sources includes: Determining the perception capability of the target object to each of the sound sources based on the positions of the sound sources and the position of the target object in the environment space; A sound source category analysis is performed on each of the sound sources according to the perception capabilities corresponding to each of the sound sources, and a foreground sound source and at least two background sound sources are determined among the sound sources.

3. The method according to claim 2, characterized in that The determining, based on the positions of the sound sources and the position of the target object in the environment space, the perception capability of the target object to each of the sound sources comprises: For each of the sound sources, determining a sound source distance between a sound source position of the sound source and a position of the target object in the environment space; Based on the sound source distance corresponding to the sound source, signal conditioning is performed on the sound signal of the sound source to determine the perception ability of the target object to the sound source.

4. The method according to claim 3, characterized in that The step of performing signal conditioning on the sound signal of the sound source based on the sound source distance corresponding to the sound source to determine the perception ability of the target object to the sound source includes: Determining a plurality of audio frames included in the sound signal of the sound source; For each of the audio frames, signal conditioning is performed on a frame signal of the audio frame based on a sound source sub-distance corresponding to the audio frame to determine an initial perception capability of the target object to the sound source under the audio frame; the sound source distance corresponding to the sound source includes a sound source sub-distance corresponding to each of the audio frames; Determine, from each of the audio frames, a current audio frame and at least one historical audio frame that satisfies an interval condition with the current audio frame; Statistical processing is performed on the initial perception capabilities corresponding to the current audio frame and the historical audio frames, respectively, to determine the perception capability of the target object to the sound source in the current audio frame.

5. The method according to claim 4, characterized in that The performing signal conditioning on the frame signal of the audio frame based on the sound source sub-distance corresponding to the audio frame to determine the initial perception capability of the target object to the sound source in the audio frame includes: Determining a gain value corresponding to the audio frame based on a sound source sub-distance corresponding to the audio frame; Signal conditioning is performed on the frame signal of the audio frame based on the gain value to determine the initial perception ability of the target object to the sound source in the audio frame.

6. The method according to claim 2, characterized in that The performing sound source category analysis on each of the sound sources according to the perception capabilities corresponding to each of the sound sources to determine a foreground sound source and at least two background sound sources in each of the sound sources includes: According to the perception capabilities corresponding to the sound sources, the perception capabilities of the sound sources are sorted; A foreground sound source whose sorting position meets the position condition and at least two background sound sources whose sorting positions do not meet the position condition are determined among the sound sources.

7. The method according to any one of claims 1 to 6, characterized in that: The mixing process is performed on the sound signals of the background sound sources based on the position of the target object in the environment space and the sound source positions of the background sound sources to obtain the mixed signal, including: According to the respective sound source positions of the background sound sources, determining the interval where each background sound source is located in the stereo reconstruction area of ​​the environment space; the area center of the stereo reconstruction area is the location of the target object in the environment space; Performing mixing processing on the sound signals of the background sound sources in the same interval respectively to obtain an interval mixed signal corresponding to each interval; The step of reconstructing the mixed signal in stereo to obtain background stereo includes: Reconstructing stereo sound for each of the interval mixed signals to obtain interval stereo sound for each of the intervals; The stereo sounds of the respective intervals are superimposed to obtain the background stereo sounds corresponding to the background sound sources.

8. The method according to claim 7, characterized in that The method further comprises: Acquire the position of the target object in the environment space and the object posture of the target object; Determine a stereo reconstruction area in the environment space with the location of the target object as the area center; The stereo reconstruction region is divided into a plurality of intervals based on the object posture.

9. The method according to claim 7, characterized in that: The step of reconstructing the mixed audio signal of each interval into stereo signals to obtain the stereo signals of each interval includes: For each of the intervals, determining a characteristic position of the interval; Based on the excitation signal corresponding to the characteristic position, stereo reconstruction is performed on the interval mixed signal of the interval to obtain the interval stereo of the interval.

10. The method according to claim 9, characterized in that The number of the characteristic positions is at least two; The step of reconstructing the interval mixed signal of the interval based on the excitation signal corresponding to the characteristic position to obtain the interval stereo of the interval includes: Superimposing the excitation signals corresponding to the characteristic positions to obtain the interval excitation signal of the interval; Based on the interval excitation signal, stereo reconstruction is performed on the interval mixing signal of the interval to obtain the interval stereo of the interval.

11. The method according to claim 9, characterized in that The excitation signal includes a left channel excitation signal and a right channel excitation signal; the interval mixing signal includes a left channel interval signal and a right channel interval signal; The step of reconstructing the interval mixed signal of the interval based on the excitation signal corresponding to the characteristic position to obtain the interval stereo of the interval includes: Based on the left channel excitation signal corresponding to the characteristic position, convolution processing is performed on the left channel interval signal of the interval to obtain the left channel signal of the interval; Based on the right channel excitation signal corresponding to the characteristic position, convolution processing is performed on the right channel interval signal of the interval to obtain the right channel signal of the interval; An interval stereo including the left channel signal and the right channel signal is determined.

12. The method according to claim 1, characterized in that The number of the foreground sound sources is at least two; the method further comprises: Reconstructing the sound signal of each foreground sound source into stereo sound to obtain the foreground stereo sound corresponding to each foreground sound source; The foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source is superimposed with the background stereo sound to obtain a stereo sound matching the position of the target object, including: The foreground stereo sound and the background stereo sound are superimposed to obtain a stereo sound that matches the position of the target object.

13. A stereo sound generating device, characterized in that: The device comprises: An acquisition module, used to acquire the sound source positions and sound signals of multiple sound sources in the environment space where the target object is located; A sound source category analysis module, used to perform sound source category analysis on each of the sound sources to determine a foreground sound source and at least two background sound sources in each of the sound sources; A mixing module, configured to perform mixing processing on the sound signals of each background sound source based on the position of the target object in the environment space and the sound source position of each background sound source to obtain a mixed signal; A stereo reconstruction module, used for performing stereo reconstruction on the mixed signal to obtain background stereo; The stereo sound generation module is used to perform superposition processing on the foreground stereo sound obtained by stereo reconstruction of the sound signal of the foreground sound source and the background stereo sound to obtain the stereo sound matching the position of the target object.

14. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images