Pulse feedback signal generation method, terminal, storage medium and program product

Abstract

The embodiments of the present application disclose a pulse feedback signal generation method, terminal, storage medium, and program product, belonging to the field of audio simulation technology. The method includes: tracking multiple directional sound waves from a sound source and determining the propagation path of each sound wave within a preset space; calculating the pulse signal when the corresponding sound wave reaches the receiver based on the sound pressure information, phase information, and propagation path of the sound wave; synthesizing the pulse signals corresponding to the multiple directional sound waves according to the arrival time to obtain a pulse feedback signal. This method not only simulates the sound pressure loss of the sound wave during propagation, but also simulates the influence of the propagation medium on the phase of the sound wave, which can improve the realism of the sound wave tracking and make the simulated reverberation closer to the real situation.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio simulation technology, and in particular to a pulse feedback signal generation method, terminal, storage medium, and program product. Background Art

[0002] As people's requirements for the realism of sound effects in entertainment scenarios such as games, movies, and television increase, it becomes particularly important to simulate the acoustic characteristics of real physical rooms in virtual rooms to improve the realism of reverberation.

[0003] Currently, in the field of room acoustic simulation, the common practice is to model the room and obtain pulse feedback of the room reverberation. This involves using wave tracing technology to calculate the arrival time and sound pressure of the sound wave from the sound source to the microphone, obtaining a digital pulse signal when the sound wave reaches the microphone. This pulse signal is then generated for each direct or reflected path according to the arrival time, ultimately synthesizing the pulse feedback received by the microphone. Because the pulse signal calculation only considers the sound pressure changes during reflection, while sound waves in real environments are more complex, the calculated pulse signal and pulse signal are inaccurate, significantly reducing the authenticity of the pulse feedback signal. Summary of the Invention

[0004] The present invention provides a method, terminal, storage medium, and program product for generating a pulse feedback signal. The technical solution is as follows:

[0005] In one aspect, an embodiment of the present application provides a method for generating a pulse feedback signal, the method comprising:

[0006] Tracking multiple directional sound waves from a sound source and determining the propagation path of each sound wave within a preset space;

[0007] Calculating a pulse signal when the corresponding sound wave reaches the microphone based on the sound pressure information, phase information and propagation path of the sound wave; the pulse signal includes: the pulse information of the sound wave and the arrival time at the microphone;

[0008] According to the arrival time, the pulse signals corresponding to the multiple directional sound waves are synthesized to obtain a pulse feedback signal; the pulse feedback signal is used to represent: the characteristic change of the reverberation sound reaching the microphone over time.

[0009] On the other hand, an embodiment of the present application provides a pulse feedback signal generating device, the device comprising:

[0010] A first determination module is used to track multiple directional sound waves of a sound source and determine the propagation path of each sound wave in a preset space;

[0011] a calculation module, configured to calculate a pulse signal when the corresponding sound wave reaches the microphone based on the sound pressure information, phase information, and propagation path of the sound wave; the pulse signal includes: the pulse information of the sound wave and the arrival time at the microphone;

[0012] The second determination module is used to synthesize the pulse signals corresponding to the multiple directional sound waves according to the arrival time to obtain a pulse feedback signal; the pulse feedback signal is used to represent: the characteristic change of the reverberation sound reaching the microphone over time.

[0013] On the other hand, an embodiment of the present application provides a terminal, which includes a processor and a memory; the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set are loaded and executed by the processor to implement the pulse feedback signal generation method as described in the above aspects.

[0014] On the other hand, an embodiment of the present application provides a computer-readable storage medium, in which at least one computer program is stored. The computer program is loaded and executed by a processor to implement the pulse feedback signal generation method as described in the above aspects.

[0015] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the terminal to perform the pulse feedback signal generation method provided in various optional implementations of the above aspects.

[0016] The technical solutions provided by the embodiments of the present application include at least the following beneficial effects:

[0017] In an embodiment of the present application, a pulse feedback signal of reverberation in a preset space is synthesized based on sound wave tracking technology. During the sound wave tracking process, not only the sound pressure loss of the sound wave during propagation is simulated, but also the influence of the propagation medium on the phase of the sound wave is simulated. Compared with the method of generating a pulse feedback signal based on the sound pressure of the sound wave in the related art, it avoids the situation where the calculated pulse feedback signal is greatly different from the pulse feedback signal in the real environment due to the default phase consistency of the sound wave emission and the sound wave arrival. It can improve the realism of the sound wave tracking and make the simulated reverberation closer to the real situation. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a schematic diagram of a pulse signal obtained by using a related technology;

[0019] Figure 2A flow chart of a method for generating a pulse feedback signal provided by an exemplary embodiment of the present application is shown;

[0020] Figure 3 A flow chart of a method for generating a pulse feedback signal provided by another exemplary embodiment of the present application is shown;

[0021] Figure 4 is a schematic diagram illustrating an exemplary embodiment of the present application for performing acoustic wave tracking in a preset space;

[0022] Figure 5 is a schematic diagram of sound wave reflection shown in an exemplary embodiment of the present application;

[0023] Figure 6 is a schematic diagram of a pulse signal shown in an exemplary embodiment of the present application;

[0024] Figure 7 A flow chart of a method for generating a pulse feedback signal provided by another exemplary embodiment of the present application is shown;

[0025] Figure 8 A structural block diagram of a pulse feedback signal generating device provided by an exemplary embodiment of the present application is shown;

[0026] Figure 9 A structural block diagram of a terminal provided by an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0028] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0029] In the related art, there are two main types of pulse feedback synthesis technology, one is the sound wave tracing technology similar to ray tracing, and the other is the digital waveguide mesh (DWG) technology. The main idea of ​​the pulse feedback synthesis technology is to estimate the arrival time and arrival sound pressure of the direct sound wave, early reflection sound wave and post-reflection sound wave from the sound source to the receiver, and then convert the arrival sound pressure into a digital pulse signal, and store the digital pulse signal in a pulse sequence according to the arrival time to obtain the final pulse feedback. When using DWG technology, in order to improve the realism of reverberation, it is often necessary to establish a large number of DWG intersections (Digital waveguide mesh junction points) in the virtual room model. For example, for a room model with a size of more than 200 cubic meters, the computer may require more than 32GB of random access memory (RAM) and a computing power of up to 20TFlops. It has high hardware requirements for the device and is usually used in scenes such as film and television post-production and large-scale games that require high reverberation realism. Wave tracing technology, however, has a relatively simple calculation process and requires minimal computation, making it less demanding on the equipment and applicable to users' personal devices. However, because the sound wave signal in conventional wave tracing technology only represents sound pressure, the impulse feedback obtained by using wave tracing techniques similar to ray tracing alone will significantly reduce the realism of the reverberation.

[0030] In order to solve the above technical problems, an embodiment of the present application provides a method for generating a pulse feedback signal. The pulse feedback signal generation method provided in the embodiment of the present application can be applied to terminals such as smart phones, tablet computers, and desktop computers. In one possible implementation, the pulse feedback generation method for reverberation provided in the embodiment of the present application can be implemented as an application or a part of an application and installed in a terminal, so that the terminal has the function of simulating room reverberation. For the convenience of description, in the following embodiments, the pulse feedback generation method for reverberation is applied to a terminal as an example for description, but this does not constitute a limitation.

[0031] The following schematically illustrates the application scenario of the pulse feedback signal generation method provided in the embodiment of the present application.

[0032] 1. Add music effects

[0033] The pulse feedback signal generation method can implement the function of adding sound effects to music in music playback applications. For example, when a user selects a room reverberation sound effect in the sound effect addition function, the terminal performs room modeling based on virtual room parameters. Then, based on the pulse feedback generation method for reverberation provided in this application, it generates pulse feedback when receiving music emitted by a sound source in the virtual room. Then, based on this pulse feedback, reverberation processing is performed on the original audio data, achieving the effect of the user being in the room and listening to music performed by a singer or instrument.

[0034] 2. Sound processing in the virtual room of the game

[0035] In addition to reverberation processing for music, the pulse feedback signal generation method can also be applied to film, television, and gaming scenarios. For example, if a terminal is running a game that supports a three-dimensional virtual environment and a user moves a virtual object controlled by the terminal into a virtual room, the terminal will create a room model based on the virtual room in the game and perform reverberation processing on the terminal's original audio data, creating a reverberation effect that simulates the user being in the virtual room of the game.

[0036] The pulse feedback signal generation method provided in the embodiment of the present application is a method based on the sound wave tracking technology. It has a small amount of calculation and does not require high terminal hardware and data processing capabilities. Therefore, it can be applied to small terminals and various application scenarios. Compared with the sound wave tracking technology in related technologies, it can improve the realism of room reverberation.

[0037] Figure 2 A flow chart of a pulse feedback signal generation method provided by an exemplary embodiment of the present application is shown. This embodiment takes the method applied to a terminal with voice collection and voice recognition functions as an example to illustrate the method, and the method includes the following steps.

[0038] Step 201 : Track multiple directional sound waves of a sound source and determine the propagation path of each sound wave in a preset space.

[0039] Using sound wave tracing technology similar to ray tracing, the terminal tracks virtual sound waves emitted by the sound source and determines the sound wave's propagation path within a preset space. The key principles of sound wave tracing are: sound waves propagate in a straight line within the same medium, change direction at the interface between different media, and some of their energy is absorbed by the medium during propagation.

[0040] The preset space can be a real space or a virtual space. In one possible implementation, the method provided in the embodiment of the present application is mainly applied to room acoustic simulation technology to generate room reverberation. Taking the virtual space as an example, the terminal first performs room modeling to obtain a virtual room, and determines the position of the sound source in the virtual room, which is a virtual sound source. Specifically, the terminal establishes a virtual room model based on the size of the virtual room in a three-dimensional coordinate system, and then determines the coordinates corresponding to the sound source based on the relative position of the sound source and the virtual room, that is, the sound source position, and uses the sound source position as the starting point of sound wave propagation for sound wave tracking. The above-mentioned room parameters include room size, material or reflection coefficient of the virtual reflective surface, temperature, humidity, etc.

[0041] The sound source in the preset space can be one or more. This embodiment of the application uses a virtual room containing one sound source as an example for schematic illustration. The sound waves emitted by the sound source in the preset space can be approximated as spherical waves. The terminal tracks the sound waves emitted by the sound source in multiple directions (for example, according to a preset 10,000 directions) and calculates their propagation path within the preset space.

[0042] Step 202 : Calculate the pulse signal when the corresponding sound wave reaches the microphone based on the sound pressure information, phase information and propagation path of the sound wave.

[0043] The pulse signal includes the pulse information of the sound wave and the arrival time of the sound wave at the receiver.

[0044] When the terminal simulates and calculates the reverberation in the preset space, it is also necessary to set a virtual microphone in the preset space. Taking the preset space as a virtual room as an example, the virtual room is equivalent to a room in the real world, and the sound source in the virtual room is equivalent to the sound source in the room in the real world (for example, a person talking in the room, a speaker playing audio, an object colliding, etc.), and the microphone in the virtual room is equivalent to a person or object in the real world that receives the sound source (for example, a human ear, a microphone, etc.). The pulse feedback signal generation method provided in the embodiment of the present application can be applied to music reverberation processing, stereo sound effects in virtual rooms in games, etc. Therefore, the microphone is equivalent to a human ear in a real environment, and the pulse signal simulates a pulse signal corresponding to one of the sound waves when a person receives a sound wave emitted by a sound source in a room.

[0045] In the real world, when a person receives a sound signal indoors, the sound waves emitted by the sound source can be considered as spherical waves, that is, the sound waves are emitted from the sound source in countless directions. When a sound wave is incident from one medium to another medium with different acoustic properties, reflection will occur at the interface between the two media (such as the wall, floor, and surface of objects in the room), causing part of the energy of the incident sound wave to return to the first medium, while the second medium will absorb the energy of the sound wave, causing the sound pressure and phase of the sound wave to change. Therefore, the sound pressure and phase of the sound wave after reflection will change relative to the sound wave before reflection. Therefore, in order to make the simulated pulse signal closer to the pulse signal of the sound wave received by the human ear in a real environment, in one possible embodiment, the terminal tracks the sound wave based on the sound pressure change and phase change when the sound wave is reflected. When the sound wave propagation path indicates that the sound wave has reached the microphone, the terminal determines the sound wave pulse signal based on the property information of the sound wave collected by the microphone (including sound pressure information and phase information) and the propagation path. The pulse signal contains the pulse information and arrival time of the sound wave, and the pulse information is calculated by fusing the sound pressure information and the phase information.

[0046] Since there is a difference between the attribute information of the sound waves collected by the microphone and the attribute information of the sound waves emitted by the sound source, this difference is generated during the propagation of the sound waves and is related to the propagation path of the sound waves. Therefore, the terminal updates the sound pressure information and phase information of the sound waves during the sound wave tracking process. When the propagation path obtained by tracking indicates that the sound wave arrives at the microphone, the pulse signal is determined based on the sound pressure information and phase information when the sound wave arrives at the microphone.

[0047] Schematically, a pulse signal can represent the arrival time of a sound wave and the amplitude of the sound wave when it reaches the microphone (pulse information). Since the method of the embodiment of the present application integrates the changes in the two acoustic characteristics of sound pressure and phase, the amplitude of the obtained pulse signal is closer to the true value than the pulse signal of the related technology.

[0048] Step 203 : synthesize the pulse signals corresponding to the sound waves in multiple directions according to the arrival time to obtain a pulse feedback signal.

[0049] Among them, the pulse feedback signal is used to represent the characteristic changes of the reverberation sound reaching the microphone over time. A pulse signal calculated in the above steps represents the pulse information and arrival time of the sound wave propagating along a propagation path when it reaches the microphone. Therefore, the terminal synthesizes all the calculated pulse signals to obtain the pulse feedback signal of the reverberation sound collected by the microphone in the preset space, and calculates the reverberation effect of the preset space. Specifically, the terminal uses time as a reference to linearly superimpose the amplitudes corresponding to the same moment to obtain the pulse feedback signal of the reverberation sound.

[0050] In one possible implementation, the terminal controls the virtual microphone to continuously sample at a certain sampling frequency, that is, to receive sound signals within a period of time (sampling duration). According to the above steps, the terminal simulates and calculates the pulse information of the sound waves received by the microphone at each sampling moment, and then generates a pulse signal according to the moment when the sound wave arrives at the microphone. Among them, a pulse signal corresponds to a sound wave that arrives at the microphone along a specific path, so the pulse feedback signal is used to represent the characteristic changes of the sound wave propagating along the same path over time (that is, the amplitude changes).

[0051] Schematically, when simulating room reverberation, the terminal determines at least two target directions, where the target direction refers to the direction in which the sound wave is emitted from the sound source. The terminal determines the sound wave propagation path based on the target direction and the principle of sound wave propagation, where the sound wave propagation paths corresponding to the sound waves emitted in different target directions are different. Finally, the terminal linearly superimposes the pulse signals of each sound wave propagation path to obtain a pulse feedback signal of the room reverberation. The subsequent terminal can perform reverberation processing based on the pulse feedback signal and the sound signal to be processed to simulate the reverberation sound effect of the room.

[0052] To sum up, in the embodiment of the present application, a pulse feedback signal of reverberation in a preset space is synthesized based on sound wave tracking technology. In the process of sound wave tracking, not only the sound pressure loss of the sound wave during propagation is simulated, but also the influence of the propagation medium on the phase of the sound wave is simulated. Compared with the method of generating a pulse feedback signal based on the sound pressure of the sound wave in the related art, it avoids the situation where the calculated pulse feedback signal is greatly different from the pulse feedback signal in the real environment due to the default phase consistency of the sound wave emission and the sound wave arrival. It can improve the authenticity of the sound wave tracking and make the simulated reverberation closer to the real situation.

[0053] During the process of implementing this application, the inventors discovered that when the related technology performs sound wave tracking to generate pulse signals, since only the sound pressure changes of the sound waves are considered, the sound pressure is always quantified by a real number to characterize the acoustic properties of the sound waves, resulting in a large deviation between the final pulse feedback signal and the pulse feedback signal in actual conditions, resulting in poor authenticity of the obtained reverberation sound. In the embodiment of the present application, sound wave tracking not only takes into account the sound pressure changes, but also the phase changes, so it is impossible to simply characterize the acoustic properties of the sound wave by a real number. In one possible implementation, the terminal fuses the sound pressure information and phase information of the sound wave in complex form to quantify the acoustic properties of the sound wave. Figure 3 A flow chart of a pulse feedback signal generation method provided by another exemplary embodiment of the present application is shown. This embodiment takes the method applied to a terminal with voice collection and voice recognition functions as an example to illustrate the method, and the method includes the following steps.

[0054] Step 301: Track multiple directional sound waves of a sound source and determine the propagation path of each sound wave in a preset space.

[0055] The specific implementation of step 301 can refer to the above step 201, and will not be repeated here in this embodiment of the present application.

[0056] Step 302: Obtain a sound wave signal of the sound wave.

[0057] The acoustic wave signal is a complex number in which the sound pressure is represented by the modulus and the phase is represented by the argument.

[0058] In a possible implementation, the initial sound wave signals of the sound waves in each target direction are the same, that is, the sound pressure is the same and the phase difference between each sound wave is 0. The initial sound wave signal refers to the sound wave signal at the sound source.

[0059] Schematically, the acoustic signal of a sound wave is represented by SIG=[real part: Xcos(Q), imaginary part: Xsin(Q)], where X represents the sound pressure and Q represents the phase. The acoustic signals of the sound waves emitted by the same sound source at the same time, that is, the initial acoustic signals of the sound waves are consistent, for example, SIG 初始 =[real part: 1cos(0), imaginary part: 1sin(0)].

[0060] Step 303: Determine the reflection parameters of the reflection surface through which the sound wave passes based on the propagation path of the sound wave.

[0061] The reflection parameters include the sound pressure attenuation coefficient and the phase variation coefficient. Specifically, the reflection parameter is a complex number that represents the degree of sound pressure attenuation using the modulus and the degree of phase variation using the argument. That is, the sound pressure attenuation coefficient is the modulus of the reflection parameter, and the phase variation coefficient is the argument of the reflection parameter.

[0062] When sound waves propagate in a preset space, changes in sound pressure and phase are mainly caused by reflection. The reflecting surface absorbs part of the energy of the sound wave, making the sound pressure of the reflected sound wave lower than that of the incident sound wave, and its phase will also change. Figure 4 As shown, the terminal models a virtual room 401 in a three-dimensional coordinate system and determines the position of the sound source 402 and the position of the microphone 403. Then, at the sound source position of the sound source 402, a sound wave propagating in the target direction is generated. The dotted line in the figure represents the sound wave propagation path, and the arrow represents the sound wave propagation direction. Some sound waves reach the microphone 403 directly without being reflected, and some sound waves reach the microphone 403 after one or more reflections from the virtual reflective surface in the virtual room 401. The figure only shows sound waves emitted along three target directions. In actual application, more target directions may be included. This embodiment of the present application is not limited to this.

[0063] In one possible implementation, when the terminal is tracking the sound source, it updates the sound wave signal while calculating the propagation path. That is, whenever the propagation path reaches the reflecting surface and the sound wave is reflected, the terminal immediately obtains the reflection parameters of the reflecting surface, and then calculates the reflected sound wave signal according to the following steps until the sound wave reaches the receiver.

[0064] Schematically, it is assumed that the room model of the virtual room is a simple shoebox room, that is, there are only 6 virtual reflective surfaces of the virtual room inside, and the reflection parameters corresponding to the 6 virtual reflective surfaces are consistent. The sound wave signal of the sound wave is expressed as SIG = [real part: Xcos(Q), imaginary part: Xsin(Q)], where X represents the sound pressure and Q represents the phase. The reflection parameter of the virtual reflective surface is expressed as REF = [real part: Rcos(P), imaginary part: Rsin(P)], where the sound pressure attenuation coefficient R represents the influence of the virtual reflective surface on the sound pressure, and the phase change coefficient P represents the degree of influence of the virtual reflective surface on the phase, that is, the phase difference between the reflected sound wave and the incident sound wave. The values ​​of R and P are determined by the material of the virtual reflective surface, that is, the user can set them according to their needs. For example, (in the real number domain) the R of the virtual reflective surface is 0.3, the sound pressure of the incident sound wave is 2, then the sound pressure of the reflected sound wave is 2*0.3=0.6.

[0065] Furthermore, in a real environment, when sound waves of different frequency bands are reflected on the same reflective surface, the loss of sound pressure and the degree of phase change are different. In order to make the simulation effect more realistic and the pulse signal closer to the actual data, in one possible embodiment, in the preset space obtained by terminal modeling, the interface reflection parameters of the same reflective surface for sound waves of different frequency bands are different. Step 303 includes the following steps:

[0066] Step 303a: Determine the reflection surface that the sound wave passes through based on the propagation path of the sound wave.

[0067] The terminal calculates the propagation path based on the sound wave's direction of propagation, its starting position within the preset space, and the dimensions of the preset space. If the propagation path indicates that the sound wave reaches an interface between two media, the terminal identifies that interface as a reflecting surface (e.g., the interface between air and a wall in the virtual room).

[0068] Step 303b: Obtain frequency band reflection parameters of the reflection surface for sound waves within a plurality of preset frequency bands as reflection parameters of the sound waves passing through the corresponding reflection surface.

[0069] In one possible implementation, if the sound wave reaches the reflecting surface and has not passed through the microphone before, the terminal continues to track the sound wave and determines interface reflection parameters of the reflecting surface for sound waves of n different frequency bands.

[0070] Schematically, the embodiment of the present application divides the sound waves in the range of 0 Hz to 16000 Hz on the same path into five frequency bands, namely, segment a (0 Hz ~ 3200 Hz), segment b (3200 Hz ~ 6400 Hz), segment c (6400 Hz ~ 9600 Hz), segment d (9600 Hz ~ 12800 Hz) and segment e (12800 Hz ~ 16000 Hz). The interface reflection parameters of the six reflecting surfaces include: REF (segment a) = [real part: R1cos(P1), imaginary part: R1sin(P1)]; REF (segment b) = [real part: R2cos(P2), imaginary part: R2sin(P2)]; REF (segment c) = [real part: R3cos(P3), imaginary part: R3sin(P3)]; REF (segment d) = [real part: R4cos(P4), imaginary part: R4sin(P4)]; REF (segment e) = [real part: R5cos(P5), imaginary part: R5sin(P5)]. If the sound wave passes through the reflecting surface, the terminal determines the reflection parameters as REF (segment a) = [real part: R1cos(P1), imaginary part: R1sin(P1)], REF (segment b) = [real part: R2cos(P2), imaginary part: R2sin(P2)], REF (segment c) = [real part: R3cos(P3), imaginary part: R3sin(P3)], REF (segment d) = [real part: R4cos(P4), imaginary part: R4sin(P4)], REF (segment e) = [real part: R5cos(P5), imaginary part: R5sin(P5)].

[0071] Step 304 : Calculate a pulse signal when the sound wave reaches the microphone based on the sound wave signal and the reflection parameters of the reflection surface passed by the sound wave.

[0072] After simulating a sound source and emitting a sound wave, the terminal calculates the propagation path of the sound wave. When the sound wave reaches the reflecting surface, the terminal calculates the sound wave signal after the first reflection based on the initial sound wave signal and the reflection parameters of the reflecting surface. When the sound wave reflects again, the terminal calculates the sound wave signal after the second reflection based on the sound wave signal after the previous reflection and the reflection parameters of the new reflecting surface. This process continues until the sound wave reaches the microphone after the nth reflection. The terminal then converts the sound wave signal after the nth reflection into a pulse signal. Where n is an integer. When n is 0, it means that the sound wave reaches the microphone directly without any reflection after emitting from the sound source. In this case, the terminal converts the initial sound wave signal into a pulse signal.

[0073] After the terminal simulates and calculates the sound wave signal of the reflected sound wave, it continues to track the reflected sound wave. If the sound wave reaches the reflecting surface again and does not pass through the microphone, it will simulate and calculate again according to the above steps to update the sound wave signal. Figure 5As shown, the incident sound wave is reflected at point O in the virtual reflection surface, and the incident angle is i. According to the principle that the reflection angle α is equal to the incident angle i, the terminal can determine the reflection direction of the reflected sound wave and then continue tracking according to the reflection direction.

[0074] The terminal updates the sound wave signal while calculating the propagation path. In another possible implementation, the terminal may also calculate the complete propagation path of the sound wave from the sound source to the microphone, and then calculate the sound wave signal when the sound wave reaches the microphone based on the initial sound wave signal and the reflection parameters of each reflection surface on the path, and then convert it into a pulse signal.

[0075] Specifically, step 304 includes the following steps:

[0076] Step 304a: Calculate the acoustic wave signal of the reflected acoustic wave based on the acoustic wave signal of the incident acoustic wave and the reflection parameters of the reflecting surface.

[0077] The sound wave signal of the incident sound wave is the sound wave signal after the sound wave passes through the last reflecting surface.

[0078] In the embodiment of the present application, the reflection parameter is a complex number that represents the degree of sound pressure attenuation with the modulus and the degree of phase change with the angle. When the sound wave reaches the reflecting surface, it will be reflected. The terminal calculates the sound wave signal of the reflected sound wave based on the sound wave signal of the incident sound wave and the reflection parameters of the reflecting surface. The sound wave signal SIG of the reflected sound wave 反射 =SIG 入射 *REF, where * indicates complex multiplication.

[0079] For example, if the acoustic wave signal of the incident sound wave is [50cos(60°), 50sin(60°)] and the reflection parameters of the reflecting surface are [0.8cos(20°), 0.8sin(20°)], the terminal calculates the acoustic wave signal of the reflected sound wave as [50cos(60°), 50sin(60°)]*[0.8cos(20°), 0.8sin(20°)], where * represents complex multiplication.

[0080] Furthermore, the terminal divides the sound waves into frequency bands. The same reflecting surface has different interface reflection parameters for sound waves of different frequency bands. Since sound waves are composite waves, meaning that multiple frequencies exist along a single path, the terminal must calculate the sound waves for each frequency band along the propagation path and update the sound wave signals for each frequency band. In other words, the terminal must perform m simulation calculations for a single sound wave reflection, calculating the sound wave signals for each of the m frequency bands in the reflected sound wave based on the sound wave signals of the incident sound wave in each frequency band and the reflection parameters of the reflecting surface for each frequency band.

[0081] Based on the example in step 303b, the sound wave signal SIG of the sound wave in the frequency band a of the reflected sound wave 反射(Section a) = SIG 入射 (segment a)*REF(segment a), the acoustic wave signal SIG of the frequency band b sound wave 反射 (Section b) = SIG 入射 (section b)*REF(section b)...and so on, the sound wave signals of the five reflected sound waves are calculated respectively.

[0082] Step 304b: Process the acoustic wave signal of the reflected acoustic wave reaching the microphone to obtain the pulse amplitude when the acoustic wave reaches the microphone.

[0083] Since the sound wave signal is a complex number signal, the terminal needs to convert it into a pulse signal in the real domain. Specifically, on the one hand, since the inverse real Fourier transform of a single complex number is meaningless, on the other hand, in order to improve the accuracy of the pulse signal, the terminal divides the sound wave into frequency bands during the process of tracking the sound wave and calculating the sound wave signal, and calculates the sound wave signal of each frequency band in each propagation path, and then obtains the pulse signal of the sound wave corresponding to the path based on the sound wave signals of the m frequency bands on the same propagation path. Step 304b includes the following steps:

[0084] Step 1: Combining the sound wave signals corresponding to the sound waves of multiple preset frequency bands in the sound wave in order from low to high frequency to obtain a sound wave spectrum. The sound wave signals corresponding to the sound waves of different frequency bands are different.

[0085] Among them, the initial sound wave signals of sound waves in different frequency bands on the same path are the same. Since the reflection parameters of the same reflection surface for sound waves in different frequency bands are different during reflection, even if the sound waves reach the microphone along the same path, different frequency bands correspond to different sound wave signals.

[0086] Schematically, the sound wave frequency range is divided into five segments: segment a (0 Hz to 3200 Hz), segment b (3200 Hz to 6400 Hz), segment c (6400 Hz to 9600 Hz), segment d (9600 Hz to 12800 Hz), and segment e (12800 Hz to 16000 Hz). The terminal concatenates REF (segment a), REF (segment b), REF (segment c), REF (segment d), and REF (segment e) in ascending frequency order to obtain the sound wave spectrum.

[0087] Step 2: Perform an inverse real number Fast Fourier Transform (IRFFT) on the sound wave spectrum to obtain multiple pulse amplitudes.

[0088] The terminal obtains multiple pulse amplitudes by performing an inverse real number Fourier transform on the sound wave spectrum. These multiple pulse amplitudes are the amplitudes generated by the vibration at the human ear when the terminal simulates the sound wave propagating along a certain path in a preset space and reaches the human ear.

[0089] In one possible implementation, step 2 includes the following steps:

[0090] 1. Perform an inverse real Fourier transform on the sound wave spectrum to obtain multiple candidate pulse amplitudes.

[0091] Since the process of tracking sound waves and calculating pulse signals in the early stage requires a large amount of calculation, if the frequency band of the sound wave is finely divided at the beginning (for example, divided into 100 frequency bands), it will lead to a large amount of calculation for the terminal. For terminals with low processor performance (such as smartphones), it may lead to long processing time and processor jamming. Therefore, in the embodiment of the present application, a rough division of the frequency band is adopted in the early stage, and the sound wave signal corresponding to the fine frequency band is estimated by spline interpolation in the later stage, so as to obtain sound wave signals of more frequency bands, improve the resolution of the spectrum, and make the later pulse feedback more accurate.

[0092] In one possible implementation, the terminal performs spline interpolation on the sound wave spectrum to obtain a target sound wave spectrum, and then performs an inverse real Fourier transform on the target sound wave spectrum to obtain multiple candidate pulse amplitudes. The number of frequency bands in the target sound wave spectrum is greater than the number of frequency bands in the sound wave spectrum, and the sound wave spectrum and the target sound wave spectrum have the same spectral range.

[0093] Schematically, the terminal uses the cubic spline interpolation method to perform spline interpolation processing on the sound wave spectrum, interpolating the real part and the imaginary part respectively. The embodiment of the present application does not limit whether the cubic spline interpolation method is adopted and how many sound wave signals are specifically inserted. Based on experience, the developers use the cubic spline interpolation method to interpolate the 5 complex sound wave signals in the above example to obtain 17 sound wave signals. Other interpolation methods can be used in the actual application stage, or fewer or more sound wave signals can be inserted. In order to facilitate the subsequent inverse Fourier transform to obtain an even number of real numbers, it is recommended here to interpolate to obtain an odd number of sound wave signals. After the terminal performs spline interpolation processing, 17 sound wave signals are obtained based on the sound wave signals of 5 frequency bands, and are combined in the order of frequency bands from low to high to obtain the target sound wave spectrum. The terminal then performs an inverse real number Fourier transform on the target sound wave spectrum to obtain a pulse signal composed of 32 real numbers.

[0094] It's worth noting that in a real Fourier transform, N real numbers are transformed into N complex numbers. However, the last N / 2-1 complex numbers are symmetrical with the first N / 2 complex numbers. Conventional real Fourier transforms and inverse real Fourier transforms omit this symmetrical complex number portion. That is, after converting N real numbers, only N / 2+1 complex numbers are written out, and after converting N / 2+1 complex numbers, only N real numbers are written out. The terminal interpolates the sound wave spectra of the n sound waves that propagate along a certain path and reach the microphone, obtaining a target sound wave spectrum consisting of N complex sound wave signals. The target sound wave spectrum is then inversely transformed to obtain 2N-2 real-domain pulse amplitudes. Specifically, the converted real number represents the amplitude of the composite sound wave. This amplitude is calculated based on a fusion of two acoustic characteristics: the sound pressure change and the phase change. Therefore, compared to the amplitude calculated based on a single acoustic property, sound pressure, in related technologies, it is closer to the true value.

[0095] 2. In response to the sound wave being an early reflected sound wave or a direct sound wave, determining the candidate pulse amplitude as the pulse amplitude.

[0096] Among them, the early reflected sound wave is the sound wave that is reflected less than n times before reaching the microphone, and the direct sound wave is the sound wave that is not reflected before reaching the microphone, and n is a positive integer.

[0097] In room acoustics, the impulse response in the room can be simply divided into direct sound waves, early reflection sound waves and post-reflection sound waves. Among them, direct sound waves refer to sound waves that are emitted by the sound source and directly reach the sound receiver without being reflected; early reflection sound waves are mainly composed of sound waves that reach the sound receiver after a few reflections, usually sound waves that arrive about 50ms to 80ms after the direct sound; post-reflection sound waves are sound waves that reach the sound receiver after more reflections. The embodiments of the present application are mainly used to simulate the human ear's perception of room reverberation, but the human auditory system has a weak perception of post-reflection sound waves. It can only distinguish the total length of the reverberation's post-reflection, and cannot distinguish post-reflection signals in different frequency bands. Therefore, in one possible implementation, in order to further improve the realism of room reverberation and better simulate the human auditory system, the terminal processes the pulse amplitudes corresponding to direct sound waves, early reflection sound waves, and post-reflection sound waves differently.

[0098] For early reflected sound waves and direct sound waves, since the human ear has a strong perception ability, the terminal directly determines the corresponding candidate pulse amplitude as the pulse amplitude.

[0099] 3. In response to the sound wave not being an early reflected sound wave or a direct sound wave, a signal filtering process is performed on the multiple candidate pulse amplitudes to obtain multiple pulse amplitudes.

[0100] For the back-reflected sound waves, since the human ear has a weak perception ability and cannot distinguish sound wave signals in different frequency bands, in order to simulate the auditory effect of the human ear, the terminal performs signal filtering processing on multiple candidate pulse amplitudes to obtain multiple pulse amplitudes.

[0101] Specifically, the process of performing signal filtering on the candidate pulse amplitude of the back-reflected sound wave is as follows:

[0102] In response to the sound wave not being an early reflected sound wave or a direct sound wave, a random sequence is generated, wherein the random sequence includes valid random numbers and invalid random numbers; a plurality of candidate pulse amplitudes are subjected to signal filtering processing (linear multiplication) through the random sequence to obtain a plurality of pulse amplitudes, wherein the candidate pulse amplitudes corresponding to the invalid random numbers are invalid.

[0103] Schematically, the random number in the random sequence is 0 (invalid random number) or 1 (valid random number), the first random number in the random sequence is 1, and the number of random numbers in the random sequence is consistent with the number of candidate pulse amplitudes. Specifically, in order to ensure that the generation probability of 0 and 1 is equal, that is, to ensure that valid random numbers and invalid random numbers appear randomly, the terminal generates an initial random sequence based on the target value range, and the number of random numbers in the initial random sequence is consistent with the number of candidate pulse signals. For 2N-2 candidate pulse amplitudes, the terminal generates an initial random sequence containing 2N-2 random numbers based on the random number value range. The terminal converts the random numbers and the first random number in the initial random sequence that are greater than the random number threshold and the first random number to 1, and converts the non-first random number in the initial random sequence that is less than the random number threshold to 0 to obtain a random sequence.

[0104] For example, the number of candidate pulse amplitudes is 32, and the terminal generates an initial random sequence of length 32, that is, the number of elements of the initial random sequence is 32, and the value range of each element is 0 to 1. Since the terminal needs to ensure that the post-reflection sound wave sequence corresponding to the sampling moment of the microphone is ordered, the first random number in the initial random sequence is converted to 1 regardless of its numerical value. For non-first random numbers less than 0.5, the terminal converts it to 0, and converts other random numbers to 1 to obtain a random sequence. Since the probability of a random number being greater than 0.5 and less than 0.5 is equal, better results can be obtained, making the reverberation closer to the actual situation of human hearing. Users can also select other values ​​according to their needs, and the embodiments of the present application are not limited to this.

[0105] The terminal linearly multiplies the random sequence and the candidate pulse amplitude to obtain the pulse amplitude corresponding to the reflected sound wave. Because the number of invalid random numbers between two consecutive valid random numbers in a random sequence is random, the random sequence can be considered a sequence with random intervals, i.e., an unordered sequence. Linearly multiplying the candidate pulse amplitude by the random sequence converts it into an unordered pulse amplitude, resulting in a similarly unordered pulse signal.

[0106] During the propagation of sound waves in a real environment, on the one hand, when incident at the interface between two media, a portion of the sound waves is absorbed by the incident medium, resulting in a reduction in the energy of the reflected sound waves. On the other hand, a portion of the sound waves in the air will be absorbed by the air, resulting in energy loss. Step 304b also includes the following steps:

[0107] Step three: determining the air absorption coefficient of the preset space to sound waves based on the environmental parameters of the preset space.

[0108] Air absorption, also known as atmospheric sound absorption, refers to the gradual loss of energy in sound waves as they propagate through the atmosphere. Therefore, the terminal must simulate and calculate the pulse amplitude based on the amount of air absorption. This calculation is dependent on the ambient air pressure, temperature, humidity, and frequency of the sound waves. The user can set the environmental parameters of the preset space based on their needs. The frequency can be selected based on the human auditory system, for example, 1 kHz.

[0109] Step 4: Process the sound wave signal of the reflected sound wave reaching the microphone based on the air absorption coefficient to obtain the pulse amplitude.

[0110] In a possible implementation, the terminal calculates the air absorption coefficient A and obtains a candidate pulse amplitude U consisting of real numbers, and then calculates the pulse amplitude U*(10^(-A / 20)).

[0111] Step 304c: Generate a pulse signal based on the pulse amplitude, arrival time, and sampling frequency of the microphone.

[0112] The arrival time corresponds to the first pulse amplitude in the pulse amplitude sequence obtained by inverse real Fourier transform, that is, the arrival time is the time when the sound wave starts to vibrate at the human ear. Specifically, step 304c includes the following steps:

[0113] Step 5: Taking the arrival time as the signal starting time, determine the signal sampling time corresponding to the sampling frequency.

[0114] Step six: Generate a pulse signal based on the correspondence between the multiple pulse amplitudes and the signal starting time and the signal sampling time.

[0115] In one possible implementation, the terminal calculates the arrival time based on the propagation path length and sound speed, and uses this time as the signal start time corresponding to the pulse signal of the sound wave. Based on the signal start time, the terminal determines the subsequent signal sampling times according to the sampling frequency of the microphone. The pulse amplitudes are then added to the time series in chronological order to generate the pulse signal.

[0116] Schematically, the terminal calculates the arrival time of a certain sound wave based on the propagation path and the speed of sound as 5 / 44100 seconds, the sampling frequency of the radio is 44100 Hz, and the sound wave spectrum of the sound wave is converted to obtain 32 pulse amplitudes. The terminal then determines that the start time of the signal is 5 / 44100 seconds, and the signal sampling moments include 31 moments such as 6 / 44100 seconds, 7 / 44100 seconds, and 8 / 44100 seconds, each moment corresponding to a pulse amplitude.

[0117] like Figure 6 As shown in FIG, it shows a graphic representation of a pulse signal received by a microphone, where 1 / 44100s is the arrival time, i.e., the signal start time, and 2 / 44100s, 3 / 44100s, 4 / 44100s, and 5 / 44100s are calculated based on the signal start time and the sampling frequency of the microphone 44100Hz. Since the pulse signal calculated by the method provided in the embodiment of the present application is calculated based on the changes in the two acoustic characteristics of the sound wave, the sound pressure and the phase, compared to Figure 1 The pulse signal generated based on the relevant technology shown in the image has an amplitude closer to the sound wave amplitude of real room reverberation and a more refined division of frequency. Therefore, the terminal can simulate a more realistic room reverberation sound effect based on this pulse signal.

[0118] Step 305 : synthesize the pulse signals corresponding to the sound waves in multiple directions according to the arrival times to obtain a pulse feedback signal.

[0119] The specific implementation of step 305 can refer to the above step 203, and will not be repeated here in this embodiment of the present application.

[0120] In an embodiment of the present application, on the basis of generating pulse feedback based on the two acoustic characteristics of sound pressure and phase, the sound waves are also processed in frequency bands. Considering that the same reflection surface has different degrees of influence on sound waves in different frequency bands, the sound wave signals of sound waves in each frequency band are calculated separately, and then a pulse signal that integrates the acoustic characteristics of multiple frequency bands is obtained, so that the pulse signal is closer to the pulse signal in the real environment, thereby improving the authenticity of the pulse feedback; and, in the sound wave tracking stage, the frequency bands of the sound waves are roughly divided, and in the pulse synthesis stage, interpolation is performed to obtain a more refined frequency band division result, thereby ensuring that the terminal has a small amount of calculation and reducing the terminal power consumption on the basis of improving the authenticity; in addition, the rear reflection signal is disordered with random interval lengths, so that the pulse sequence of the rear reflection signal is closer to the human auditory system's resolution of the rear reflection signal, thereby further improving the authenticity of the reverberation.

[0121] Figure 7 A flow chart of a pulse feedback signal generation method provided by another exemplary embodiment of the present application is shown. This embodiment takes the method applied to a terminal with voice collection and voice recognition functions as an example to illustrate the method, and the method includes the following steps.

[0122] Step 701 : In response to the sound pressure information indicating that the sound pressure of the sound wave is higher than a sound pressure threshold, determining a propagation path of the sound wave in a preset space.

[0123] Since the human ear has limited sensitivity to sound and cannot perceive sound signals with weaker energy, the terminal determines whether to continue tracking the sound wave based on the sound wave signal of the reflected sound wave during the process of tracking the virtual sound wave. When the terminal determines that the sound pressure of the sound wave is higher than the sound pressure threshold during the sound wave tracking process, it continues to track the sound wave. Specifically, when the terminal determines that the sound wave is reflected and the calculated sound pressure of the reflected sound wave is higher than the sound pressure threshold, the terminal continues to track based on the reflection direction to determine the propagation path of the sound wave.

[0124] Users can set a sound pressure threshold based on the human ear's ability to perceive sound and the terminal's quantitative representation of sound pressure. After calculating the sound wave signal of each reflected sound wave, the terminal compares the sound pressure of the sound waves in n frequency bands with the sound pressure threshold. If the sound pressure of a frequency band exceeds the sound pressure threshold, the terminal continues to track the sound wave in that frequency band on that path until the next reflection.

[0125] The specific implementation of the acoustic wave tracking in step 701 can refer to the above-mentioned step 201, and will not be repeated here in this embodiment of the present application.

[0126] Step 702 : In response to the sound pressure information indicating that the sound pressure of the sound wave is lower than the sound pressure threshold during the tracking process, the tracking of the sound wave is stopped.

[0127] Correspondingly, for sound waves with a sound pressure lower than the sound pressure threshold, their energy is too low and the human ear is usually unable to perceive them, so the terminal stops tracking this type of sound wave. Specifically, when the terminal determines that the sound wave is reflected and the calculated sound pressure of the reflected sound wave is lower than the sound pressure threshold, the terminal stops calculating the propagation path of the sound wave, and the pulse signal finally obtained does not contain the pulse signal of the sound wave. Specifically, in response to the sound pressure of a certain frequency band in the reflected sound wave being lower than the sound pressure threshold, the terminal stops tracking the sound wave of this frequency band on the path. If the sound pressure of n frequency bands is all lower than the sound pressure threshold, the terminal stops tracking the sound wave on the path.

[0128] Step 703 : Calculate the pulse signal when the corresponding sound wave reaches the microphone based on the sound pressure information, phase information and propagation path of the sound wave.

[0129] The specific implementation of step 703 can refer to the above step 202, and will not be repeated here in this embodiment of the present application.

[0130] Step 704 : According to the arrival time of the sound waves in each direction, the pulse signals with the same arrival time are linearly superimposed to obtain a pulse feedback signal.

[0131] For the pulse signals of all sound waves received by the microphone, the terminal synthesizes the pulse signals based on time, that is, linearly adds the pulse amplitudes corresponding to the same moment to obtain a pulse feedback signal.

[0132] In an embodiment of the present application, the terminal synthesizes the pulse signal strictly according to the arrival time of the sound wave to generate an accurate pulse feedback signal, thereby ensuring the authenticity of the pulse feedback signal from the perspective of the signal reception time; in the process of sound wave tracking, the terminal stops tracking the sound wave whose sound pressure is lower than the sound pressure threshold based on the human ear's perception of the sound wave. On the one hand, this can further improve the authenticity of the pulse feedback signal, and on the other hand, it can reduce unnecessary calculations and improve the efficiency of pulse feedback signal generation.

[0133] Figure 8 : is a structural block diagram of a pulse feedback signal generating device provided by an exemplary embodiment of the present application, the device comprising:

[0134] The first determination module 801 is used to track the sound source and determine the sound wave propagation path within a preset space;

[0135] A calculation module 802 is configured to determine a pulse signal when the sound wave reaches the microphone based on initial attribute information of the sound wave at the sound source and the sound wave propagation path, wherein the attribute information includes sound pressure information and phase information;

[0136] The second determination module 803 is used to generate a pulse sequence corresponding to the sound wave propagation path based on the arrival time of the sound wave at the microphone and the pulse signal, and the pulse sequence is used to represent the characteristic changes of the sound wave propagating along the same path over time.

[0137] Optionally, the calculation module 802 includes:

[0138] an acquisition unit, configured to obtain a sound wave signal of the sound wave; the sound wave signal is a complex number in which the sound pressure is represented by a modulus and the phase is represented by an argument;

[0139] a first determining unit, configured to determine, based on a propagation path of the sound wave, reflection parameters of a reflection surface through which the sound wave passes; the reflection parameters including a sound pressure attenuation coefficient and a phase variation coefficient;

[0140] The first calculation unit is used to calculate the pulse signal when the corresponding sound wave reaches the microphone based on the sound wave signal of the sound wave and the reflection parameter of the reflection surface passed by the sound wave.

[0141] Optionally, the first determining unit is further configured to:

[0142] Determining, based on a propagation path of the sound wave, a reflection surface through which the sound wave passes;

[0143] Frequency band reflection parameters of the reflection surface for sound waves within a plurality of preset frequency bands are obtained as reflection parameters of the sound waves passing through the corresponding reflection surface.

[0144] Optionally, the first computing unit is further configured to:

[0145] Calculating a sound wave signal of a reflected sound wave based on a sound wave signal of the incident sound wave and the reflection parameter of the reflecting surface, wherein the sound wave signal of the incident sound wave is a sound wave signal of the sound wave after the sound wave passes through the previous reflecting surface;

[0146] performing signal processing on a sound wave signal of the reflected sound wave reaching the microphone to obtain a pulse amplitude when the sound wave reaches the microphone;

[0147] The pulse signal is generated based on the pulse amplitude, the arrival time, and the sampling frequency of the microphone.

[0148] Optionally, the first computing unit is further configured to:

[0149] Combining the sound wave signals corresponding to the sound waves of multiple preset frequency bands in the sound wave in order from low to high frequency to obtain a sound wave spectrum, wherein the sound wave signals corresponding to the sound waves of different frequency bands are different;

[0150] Performing an inverse real Fourier transform on the sound wave spectrum to obtain a plurality of pulse amplitudes;

[0151] Taking the arrival time as the signal starting time, determining the signal sampling time corresponding to the sampling frequency;

[0152] The pulse signal is generated based on the correspondence between the multiple pulse amplitudes, the signal starting time, and the signal sampling time.

[0153] Optionally, the first computing unit is further configured to:

[0154] Performing an inverse real Fourier transform on the acoustic wave spectrum to obtain a plurality of candidate pulse amplitudes;

[0155] In response to the sound wave being an early reflected sound wave or a direct sound wave, determining the candidate pulse amplitude as the pulse amplitude, wherein the early reflected sound wave is a sound wave that is reflected less than n times before reaching the microphone, and the direct sound wave is a sound wave that is not reflected before reaching the microphone, where n is a positive integer;

[0156] In response to the sound wave not belonging to the early reflected sound wave and the direct sound wave, signal filtering processing is performed on the multiple candidate pulse amplitudes to obtain the multiple pulse amplitudes.

[0157] Optionally, the first computing unit is further configured to:

[0158] In response to the sound wave not belonging to the early reflected sound wave and the direct sound wave, generating a random sequence, wherein the random sequence includes valid random numbers and invalid random numbers;

[0159] The signal filtering process is performed on the multiple candidate pulse amplitudes through the random sequence to obtain the multiple pulse amplitudes, wherein the candidate pulse amplitudes corresponding to the invalid random numbers are invalid.

[0160] Optionally, the first computing unit is further configured to:

[0161] determining an air absorption coefficient of the preset space to the sound wave based on environmental parameters of the preset space;

[0162] The signal processing is performed on the sound wave signal of the reflected sound wave reaching the microphone based on the air absorption coefficient to obtain the pulse amplitude.

[0163] Optionally, the first determining module 801 includes:

[0164] a second determining unit, configured to determine a propagation path of the sound wave in the preset space in response to the sound pressure information indicating that the sound pressure of the sound wave is higher than a sound pressure threshold;

[0165] The device further comprises:

[0166] The tracking stopping module is configured to stop tracking the sound wave in response to the sound pressure information indicating that the sound pressure of the sound wave is lower than the sound pressure threshold during the tracking process.

[0167] Optionally, the second determining module 803 includes:

[0168] The second calculation unit is used to linearly superimpose the pulse signals with the same arrival time according to the arrival time of the sound waves in each direction to obtain the pulse feedback signal.

[0169] To sum up, in the embodiment of the present application, a pulse feedback signal of reverberation in a preset space is synthesized based on sound wave tracking technology. In the process of sound wave tracking, not only the sound pressure loss of the sound wave during propagation is simulated, but also the influence of the propagation medium on the phase of the sound wave is simulated. Compared with the method of generating a pulse feedback signal based on the sound pressure of the sound wave in the related art, it avoids the situation where the calculated pulse feedback signal is greatly different from the pulse feedback signal in the real environment due to the default phase consistency of the sound wave emission and the sound wave arrival. It can improve the authenticity of the sound wave tracking and make the simulated reverberation closer to the real situation.

[0170] It should be noted that the apparatus provided in the above embodiments is merely exemplified by the division of the above functional modules. In actual applications, the above functions can be distributed among different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The implementation process is detailed in the method embodiments and will not be repeated here.

[0171] Please refer to Figure 9 , which shows a block diagram of the structure of a terminal provided by an exemplary embodiment of the present application. The terminal 900 can be a smartphone, tablet computer, laptop computer, etc. The terminal 900 in the present application can include one or more of the following components: a processor 910, a memory 920, and a display screen 930.

[0172] The processor 910 may include one or more processing cores. The processor 910 uses various interfaces and lines to connect the various parts of the entire terminal 900. It performs various functions of the terminal 900 and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 920, and calling data stored in the memory 920. Optionally, the processor 910 can be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 910 can integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), a neural network processing unit (NPU), and a modem. Among them, the CPU mainly processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing the content required to be displayed on the touch screen display 930; the NPU is used to implement artificial intelligence (AI) functions; and the modem is used to handle wireless communication. It is understandable that the above-mentioned modem may not be integrated into the processor 910, but may be implemented by a separate chip.

[0173] The memory 920 may include a random access memory (RAM) or a read-only memory (ROM). Optionally, the memory 920 includes a non-transitory computer-readable storage medium. The memory 920 may be used to store instructions, programs, codes, code sets, or instruction sets. The memory 920 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc.; the data storage area may store data created according to the use of the terminal 900 (such as audio data, a phone book), etc.

[0174] The display screen 930 is a display component for displaying a user interface. Optionally, the display screen 930 also has a touch function, through which a user can perform touch operations on the display screen 930 using any suitable object such as a finger or a touch pen.

[0175] The display screen 930 is typically provided on the front panel of the terminal 930. The display screen 930 can be designed as a full screen, a curved screen, a special-shaped screen, a double-sided screen, or a foldable screen. The display screen 930 can also be designed as a combination of a full screen and a curved screen, or a combination of a special-shaped screen and a curved screen, which is not limited in this embodiment.

[0176] In addition, those skilled in the art will understand that the structure of the terminal 900 shown in the above drawings does not constitute a limitation of the terminal 900. The terminal may include more or fewer components than shown, or a combination of certain components, or a different arrangement of components. For example, the terminal 900 also includes a camera assembly, a microphone, a speaker, a radio frequency circuit, an input unit, sensors (such as an accelerometer, an angular velocity sensor, a light sensor, etc.), an audio circuit, a WiFi module, a power supply, a Bluetooth module, and other components, which are not described in detail here.

[0177] An embodiment of the present application further provides a computer-readable storage medium, which stores at least one instruction, and the at least one instruction is loaded and executed by a processor to implement the pulse feedback signal generation method described in the above embodiments.

[0178] According to one aspect of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a terminal reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the terminal to perform the pulse feedback signal generation method provided in various optional implementations of the above aspects.

[0179] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the embodiments of the present application can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable storage medium or transmitted as one or more instructions or codes on a computer-readable storage medium. Computer-readable storage media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0180] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for generating a pulse feedback signal, characterized in that: The method comprises: Tracking multiple directional sound waves from a sound source and determining a propagation path of each sound wave within a preset virtual space; Calculating a pulse signal when the corresponding sound wave reaches the microphone based on the sound pressure information, phase information and propagation path of the sound wave; the pulse signal includes: the pulse information of the sound wave and the arrival time at the microphone; According to the arrival time, the pulse signals corresponding to the multiple directional sound waves are synthesized to obtain a pulse feedback signal; the pulse feedback signal is used to represent: the characteristic change of the reverberation sound reaching the microphone over time; The step of calculating the pulse signal when the sound wave reaches the receiver based on the sound pressure information, phase information, and propagation path of the sound wave includes: Obtaining a sound wave signal of the sound wave, wherein the sound wave signal is a complex number in which the sound pressure is represented by a modulus and the phase is represented by an argument; Determining, based on the propagation path of the sound wave, reflection parameters of a reflection surface through which the sound wave passes, the reflection parameters including a sound pressure attenuation coefficient and a phase variation coefficient; Calculating a sound wave signal of a reflected sound wave based on a sound wave signal of the incident sound wave and the reflection parameter of the reflecting surface, wherein the sound wave signal of the incident sound wave is a sound wave signal of the sound wave after the sound wave passes through the previous reflecting surface; Combining the sound wave signals corresponding to the sound waves of multiple preset frequency bands in the sound wave in order from low to high frequency to obtain a sound wave spectrum, wherein the sound wave signals corresponding to the sound waves of different frequency bands are different; Performing an inverse real Fourier transform on the sound wave spectrum to obtain a plurality of pulse amplitudes; Taking the arrival time as the signal starting time, determining the signal sampling time corresponding to the sampling frequency of the microphone; Based on the correspondence between the multiple pulse amplitudes and the signal starting time and the signal sampling time, a pulse signal is generated when the corresponding sound wave reaches the microphone.

2. The method according to claim 1, characterized in that The determining, based on the propagation path of the sound wave, a reflection parameter of a reflection surface through which the sound wave passes, includes: Determining, based on the propagation path of the sound wave, a reflection surface through which the sound wave passes; Frequency band reflection parameters of the reflection surface for sound waves within a plurality of preset frequency bands are obtained as reflection parameters of the sound waves passing through the corresponding reflection surface.

3. The method according to claim 1, characterized in that The inverse real number Fourier transform is performed on the sound wave spectrum to obtain multiple pulse amplitudes, including: Performing an inverse real Fourier transform on the acoustic wave spectrum to obtain a plurality of candidate pulse amplitudes; In response to the sound wave being an early reflected sound wave or a direct sound wave, determining the candidate pulse amplitude as the pulse amplitude, wherein the early reflected sound wave is a sound wave that is reflected less than n times before reaching the microphone, and the direct sound wave is a sound wave that is not reflected before reaching the microphone, where n is a positive integer; In response to the sound wave not belonging to the early reflected sound wave and the direct sound wave, signal filtering processing is performed on the multiple candidate pulse amplitudes to obtain the multiple pulse amplitudes.

4. The method according to claim 3, characterized in that In response to the sound wave not belonging to the early reflected sound wave and the direct sound wave, performing signal filtering processing on the multiple candidate pulse amplitudes to obtain the multiple pulse amplitudes includes: In response to the sound wave not belonging to the early reflected sound wave and the direct sound wave, generating a random sequence, wherein the random sequence includes valid random numbers and invalid random numbers; The signal filtering process is performed on the multiple candidate pulse amplitudes through the random sequence to obtain the multiple pulse amplitudes, wherein the candidate pulse amplitudes corresponding to the invalid random numbers are invalid.

5. The method according to any one of claims 1 to 4, characterized in that: The method comprises: determining, based on environmental parameters of the preset virtual space, an air absorption coefficient of the preset virtual space to the sound wave; The plurality of pulse amplitudes are obtained by performing signal processing on the sound wave signal of the reflected sound wave reaching the microphone based on the air absorption coefficient.

6. The method according to any one of claims 1 to 4, characterized in that: Tracking multiple directional sound waves of a sound source and determining a propagation path of each sound wave in a preset virtual space includes: In response to the sound pressure information indicating that the sound pressure of the sound wave is higher than a sound pressure threshold, determining a propagation path of the sound wave in the preset virtual space; The method further comprises: In response to the sound pressure information indicating that the sound pressure of the sound wave is lower than the sound pressure threshold during the tracking process, the tracking of the sound wave is stopped.

7. The method according to any one of claims 1 to 4, characterized in that: The step of synthesizing the pulse signals corresponding to the sound waves in the multiple directions according to the arrival time to obtain the pulse feedback signal includes: According to the arrival time of the sound waves in each direction, the pulse signals with the same arrival time are linearly superimposed to obtain the pulse feedback signal.

8. A pulse feedback signal generating device, characterized in that: The device comprises: A first determination module is configured to track multiple directional sound waves of a sound source and determine a propagation path of each sound wave in a preset virtual space; a calculation module, configured to calculate a pulse signal when the corresponding sound wave reaches the microphone based on the sound pressure information, phase information, and propagation path of the sound wave; the pulse signal includes: the pulse information of the sound wave and the arrival time at the microphone; The second determining module is configured to synthesize the pulse signals corresponding to the multiple directional sound waves according to the arrival time to obtain a pulse feedback signal; the pulse feedback signal is used to represent the characteristic change of the reverberation sound reaching the receiver over time; The calculation module is specifically configured to: obtain a sound wave signal of the sound wave, wherein the sound wave signal is a complex number in which the sound pressure is represented by the modulus and the phase is represented by the argument; Determining, based on the propagation path of the sound wave, reflection parameters of a reflection surface through which the sound wave passes, the reflection parameters including a sound pressure attenuation coefficient and a phase variation coefficient; Calculating a sound wave signal of a reflected sound wave based on a sound wave signal of the incident sound wave and the reflection parameter of the reflecting surface, wherein the sound wave signal of the incident sound wave is a sound wave signal of the sound wave after the sound wave passes through the previous reflecting surface; Combining the sound wave signals corresponding to the sound waves of multiple preset frequency bands in the sound wave in order from low to high frequency to obtain a sound wave spectrum, wherein the sound wave signals corresponding to the sound waves of different frequency bands are different; Performing an inverse real Fourier transform on the sound wave spectrum to obtain a plurality of pulse amplitudes; Taking the arrival time as the signal starting time, determining the signal sampling time corresponding to the sampling frequency of the microphone; Based on the correspondence between the multiple pulse amplitudes and the signal starting time and the signal sampling time, a pulse signal is generated when the corresponding sound wave reaches the microphone.

9. A terminal, characterized in that: The terminal includes a processor and a memory; the memory stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the pulse feedback signal generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the computer program is loaded and executed by the processor to implement the pulse feedback signal generating method according to any one of claims 1 to 7.

11. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium; the processor of the terminal reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the terminal executes the pulse feedback signal generation method as described in any one of claims 1 to 7.

Images