A sound source localization system and method in a virtual environment

Abstract

The embodiments of the present invention provide a sound source localization system and method in a virtual environment, which are applied to the sound source localization of an intelligent body in a virtual environment. The sound source simulation module is used to simulate and generate sound signals in a virtual environment, and simulate the propagation of sound signals in the virtual environment; the sound source cache module is used to set multiple buffers, cache the sound signals generated by the sound source simulation module, and set the frequency at which the sound signals are sent to the sound source localization module for processing; the sound source localization module is used to detect the location of the sound source according to the sound signal sent by the sound source cache module, and perform sound source integration. The present invention designs a unified system architecture to support the hearing ability and sound source localization ability of virtual people, and can adapt to different sound source localization algorithms by designing a highly versatile interface. In addition, a buffer is used to cache sound fragments, increase the data length processed by the sound source localization algorithm at a single time, and improve the accuracy of the positioning algorithm.

Description

Technical Field

[0001] The present invention relates to virtual human technology, and in particular to a sound source localization system and method for a virtual human in a virtual environment. Background Art

[0002] Virtual intelligent people are those who use computer software to simulate the real physical world and establish an intelligent body in it. This intelligent body has the same abilities as people in real life and can sense various sound, light, and electrical signals and interact with the environment. These abilities are realized by various complex systems and algorithms.

[0003] However, existing virtual human technology mainly focuses on the image display and voice interaction of virtual humans, and lacks perception of environmental information such as sound sources. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a sound source localization system and method in a virtual environment, so as to solve the problem that the existing technical solutions cannot realize that the virtual human cannot perceive environmental information such as the sound source in the virtual environment.

[0005] In order to solve the above technical problems, the present invention discloses a sound source localization system in a virtual environment, which is applied to the sound source localization of an intelligent body in a virtual environment, and includes a sound source simulation module, a sound source cache module and a sound source localization module, wherein: the sound source simulation module is used to simulate and generate sound signals in the virtual environment, and simulate the propagation of sound signals in the virtual environment; the sound source cache module is used to set multiple buffers, cache the sound signals generated by the sound source simulation module, and set the frequency at which the sound signals are sent to the sound source localization module for processing; the sound source localization module is used to detect the sound source position according to the sound signals sent by the sound source cache module, and perform sound source integration.

[0006] In one embodiment, the sound source simulation module is used to simulate and generate sound signals in a virtual environment and simulate the propagation of sound signals in the virtual environment, including: in the virtual environment, treating sound waves as rays emitted from a point sound source, and simulating and modeling the direct radiation, reflection, diffraction and absorption phenomena of the sound waves to simulate the propagation of sound waves in the virtual environment.

[0007] In one embodiment, the multiple buffers in the sound source cache module are a queue. When the sound segment reaches the set threshold value n, it is sent to the sound source localization module for processing, and the k sound segments at the head of the queue are discarded. When the number of sound segments in the queue reaches the set threshold value n again, it is sent to the sound source localization module for processing again, and the k sound segments at the head of the queue processed this time are discarded again, and continuous processing is performed according to n and k; by adjusting k, the frequency of sending to the sound source localization module for processing is controlled.

[0008] In one embodiment, the sound source localization module is used to detect the sound source position according to the sound signal sent by the sound source buffer module, including: ; Wherein, data is the sound data sent by the sound source cache module, azi, ele, dist are the horizontal angle, pitch angle and distance of the current positioning relative to the intelligent body, respectively, , , model is the sound source localization model.

[0009] In one implementation, the sound source localization module is further used to further calculate the sound source coordinates of the world coordinate system, including:

[0010] The current position of the agent in the world coordinate system is , the orientation relative to the world coordinate system is ,in ;

[0011] The position of the sound source in the world coordinate system is calculated based on the coordinate information of the intelligent agent and the output result of the sound source localization model. ;

[0012] for ,have:

[0013] ;

[0014] To calculate , , first calculate the projection of the distance dist on the horizontal plane:

[0015] ;

[0016] Calculate the direction of the sound source in the world coordinate system based on the direction of the agent:

[0017] ;

[0018] Calculate the world coordinates of the sound source , :

[0019] ;

[0020] ;

[0021] The world coordinates of the sound source can be obtained .

[0022] In one embodiment, the sound source localization module is further used to determine whether the sound source is an existing sound source and perform sound source integration, including: assuming that a sound source has been detected m The location of the sound source , given a threshold , if it exists So that:

[0023]

[0024] The detected sound source is considered to be For the same sound source, represents the two-norm, which is the Euclidean distance between two points when applied to the above coordinates;

[0025] If there is no such j , then the detected sound source Add to the list of sound sources.

[0026] In order to solve the above technical problems, the present invention also discloses a sound source localization method in a virtual environment, which is applied to the sound source localization of an intelligent agent in a virtual environment, including: simulating and generating sound signals in the virtual environment, and simulating the propagation of sound signals in the virtual environment; setting multiple buffers to cache the simulated sound signals and setting the frequency for processing the sound signals; detecting the sound source position according to the sound signals sent from the cache, and performing sound source integration.

[0027] Another aspect of an embodiment of the present invention provides an electronic device, including one or more processors and a storage device, wherein the storage device is used to store executable instructions, and when the executable instructions are executed by the processor, the method of the embodiment of the present invention is implemented.

[0028] Another aspect of an embodiment of the present invention provides a computer program, wherein the computer program includes computer executable instructions, and the instructions are used to implement the method of the embodiment of the present invention when being executed.

[0029] The present invention implements a system and method for supporting a virtual person to determine the location of a sound source in a virtual world. A unified system architecture is designed to support the virtual person's hearing ability and sound source localization ability. By designing a highly versatile interface, different sound source localization algorithms can be adapted. In addition, a buffer is used to cache sound fragments, increase the data length processed by the sound source localization algorithm at a time, and improve the accuracy of the localization algorithm. In order to allow users to visually perceive where the sound is coming from, the system will maintain the sound source location for a period of time and perform sound source integration. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings below are part of the specification of the present invention, and illustrate exemplary embodiments of the present invention. Together with the description of the specification, the accompanying drawings are used to explain the principles of the present invention.

[0031] Figure 1 A schematic diagram of the structure of a sound source localization system for a virtual person in a virtual environment according to an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of setting a buffer zone according to an embodiment of the present invention;

[0033] Figure 3 The present invention is a flowchart of a method for localizing a sound source of a virtual human in a virtual environment according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0035] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0036] The terms “first”, “second”, etc. used in this document do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0037] The directional terms used in this article, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention. The terms "include", "including", "have", "contain", etc. used in this article are open terms, which means including but not limited to.

[0038] As used herein, "and / or" includes any or all combinations of the items described.

[0039] As used herein, “plurality” includes “two” and “more than two”; as used herein, “plurality groups” includes “two groups” and “more than two groups”.

[0040] The terms "substantially" and "approximately" used herein are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0041] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0042] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0043] Current sound source localization technology mainly focuses on the real physical world, such as collecting sound data through microphone arrays and then positioning. The goal of sound source localization is to determine the location of a sound source. The location information of the sound source includes axial angle, elevation angle and distance. The axial angle can determine the two-dimensional direction of the sound source, the axial angle and elevation angle can determine the three-dimensional direction of the sound source, and the axial angle, elevation angle and distance can determine the three-dimensional position of the sound source.

[0044] The present invention is based on the background of realizing virtual intelligent people (hereinafter referred to as virtual people or intelligent bodies). The so-called virtual intelligent people are to simulate the real physical world through computer software, and establish an intelligent person in it. This intelligent person has the same abilities as people in real life, can feel various sound, light, and electrical signals, and interact with the environment. These abilities are realized by various complex systems and algorithms. The present invention is to realize a method and system for supporting virtual people to judge the location of sound sources in the virtual world. The goal is to design a unified architecture to support the sound source localization ability of virtual people, and to adapt to different sound source localization algorithms by designing highly versatile interfaces.

[0045] Figure 1 FIG. 1 is a schematic diagram of a sound source localization system for a virtual person in a virtual environment according to an embodiment of the present invention. Figure 1 As shown, the sound source localization system of the embodiment of the present invention includes a sound source simulation module 101, a sound source buffer module 102 and a sound source localization module 103. The sound source simulation module 101 is used to simulate and generate sound signals in a virtual environment and simulate the propagation of sound signals in the virtual environment; the sound source buffer module 102 is used to set multiple buffers, buffer the sound signals generated by the sound source simulation module, and set the frequency at which the sound signals are sent to the sound source localization module for processing; the sound source localization module 103 is used to detect the sound source position according to the sound signals sent by the sound source buffer module and perform sound source integration.

[0046] First, consider a scenario where humans respond to sound: when a person hears a cup fall to the ground and break, the person will look at where the cup fell. In a virtual environment, the above process is simulated, and the sound information is sent from the sound source simulation module 101 to the sound source localization module 103. After the sound source localization module 103 detects the sound source location, it sends it to a downstream module that needs the information, such as a module that simulates the brain. The module issues a command to look at this location based on the sound source location information.

[0047] The purpose of sound simulation is to simulate real acoustic phenomena in a virtual environment, so that the virtual agent can "realistically hear" the sounds emitted in the virtual environment as much as possible. Therefore, in addition to generating sound source data, the sound source simulation module 101 also simulates the propagation of sound in the virtual space.

[0048] In the embodiment of the present invention, in order to simulate the propagation of sound in a virtual space, a ray tracing method is used. This method is based on the principle of geometric optics, regards sound waves as rays emitted from a point sound source, and models the reflection, diffraction and absorption phenomena of sound waves, which can better simulate the propagation of sound in a closed space. The first step of ray tracing is to start from the sound source and emit a large number of rays in three-dimensional space. These rays represent the directions and paths that sound waves may propagate in the room. These rays will reflect when they encounter obstructions such as walls and floors in the virtual environment, and continue to propagate, and multiple reflections may occur.

[0049] In the above modeling process, the following two situations are mainly considered:

[0050] 1. Direct sound. If there is no obstruction between the listener and the sound source, and there is a ray that can reach the listener directly, then this ray represents the direct sound. The attenuation of direct sound is mainly due to air absorption. The attenuated sound heard by the listener can be calculated based on the position of the sound source and the listener.

[0051] 2. Reflected rays. If the listener is not on the direct propagation path of the ray, it means that the ray needs to be reflected. According to the law of reflection, the direction and angle after reflection can be calculated. The reflected ray may be reflected again until it reaches the listener after a certain reflection. The energy of the ray after reflection will decay, which is related to the properties of the reflective material, such as absorptivity, scattering rate, etc. After multiple reflections, the energy of the ray may decay to a very small value. Usually a threshold is set, and when the energy is less than the threshold, it will no longer be considered.

[0052] The sound that the listener finally receives is the sum of all the original rays or reflected rays that can reach the listener directly. Based on all the ray information, the binaural room impulse response (BRIR) can be calculated. BRIR describes the impulse response from the sound source to the listener, including the information of the direct sound and all the reflected sounds. According to the arrival time and number of reflections of the ray, BRIR is usually divided into two parts: early reflections and late reverberation. Among them, early reflections are rays with fewer reflections and shorter arrival times, which mainly affect the clarity and spatial sense of the sound; late reverberation is a ray with more reflections and longer arrival times, which mainly affects the reverberation and environmental sense of the sound. After integrating the direct sound, early reflections and late reverberation together, the complete BRIR signal is obtained. After convolving the complete BRIR signal with the original audio number, the propagation and reflection of the sound in the room are simulated, and an audio signal with room acoustic characteristics is generated, which will be output to the listener as the final rendering result.

[0053] In actual engineering practice, due to the large number of late reverberation reflections and the large amount of calculation, in order to meet the real-time requirements, the late reverberation BRIR will be generated in advance for the specified scene, and real-time query will be performed based on the sound position and the listener position during operation; for direct sound and early reflections, online calculation is used. The generated BRIR signal and the original sound data are fused in the rendering module, and the fused result is sent to the player for playback.

[0054] The above is an implementation method of the sound source simulation module 101 of the present invention to simulate and generate sound signals in a virtual environment and simulate the propagation of sound signals in the virtual environment. After obtaining the sound source data, the embodiment of the present invention also uses the sound source cache module 102 to set a buffer to cache sound fragments, increase the data length processed by the sound source localization algorithm at a single time, so as to improve accuracy.

[0055] In the embodiment of the present invention, in the current sound source simulation module 101, the sound data of each sound source is independent, so N sound sources require N buffers. The current sound transmission protocol used is webrtc, and the sound data segment of the protocol is 10ms, and the sound source localization algorithm requires the input sound segment to reach a certain length to be processed correctly, so it is necessary to buffer the data sent by the sound source simulation module 101 and send it to the sound source localization module 103 after reaching a certain length.

[0056] In a specific implementation, the buffer is set as follows Figure 2 As shown in the figure, the entire buffer is a queue. When the number of sound clips reaches a given threshold (n), it is processed once and the k data at the head of the queue are discarded. When the number of sound clips in the queue reaches n again, it is processed a second time and the k data at the head of the queue are discarded. And so on, continuous processing is performed according to the values ​​of n and k. Figure 2 The first and second processed sound clips are shown in Figure 2 when k=2. Figure 2 It can be seen that by adjusting k, the frequency at which the sound clips are sent to the sound source localization module 103 for processing can be controlled. By reducing the value of n or k, the processing frequency can be accelerated, and conversely, by increasing the value of n or k, the processing frequency can be slowed down.

[0057] After receiving the sound data sent by the sound source buffer module 102, the sound source localization module 103 will perform sound source localization and sound source integration according to the existing sound source localization model.

[0058] In one embodiment of the present invention, the sound source localization model may be:

[0059]

[0060] Wherein, data is the sound data sent by the sound source buffer module 102, that is, a piece of binary audio data in PCM format, azi, ele, and dist are the horizontal angle, pitch angle, and distance of the current positioning relative to the intelligent body, respectively. , model is the sound source localization model.

[0061] When the present invention is implemented, the specific positioning algorithm of the sound source localization model is not limited. As long as the sound source localization algorithm can meet the input and output of the system architecture, it can be connected to the system designed by the present invention.

[0062] In the embodiment of the present invention, based on the output result of the sound source localization model and based on the current position and orientation of the agent, the sound source coordinates in the world coordinate system can be calculated in the following manner.

[0063] It is known that the current position of the intelligent agent in the world coordinate system is , the orientation relative to the world coordinate system is ,in ;

[0064] Based on the above information of the agent and the output of the sound source localization model, the position of the sound source in the world coordinate system is calculated. .

[0065] for ,have:

[0066] ;

[0067] To calculate , , first calculate the projection of the distance dist on the horizontal plane:

[0068] ;

[0069] Calculate the direction of the sound source in the world coordinate system based on the direction of the agent:

[0070] ;

[0071] Calculate the world coordinates of the sound source , :

[0072] ;

[0073] ;

[0074] The world coordinates of the sound source can be obtained .

[0075] In the embodiment of the present invention, the world coordinates of the identified sound source may fluctuate within a very small range, resulting in the existence of many sound sources at the same time, so it is necessary to integrate the sound sources with close positions into one point. Each time a sound source is received, it will be compared with the existing sound sources. If the distance is less than a certain threshold, it is considered that the sound source represents an existing sound source, otherwise it is added to the list of known sound sources.

[0076] In the specific implementation, it is assumed that m The location of the sound source , given a threshold , if it exists So that:

[0077]

[0078] It is considered that the detected sound source and are the same sound source, where represents the two-norm, that is, applied in the above coordinates, it is the Euclidean distance between the two points;

[0079] If there is no such j , then the detected sound source Add to the list of sound sources.

[0080] In the sound source localization system of the virtual environment of the embodiment of the present invention, the sound source cache module 102 and the sound source localization module 103 are both receiving threads, which are used to receive and process the sound source data simulated and generated in the virtual environment. After generating the sound source coordinates and the sound source list, they enter the sending thread of the system.

[0081] In some embodiments, the receiving thread workflow is as follows: accept the current sound clip and cache it. If the length of the current sound clip exceeds the set time threshold, it is input into the sound source localization algorithm for positioning, and the earliest received sound clip is popped up at the same time. The sound source localization algorithm performs sound source localization and outputs the pitch angle, horizontal angle and distance; based on the output result of the sound source localization, based on the current position and orientation of the intelligent person, a coordinate point is obtained, and the distance between the coordinate point and all known coordinate points is calculated. If it is less than the set distance threshold, it is considered to be an existing sound source, otherwise it is added to the known sound source list.

[0082] The sending thread continuously polls the list of known sound sources and sends them to downstream modules. For example, a module that simulates the brain can issue instructions to look at the location based on the location of the sound source, or trigger changes in the body movements of the virtual human.

[0083] In the embodiment of the present invention, the entire system is a distributed system, and each module can be an independent server or client, or both a server and a client, or multiple modules can be on one server. There are many kinds of sound source localization algorithms, and the principles of each are different. As part of the perception ability of virtual people, this system has designed a highly abstract architecture to support the hearing ability of intelligent people. As long as the sound source localization algorithm can meet the input and output of the system architecture, it can be connected to the system. In addition, the duration of a sound is very short, such as a broken cup. In order to allow the user to visually perceive where the sound comes from, the system will maintain the sound source location for a period of time, such as 1 second.

[0084] Based on the same concept, after introducing the sound source localization system of the virtual environment of the exemplary embodiment of the present invention, next, refer to Figure 3 The method of the exemplary embodiment of the present invention is introduced. The implementation of the method can refer to the implementation of the above-mentioned system, and the repeated parts are not repeated. The term "module" or "unit" used in the sound source localization system of the virtual environment described in the present invention can be software and / or hardware that implements the predetermined function. Although the modules described in the above-mentioned system embodiment are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0085] Figure 3 FIG. 1 is a processing flow chart of a method for localizing a sound source of an intelligent agent in a virtual environment according to an embodiment of the present invention. Figure 3 As shown, it includes: step S301, simulating the generation of sound signals in a virtual environment, and simulating the propagation of the sound signals in the virtual environment; step S302, setting multiple buffers, caching the simulated sound signals, and setting the frequency for processing the sound signals; step S303, detecting the sound source position according to the sound signals sent from the cache, and performing sound source integration.

[0086] In specific implementation, in step S301, in a virtual environment, sound waves are regarded as rays emitted from a point sound source, and simulation modeling is performed on the direct radiation, reflection, diffraction and absorption phenomena of the sound waves to simulate the propagation of the sound waves in the virtual environment.

[0087] In the specific implementation, in step S302, multiple buffers are set as a queue. When the sound clip reaches the set threshold n, it is sent for processing, and the k sound clips at the head of the queue are discarded. When the number of sound clips in the queue reaches the set threshold n again, it is sent again, and the k sound clips at the head of the queue processed this time are discarded again. Continuous processing is performed according to n and k; k is adjusted to control the frequency of sending processing.

[0088] In specific implementation, in step S303, detecting the sound source position according to the sound signal sent by the buffer includes: ;

[0089] Wherein, data is the sound data sent by the buffer, azi, ele, and dist are the horizontal angle, pitch angle, and distance of the current positioning relative to the intelligent body, respectively. , , model is the sound source localization model.

[0090] In specific implementation, in step S303, the sound source position is detected according to the sound signal sent by the buffer, and the sound source coordinates of the world coordinate system are further calculated, including:

[0091] The current position of the agent in the world coordinate system is , the orientation relative to the world coordinate system is ,in , , ;

[0092] Based on the above information and the output of the model, the position of the sound source in the world coordinate system is calculated. ;

[0093] for ,have:

[0094] ;

[0095] To calculate , , first calculate the projection of the distance dist on the horizontal plane:

[0096] ;

[0097] Calculate the direction of the sound source in the world coordinate system based on the direction of the agent:

[0098] ;

[0099] Calculate the world coordinates of the sound source , :

[0100] ;

[0101] ;

[0102] The world coordinates of the sound source can be obtained .

[0103] In specific implementation, in step S303, after obtaining the sound source coordinates, it also includes determining whether the sound source is an existing sound source and performing sound source integration, which specifically includes:

[0104] Assuming that it has been detected m The location of the sound source , given a threshold , if it exists So that:

[0105]

[0106] The detected sound source is considered to be For the same sound source, represents the two-norm, which is the Euclidean distance between two points when applied to the above coordinates;

[0107] If there is no such j , then the detected sound source Add to the list of sound sources.

[0108] The present invention implements a method and system for supporting a virtual person to determine the location of a sound source in a virtual world. A unified system architecture is designed to support the hearing ability and sound source localization ability of the virtual person. By designing a highly versatile interface, different sound source localization algorithms can be adapted. In addition, a buffer is used to cache sound fragments, increase the data length processed by the sound source localization algorithm at a time, and improve the accuracy of the localization algorithm. In order to allow users to visually perceive where the sound is coming from, the system will maintain the sound source location for a period of time and perform sound source integration.

[0109] According to an embodiment of the present invention, the method flow according to an embodiment of the present invention can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. According to an embodiment of the present invention, the electronic device, device, apparatus, module, unit, etc. described above can be implemented by a computer program module.

[0110] The present invention also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiment; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present invention is implemented.

[0111] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus or device.

[0112] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0114] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A sound source localization system in a virtual environment, characterized in that: It is used for sound source localization of intelligent agents in virtual environments, including a sound source simulation module, a sound source cache module and a sound source localization module, where: The sound source simulation module is used to simulate and generate sound signals in a virtual environment and simulate the propagation of sound signals in the virtual environment; The sound source cache module is used to set multiple buffers, cache the sound signal generated by the sound source simulation module, and set the frequency at which the sound signal is sent to the sound source localization module for processing, wherein the multiple buffers are a queue, and when the sound segment reaches a set threshold value n, it is sent to the sound source localization module for processing, and the k sound segments at the head of the queue are discarded, and when the number of sound segments in the queue reaches the set threshold value n again, it is sent to the sound source localization module for processing again, and the k sound segments at the head of the queue processed this time are discarded again, and continuous processing is performed according to n and k, and k is adjusted to control the frequency of sending to the sound source localization module for processing; The sound source localization module is used to detect the sound source position according to the sound signal sent by the sound source buffer module and perform sound source integration.

2. The sound source localization system in a virtual environment according to claim 1, characterized in that: The sound source simulation module is used to simulate and generate sound signals in a virtual environment and simulate the propagation of sound signals in the virtual environment, including: In a virtual environment, sound waves are regarded as rays emitted from a point sound source, and the direct radiation, reflection, diffraction and absorption phenomena of sound waves are simulated and modeled to simulate the propagation of sound waves in the virtual environment.

3. The sound source localization system in a virtual environment according to claim 1, characterized in that: The sound source localization module is used to detect the sound source position according to the sound signal sent by the sound source buffer module, including: ； Wherein, data is the sound data sent by the sound source cache module, azi, ele, and dist are the horizontal angle, pitch angle, and distance of the current positioning relative to the intelligent body, respectively. , , model is the sound source localization model.

4. The sound source localization system in a virtual environment according to claim 3, characterized in that: The sound source localization module is also used to further calculate the sound source coordinates of the world coordinate system, including: The current position of the agent in the world coordinate system is , with respect to the world coordinate system, ,in ; The position of the sound source in the world coordinate system is calculated based on the coordinate information of the intelligent agent and the output result of the sound source localization model. ; for ,have: ； To calculate , , first calculate the projection of the distance dist on the horizontal plane: ； Calculate the direction of the sound source in the world coordinate system based on the direction of the agent: ； Calculate the world coordinates of the sound source : ； ； The world coordinates of the sound source can be obtained .

5. The sound source localization system in a virtual environment according to claim 4, characterized in that: The sound source localization module is also used to determine whether the sound source is an existing sound source and perform sound source integration, including: Assuming that it has been detected m The location of the sound source , given a threshold , if it exists So that: ； The detected sound source is considered to be For the same sound source, represents the two-norm, which is the Euclidean distance between two points when applied to the above coordinates; If there is no such j , then the detected sound source Add to the list of sound sources.

6. A method for localizing a sound source in a virtual environment, characterized in that: Applied to the sound source localization of intelligent agents in virtual environments, including: Simulate and generate sound signals in a virtual environment, and simulate the propagation of sound signals in the virtual environment; Setting multiple buffers to cache the simulated sound signal and setting the frequency of processing the sound signal, wherein the multiple buffers are set as a queue, and when the sound segment reaches a set threshold value n, it is sent for processing, and k sound segments at the head of the queue are discarded. When the number of sound segments in the queue reaches the set threshold value n again, it is sent again, and the k sound segments at the head of the queue processed this time are discarded again. Continuous processing is performed according to n and k, and the frequency of sending processing is controlled by adjusting k; The sound source position is detected according to the sound signal sent by the buffer, and the sound source is integrated.

7. The method for localizing a sound source in a virtual environment according to claim 6, characterized in that: The simulation generates a sound signal in a virtual environment and simulates the propagation of the sound signal in the virtual environment, including: In a virtual environment, sound waves are regarded as rays emitted from a point sound source, and the direct radiation, reflection, diffraction and absorption phenomena of sound waves are simulated and modeled to simulate the propagation of sound waves in the virtual environment.

8. The method for localizing a sound source in a virtual environment according to claim 6, characterized in that: Detecting the location of a sound source according to the sound signal sent by the buffer comprises: ； Wherein, data is the sound data sent by the buffer, azi, ele, and dist are the horizontal angle, pitch angle, and distance of the current positioning relative to the intelligent body, respectively. , , model is the sound source localization model.

9. The method for localizing a sound source in a virtual environment according to claim 8, characterized in that: Detecting the sound source position according to the sound signal sent by the buffer, further calculating the sound source coordinates of the world coordinate system, including: The current position of the agent in the world coordinate system is , with respect to the world coordinate system, ,in , , ; The position of the sound source in the world coordinate system is calculated based on the coordinate information of the intelligent agent and the output result of the sound source localization model. ; for ,have: ； To calculate , first calculate the projection of the distance dist on the horizontal plane: ； Calculate the direction of the sound source in the world coordinate system based on the direction of the agent: ； Calculate the world coordinates of the sound source : ； ； The world coordinates of the sound source can be obtained .

10. The method for localizing a sound source in a virtual environment according to claim 9, characterized in that: After obtaining the coordinates of the sound source, it is also included to determine whether the sound source is an existing sound source and perform sound source integration, which specifically includes: Assuming that it has been detected m The location of the sound source , given a threshold , if it exists So that: ； The detected sound source is considered to be For the same sound source, represents the two-norm, which is the Euclidean distance between two points when applied to the above coordinates; If there is no such j , then the detected sound source Add to the list of sound sources.

11. An electronic device comprising one or more processors and a storage device, characterized in that: The storage device is used to store executable instructions, and when the executable instructions are executed by the above-mentioned processor, the sound source localization method in a virtual environment as described in any one of claims 6 to 10 is implemented.

12. A computer program, characterized in that The computer program includes computer executable instructions, and when the instructions are executed, the instructions are used to implement the sound source localization method in a virtual environment as described in any one of claims 6 to 10.

Images