Sound source positioning method, head-mounted display device and computer readable medium

Abstract

The embodiment of the invention discloses a sound source positioning method, head-mounted display equipment and a computer readable medium. One specific embodiment of the method comprises the following steps: in response to an audio signal detected by at least one audio acquisition device included in the head-mounted display equipment, generating first sound source related information corresponding to the audio signal at a first moment; second sound source related information corresponding to the audio signal detected at a second moment is generated, and the position of the head-mounted display device at the first moment is different from the position of the head-mounted display device at the second moment; and according to the first sound source related information and the second sound source related information, sound source position information corresponding to the audio signal is generated, and the sound source position information comprises a sound source distance and a sound source direction. According to the embodiment, the sound source can be positioned in the three-dimensional space by using the basic software and hardware configuration of the head-mounted display device under the condition of not additionally adding hardware equipment, so that the direction and the distance of the sound source can be accurately determined.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and in particular to a sound source localization method, a head-mounted display device, and a computer-readable medium. Background Art

[0002] Sound source localization is a technology used to determine the location of a sound source in three-dimensional space. After determining the specific location of the sound source in three-dimensional space, the head-mounted display device can simulate the occlusion effect of virtual objects in three-dimensional space on the real sound source, thereby providing users with a more advanced audio-visual experience. At present, when performing sound source localization, the commonly used methods include: using two microphones to estimate the direction of the sound source through phase difference or time difference or determining the location of the sound source through a multi-microphone array.

[0003] However, when using the above method to locate the sound source, the following technical problems often occur: sound source localization based on time difference or phase difference can often only determine the direction of the sound source, but cannot determine the distance between the head-mounted display device and the sound source on the central axis. In addition, the use of a multi-microphone array method requires the number of hardware devices, and the number of audio collection devices that a head-mounted display device can carry is relatively limited.

[0004] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0005] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0006] Some embodiments of the present disclosure propose a sound source localization method, a head-mounted display device, and a computer-readable medium to solve one or more of the technical problems mentioned in the above background technology section.

[0007] In a first aspect, some embodiments of the present disclosure provide a sound source localization method based on a head-mounted display device, the method comprising: in response to at least one audio acquisition device included in the head-mounted display device detecting an audio signal, generating first sound source related information corresponding to the audio signal at a first moment; generating second sound source related information corresponding to the audio signal detected at a second moment, wherein a position of the head-mounted display device at the first moment is different from a position at the second moment; generating sound source position information corresponding to the audio signal based on the first sound source related information and the second sound source related information, wherein the sound source position information includes the sound source distance and the sound source direction.

[0008] Optionally, the above-mentioned generating the sound source position information of the corresponding audio signal according to the above-mentioned first sound source related information and the above-mentioned second sound source related information includes: determining the device moving distance according to the first device posture information included in the above-mentioned first sound source related information and the second device posture information included in the above-mentioned second sound source related information, wherein the above-mentioned device moving distance is the distance between the position of the above-mentioned head-mounted display device at the above-mentioned first moment and the position at the above-mentioned second moment; generating the sound source position information according to the first sound source direction included in the above-mentioned first sound source related information, the second sound source direction included in the above-mentioned second sound source related information and the above-mentioned device moving distance.

[0009] Optionally, the above-mentioned generation of first sound source related information corresponding to the audio signal at the first moment includes: determining the first sound source direction corresponding to the above-mentioned first moment through the above-mentioned at least one audio acquisition device; generating the position information of the above-mentioned head-mounted display device at the above-mentioned first moment as the first device position information; and determining the above-mentioned first device position information and the above-mentioned first sound source direction as the first sound source related information.

[0010] Optionally, the above-mentioned generation of the second sound source related information corresponding to the audio signal detected at the second moment includes: determining the second sound source direction corresponding to the above-mentioned second moment through the above-mentioned at least one audio acquisition device; generating the position information of the above-mentioned head-mounted display device at the above-mentioned second moment as the second device position information; and determining the above-mentioned second device position information and the above-mentioned second sound source direction as the second sound source related information.

[0011] Optionally, the sound source position information is generated according to the first sound source direction included in the first sound source related information, the second sound source direction included in the second sound source related information and the device moving distance, including: generating a sound source distance of a corresponding audio signal according to the first sound source direction, the second sound source direction and the device moving distance; determining the second sound source direction as the sound source direction of the corresponding audio signal; and determining the sound source distance and the sound source direction as the sound source position information.

[0012] In a second aspect, some embodiments of the present disclosure provide a head-mounted display device, comprising: one or more processors; at least one optical module, the optical module comprising at least one display screen and an optical element; at least one audio acquisition device, the audio acquisition device being used to detect and acquire audio signals; a storage device, on which one or more programs are stored; when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0013] In a third aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the first aspect is implemented.

[0014] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the sound source localization method based on the head-mounted display device of some embodiments of the present disclosure, the basic hardware and software configuration of the head-mounted display device can be used to locate the sound source in three-dimensional space without adding additional hardware devices, so as to more accurately determine the direction and distance of the sound source. Specifically, the reason for the inaccuracy of the sound source position and the sound source distance is that the sound source localization based on the time difference or phase difference can often only determine the direction of the sound source, but cannot determine the distance between the head-mounted display device and the sound source on the central axis. In addition, the use of a multi-microphone array method requires the number of hardware devices, and the number of audio acquisition devices that the head-mounted display device can carry is relatively limited. Based on this, the sound source localization method based on the head-mounted display device of some embodiments of the present disclosure, first, in response to the at least one audio acquisition device included in the head-mounted display device detecting an audio signal, generates the first sound source related information corresponding to the audio signal at the first moment. Therefore, when the at least one audio acquisition device included in the head-mounted display device detects an audio signal, the direction of the sound source at the first moment can be determined by each audio acquisition device and the phase difference method, and the corresponding posture information of the head-mounted display device at the first moment can be determined. Then, the second sound source related information corresponding to the audio signal detected at the second moment is generated. Wherein, the position of the head-mounted display device at the first moment is different from the position at the second moment. Therefore, by changing the position of the head-mounted display device, an additional observation point is added, so the direction of the sound source at the second moment can be determined by various audio acquisition devices and phase difference methods, and the corresponding posture information of the head-mounted display device at the second moment can be determined. Finally, according to the first sound source related information and the second sound source related information, the sound source position information of the corresponding audio signal is generated. Wherein, the sound source position information includes the sound source distance and the sound source direction. Therefore, a geometric model can be constructed by using the sound source related information generated by the head-mounted display device at two different positions at the first moment and the second moment, so as to determine a more accurate sound source distance and sound source direction, and realize complete three-dimensional spatial sound source positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0016] Figure 1 is an architectural diagram of an exemplary system in which some embodiments of the present disclosure may be applied;

[0017] Figure 2is a flowchart of some embodiments of a sound source localization method based on a head mounted display device according to the present disclosure;

[0018] Figure 3 is a flowchart of other embodiments of a sound source localization method based on a head mounted display device according to the present disclosure;

[0019] Figure 4 is a schematic diagram of a scene at the first moment according to the sound source localization method based on a head mounted display device disclosed in the present invention;

[0020] Figure 5 is a schematic diagram of a scene at a second moment according to the sound source localization method based on a head mounted display device disclosed in the present invention;

[0021] Figure 6 It is a schematic diagram of the principle of determining the distance of a sound source according to the sound source localization method based on a head mounted display device disclosed in the present invention;

[0022] Figure 7 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0024] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0025] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0026] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0027] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0028] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0029] Figure 1 An exemplary system architecture 100 of a sound source localization method based on a head mounted display device to which some embodiments of the present disclosure may be applied is shown.

[0030] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as web browser applications, program development applications, search applications, instant messaging tools, email clients, social platform software, etc. Terminal device 103 can be a head-mounted display device. Various screen projection applications can be installed on terminal device 103.

[0032] Terminal devices 101 and 102 can be hardware or software. When terminal devices 101 and 102 are hardware, they can be various electronic devices with display screens and supporting information display, including but not limited to smart phones, tablet computers, e-book readers, laptop computers, desktop computers, etc. When terminal devices 101 and 102 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules for providing distributed services, or they can be implemented as a single software or software module. No specific limitation is made here.

[0033] The server 105 may be a server that provides various services, such as a background server that provides support for information displayed on the terminal devices 101, 102, and 103. The background server may analyze and process the received request and other data, and feed back the processing results to the terminal device.

[0034] It should be noted that the sound source localization method based on the head mounted display device provided in the embodiment of the present disclosure can be executed by the smart terminal included in the terminal device 103 .

[0035] It should be noted that the server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules for providing distributed services, or it can be implemented as a single software or software module. No specific limitation is made here.

[0036] Continue to refer Figure 2 , shows a process 200 of some embodiments of a sound source localization method based on a head mounted display device according to the present disclosure. The sound source localization method based on a head mounted display device comprises the following steps:

[0037] Step 201: In response to at least one audio acquisition device included in a head mounted display device detecting an audio signal, first sound source related information corresponding to the audio signal at a first moment is generated.

[0038] In some embodiments, an execution body (eg, Figure 1 The computing device 103 shown in the figure can generate the first sound source related information corresponding to the audio signal at the first moment in response to at least one audio acquisition device included detecting the audio signal. The above-mentioned execution subject can be a head-mounted display device. The above-mentioned head-mounted display device can be a device for the wearing user to watch the virtual scene. The above-mentioned head-mounted display device can be but not limited to one of the following: AR glasses, MR glasses, VR glasses. The above-mentioned audio acquisition device can be but not limited to other devices with the function of detecting and collecting audio signals, such as microphones, audio acquisition cards or condenser microphones. As an example, the above-mentioned head-mounted display device can include a preset number of microphones. The above-mentioned preset number can be 2. The above-mentioned first sound source related information can include: the first sound source direction and the first device posture information. The above-mentioned first sound source direction can be the direction of the sound source position relative to the central axis of the above-mentioned head-mounted display device at the above-mentioned first moment. The above-mentioned first sound source direction can be represented by the angle of the first angle. The above-mentioned first angle can be the angle between the central axis of the above-mentioned head-mounted display device and the straight line formed by connecting the head-mounted display device and the sound source position. The first device posture information may represent the position and posture of the head mounted display device in the three-dimensional space at the first moment. The first device posture information may include the position coordinates of the head mounted display device in the three-dimensional space and the posture information in the three-dimensional space. The posture information may be, but is not limited to, one of the following: Euler angles, quaternions, and rotation matrices.

[0039] like Figure 4As shown, the first sound source direction can be represented by the angle θ between the central axis of the head mounted display device 402 and the straight line formed by the head mounted display device 402 and the sound source position 401. In addition, the execution subject can generate the first device posture information by using the camera device included in the head mounted display device and executing the posture estimation algorithm (i.e., 6DOF algorithm). Figure 4 The two audio collection devices included in the head-mounted display device 402 shown in the figure are distributed left and right (that is, the two audio collection devices are respectively located on the left and right sides of the head-mounted display device), so the central axis is a vertical central axis. In addition, the two audio collection devices included in the head-mounted display device can also be distributed up and down, that is, the two audio collection devices are arranged up and down in the head-mounted display device, so the central axis can be a horizontal central axis. Here, the specific position of the audio collection device in the head-mounted display device is not limited.

[0040] In some optional implementations of some embodiments, the execution subject may generate the first sound source related information corresponding to the audio signal at the first moment through the following steps:

[0041] The first step is to determine the first sound source direction corresponding to the first moment through the at least one audio collection device. The distance between the two audio collection devices included in the head-mounted display device is a preset device spacing distance. In practice, the execution subject can determine the first sound source direction through the time difference of the audio signals detected by the two microphones and the TDOA (Time Difference of Arrival) algorithm.

[0042] The second step is to generate the posture information of the head mounted display device at the first moment as the first device posture information. In practice, the execution subject may generate the second device posture information through the camera device and the posture estimation algorithm (i.e., 6DOF algorithm) included in the head mounted display device.

[0043] In a third step, the first device posture information and the first sound source direction are determined as first sound source related information.

[0044] Step 202: Generate second sound source related information corresponding to the audio signal detected at a second moment.

[0045] In some embodiments, the execution subject may generate the second sound source related information corresponding to the audio signal detected at the second moment. The position of the head mounted display device at the first moment is different from the position at the second moment. The second sound source related information may include: the second sound source direction and the second device posture information. The second sound source direction may be the direction of the sound source position relative to the central axis of the head mounted display device at the second moment. The second sound source direction may be represented by the angle of the second angle. The second angle may be the angle between the central axis of the head mounted display device and the straight line formed by the head mounted display device and the sound source position. The second device posture information may characterize the position and posture of the head mounted display device in the three-dimensional space at the second moment. The second device posture information may include the position coordinates of the head mounted display device in the three-dimensional space and the posture information in the three-dimensional space. The posture information may be, but is not limited to, one of the following: Euler angles, quaternions, and rotation matrices.

[0046] In some optional implementations of some embodiments, the execution subject may generate the second sound source related information corresponding to the audio signal detected at the second moment through the following steps:

[0047] The first step is to determine the second sound source direction corresponding to the second moment through at least one of the above-mentioned audio collection devices. In practice, the above-mentioned execution entity can perform Fourier transform processing on the audio signals collected by the two audio collection devices at the second moment respectively to obtain the phase information of the two audio signals. Then, the two audio signal phase information can be subtracted to obtain the sound source phase difference. Then, the above-mentioned execution entity can establish a triangular model through the sound source phase difference, the time difference between the audio signals detected by the two audio collection devices at the first moment, the sound propagation speed and the preset device spacing distance to determine the second sound source direction corresponding to the second moment. In addition, the above-mentioned first sound source direction and the above-mentioned second sound source direction can also be determined by a variety of sound source direction estimation algorithms such as beamforming algorithms.

[0048] The second step is to generate the posture information of the head mounted display device at the second moment as the second device posture information. In practice, the execution subject can generate the second device posture information through the camera device and posture estimation algorithm (ie, 6DOF algorithm) included in the head mounted display device.

[0049] In a third step, the second device posture information and the second sound source direction are determined as second sound source related information.

[0050] Step 203: Generate sound source position information corresponding to the audio signal according to the first sound source related information and the second sound source related information.

[0051] In some embodiments, the execution entity may generate the sound source position information of the corresponding audio signal based on the first sound source related information and the second sound source related information. The sound source position information includes the sound source distance and the sound source direction. The sound source distance may be the straight-line distance between the head-mounted display device and the sound source. The sound source direction may be the second sound source direction included in the second sound source related information. In practice, the execution entity may use the sound source position, the head-mounted display device position represented by the first device posture information included in the first sound source related information, and the head-mounted display device position represented by the second sound source related information as three vertices, respectively, and establish a geometric (triangle) model through the first sound source direction, the second sound source direction and the distance between the position of the head-mounted display device at different times to generate the sound source position information of the corresponding audio signal.

[0052] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the sound source localization method based on the head-mounted display device of some embodiments of the present disclosure, the basic hardware and software configuration of the head-mounted display device can be used to locate the sound source in three-dimensional space without adding additional hardware devices, so as to more accurately determine the direction and distance of the sound source. Specifically, the reason for the inaccuracy of the sound source position and the sound source distance is that the sound source localization based on the time difference or phase difference can often only determine the direction of the sound source, but cannot determine the distance between the head-mounted display device and the sound source on the central axis. In addition, the use of a multi-microphone array method requires the number of hardware devices, and the number of audio acquisition devices that the head-mounted display device can carry is relatively limited. Based on this, the sound source localization method based on the head-mounted display device of some embodiments of the present disclosure, first, in response to the at least one audio acquisition device included in the head-mounted display device detecting an audio signal, generates the first sound source related information corresponding to the audio signal at the first moment. Therefore, when the at least one audio acquisition device included in the head-mounted display device detects an audio signal, the direction of the sound source at the first moment can be determined by each audio acquisition device and the phase difference method, and the corresponding posture information of the head-mounted display device at the first moment can be determined. Then, the second sound source related information corresponding to the audio signal detected at the second moment is generated. Wherein, the position of the head-mounted display device at the first moment is different from the position at the second moment. Therefore, by changing the position of the head-mounted display device, an additional observation point is added, so the direction of the sound source at the second moment can be determined by various audio acquisition devices and phase difference methods, and the corresponding posture information of the head-mounted display device at the second moment can be determined. Finally, according to the first sound source related information and the second sound source related information, the sound source position information of the corresponding audio signal is generated. Wherein, the sound source position information includes the sound source distance and the sound source direction. Therefore, a geometric model can be constructed by using the sound source related information generated by the head-mounted display device at two different positions at the first moment and the second moment, so as to determine a more accurate sound source distance and sound source direction, and realize complete three-dimensional spatial sound source positioning.

[0053] Further references Figure 3 , shows a process 300 of other embodiments of a sound source localization method based on a head mounted display device according to the present disclosure. The sound source localization method based on a head mounted display device comprises the following steps:

[0054] Step 301: In response to at least one audio acquisition device included in a head mounted display device detecting an audio signal, first sound source related information corresponding to the audio signal at a first moment is generated.

[0055] Step 302: Generate second sound source related information corresponding to the audio signal detected at a second moment.

[0056] In some embodiments, the specific implementation of steps 301-302 and the technical effects brought about can be referred to Figure 2 The corresponding steps 201-202 in the embodiments are not described in detail here.

[0057] Step 303 : determining the moving distance of the device according to the first device posture information included in the first sound source related information and the second device posture information included in the second sound source related information.

[0058] In some embodiments, the execution subject may determine the device movement distance based on the first device posture information included in the first sound source related information and the second device posture information included in the second sound source related information. The device movement distance is the distance between the position of the head mounted display device at the first moment and the position at the second moment. In practice, the execution subject may determine the device movement distance using the following formula:

[0059]

[0060] Wherein, x2 may be the X-axis coordinate value in the position coordinates included in the second device posture information, and y2 may be the Y-axis coordinate value in the position coordinates included in the second device posture information. x1 may be the X-axis coordinate value in the position coordinates included in the first device posture information, and y1 may be the Y-axis coordinate value in the position coordinates included in the first device posture information. d may be the device movement distance.

[0061] Step 304 , generating sound source position information according to the first sound source direction included in the first sound source related information, the second sound source direction included in the second sound source related information, and the device moving distance.

[0062] In some embodiments, the execution entity may generate the sound source location information according to the first sound source direction included in the first sound source related information, the second sound source direction included in the second sound source related information, and the moving distance of the device.

[0063] like Figure 5 As shown, Figure 5A triangular model is formed by including the sound source 401' at the second moment, the position of the head-mounted display device 402' at the first moment, and the position of the head-mounted display device 501 at the second moment as a vertex. The distance d between the head-mounted display devices 402' and 501 is the device movement distance. θ1 can be the direction of the first sound source. θ2 can be the direction of the second sound source. β1 can be an inner angle of a triangle formed by the sound source 401', the position of the head-mounted display device 402' at the first moment, and the position of the head-mounted display device 501 at the second moment. β2 can be another inner angle of a triangle formed by the sound source 401', the head-mounted display device 501 at the second moment, and the position of the head-mounted display device 402' at the first moment.

[0064] In some optional implementations of some embodiments, the execution subject may generate the sound source position information according to the first sound source direction included in the first sound source related information, the second sound source direction included in the second sound source related information, and the device moving distance through the following steps:

[0065] The first step is to generate a sound source distance corresponding to the audio signal according to the first sound source direction, the second sound source direction and the moving distance of the device. Figure 6 The schematic diagram of the sound source distance determination principle shown in the figure, in practice, firstly, the above-mentioned execution subject can determine the triangle internal angle β1 and another internal angle β2 by the following formula:

[0066]

[0067] Among them, x2 can be the X-axis coordinate value in the position coordinate included in the second device posture information, and y2 can be the Y-axis coordinate value in the position coordinate included in the second device posture information. x1 can be the X-axis coordinate value in the position coordinate included in the first device posture information, and y1 can be the Y-axis coordinate value in the position coordinate included in the first device posture information. β1 can be the internal angle of a triangle formed by the sound source position at the first moment, the position of the head-mounted display device at the first moment, and the position of the head-mounted display device at the second moment. β2 can be another internal angle of a triangle formed by the sound source position at the second moment, the head-mounted display device at the second moment, and the position of the head-mounted display device at the first moment. α1 can be the representation of the posture information at the first moment on the 2D plane. α2 can be the representation of the posture information at the second moment on the 2D plane. In addition, Figure 6 Where P1 represents the position of the head mounted display device at the first moment. P2 represents the position of the head mounted display device at the second moment. d represents the moving distance of the device.

[0068] Then, the execution entity can determine the distance between the head mounted display device and the sound source position at the second moment (ie, the length of one side of the formed triangle) through the internal angle β1, the internal angle β2, the device movement distance d, and the cosine theorem.

[0069] The second step is to determine the second sound source direction as the sound source direction of the corresponding audio signal.

[0070] In the third step, the sound source distance and the sound source direction are determined as the sound source position information.

[0071] from Figure 3 It can be seen that Figure 2 Compared with the description of some corresponding embodiments, the present disclosure first refines the specific method of determining the sound source distance. By introducing the displacement of the head-mounted display device, the distance between two displacements of the head-mounted display device can be determined even if the displacement distance is small, so that the geometric model (triangular model) can be constructed using the sound source position, the position of the head-mounted display device at the first moment, and the position of the head-mounted display device at the second moment. Then, the sound source distance is solved by the geometric model (triangular model). In this way, not only the direction of the sound source can be determined, but also the distance of the sound source can be determined more accurately based on the determination of the sound source direction.

[0072] Reference below Figure 7 , which shows a head mounted display device 700 (eg, Figure 1 A schematic diagram of the structure of the terminal device 103). Figure 6 The head mounted display device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0073] like Figure 7 As shown, the head mounted display device 700 may include a processing device 701 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the head mounted display device 700 are also stored. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0074] Typically, the following devices may be connected to the I / O interface 705: input devices 706 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 707 including, for example, at least one display screen, a speaker, a vibrator, etc.; and communication devices 709. At least one display screen may be combined with optical elements to image the display content in front of the user. The communication device 609 may allow the head mounted display device 700 to communicate with other devices wirelessly or wired to exchange data. Although Figure 6 The head mounted display device 700 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown. More or fewer devices may be implemented or have instead. Figure 7 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0075] The head mounted display device may further include an optical module. The optical module includes at least one display screen and an optical element, and is used to form an image in front of the user's eyes. The head mounted display device may further include at least one audio acquisition device. The audio acquisition device is used to detect and acquire audio signals.

[0076] Optionally, the head-mounted display device may include a head-mounted display device body and a smart terminal, and the smart terminal is communicatively connected to the head-mounted display device body.

[0077] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 709, or installed from the storage device 708, or installed from the ROM 702. When the computer program is executed by the processing device 701, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0078] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0079] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0080] The computer-readable medium may be included in the head-mounted display device; or it may exist independently without being assembled into the head-mounted display device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the head-mounted display device, the head-mounted display device: generates first sound source related information corresponding to the audio signal at a first moment in response to at least one audio acquisition device included in the head-mounted display device detecting an audio signal; generates second sound source related information corresponding to the audio signal detected at a second moment, wherein the position of the head-mounted display device at the first moment is different from the position at the second moment; generates sound source position information corresponding to the audio signal based on the first sound source related information and the second sound source related information, wherein the sound source position information includes the sound source distance and the sound source direction.

[0081] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0082] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0083] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0084] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. A sound source localization method based on a head mounted display device, comprising: In response to at least one audio acquisition device included in the head mounted display device detecting an audio signal, generating first sound source related information corresponding to the audio signal at a first moment; generating second sound source related information corresponding to the audio signal detected at a second moment, wherein a position of the head mounted display device at the first moment is different from a position of the head mounted display device at the second moment; Sound source position information of a corresponding audio signal is generated according to the first sound source related information and the second sound source related information, wherein the sound source position information includes a sound source distance and a sound source direction.

2. The method according to claim 1, wherein: Generating the sound source position information of the corresponding audio signal according to the first sound source related information and the second sound source related information includes: Determine a device movement distance according to first device posture information included in the first sound source related information and second device posture information included in the second sound source related information, wherein the device movement distance is a distance between a position of the head mounted display device at the first moment and a position of the head mounted display device at the second moment; Sound source position information is generated according to a first sound source direction included in the first sound source related information, a second sound source direction included in the second sound source related information, and a moving distance of the device.

3. The method according to claim 1, wherein: The generating of the first sound source related information corresponding to the audio signal at the first moment includes: Determining, by means of the at least one audio collection device, a first sound source direction corresponding to the first moment; Generate the position information of the head mounted display device at the first moment as the first device position information; The first device posture information and the first sound source direction are determined as first sound source related information.

4. The method according to claim 1, wherein: The generating the second sound source related information corresponding to the audio signal detected at the second moment includes: Determining, by means of the at least one audio collection device, a second sound source direction corresponding to the second moment; Generate the position and posture information of the head mounted display device at the second moment as the second device position and posture information; The second device position information and the second sound source direction are determined as second sound source related information.

5. The method according to claim 2, wherein: The generating the sound source position information according to the first sound source direction included in the first sound source related information, the second sound source direction included in the second sound source related information, and the moving distance of the device includes: Generate a sound source distance of a corresponding audio signal according to the first sound source direction, the second sound source direction, and the moving distance of the device; Determining the second sound source direction as the sound source direction of the corresponding audio signal; The sound source distance and the sound source direction are determined as the sound source position information.

6. A head mounted display device, comprising: one or more processors; at least one optical module, the optical module comprising at least one display screen and an optical element; At least one audio acquisition device, the audio acquisition device is used to detect and acquire audio signals; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 5.

7. The head mounted display device according to claim 6, wherein: The head mounted display device comprises a head mounted display device body and a smart terminal, and the smart terminal is communicatively connected with the head mounted display device body.

8. A computer readable medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images