Perspective area display method based on sound source position and VR equipment

Abstract

The invention relates to the technical field of VR, provides a perspective area display method based on a sound source position and VR equipment, and is used for improving the display flexibility of a perspective area on a safe area. The method comprises the steps of detecting a sound source in a real environment and determining a position coordinate of the sound source relative to a head based on environment sound received by at least two microphones on the VR equipment, and when the sound source is determined to be located outside a safe area according to the position coordinate, determining that the sound source is located outside the safe area according to the intensity of target environment sound received by the at least two microphones and the position of the sound source. And the size and the position of the perspective area on the safe area are calculated, and the real environment image is displayed by opening the perspective area, so that accidents and dangers possibly occurring in an immersive experience process are avoided, and the safety of a user is protected.

Description

Background Art

[0002] In VR devices, the safety zone is an important interactive function. It refers to a safe range or area set in the physical space to ensure that users do not collide with surrounding objects, hurt themselves or others, and other unexpected situations during the user experience of VR technology. It aims to ensure that users enjoy an immersive experience in a safe and controllable environment.

[0003] Currently, most security zones are usually defined or customized by users, such as Figure 1 As shown in the figure, the safety zone is a sphere centered on the user. The safety zone can be designed to be dynamically triggered and change as the user moves. When the user is in a dangerous situation, the perspective area in the safety zone is opened to display the real environment, so that the user can perceive the real world, avoid accidents and feelings of being out of touch with reality that may occur when immersed in a virtual environment, and protect the user's safety.

[0004] However, in the related art, the position and size of the perspective area are often fixed and triggered by a handle ray or a specific button of the VR device, and the display method is not flexible enough. Summary of the invention

[0005] The embodiments of the present application provide a method for displaying a perspective area based on a sound source position and a VR device, which are used to improve the flexibility of displaying a perspective area in a safe zone.

[0006] On the one hand, an embodiment of the present application provides a perspective area display method based on a sound source position, which is applied to a VR device, including:

[0007] Obtaining the volume of the ambient sound received by at least two microphones on the VR device at a preset time interval;

[0008] If at least one volume is greater than a preset volume threshold, detecting the type of sound source in the environment based on the target environmental sound received by the at least two microphones during a period greater than the preset volume threshold;

[0009] When the sound source type is a target type, determining the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target environmental sound and the position information of the at least two microphones;

[0010] If it is determined according to the position coordinates that the sound source is outside the safety zone of the VR device, then according to the position coordinates and the intensity of the target environmental sound received by the at least two microphones, the size and position of the perspective area of ​​the sound source on the spherical surface of the safety zone are calculated, and the perspective area is opened;

[0011] The real environment image viewed by the perspective area is displayed in the virtual environment.

[0012] On the other hand, an embodiment of the present application provides a VR device, including a processor, a memory, a display, and at least two microphones, wherein the at least two microphones, the display, the memory, and the processor are connected via a bus;

[0013] The memory stores a computer program, and the processor performs the following operations according to the computer program:

[0014] Acquiring the volume of the ambient sound received by the at least two microphones at a preset time interval;

[0015] If at least one volume is greater than a preset volume threshold, detecting the type of sound source in the environment based on the target environmental sound received by the at least two microphones during a period greater than the preset volume threshold;

[0016] When the sound source type is a target type, determining the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target environmental sound and the position information of the at least two microphones;

[0017] If it is determined according to the position coordinates that the sound source is outside the safety zone of the VR device, then according to the position coordinates and the intensity of the target environmental sound received by the at least two microphones, the size and position of the perspective area of ​​the sound source on the spherical surface of the safety zone are calculated, and the perspective area is opened;

[0018] The real environment image viewed by the perspective area is displayed in the virtual environment through the display.

[0019] Optionally, the processor calculates the size and position of the perspective area projected on the safety zone according to the position coordinates and the intensity of the ambient sound received by the at least two microphones, combined with the spherical radius of the safety zone, and the specific operation is:

[0020] Convert the position coordinates from a rectangular coordinate system to a spherical coordinate system to obtain the spherical coordinates of the sound source;

[0021] Isometrically projecting the spherical coordinates onto the spherical surface of the safety zone to obtain the projection point, and using the polar angle and azimuth of the spherical coordinates as the polar angle and azimuth of the projection point;

[0022] Obtaining a target sound intensity according to the intensity of the target environmental sound received by each of the at least two microphones;

[0023] Determine the radius of the projection point according to the target sound intensity and the reference sound intensity; wherein the reference sound intensity is the average intensity of the background sound received by the at least two microphones;

[0024] The size and position of the perspective area are obtained according to the radius, polar angle and azimuth angle of the projection point.

[0025] Optionally, the radius of the projection point is calculated as follows:

[0026]

[0027] R=α*lg(L / L0)

[0028] Where i = 1, 2, ..., n, L i represents the intensity of the target environment sound received by the ith microphone, L represents the target sound intensity, L0 represents the reference sound intensity, and R represents the radius of the projection point.

[0029] Optionally, the processor detects the type of sound source in the environment according to the target environmental sound received by the at least two microphones within a period of time greater than the preset volume threshold, and the specific operation is:

[0030] Performing normalization and noise reduction processing on the target environmental sound received by the at least two microphones within a period of time greater than the preset volume threshold;

[0031] The sound source type of the environment is determined based on the waveform and frequency domain characteristics of the processed target environmental sound.

[0032] Optionally, the processor determines the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target ambient sound and the position information of the at least two microphones, and the specific operation is:

[0033] Obtaining a signal receiving time difference between every two microphones according to the start time or the end time when the at least two microphones respectively receive the target ambient sound;

[0034] Obtaining the initial position coordinates of the sound source relative to the head according to the signal receiving time difference between each two microphones, the distance between the corresponding two microphones, and the relative positions of the corresponding two microphones to the center of the eyebrows;

[0035] According to each initial position coordinate, the target position coordinate of the sound source relative to the head is obtained.

[0036] Optionally, the background sound includes at least one of ambient noise and noise generated by the operation of the VR device itself, and the preset volume threshold is greater than the average volume of the background sound.

[0037] On the other hand, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are used to enable a computer device to execute any one of the perspective area display methods based on the sound source position provided in the embodiments of the present application.

[0038] The beneficial effects of a perspective area display method based on sound source position and a VR device provided by the embodiments of the present application are as follows:

[0039] Based on the ambient sound received by at least two microphones on the VR device, the sound source in the real environment is detected and the position coordinates of the sound source relative to the head are determined. When the sound source is determined to be outside the safety zone according to the position coordinates, the size and position of the perspective area on the safety zone are calculated according to the intensity of the target ambient sound received by at least two microphones and the position of the sound source. The perspective area is opened to display the real environment image, thereby avoiding accidents and dangers that may occur during the immersive experience and protecting user safety.

[0040] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0042] Figure 1 A schematic diagram of a safe area of ​​a VR device provided in an embodiment of the present application;

[0043] Figure 2 A flow chart of a method for displaying a perspective area based on a sound source position provided in an embodiment of the present application;

[0044] Figure 3 A flow chart of a method for estimating a sound source position provided in an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the positional relationship between a sound source and two microphones provided in an embodiment of the present application;

[0046] Figure 5 A schematic diagram of the principle of sound source position estimation provided in an embodiment of the present application;

[0047] Figure 6 A flow chart of a method for calculating a perspective area based on a sound source position provided in an embodiment of the present application;

[0048] Figure 7 A schematic diagram of a perspective area provided in an embodiment of the present application;

[0049] Figure 8 A display effect diagram of the perspective area provided for the implementation of this application;

[0050] Fig. 9 A flow chart of the VR experience method provided in the embodiment of the present application;

[0051] Fig.10 A structural diagram of a VR device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0053] Based on the exemplary embodiments shown in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application. In addition, although the disclosure in this application is introduced according to one or several exemplary examples, it should be understood that each aspect of these disclosures can also constitute a complete technical solution separately.

[0054] In addition, the terms "including" and "having" and any variations thereof are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to those components expressly listed but may include other components not expressly listed or inherent to such products or devices.

[0055] The term "module" as used in this application refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0056] The following is an overview of the design concept of the embodiments of the present application in conjunction with application scenarios.

[0057] VR devices protect user safety by opening a perspective area within a safe zone to display the real environment. However, in related technologies, the position and size of the perspective area are often fixed and triggered by a handle ray or a specific button of the VR device, making the display method inflexible.

[0058] With the development of artificial intelligence (AI) technology, deep learning has been widely developed in the field of VR. The model trained by the deep learning algorithm detects the images collected by the VR device, and when the target object is detected, the perspective area is automatically opened. However, this method usually requires a given data set and prior knowledge for model training, which is time-consuming and has certain requirements on the processing performance of the VR device. In addition, although this method realizes the dynamic display of the perspective area, the position and size of the perspective area remain fixed.

[0059] In view of this, an embodiment of the present application provides a perspective area display method based on the sound source position and a VR device. Based on the ambient sound received by at least two microphones on the VR device, the sound source in the real environment is detected and the position coordinates of the sound source relative to the head are determined, and then the position coordinates are used to determine whether the sound source is outside the safety zone. If so, the size and position of the perspective area on the safety zone are calculated based on the ambient sound received by at least two microphones, and the perspective area is turned on to display the real environment image, thereby avoiding accidents and dangers that may occur during the immersive experience and protecting user safety.

[0060] See also Figure 2 , is an implementation process of a perspective area display method based on a sound source position provided in an embodiment of the present application, which is executed by a VR device and mainly includes the following steps:

[0061] S201: Obtaining the volume of ambient sound received by at least two microphones on a VR device at a preset time interval.

[0062] Typically, VR devices have two or more microphone arrays to achieve spatial stereo sound effects. When users wear VR devices for immersive experiences such as gaming, entertainment, and socializing, the users and their real environment will constantly change, such as waving arms, running babies, pedestrians, passing toy cars, etc. This dynamic change may endanger the user and others. When experiencing an immersive experience, the microphone can be turned on to receive the sound in the user's real environment in real time, and the volume of the sound can reflect the distance between the user and the sound source, so as to analyze potential dangers in the real environment through sound.

[0063] In specific implementation, during the immersive experience, the VR device synchronously obtains the ambient sounds received by at least two microphones at a preset time interval, that is, the timestamps of the two ambient sounds obtained by the VR device are the same, and for each microphone, the amplitude of the ambient sound received by it is analyzed to obtain the volume of the ambient sound received by it.

[0064] Taking two microphones as an example, assuming that the time interval is 1S, then in the first second, the VR device analyzes the ambient volume based on the ambient sound received by microphone 1 at timestamp S1, and analyzes the ambient volume based on the ambient sound received by microphone 2 at timestamp S1; in the second second, the VR device analyzes the ambient volume based on the ambient sound received by microphone 1 at timestamp S2, and analyzes the ambient volume based on the ambient sound received by microphone 2 at timestamp S2.

[0065] S202: Determine whether at least one volume is greater than a preset volume threshold, if so, execute S203, otherwise, return to S201.

[0066] Considering that the ambient sound received by the microphone may contain background sounds such as environmental noise and noise generated by the operation of the VR device itself (such as CPU operation noise, VR device fan noise, etc.), in order to avoid the noise in the background sound interfering with the sound source in the environment, a volume threshold can be set in advance to distinguish between real sound and noise.

[0067] In an optional embodiment, the process of setting the preset volume threshold includes:

[0068] First, background sounds received by at least two microphones when there is no sound source are acquired, and the volume of each background sound is analyzed.

[0069] Then, the volume average of each background sound is calculated, and a preset volume threshold greater than the volume average is set.

[0070] Taking two microphones as an example, assuming that the volume of the background sound received by microphone 1 is noise1, and the volume of the background sound received by microphone 2 is noise2, the average volume is (noise1+noise2) / 2, and the preset volume threshold is: (noise1+noise2) / 2+σ.

[0071] Optionally, σ is 0.5, 0.8, 1, etc. decibels (dB), which can be flexibly adjusted according to actual needs.

[0072] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the setting method of the preset volume threshold. For example, the volume threshold can be set to be greater than the maximum volume of each background sound, and the volume threshold can also be set to be greater than the minimum volume of each background sound.

[0073] S203: Detect the type of sound source in the environment according to the target environmental sound received by at least two microphones within a period of time greater than a preset volume threshold.

[0074] When at least one of the volumes of the ambient sounds received by at least two microphones is greater than a preset volume threshold, it indicates that the real environment where the user is located has other sound sources in addition to the background sound. At this time, sound source detection can be performed on the ambient sound.

[0075] In an optional embodiment, the sound source detection process includes the following steps:

[0076] S2031: performing normalization and noise reduction processing on the target environment sound received by at least two microphones within a period of time when the volume is greater than a preset volume threshold.

[0077] Among them, noise reduction processing includes but is not limited to low-pass filtering, high-pass filtering, band-pass filtering, frequency domain filtering, time domain filtering, etc.

[0078] S2032: Determine the type of sound source in the environment according to the processed waveform and frequency domain characteristics of the target environmental sound.

[0079] In one example, a series of sound sources that need to be paid attention to can be preset, and these sound sources usually have unique sound characteristics. For example, the frequency of the sound of an object breaking is low, and the pitch of a baby crying is low and has a certain rhythm. Therefore, the type of sound source in the environment can be determined by analyzing the waveform and frequency domain characteristics of the target environment sound.

[0080] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the method for detecting the sound source type. For example, neural network, machine learning and other algorithms can also be used for detection.

[0081] S204: Determine whether the sound source type is the target type, if so, execute S205, otherwise return to S201.

[0082] Typically, target types include sound sources that may affect the safety of users and others during VR experiences, such as the sound of objects breaking, things falling, whistles, babies crying, etc.

[0083] S205: Determine the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target ambient sound and the position information of the at least two microphones.

[0084] When the sound source type is a target type, it indicates that the user or others may be in danger, so it is necessary to estimate the relative position of the sound source and the user.

[0085] Sound source estimation refers to the process of determining the location, intensity, frequency components and other related properties of a sound source by analyzing the received sound signal in an acoustic environment. Common sound source estimation includes sound source location estimation to confirm the three-dimensional coordinates of the sound source, and sound source quantity estimation to confirm the number of sound sources. Among them, when using dual microphones to estimate the location of indoor sound sources, most of them are achieved by using the time difference (TDOA) and / or level difference (LDOA) ​​of the sound waves arriving at the two microphones.

[0086] Taking the sound source position estimation based on TDOA as an example, in an optional implementation, the sound source position estimation process is as follows: Figure 3 As shown, it mainly includes the following steps:

[0087] S2051: Obtain a signal receiving time difference between every two microphones according to the start time or the end time when the at least two microphones respectively receive the target ambient sound.

[0088] Take two microphones (MIC) as an example. Figure 4 As shown in the figure, since the distances of the sound source in the environment from MIC 1 and MIC 2 on the VR device are different, MIC 1 and MIC 2 start receiving the target environment sound from the same sound source at different times, or end receiving the target environment sound at different times, that is, t1≠t2. Therefore, the signal receiving time difference between MIC 1 and MIC 2 is |t1-t2|.

[0089] S2052: Obtain the initial position coordinates of the sound source relative to the head according to the reception time difference between each two microphones, the distance between the corresponding two microphones, and the relative positions of the corresponding two microphones to the center of the eyebrows.

[0090] Since the position of each microphone on the VR device is fixed, the distance between each two microphones is also fixed, and the signal reception time difference between each two microphones is also fixed, and since the propagation speed of sound is fixed, the distance difference between the same sound source and the two microphones is a constant. Therefore, for each two microphones, a hyperbolic model representing the position of the sound source can be constructed based on the distance between the two microphones, and the signal reception time difference between the two microphones can be used to determine the position coordinates of the sound source on the hyperbolic model. Then, with the center of the user's eyebrows as the origin, the position coordinates of the sound source are converted to the head coordinate system according to the relative positions of the two microphones and the center of the eyebrows, and the position coordinates of the sound source relative to the head are obtained.

[0091] Still taking two microphones as an example, Figure 5 As shown, Figure 4A schematic diagram of the position of the sound source in the target ambient sound received by MIC 1 and MIC 2 in FIG. 1 , according to the sequence of the start time or the end time of the target ambient sound received by MIC 1 and MIC 2, the quadrant of the curve where the sound source is located is determined, wherein the sound source is located on the curve of the quadrant corresponding to the microphone that first receives the target ambient sound; then, according to the distance between MIC 1 and MIC 2 and the signal reception time difference, the position coordinates of the sound source on the curve are calculated.

[0092] S2053: According to each initial position coordinate, obtain the target position coordinate of the sound source relative to the head.

[0093] When the VR device only contains two microphones, the initial position coordinates between the two microphones are the target position coordinates of the sound source relative to the head; when the VR device only contains more than two microphones, the average of the initial position coordinates between every two microphones is used as the target position coordinates of the sound source relative to the head.

[0094] It should be noted that Figure 4 This is only an example, and the embodiments of the present application do not impose any restrictive requirements on the number and location of microphones on the VR device.

[0095] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the sound source position estimation method. For example, for microphone arrays, beam forming methods, minimum variance distortion response (MVDR) methods, subspace decomposition-based methods (such as the MUSIC algorithm), and deep learning methods can also be used.

[0096] S206: Determine whether the sound source is outside the safety zone of the VR device based on the position coordinates. If so, execute S207; otherwise, return to S201.

[0097] After determining the position coordinates of the sound source relative to the head, the positional relationship between the sound source and the safety zone may be determined based on the position coordinates.

[0098] For example, when it is determined based on the position coordinates that the distance between the sound source and the head is greater than the radius of the VR device safety zone, it is determined that the sound source is outside the safety zone.

[0099] S207: Calculate the size and position of a perspective area of ​​the sound source on the spherical surface of the safety zone according to the position coordinates and the intensity of the target ambient sound received by at least two microphones, and open the perspective area.

[0100] When the sound source is outside the safe zone, the spherical design of the safe zone is used to perform projection mapping based on the direction indicated by the position coordinates, and the position and size of the perspective area are calculated based on the intensity of the target ambient sound received by at least two microphones. The perspective area is then opened based on the calculated position and size.

[0101] like Figure 6 As shown in the figure, the calculation process of the perspective area mainly includes the following steps:

[0102] S2071: Convert the position coordinates from the rectangular coordinate system to the spherical coordinate system to obtain the spherical coordinates of the sound source.

[0103] Assuming that the three-dimensional position coordinates of the sound source in the rectangular coordinate system are (x, y, z), the spherical coordinates of the sound source are Among them, the radius Polar angle θ = arccos(x = z / r), azimuth angle

[0104] S2072: Project the spherical coordinates equidistantly onto the spherical surface of the safety zone to obtain a projection point, and use the polar angle and azimuth of the spherical coordinates as the polar angle and azimuth of the projection point.

[0105] The projection point can be used as the center of the perspective area on the spherical surface. The orientation of the perspective area on the spherical surface can be determined by the polar angle and azimuth of the sound source.

[0106] S2073: Obtain target sound intensity according to the intensity of the target environmental sound received by each of the at least two microphones.

[0107] Specifically, the formula of the target sound intensity L is expressed as:

[0108]

[0109] Where n represents the number of microphones, n is an integer greater than or equal to 2, i = 1, 2, ..., n, L i Represents the intensity of the target environment sound received by the i-th microphone.

[0110] S2074: Determine the radius of the projection point according to the target sound intensity and the reference sound intensity.

[0111] Specifically, the calculation formula for the radius R of the projection point is:

[0112] R = α * lg (L / L0) Formula 2

[0113] Among them, L0 represents the reference sound intensity, which in the embodiment of the present application is the average intensity of the background sound received by at least two microphones, α is an empirical constant confirmed based on the radius R0 of the safety zone and test data, and R represents the range of the perspective area on the sphere.

[0114] S2075: Obtain the size and position of the perspective area according to the radius, polar angle and azimuth angle of the projection point.

[0115] like Figure 7 As shown in the figure, it is a schematic diagram of the calculated perspective area, where the spherical radius of the safe zone is R0, and the spherical coordinates of the sound source are The spherical coordinates of the perspective area are Located on the sphere where the sound source is projected onto the safe zone.

[0116] S208: Displaying the real environment image viewed by the perspective area in the virtual environment.

[0117] In an optional embodiment, after the perspective area is turned on, a new layer is created on top of the layer of the virtual screen according to the size of the perspective area, and the real environment image viewed by the perspective area is rendered and displayed on the new layer. This allows the user to see the real scene in the direction of the sound source when immersed in the virtual environment, avoid possible collisions, and protect himself and others.

[0118] like Figure 8 As shown, this is a perspective area rendering provided for the implementation of the present application. When the VR device detects the crying of a baby based on the ambient sound received by the microphone, the position and size of the perspective area are calculated based on the position of the sound source and the intensity of the ambient sound. The real environment picture in the direction of the sound source is displayed in the virtual picture through the perspective area.

[0119] It should be noted that the embodiments of the present application do not impose any restrictive requirements on the display method of the perspective area. For example, in addition to displaying it in an overlaid new layer, it is also possible to divide an area on the layer of the virtual screen by digging a hole based on the spherical coordinates of the perspective area, and display the real environment screen in the area.

[0120] Take a VR device with two microphones as an example, see Fig. 9 , which is a flow chart of the VR experience method provided in the embodiment of the present application, mainly includes the following steps:

[0121] S901: Start the VR application and enter the VR scene.

[0122] S902: Detect whether the switch of the sound source controlling the perspective area is turned on, if so, execute S903, otherwise, execute S913.

[0123] S903: Obtain a configuration file of the sound source type.

[0124] This configuration file contains a series of target types of target sound sources that affect user safety, such as breaking sounds, falling sounds, baby crying, whistle sounds, etc.

[0125] S904: Detect whether the sound source type switch is turned on, if so, execute S905, otherwise, execute S913.

[0126] In an optional example, the detection of various sound sources in the configuration file can be controlled by a type switch, that is, when the sound source type switch is turned on, all types of sound sources in the configuration file can be detected, thereby preventing the problem of opening the perspective area caused by false detection of the sound source.

[0127] S905: Obtain the volume of the ambient sound received by the two microphones at a preset time interval.

[0128] S906: Determine whether there is a volume greater than a preset volume threshold, if so, execute S907, otherwise, return to S905.

[0129] S907: Detect the type of sound source in the environment according to the target environment sound received by the two microphones within a period of time when the volume is greater than a preset volume threshold.

[0130] S908: Determine whether the sound source type is the target type, if so, execute S909, otherwise, return to S905.

[0131] S909: Determine the position coordinates of the sound source relative to the head according to the time information when the two microphones receive the target ambient sound and the position information of the two microphones.

[0132] S910: Determine whether the sound source is outside the safety zone of the VR device based on the position coordinates. If so, execute S911; otherwise, return to S905.

[0133] S911: Calculate the size and position of the perspective area of ​​the sound source on the spherical surface of the safety zone according to the position coordinates and the intensity of the target ambient sound received by the two microphones.

[0134] S912: Open the perspective area, and display the real environment image viewed by the perspective area in the virtual environment.

[0135] S913: Perform normal immersive interactions in a virtual environment.

[0136] It should be noted that the embodiment of the present application does not impose any restrictive requirements on the steps of determining the location and type of the sound source. For example, it is also possible to first detect whether the sound source is outside the safe zone, and then detect whether the sound source is of the target type.

[0137] In another optional embodiment, the sound source type switch is used only to enable detection of some types of sound sources in the configuration file.

[0138] For example, taking the configuration file containing breaking sounds, falling sounds, and baby crying sounds as an example, when the sound source type switch is not turned on, only breaking sounds and falling sounds can be detected. When the sound source type switch is turned on, in addition to detecting breaking sounds and falling sounds, baby crying sounds can also be detected.

[0139] In an embodiment of the present application, based on the ambient sound received by at least two microphones on a VR device, a specific type of sound source in the real environment is detected and the position coordinates of the sound source relative to the head are determined. When the sound source is determined to be outside the safe zone by the position coordinates, the size and position of the perspective area on the safe zone are calculated based on the ambient sound received by at least two microphones, and the perspective area is opened to display the real environment image, thereby avoiding accidents and dangers that may occur during the immersive experience and protecting the user's safety. The entire process does not require additional hardware costs, and compared with existing AI detection algorithms, it does not require model training based on pre-built data sets, saving resource costs.

[0140] Based on the same technical concept, an embodiment of the present application provides a VR device that can implement the steps of the above-mentioned perspective area display method based on the sound source position and achieve the same technical effect.

[0141] See also Fig.10 , the VR device includes a processor 1001, a memory 100, a display 1003 and at least two microphones 1004, and the microphones 1004, the display 1003, the memory 1002 and the processor 1001 are connected via a bus 1005;

[0142] The memory 1002 stores a computer program, and the processor 1001 performs the following operations according to the computer program:

[0143] Acquire the volume of the ambient sound received by at least two microphones 1004 at a preset time interval;

[0144] If at least one volume is greater than a preset volume threshold, detecting the type of sound source in the environment based on the target environmental sound received by at least two microphones 1004 during a period greater than the preset volume threshold;

[0145] When the sound source type is a target type, determining the position coordinates of the sound source relative to the head according to the time information when the at least two microphones 1004 receive the target environment sound and the position information of the at least two microphones;

[0146] If the sound source is determined to be outside the safety zone of the VR device according to the position coordinates, the size and position of the perspective area of ​​the sound source on the spherical surface of the safety zone are calculated according to the position coordinates and the intensity of the target environment sound received by at least two microphones 1004, and the perspective area is opened;

[0147] The real environment image viewed by the perspective area is displayed in the virtual environment through the display 1003.

[0148] Optionally, the processor 1001 calculates the size and position of the perspective area projected on the safety zone according to the position coordinates and the intensity of the ambient sound received by at least two microphones 1004, combined with the spherical radius of the safety zone, and the specific operation is:

[0149] Convert the position coordinates from the rectangular coordinate system to the spherical coordinate system to obtain the spherical coordinates of the sound source;

[0150] Project the spherical coordinates equidistantly onto the spherical surface of the safety zone to obtain the projection point, and use the polar angle and azimuth of the spherical coordinates as the polar angle and azimuth of the projection point;

[0151] Obtaining a target sound intensity according to the intensity of the target environmental sound received by each of the at least two microphones 1004;

[0152] Determine the radius of the projection point according to the target sound intensity and the reference sound intensity; wherein the reference sound intensity is the average intensity of the background sound received by at least two microphones;

[0153] The size and position of the perspective area are obtained according to the radius, polar angle and azimuth of the projection point.

[0154] Optionally, the radius of the projection point is calculated as:

[0155]

[0156] R=α*lg(L / L0)

[0157] Where i = 1, 2, ..., n, L i represents the intensity of the target environmental sound received by the i-th microphone 1004, L represents the target sound intensity, L0 represents the reference sound intensity, and R represents the radius of the projection point.

[0158] Optionally, the processor 1001 detects the type of sound source in the environment according to the target environmental sound received by at least two microphones 1004 within a period of time greater than a preset volume threshold, and the specific operation is:

[0159] Normalizing and noise reduction processing are respectively performed on the target environment sound received by at least two microphones 1004 within a period of time when the volume is greater than a preset volume threshold;

[0160] The sound source type of the environment is determined based on the waveform and frequency domain characteristics of the processed target environmental sound.

[0161] Optionally, the processor 1001 determines the position coordinates of the sound source relative to the head according to the time information when the at least two microphones 1004 receive the target environmental sound and the position information of the at least two microphones 1004, and the specific operation is:

[0162] Obtaining a signal receiving time difference between every two microphones 1004 according to the start time or the end time when the at least two microphones 1004 receive the target ambient sound respectively;

[0163] The initial position coordinates of the sound source relative to the head are obtained according to the signal receiving time difference between each two microphones 1004, the distance between the corresponding two microphones 1004, and the relative positions of the corresponding two microphones 1004 and the center of the eyebrows;

[0164] According to each initial position coordinate, the target position coordinate of the sound source relative to the head is obtained.

[0165] Optionally, the background sound includes at least one of ambient noise and noise generated by the operation of the VR device itself, and the preset volume threshold is greater than the average volume of the background sound.

[0166] In the embodiment of the present application, the memory 1002 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, and programs required for running an instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc. The memory 1002 may be a volatile memory (volatile memory), such as a random-access memory (RAM); the memory 1002 may also be a non-volatile memory (non-volatile memory), such as a read-only memory, a flash memory (flash memory), a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1002 may be any other medium that can be used to carry or store a desired computer program in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 1002 may be a combination of the above memories.

[0167] The processor 1001 may include one or more central processing units (CPU), GPU or digital processing unit, etc. The processor 1001 is used to implement any of the above-mentioned steps of the perspective area display method based on the sound source position when calling the computer program stored in the memory 1002.

[0168] It should be noted that Fig.10 This is just an example, and provides the necessary hardware for the VR device to execute the steps of the perspective area display method based on the sound source position provided in the embodiment of the present application. If not shown, the VR device may also include conventional hardware such as a microphone, a communication interface, a power supply, and a handle.

[0169] In the embodiment of the present application, the specific connection medium between the microphone 1004, the display 1003, the memory 1002 and the processor 1001 is not limited. In the embodiment of the present application, the microphone 1004 and the display 1003 are connected to the bus 1005 between the memory 1002 and the processor 1001. Fig.10 The connections between the other components are only for illustration and are not intended to be limiting. The bus 1005 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.10 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or one type of bus.

[0170] For the convenience of description, the VR device can be divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0171] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0172] The embodiment of the present application also provides a computer-readable storage medium for storing some instructions. When these instructions are executed, the steps of any one of the perspective area display methods based on the sound source position in the aforementioned embodiments can be completed.

[0173] An embodiment of the present application also provides a computer program product for storing a computer program, wherein the computer program is used to execute the steps of any one of the perspective area display methods based on the sound source position in the aforementioned embodiments.

[0174] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0175] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0176] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0178] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A perspective area display method based on sound source position, characterized in that: Applied to VR equipment, the method includes: Obtaining the volume of the ambient sound received by at least two microphones on the VR device at a preset time interval; If at least one volume is greater than a preset volume threshold, detecting the type of sound source in the environment based on the target environmental sound received by the at least two microphones during a period greater than the preset volume threshold; When the sound source type is a target type, determining the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target environmental sound and the position information of the at least two microphones; If it is determined according to the position coordinates that the sound source is outside the safety zone of the VR device, then according to the position coordinates and the intensity of the target environmental sound received by the at least two microphones, the size and position of the perspective area of ​​the sound source on the spherical surface of the safety zone are calculated, and the perspective area is opened; The real environment image viewed by the perspective area is displayed in the virtual environment.

2. The method according to claim 1, characterized in that The calculating the size and position of the perspective area projected on the safety zone according to the position coordinates and the intensity of the ambient sound received by the at least two microphones in combination with the spherical radius of the safety zone comprises: Convert the position coordinates from a rectangular coordinate system to a spherical coordinate system to obtain the spherical coordinates of the sound source; Isometrically projecting the spherical coordinates onto the spherical surface of the safety zone to obtain the projection point, and using the polar angle and azimuth of the spherical coordinates as the polar angle and azimuth of the projection point; Obtaining a target sound intensity according to the intensity of the target environmental sound received by each of the at least two microphones; Determine the radius of the projection point according to the target sound intensity and the reference sound intensity; wherein the reference sound intensity is the average intensity of the background sound received by the at least two microphones; The size and position of the perspective area are obtained according to the radius, polar angle and azimuth angle of the projection point.

3. The method according to claim 2, characterized in that The calculation formula of the radius of the projection point is: R=α*lg(L / L0) Where i = 1, 2, ..., n, L i represents the intensity of the target environment sound received by the ith microphone, L represents the target sound intensity, L0 represents the reference sound intensity, and R represents the radius of the projection point.

4. The method according to claim 1, characterized in that The detecting the type of sound source in the environment according to the target environmental sound received by the at least two microphones within a period greater than the preset volume threshold comprises: Performing normalization and noise reduction processing on the target environmental sound received by the at least two microphones within a period of time greater than the preset volume threshold; The sound source type of the environment is determined based on the waveform and frequency domain characteristics of the processed target environmental sound.

5. The method according to any one of claims 1 to 4, characterized in that The determining the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target ambient sound and the position information of the at least two microphones includes: Obtaining a signal receiving time difference between every two microphones according to the start time or the end time when the at least two microphones respectively receive the target ambient sound; Obtaining the initial position coordinates of the sound source relative to the head according to the signal receiving time difference between each two microphones, the distance between the corresponding two microphones, and the relative positions of the corresponding two microphones to the center of the eyebrows; According to each initial position coordinate, the target position coordinate of the sound source relative to the head is obtained.

6. The method according to any one of claims 1 to 4, characterized in that The background sound includes at least one of environmental noise and noise generated by the operation of the VR device itself, and the preset volume threshold is greater than the average volume of the background sound.

7. A VR device, characterized in that: The device comprises a processor, a memory, a display and at least two microphones, wherein the microphones, the display, the memory and the processor are connected via a bus; The memory stores a computer program, and the processor performs the following operations according to the computer program: Acquiring the volume of the ambient sound received by the at least two microphones at a preset time interval; If at least one volume is greater than a preset volume threshold, detecting the type of sound source in the environment based on the target environmental sound received by the at least two microphones during a period greater than the preset volume threshold; When the sound source type is a target type, determining the position coordinates of the sound source relative to the head according to the time information when the at least two microphones receive the target environmental sound and the position information of the at least two microphones; If it is determined according to the position coordinates that the sound source is outside the safety zone of the VR device, then according to the position coordinates and the intensity of the target environmental sound received by the at least two microphones, the size and position of the perspective area of ​​the sound source on the spherical surface of the safety zone are calculated, and the perspective area is opened; The real environment image viewed by the perspective area is displayed in the virtual environment through the display.

8. The VR device according to claim 7, characterized in that: The processor calculates the size and position of the perspective area projected on the safety zone according to the position coordinates and the intensity of the ambient sound received by the at least two microphones, combined with the spherical radius of the safety zone, and the specific operation is: Convert the position coordinates from a rectangular coordinate system to a spherical coordinate system to obtain the spherical coordinates of the sound source; Isometrically projecting the spherical coordinates onto the spherical surface of the safety zone to obtain the projection point, and using the polar angle and azimuth of the spherical coordinates as the polar angle and azimuth of the projection point; Obtaining a target sound intensity according to the intensity of the target environmental sound received by each of the at least two microphones; Determine the radius of the projection point according to the target sound intensity and the reference sound intensity; wherein the reference sound intensity is the average intensity of the background sound received by the at least two microphones; The size and position of the perspective area are obtained according to the radius, polar angle and azimuth angle of the projection point.

9. The VR device according to claim 8, characterized in that: The calculation formula of the radius of the projection point is: R=α*lg(L / L0) Where i = 1, 2, ..., n, L i represents the intensity of the target environment sound received by the ith microphone, L represents the target sound intensity, L0 represents the reference sound intensity, and R represents the radius of the projection point.

10. The VR device according to claim 7, wherein: The processor detects the type of sound source in the environment according to the target environmental sound received by the at least two microphones within a period greater than the preset volume threshold, and the specific operation is: Performing normalization and noise reduction processing on the target environmental sound received by the at least two microphones within a period of time greater than the preset volume threshold; The sound source type of the environment is determined based on the waveform and frequency domain characteristics of the processed target environmental sound.

Images