Audio rendering method and device for virtual reality space

Abstract

The present application relates to the field of virtual reality technology and provides an audio rendering method and device for a virtual reality space. The method includes: obtaining the forehead position and the back of the head position of a user, and calculating the user's head position based on the forehead position and the back of the head position; obtaining the sound source position of a sound source in a target virtual space, and calculating the azimuth and elevation angles between the sound source and the user based on the sound source position and the head position; constructing an octree for the target virtual space, and obtaining a head-related transfer function based on the octree, the azimuth angle, and the elevation angle; generating the left-ear spatial audio and right-ear spatial audio of the sound source to the user based on the head-related transfer function; obtaining the left-ear ambient audio and the right-ear ambient audio of the user, and mixing the left-ear spatial audio and the left-ear ambient audio to obtain left-ear mixed audio, and mixing the right-ear spatial audio and the right-ear ambient audio to obtain right-ear mixed audio. The method of the present application can improve the quality of audio rendering.

Description

Technical Field

[0001] The present application relates to the field of virtual reality technology, and in particular to an audio rendering method and device for a virtual reality space. Background Art

[0002] VR devices, which utilize virtual reality (VR) technology, can create three-dimensional virtual scenes, simulating users' senses like vision, hearing, and touch, allowing them to experience the virtual world in real time and without limitations. With the continuous development of VR, the demand for realistic audio is greater than ever. High-quality audio can provide VR users with a sufficient sense of immersion.

[0003] However, there are currently many problems with VR audio rendering. There is a lack of real-time spatial audio system solutions that combine VR virtual scenes and real scenes. Existing VR audio solutions mainly reflect the sense of space through virtual scene audio production, lack real-time audio interaction effects, and have a poor sense of immersion. The existing VR headsets' built-in positioning solutions are generally based on vision. Visual solutions are prone to errors when the scene visual elements are very complex. In addition, the positioning accuracy cannot be guaranteed in dark scenes, resulting in poor quality of positioning-based spatial audio rendering. It can be seen that the existing audio rendering methods have the problem of poor quality of audio rendering in virtual space. Summary of the Invention

[0004] The present application provides a method and device for audio rendering in a virtual reality space, which can solve the problem of poor quality of audio rendering in the virtual space.

[0005] In a first aspect, an embodiment of the present application provides an audio rendering method for a virtual reality space, the audio rendering method comprising:

[0006] Obtaining a forehead position and a back-of-the-head position of a user in a real-life space corresponding to the target virtual space, and calculating a head position of the user based on the forehead position and the back-of-the-head position;

[0007] Obtaining a sound source position of the sound source in the target virtual space, and calculating an azimuth and elevation angle between the sound source and the user based on the sound source position and the user's head position;

[0008] An octree is constructed for the target virtual space, and a head-related transfer function is obtained based on the octree, azimuth, and elevation angles. The octree is used to describe the spatial structure of the target virtual space, and the head-related transfer function is used to describe the audio transmission relationship between the sound source and the user.

[0009] Generate left-ear spatial audio and right-ear spatial audio of the sound source for the user according to the head-related transfer function;

[0010] Obtain the user's left ear ambient audio and right ear ambient audio, mix the left ear spatial audio and the left ear ambient audio to obtain left ear mixed audio, and mix the right ear spatial audio and the right ear ambient audio to obtain right ear mixed audio; the left ear ambient audio is the ambient audio transmitted to the user's left ear in the real space, and the right ear ambient audio is the ambient audio transmitted to the user's right ear in the real space.

[0011] Optionally, the user's head position at the current moment is calculated based on the forehead position and the back of the head position, including:

[0012] By formula:

[0013]

[0014] Calculate the user's head position P at the current moment c ;

[0015] Among them, w f represents the frontal weight, w b represents the weight of the hindbrain, x f Indicates the horizontal coordinate position of the forehead, y f Indicates the vertical coordinate position of the forehead, z f Indicates the vertical coordinate position of the forehead, x b Indicates the horizontal coordinate position of the back of the brain, y b Indicates the vertical coordinate position of the hindbrain, z b Indicates the vertical coordinate position of the hindbrain.

[0016] Optionally, based on the sound source position and the user's head position, the azimuth and elevation angles between the sound source and the user are calculated, including:

[0017] Calculate the azimuth angle between the sound source and the user based on the sound source position and the user's head position;

[0018] Based on the sound source position and the user's head position, a direction vector between the sound source and the user is calculated, and based on the direction vector, an elevation angle between the sound source and the user is calculated.

[0019] Optionally, based on the sound source position and the user's head position, the azimuth angle between the sound source and the user is calculated, including:

[0020] By formula:

[0021] θ=arctan2(y s -y l ,x s -x l )

[0022] Calculate the azimuth angle θ;

[0023] Among them, y s Indicates the vertical coordinate position of the sound source, yl Indicates the vertical coordinate position of the user's head, x s Indicates the horizontal coordinate position of the sound source, x l Indicates the horizontal coordinate position of the user's head.

[0024] Optionally, based on the sound source position and the user's head position, a direction vector between the sound source and the user is calculated, including:

[0025] By formula:

[0026]

[0027] Calculate the direction vector

[0028] Among them, z s Indicates the vertical coordinate position of the sound source, z l Indicates the vertical coordinate position of the user's head;

[0029] Calculate the elevation angle between the sound source and the user based on the direction vector, including:

[0030] By formula:

[0031]

[0032] Calculating elevation angle

[0033] in, Represents the magnitude of the direction vector.

[0034] Optionally, obtain the head-related transfer function based on the octree, azimuth, and elevation, including:

[0035] Construct a head-related transfer function for each node of the octree to obtain a head-related transfer function library;

[0036] The user's head-related transfer function is obtained by querying the head-related transfer function library according to the azimuth angle and the elevation angle.

[0037] Optionally, generating the left-ear spatial audio and the right-ear spatial audio of the user from the sound source according to the head-related transfer function includes:

[0038] Obtain a left-ear mono signal transmitted from the sound source to the user's left ear, and obtain a right-ear mono signal transmitted from the sound source to the user's right ear;

[0039] Convolve the left ear mono signal with the head-related transfer function to obtain the left ear spatial audio;

[0040] The right ear mono signal is convolved with the head-related transfer function to obtain the right ear spatial audio.

[0041] Optionally, mixing the left ear spatial audio and the left ear ambient audio to obtain left ear mixed audio includes:

[0042] Perform weighted fusion of the left ear spatial audio and the left ear ambient audio to obtain the left ear weighted audio;

[0043] The left ear weighted audio is adjusted using Fourier transform to obtain the left ear mixed audio.

[0044] Optionally, weighted fusion of the left-ear spatial audio and the left-ear ambient audio to obtain left-ear weighted audio includes:

[0045] By formula:

[0046] L mix (t)=α(t)L' ent (t)+β(t)L' hrtf (t)

[0047] Calculate the left ear weighted audio L mix (t);

[0048] Among them, α(t) and β(t) are weight coefficients, L' ent (t) represents the left ear ambient audio, L' hrtf (t) represents the left ear spatial audio;

[0049] The left ear weighted audio is adjusted using Fourier transform to obtain the left ear mixed audio, including:

[0050] By formula:

[0051]

[0052]

[0053]

[0054] Get the left ear mixed audio L final (t);

[0055] in, represents the inverse Fourier transform, L' mix (f) represents the normalized mixed audio, L mix (f) represents the initial left ear mixed audio, represents the Fourier transform.

[0056] In a second aspect, an embodiment of the present application provides an audio rendering device for a virtual reality space, comprising:

[0057] A first calculation module obtains a forehead position and a back-of-the-head position of a user in a real-life space corresponding to a target virtual space, and calculates a head position of the user based on the forehead position and the back-of-the-head position;

[0058] The second calculation module obtains the sound source position of the sound source in the target virtual space and calculates the azimuth and elevation angles between the sound source and the user based on the sound source position and the user's head position;

[0059] The construction module constructs an octree for the target virtual space and obtains the head-related transfer function based on the octree, azimuth, and elevation angles. The octree is used to describe the spatial structure of the target virtual space, and the head-related transfer function is used to describe the audio transmission relationship between the sound source and the user.

[0060] A generation module generates left-ear spatial audio and right-ear spatial audio for the user based on the head-related transfer function;

[0061] The mixing module obtains the user's left ear ambient audio and right ear ambient audio, and mixes the left ear spatial audio and the left ear ambient audio to obtain left ear mixed audio, and mixes the right ear spatial audio and the right ear ambient audio to obtain right ear mixed audio; the left ear ambient audio is the ambient audio transmitted to the user's left ear in the real scene space, and the right ear ambient audio is the ambient audio transmitted to the user's right ear in the real scene space.

[0062] In a third aspect, an embodiment of the present application provides a terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned audio rendering method for the virtual reality space when executing the above-mentioned computer program.

[0063] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned audio rendering method for virtual reality space.

[0064] The above solution of the present application has the following beneficial effects:

[0065] In an embodiment of the present application, the forehead position and the back of the head position of the user in the real space corresponding to the target virtual space are obtained, and the head position of the user is calculated based on the forehead position and the back of the head position, and then the sound source position of the sound source in the target virtual space is obtained, and based on the sound source position and the user's head position, the azimuth and elevation angles between the sound source and the user are calculated, and then an octree is constructed for the target virtual space, and a head-related transfer function is obtained according to the octree, the azimuth angle and the elevation angle, and then the left ear spatial audio and right ear spatial audio of the sound source to the user are generated according to the head-related transfer function, and finally the left ear environmental audio and the right ear environmental audio of the user are obtained, and the left ear spatial audio and the left ear environmental audio are mixed to obtain the left ear mixed audio, and the right ear spatial audio and the right ear environmental audio are mixed to obtain the right ear mixed audio. Among them, calculating the head position based on the forehead position and the back of the head position can improve the accuracy of the head position. The octree can accurately describe the spatial structure of the target virtual space. The head-related transfer function obtained by using the accurate head position and octree can accurately express the audio transmission relationship between the sound source and the head position, thereby improving the quality of the obtained spatial audio. At the same time, mixing the spatial audio and the ambient audio can improve the authenticity of the mixed audio and effectively improve the rendering quality of the audio.

[0066] Other beneficial effects of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0068] Figure 1 A flowchart of an audio rendering method for a virtual reality space provided in one embodiment of the present application;

[0069] Figure 2 A schematic diagram of UWB positioning provided in one embodiment of the present application;

[0070] Figure 3 A schematic diagram of the structure of an audio rendering device for a virtual reality space provided in one embodiment of the present application;

[0071] Figure 4 A schematic diagram of the structure of a terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0072] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0073] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0074] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0075] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0076] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0077] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized.

[0078] To address the problem of poor audio quality in existing virtual spaces, an embodiment of the present application provides an audio rendering method for a virtual reality space. The audio rendering method obtains the forehead position and back of the head position of a user in a real space corresponding to a target virtual space, and calculates the user's head position based on the forehead position and back of the head position. Then, the sound source position of the sound source in the target virtual space is obtained, and the azimuth and elevation angles between the sound source and the user are calculated based on the sound source position and the user's head position. An octree is then constructed for the target virtual space, and a head-related transfer function is obtained based on the octree, azimuth, and elevation angles. Then, based on the head-related transfer function, left-ear spatial audio and right-ear spatial audio of the sound source to the user are generated. Finally, the left-ear ambient audio and right-ear ambient audio of the user are obtained, and the left-ear spatial audio and the left-ear ambient audio are mixed to obtain left-ear mixed audio, and the right-ear spatial audio and the right-ear ambient audio are mixed to obtain right-ear mixed audio. Among them, calculating the head position based on the forehead position and the back of the head position can improve the accuracy of the head position. The octree can accurately describe the spatial structure of the target virtual space. The head-related transfer function obtained by using the accurate head position and octree can accurately express the audio transmission relationship between the sound source and the head position, thereby improving the quality of the obtained spatial audio. At the same time, mixing the spatial audio and the ambient audio can improve the authenticity of the mixed audio and effectively improve the rendering quality of the audio.

[0079] Next, the audio rendering method for the virtual reality space provided by this application is exemplified.

[0080] like Figure 1 As shown, the audio rendering method for a virtual reality space provided by this application includes the following steps:

[0081] Step 11: Obtain the forehead position and the back of the head position of the user in the real space corresponding to the target virtual space, and calculate the head position of the user based on the forehead position and the back of the head position.

[0082] The above-mentioned target virtual space is a three-dimensional virtual scene that requires audio rendering. The user is a person who experiences the three-dimensional virtual scene in the real space through VR glasses and other devices. The reverberation effect of the target virtual space and the real space is consistent, that is, the size of the real space is consistent with the size of the target virtual space, and the position and size of obstructions such as walls that affect sound reverberation are consistent with those of the target virtual space.

[0083] In some embodiments of the present application, ultra-wideband (UWB) positioning may be used to obtain the forehead position and the back of the head position of the user. The steps of calculating the user's head position based on the forehead position and the back of the head position are specifically as follows:

[0084] By formula:

[0085]

[0086] Calculate the user's head position P at the current moment c .

[0087] Among them, w f represents the frontal weight, w b represents the weight of the hindbrain, x f Indicates the horizontal coordinate position of the forehead, y f Indicates the vertical coordinate position of the forehead, z f Indicates the vertical coordinate position of the forehead, x b Indicates the horizontal coordinate position of the back of the brain, y b Indicates the vertical coordinate position of the hindbrain, z b Indicates the vertical coordinate position of the hindbrain.

[0088] It should be noted that the above coordinate positions are all coordinate positions in a three-dimensional rectangular coordinate system, which is a three-dimensional rectangular coordinate system with the center point of the real space as the origin. The head position is used to represent the user's position in the real space. The position tracking system in the VR device can be used to map the head position to the position of the user's corresponding virtual character in the target virtual space to synchronize the positions of the user and the virtual character. After obtaining the head position, algorithms such as sliding windows and long short-term memory network models (LSTM) can be used to perform anomaly detection and filtering on the head position based on the user's head position data at multiple historical moments to improve the accuracy of the head position. Specifically, by judging the following conditions:

[0089] lower_bound≤x o ≤upper_bound

[0090] Determine the horizontal coordinate position x of the head position o Is it an outlier? If x o If the above judgment conditions are met, then x o For normal data, if x o If the above conditions are not met, then x o For abnormal data, it needs to be retrieved. Among them, lower_bound represents the lower bound of the abnormal value, and upper_bound represents the upper bound of the abnormal value:

[0091] lower_bound=u i -3σ i -v i

[0092] upper_bound=u i +3σ i +v i

[0093] Among them, u i represents the window mean, σ i represents the window standard deviation, v i Indicates the window speed:

[0094]

[0095]

[0096]

[0097] Among them, x j represents the jth head position, i+19 represents the total number of head positions, x j+1 Indicates the j+1th head position.

[0098] Then LSTM is used to filter the data after anomaly detection to obtain the filtered head position.

[0099] It is worth mentioning that when positioning the user, the positioning accuracy may be low due to factors such as the positioning system and the environment. Calculating the head position based on the forehead position and the back of the head can improve the accuracy of the head position.

[0100] The following is an illustrative example of UWB positioning.

[0101] UWB node deployment Figure 2 As shown in the figure, a hexagonal array layout is adopted, with the six sides as equal as possible. The UWB nodes in the entire column are installed at a height of about 2m. The circle in the center of the figure is the UWB mobile node, that is, the user, and the other circles are UWB fixed nodes. The UWB mobile node sends pulse signals to the surrounding area. According to the time difference of the six UWB fixed nodes receiving the pulse signals, the arrival time difference positioning method is used to calculate the position of the UWB mobile node.

[0102] Step 12: Obtain the sound source position of the sound source in the target virtual space, and calculate the azimuth and elevation angles between the sound source and the user based on the sound source position and the user's head position.

[0103] The above-mentioned sound source is a virtual speaker in the target virtual space, such as a virtual sound system.

[0104] In some embodiments of the present application, before performing this step, it is necessary to synchronize the user's head position with the head position of the virtual character corresponding to the user, so as to calculate the azimuth and elevation between the sound source and the user in the target virtual space. Since the target virtual space and the real scene space are of the same size, the two can use corresponding three-dimensional coordinate systems. For example, the origin of the coordinate system in the target virtual space is the center of the ground, and the origin of the coordinate system in the real scene space is also the center of the ground. Therefore, when the coordinates of the real scene space are synchronized to the target virtual space, the coordinates remain unchanged, and the sound source position of the sound source can be obtained by constructing a three-dimensional rectangular coordinate system of the target virtual space. The above steps of calculating the azimuth and elevation between the sound source and the user based on the sound source position and the user's head position are specifically as follows:

[0105] The first step is to calculate the azimuth angle between the sound source and the user based on the sound source position and the user's head position.

[0106] Specifically, through the formula:

[0107] θ=arctan2(y s -y l ,x s -x l )

[0108] Calculate the azimuth angle θ.

[0109] Among them, y s Indicates the vertical coordinate position of the sound source, y l Indicates the vertical coordinate position of the user's head, x s Indicates the horizontal coordinate position of the sound source, x l Indicates the horizontal coordinate position of the user's head.

[0110] In the second step, based on the sound source position and the user's head position, the direction vector between the sound source and the user is calculated, and the elevation angle between the sound source and the user is calculated based on the direction vector.

[0111] Specifically, through the formula:

[0112]

[0113] Calculate the direction vector

[0114] Among them, z s Indicates the vertical coordinate position of the sound source, z l Indicates the vertical coordinate position of the user's head.

[0115] By formula:

[0116]

[0117] Calculating elevation angle

[0118] in, Represents the magnitude of the direction vector.

[0119] For example, the above calculation formula can be run using computer software such as MATLAB to obtain the azimuth and elevation angles between the sound source and the user.

[0120] It is worth mentioning that the accuracy of the elevation and azimuth angles calculated using the accurate user head position is improved, which can well describe the relative position between the sound source and the user.

[0121] Step 13: construct an octree for the target virtual space, and obtain a head-related transfer function based on the octree, azimuth angle, and elevation angle.

[0122] The octree is used to describe the spatial structure information of the target virtual space, and the head-related transfer function is used to describe the audio transmission relationship between the sound source and the user.

[0123] In some embodiments of the present application, the step of obtaining a head-related transfer function based on the octree, azimuth, and elevation angles is specifically as follows: constructing a head-related transfer function for each node of the octree to obtain a head-related transfer function library, and then querying the head-related transfer function library based on the azimuth and elevation angles to obtain the user's head-related transfer function. In the free field, the head-related transfer function is defined as:

[0124]

[0125]

[0126] Among them, PL and PR are the complex sound pressures generated by the sound source at the left and right ears of the user respectively. P0 is the complex sound pressure at the center of the head when the head is not present, and α is a parameter. In general, HL and HR are the horizontal azimuth angle θ and elevation angle of the sound source. A function of the distance r from the sound source to the center of the head and the angular frequency ω of the sound wave.

[0127] It can be seen that the head-related transfer function is related to the azimuth and elevation angles. When searching in the head-related transfer function library, the head-related transfer function corresponding to the azimuth and elevation angles between the user and the sound source is searched.

[0128] For example, the nodes of the octree are divided into two levels, the step length of the first level is 2 meters, and the step length of the second level is 0.5 meters. The octree can be constructed by running the octree pseudo code, which is:

[0129] OctreeNode = {bounds, data, children}; build an octree node;

[0130] Build_octree(data,bounds,step_size,depth,max_depth)→OctreeNode; Octree construction;

[0131] Build_first_level_octree(data,bounds,step_size=2,depth,max_depth)→Octree Node; where children_data[i]={data[j]|data[j][k]>mid_point[k]for k=0,1,2}; first level octree construction;

[0132] Buid_second_level_octree(data,bounds,step=0.5,depth,max,deth=2)→Octree Node;Second level octree construction;

[0133] Hrtf_data=query_octree(first_level_octree,point,step=2); octree query;

[0134] Second_level_octree=first_level_octree,children[index];

[0135] Hrtf_data=query_octree(second_level_octree,point,step=0.5).

[0136] The query formula in the header-related transfer function library is: cache.put(key,value).

[0137] Exemplarily, the above steps can be implemented using computer software such as Python to obtain the user's head-related transfer function.

[0138] It is worth mentioning that the octree can accurately describe the spatial structure of the target virtual space. The head-related transfer function obtained by using the accurate head position and the octree can accurately express the audio transmission relationship between the sound source and the head position.

[0139] Step 14: Generate left-ear spatial audio and right-ear spatial audio for the user according to the head-related transfer function.

[0140] In some embodiments of the present application, the step of generating the left-ear spatial audio and the right-ear spatial audio of the user according to the head-related transfer function specifically includes:

[0141] The first step is to obtain a left-ear mono signal transmitted from the sound source to the user's left ear, and obtain a right-ear mono signal transmitted from the sound source to the user's right ear.

[0142] It should be noted that the left-ear mono audio and the right-ear mono audio are in the target virtual space, and only the sound source is transmitted to the user. They can be generated by calculation or other methods using relevant computer software for VR audio.

[0143] In the second step, the left ear mono signal is convolved with the head-related transfer function to obtain the left ear spatial audio.

[0144] In the third step, the right ear mono signal is convolved with the head-related transfer function to obtain the right ear spatial audio.

[0145] For example, computer software such as Matlab can be used to convolve a mono signal with a head-related transfer function to obtain spatial audio.

[0146] It is worth mentioning that the improved quality of spatial audio obtained through precise head-related transfer functions can bring users a sense of immersion.

[0147] Step 15: Obtain the user's left ear ambient audio and right ear ambient audio, mix the left ear spatial audio and the left ear ambient audio to obtain left ear mixed audio, and mix the right ear spatial audio and the right ear ambient audio to obtain right ear mixed audio.

[0148] The above-mentioned left-ear ambient audio is the ambient audio transmitted to the user's left ear in the real space, and the right-ear ambient audio is the ambient audio transmitted to the user's right ear in the real space. The user's left-ear ambient audio and right-ear ambient audio can be obtained by setting a microphone or other sound pickup device on the user's left ear and right ear respectively.

[0149] In some embodiments of the present application, the step of mixing the left-ear spatial audio and the left-ear ambient audio to obtain the left-ear mixed audio specifically includes:

[0150] The first step is to perform weighted fusion of the left ear spatial audio and the left ear ambient audio to obtain the left ear weighted audio.

[0151] Specifically, through the formula:

[0152] L mix (t)=α(t)L' ent (t)+β(t)L' hrtf (t)

[0153] Calculate the left ear weighted audio L mix (t).

[0154] Among them, α(t) and β(t) are weight coefficients, L' ent (t) represents the left ear ambient audio, L' hrtf (t) represents the left ear spatial audio.

[0155] It should be noted that if there is more than one sound source in the target virtual space, steps 12, 13, and 14 are performed for each sound source, and the left ear spatial audio corresponding to all sound sources is added to the formula for calculating the left ear weighted audio in this step for weighted fusion.

[0156] In the second step, the left ear weighted audio is adjusted using Fourier transform to obtain the left ear mixed audio.

[0157] Specifically, through the formula:

[0158] L final (t) = F -1 {L' mix (f)}

[0159]

[0160]

[0161] Get the left ear mixed audio L final (t).

[0162] in, represents the inverse Fourier transform, L' mix (f) represents the normalized mixed audio, L mix (f) represents the initial left ear mixed audio, represents the Fourier transform.

[0163] It should be noted that the process of mixing the right ear spatial audio and the right ear ambient audio to obtain the right ear mixed audio is the same as the process of obtaining the left ear mixed audio mentioned above, that is, the right ear spatial audio and the right ear ambient audio are weighted and fused to obtain the right ear weighted audio, and then the right ear weighted audio is adjusted using Fourier transform to obtain the right ear mixed audio.

[0164] For example, after obtaining the left-ear mixed audio and the right-ear mixed audio, the left-ear mixed audio and the right-ear mixed audio can be played for the user using an audio playback device such as headphones or speakers used by the user.

[0165] It is worth mentioning that mixing spatial audio and ambient audio can improve the authenticity of the mixed audio, effectively improve the rendering quality of the audio, and enhance the user's sense of immersion.

[0166] The following is an exemplary description of the audio rendering device for the virtual reality space provided by this application.

[0167] like Figure 3 As shown, an embodiment of the present application provides an audio rendering device for a virtual reality space. The audio rendering device 300 for a virtual reality space includes:

[0168] A first calculation module 301 obtains a forehead position and a back-of-the-head position of a user in a real space corresponding to a target virtual space, and calculates a head position of the user based on the forehead position and the back-of-the-head position;

[0169] A second calculation module 302 obtains a sound source position of a sound source in the target virtual space and calculates an azimuth and elevation angle between the sound source and the user based on the sound source position and the user's head position;

[0170] Construction module 303 constructs an octree for the target virtual space and obtains a head-related transfer function based on the octree, azimuth, and elevation. The octree is used to describe the spatial structure of the target virtual space, and the head-related transfer function is used to describe the audio transmission relationship between the sound source and the user.

[0171] A generating module 304 generates a left-ear spatial audio and a right-ear spatial audio of the user according to the head-related transfer function;

[0172] The mixing module 305 obtains the user's left ear ambient audio and right ear ambient audio, and mixes the left ear spatial audio and the left ear ambient audio to obtain left ear mixed audio, and mixes the right ear spatial audio and the right ear ambient audio to obtain right ear mixed audio; the left ear ambient audio is the ambient audio transmitted to the user's left ear in the real scene space, and the right ear ambient audio is the ambient audio transmitted to the user's right ear in the real scene space.

[0173] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0174] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0175] like Figure 4 As shown, an embodiment of the present application provides a terminal device, and the terminal device D10 of this embodiment includes: at least one processor D100 ( Figure 4 Only one processor is shown in the figure), a memory D101, and a computer program D102 stored in the memory D101 and executable on the at least one processor D100, wherein the processor D100 implements the steps of any of the above method embodiments when executing the computer program D102.

[0176] Specifically, when the processor D100 executes the computer program D102, it obtains the forehead position and back of the head position of the user in the real space corresponding to the target virtual space, and calculates the user's head position based on the forehead position and back of the head position, then obtains the sound source position of the sound source in the target virtual space, and calculates the azimuth and elevation between the sound source and the user based on the sound source position and the user's head position, then constructs an octree for the target virtual space, and obtains the head-related transfer function based on the octree, azimuth and elevation, and then generates the left ear spatial audio and right ear spatial audio of the sound source to the user based on the head-related transfer function, finally obtains the left ear environmental audio and right ear environmental audio of the user, and mixes the left ear spatial audio and the left ear environmental audio to obtain the left ear mixed audio, and mixes the right ear spatial audio and the right ear environmental audio to obtain the right ear mixed audio. Among them, calculating the head position based on the forehead position and the back of the head position can improve the accuracy of the head position. The octree can accurately describe the spatial structure of the target virtual space. The head-related transfer function obtained by using the accurate head position and octree can accurately express the audio transmission relationship between the sound source and the head position, thereby improving the quality of the obtained spatial audio. At the same time, mixing the spatial audio and the ambient audio can improve the authenticity of the mixed audio and effectively improve the rendering quality of the audio.

[0177] The processor D100 may be a central processing unit (CPU), or may be another general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor.

[0178] In some embodiments, the memory D101 may be an internal storage unit of the terminal device D10, such as a hard disk or memory of the terminal device D10. In other embodiments, the memory D101 may also be an external storage device of the terminal device D10, such as a plug-in hard disk, a smart memory card (SMC, SmartMedia Card), a secure digital (SD, Secure Digital) card, a flash card, etc. equipped on the terminal device D10. Furthermore, the memory D101 may also include both an internal storage unit of the terminal device D10 and an external storage device. The memory D101 is used to store an operating system, an application program, a boot loader (BootLoader), data, and other programs, such as the program code of the computer program. The memory D101 may also be used to temporarily store data that has been output or is to be output.

[0179] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented.

[0180] An embodiment of the present application provides a computer program product. When the computer program product is run on a terminal device, the terminal device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0181] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the process in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it can implement the steps of each of the above-mentioned method embodiments. Among them, the computer program includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the audio rendering method device / terminal device of the virtual reality space, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium. For example, USB flash drive, mobile hard disk, magnetic disk or optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0182] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0183] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0184] The above is a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles described in the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for audio rendering in a virtual reality space, characterized in that: include: Obtaining a forehead position and a back-of-the-head position of a user in a real-life space corresponding to a target virtual space, and calculating a head position of the user based on the forehead position and the back-of-the-head position; Obtaining a sound source position of a sound source in the target virtual space, and calculating an azimuth and an elevation angle between the sound source and the user based on the sound source position and the head position of the user; An octree is constructed for the target virtual space, and a head-related transfer function is obtained based on the octree, the azimuth angle, and the elevation angle; the octree is used to describe spatial structure information of the target virtual space, and the head-related transfer function is used to describe an audio transmission relationship between the sound source and the user; generating, according to the head-related transfer function, left-ear spatial audio and right-ear spatial audio of the sound source for the user; Obtaining the left ear ambient audio and right ear ambient audio of the user, and mixing the left ear spatial audio and the left ear ambient audio to obtain left ear mixed audio, and mixing the right ear spatial audio and the right ear ambient audio to obtain right ear mixed audio; the left ear ambient audio is the ambient audio transmitted to the left ear of the user in the real scene space, and the right ear ambient audio is the ambient audio transmitted to the right ear of the user in the real scene space; The acquiring a head-related transfer function according to the octree, the azimuth angle, and the elevation angle includes: Constructing a head-related transfer function for each node of the octree to obtain a head-related transfer function library; The head-related transfer function of the user is obtained by querying the head-related transfer function library according to the azimuth angle and the elevation angle.

2. The audio rendering method according to claim 1, wherein The calculating the head position of the user at the current moment based on the forehead position and the back of the head position includes: By formula: Calculate the user's head position P at the current moment c ; Among them, w f represents the forehead weight, w b represents the weight of the hindbrain, x f Indicates the horizontal coordinate position of the forehead, y f Indicates the vertical coordinate position of the forehead, z f Indicates the vertical coordinate position of the forehead, x b Indicates the horizontal coordinate position of the back of the brain, y b Indicates the vertical coordinate position of the hindbrain, z b Indicates the vertical coordinate position of the hindbrain.

3. The audio rendering method according to claim 1, wherein The calculating, based on the sound source position and the head position of the user, the azimuth and elevation angles between the sound source and the user, includes: Calculating an azimuth angle between the sound source and the user based on the sound source position and the user's head position; A direction vector between the sound source and the user is calculated based on the sound source position and the head position of the user, and an elevation angle between the sound source and the user is calculated based on the direction vector.

4. The audio rendering method according to claim 3, wherein: The calculating the azimuth angle between the sound source and the user based on the sound source position and the head position of the user includes: By formula: θ=arctan2(y s -y l ,x s -x l ) Calculate the azimuth angle θ; Among them, y s Indicates the vertical coordinate position of the sound source, y l Indicates the vertical coordinate position of the user's head, x s Indicates the horizontal coordinate position of the sound source, x l Indicates the horizontal coordinate position of the user's head.

5. The audio rendering method according to claim 4, characterized in that The calculating, based on the sound source position and the head position of the user, a direction vector between the sound source and the user, includes: By formula: Calculate the direction vector Among them, z s Indicates the vertical coordinate position of the sound source, z l Indicates the vertical coordinate position of the user's head; The calculating the elevation angle between the sound source and the user based on the direction vector includes: By formula: Calculating elevation angle in, Represents the magnitude of the direction vector.

6. The audio rendering method according to claim 1, wherein: Generating the left-ear spatial audio and the right-ear spatial audio of the sound source for the user according to the head-related transfer function includes: Acquire a left-ear monophonic signal transmitted by the sound source to the left ear of the user, and acquire a right-ear monophonic signal transmitted by the sound source to the right ear of the user; Convolving the left ear monaural signal with the head-related transfer function to obtain the left ear spatial audio; The right-ear monophonic signal is convolved with the head-related transfer function to obtain the right-ear spatial audio.

7. The audio rendering method according to claim 1, wherein: The mixing of the left-ear spatial audio and the left-ear ambient audio to obtain the left-ear mixed audio includes: Performing weighted fusion on the left-ear spatial audio and the left-ear ambient audio to obtain left-ear weighted audio; The left-ear weighted audio is adjusted using Fourier transform to obtain left-ear mixed audio.

8. The audio rendering method according to claim 7, characterized in that: The step of weightedly fusing the left-ear spatial audio and the left-ear ambient audio to obtain left-ear weighted audio includes: By formula: L mix (t)=α(t)L' ent (t)+β(t)L' hrtf (t) Calculate the left ear weighted audio L mix (t); Among them, α(t) and β(t) are weight coefficients, L' ent (t) represents the left ear ambient audio, L' hrtf (t) represents the left ear spatial audio; The method of adjusting the left-ear weighted audio by using Fourier transform to obtain the left-ear mixed audio includes: By formula: Get the left ear mixed audio L final (t); in, represents the inverse Fourier transform, L' mix (f) represents the normalized mixed audio, L mix (f) represents the initial left ear mixed audio, represents the Fourier transform.

9. A terminal device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the audio rendering method for a virtual reality space according to any one of claims 1 to 8 is implemented.

Images