Sound effect rendering method and device, electronic equipment and readable storage medium

Abstract

The application discloses an audio effect rendering method and device, electronic equipment and a readable storage medium. The audio effect rendering method comprises the following steps: dynamically monitoring a current virtual reality environment, and determining an audio propagation correlation attribute corresponding to the virtual reality environment; performing audio parameter adjustment on original audio to be output at present according to the audio propagation correlation attribute, so as to obtain target audio; and rendering the target audio to the virtual reality environment. The application improves the accuracy of audio effect rendering of an extended reality application.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of extended reality application, and particularly relates to an audio effect rendering method and device, electronic equipment and a readable storage medium. BACKGROUND

[0002] Virtual reality ("VR"), augmented reality ("AR"), mixed reality ("MR"), and related extended reality technologies (collectively "XR") share the ability to present sensory information to a user of an XR system that corresponds to a virtual environment represented by data in a computer system, providing a truly immersive virtual reality experience by synchronously presenting virtual visuals and audio effects that match the virtual visuals.

[0003] However, the existing audio effect rendering method of the XR system only considers obtaining pre-stored original audio corresponding to an image frame of a current virtual reality environment from a storage area, and then rendering the current virtual reality environment using the pre-stored original audio, without considering that the virtual reality environment is variable, and due to differences in propagation effects or audio parameters of the same audio in different virtual reality environments, the audio effect rendered in the virtual reality environment often cannot match the virtual reality environment, and thus the sound heard by the user in the extended reality application is not real enough, and a real auditory experience cannot be obtained in the virtual reality environment, that is, the accuracy of the audio effect rendering in the extended reality application in the prior art is low, which seriously affects the immersion of the user in the virtual reality environment. SUMMARY

[0004] The main purpose of the present application is to provide an audio effect rendering method, device, electronic equipment and readable storage medium, aiming to solve the technical problem of how to improve the accuracy of the audio effect rendering in the extended reality application.

[0005] To achieve the above-mentioned purpose, the present application provides an audio effect rendering method, which is applied to a smart wearable device, and the steps of the audio effect rendering method include:

[0006] dynamically monitoring a current virtual reality environment and determining audio propagation related attributes corresponding to the virtual reality environment;

[0007] adjusting audio parameters of original audio to be output currently according to the audio propagation related attributes, to obtain target audio;

[0008] rendering the target audio to the virtual reality environment.

[0009] Optionally, the step of determining the audio propagation correlation attribute corresponding to the virtual reality environment comprises:

[0010] determining a sound source position of the original audio in the virtual reality environment, and a pickup position of the original audio;

[0011] determining an audio propagation path of the original audio in the virtual reality environment according to the sound source position and the pickup position;

[0012] identifying an occlusion attribute corresponding to a virtual occlusion in the audio propagation path, and taking the occlusion attribute as the audio propagation correlation attribute corresponding to the virtual reality environment.

[0013] Optionally, the step of adjusting an audio parameter of the original audio to be currently output according to the audio propagation correlation attribute to obtain target audio comprises:

[0014] querying an audio occlusion coefficient corresponding to the occlusion attribute from a first preset mapping data table, wherein the occlusion attribute is at least one of an occlusion material, an occlusion thickness, and an occlusion shape;

[0015] determining an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio occlusion coefficient, and adjusting an audio parameter of the original audio to be currently output according to the audio attenuation coefficient to obtain target audio.

[0016] Optionally, the step of querying the audio occlusion coefficient corresponding to the occlusion attribute from the first preset mapping data table comprises:

[0017] if there is no virtual occlusion in the audio propagation path, determining an audio propagation distance between the sound source position and the pickup position, determining an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio propagation distance, and performing the step of adjusting an audio parameter of the original audio to be currently output according to the audio attenuation coefficient to obtain target audio;

[0018] if there is a virtual occlusion in the audio propagation path, performing the step of querying the audio occlusion coefficient corresponding to the occlusion attribute from the first preset mapping data table.

[0019] Optionally, the occlusion attribute further comprises an occlusion range of the occlusion formed in a direction perpendicular to the audio propagation path, and the step of adjusting an audio parameter of the original audio to be currently output according to the audio propagation correlation attribute to obtain target audio comprises:

[0020] acquire an audio propagation area covered by the original audio in the audio propagation path, and determine an audio occlusion area covered by the virtual occlusion in the audio propagation path according to the occlusion range;

[0021] determine an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio occlusion area and the audio propagation area, and perform audio parameter adjustment on the original audio to be currently output according to the audio attenuation coefficient to obtain target audio.

[0022] Optionally, the step of determining the audio attenuation coefficient of the original audio in the virtual reality environment according to the audio occlusion area and the audio propagation area comprises:

[0023] calculating a ratio of the audio occlusion area to the audio propagation area to obtain an audio blocked area ratio; querying, from a second preset mapping data table, an audio attenuation coefficient associated with the audio blocked area ratio according to the audio blocked area ratio, and taking the associated audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment; or,

[0024] subtracting the audio occlusion area from the audio propagation area to obtain an actual sound propagation area of the original audio in the audio propagation path; determining a sound propagation angle of the original audio propagated from the sound source position to the pickup position through the sound propagation area, and determining a pickup offset angle of the original audio picked up by the pickup position according to the sound propagation angle; querying, from a third preset mapping data table, an audio attenuation coefficient corresponding to the pickup offset angle, and taking the corresponding audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment.

[0025] Optionally, the occlusion attribute is an occlusion material, and the step of identifying the occlusion attribute corresponding to the virtual occlusion in the audio propagation path comprises:

[0026] acquiring a surface texture map rendered on the virtual occlusion in the audio propagation path, determining an occlusion material mapped by the surface texture map, and taking the mapped occlusion material as the occlusion material corresponding to the virtual occlusion; or,

[0027] determining a current scene environment corresponding to the virtual reality environment, determining an occlusion material matched with the current scene environment, and taking the matched occlusion material as the occlusion material corresponding to the virtual occlusion.

[0028] The application further provides an audio effect rendering device applied to a smart wearable device, and the audio effect rendering device comprises:

[0029] a monitoring module configured to dynamically monitor a current virtual reality environment and determine an audio propagation correlation attribute corresponding to the virtual reality environment;

[0030] an adjusting module configured to perform audio parameter adjustment on original audio to be output according to the audio propagation correlation attribute, to obtain target audio;

[0031] a rendering module configured to render the target audio to the virtual reality environment.

[0032] The application further provides an electronic device, which is a physical device. The electronic device comprises a memory, a processor, and a program of the sound effect rendering method stored in the memory and executable on the processor. When the program of the sound effect rendering method is executed by the processor, the steps of the sound effect rendering method described above can be implemented.

[0033] The application further provides a readable storage medium, which is a computer readable storage medium. The computer readable storage medium stores a program for implementing a sound effect rendering method. When the program for implementing the sound effect rendering method is executed by a processor, the steps of the sound effect rendering method described above can be implemented.

[0034] The application further provides a computer program product, which comprises a computer program. When the computer program is executed by a processor, the steps of the sound effect rendering method described above can be implemented.

[0035] The application dynamically monitors the current virtual reality environment, determines the audio propagation correlation attribute corresponding to the virtual reality environment, wherein the audio propagation correlation attribute can include the distance between the user and the virtual sound source in the virtual reality environment, the sound direction of the virtual sound source, the occlusion material of the occlusion existing between the user and the virtual sound source, and the virtual indoor space type in which the user is located, and then the application adjusts the audio parameters of the original audio to be output according to the audio propagation correlation attribute corresponding to the virtual reality environment to obtain target audio, and then renders the target audio to the current virtual reality environment, so that the actual played audio signal is adapted to the current virtual reality environment, so that the virtual sound is like a real sound naturally generated in the virtual reality environment, and then the audio signal sounds more real and natural, and the immersion of the user to the virtual reality environment is improved. Compared with the existing XR system which extracts the pre-stored audio corresponding to the image frame of the current virtual reality environment from the storage area for synchronous playback to render the current virtual reality environment, the application also considers the influence of environmental changes on the propagation effect of sound or audio parameters, identifies the virtual reality environment in the XR scene, and adjusts the audio parameters of the pre-stored audio (i.e. the original audio to be output) according to the audio propagation correlation attribute corresponding to the virtual reality environment. After the audio parameter adjustment, the user is played, so that the user can hear the sound that matches the XR scene more, the sound effect rendered in the virtual reality environment is accurately matched with the virtual reality environment, the immersion of the user to the XR scene is enhanced, a more real and more immersive spatial audio experience is provided for the user, and thus the technical problem of how to improve the accuracy of sound effect rendering of the extended reality application is solved. BRIEF DESCRIPTION OF DRAWINGS

[0036] The accompanying drawings, which are incorporated into and form a part of the specification, illustrate an embodiment consistent with the present application and, together with the description, serve to explain the principles of the application.

[0037] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor.

[0038] Figure 1 The flowchart of the first embodiment of the sound effect rendering method of the present application;

[0039] Figure 2 The flowchart of the second embodiment of the sound effect rendering method of the present application;

[0040] Figure 3 The detailed flowchart of step S200 in the third embodiment of the sound effect rendering method of the present application;

[0041] Figure 4 The device structure diagram of the hardware running environment of the electronic device in the embodiments of the present application.

[0042] The purposes, functional features and advantages of the present application will be further described with reference to the embodiments and the accompanying drawings. DETAILED DESCRIPTION

[0043] In order to make the above objectives, characteristics and advantages of the present application more apparent, clear and easy to understand, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0044] In the embodiments, the smart wearable device of the present application can be, for example, a Mixed Reality (MR) device (such as MR glasses or a MR helmet), an Augmented Reality (AR) device (such as AR glasses or an AR helmet), a Virtual Reality (VR) device (such as VR glasses or a VR helmet), an Extended Reality (XR) device or some combination thereof.

[0045] The sound effect rendering method of the existing XR system only considers obtaining pre-stored audio corresponding to the image frame of the current virtual reality environment from a storage area and then using the pre-stored audio for rendering the current virtual reality environment, without considering that the virtual reality environment is changeable. Since the propagation effect or audio parameter of the same audio in different virtual reality environments is different, the sound effect rendered in the virtual reality environment often cannot match the virtual reality environment, and thus the sound heard by the user in the extended reality application is not real enough, and the user cannot obtain a real auditory experience in the virtual reality environment.

[0046] Embodiment One

[0047] Based on this, please refer to Figure 1 The present embodiment provides a sound effect rendering method, which is applied to a smart wearable device, and the sound effect rendering method comprises the following steps:

[0048] Step S100: dynamically monitoring a current virtual reality environment and determining an audio propagation associated attribute corresponding to the virtual reality environment;

[0049] The audio propagation related attribute can include, but is not limited to, a position distance between the user and a virtual sound emitter in the virtual reality environment, a sound emission orientation of the virtual sound emitter, a blocking object material between the user and the virtual sound emitter, and a virtual indoor space type in which the user is located.

[0050] As an example, when the virtual reality environment is a virtual reality scene in which the user is in conversation with a virtual object (e.g., a speaker A), the audio propagation related attribute corresponding to the virtual reality environment can be a position distance between the user and the speaker A in the virtual scene. It can be understood that, with other factors being the same, the farther the position distance, the smaller the loudness of the sound received by the user from the speaker A, and the more diffuse the sound field generated by the speaker A. The closer the position distance, the greater the loudness of the sound received by the user from the speaker A, and the more focused the sound field generated by the speaker A. Therefore, the embodiment can adjust the audio parameters of the original audio to be output according to the audio propagation related attribute corresponding to the virtual reality environment, i.e., the position distance to the virtual communication object, to obtain target audio matching the position distance.

[0051] As another example, when the virtual reality environment is a virtual reality scene in which the user hears a sound emitted by a virtual sound emitter (e.g., a sound emitter B), the audio propagation related attribute corresponding to the virtual reality environment can be a position distance between the user and the sound emitter B in the virtual scene, and a sound emission orientation of the sound emitter B. It can be understood that, with other factors being the same, the farther the position distance, the smaller the loudness of the sound received by the user from the sound emitter B, and the more diffuse the sound field generated by the sound emitter B. Meanwhile, the sound emission orientation of the sound emitter B directly facing the user is referred to as a target sound emission orientation. It can be understood by those skilled in the art that, with other factors being the same, the smaller the angle between the sound emission orientation of the sound emitter B and the target sound emission orientation, the greater the loudness and the clearer the sound received by the user from the sound emitter B. The greater the angle between the sound emission orientation of the sound emitter B and the target sound emission orientation, the smaller the loudness and the more blurred the sound received by the user from the sound emitter B. Therefore, the embodiment can adjust the audio parameters of the original audio to be output according to the audio propagation related attribute corresponding to the virtual reality environment, i.e., the position distance to the virtual sound emitter and the sound emission orientation of the virtual sound emitter, to obtain target audio matching the position distance and the sound emission orientation.

[0052] As still another example, when the virtual reality environment is a virtual reality scene in which the user hears a sound emitted by a virtual sound emitter (e.g., sound emitter C). In this case, the audio propagation related attribute corresponding to the virtual reality environment can be the occlusion material of an occlusion object existing between the user and the sound emitter C. It is easy to understand that, in the case of other factors being the same, different occlusion materials correspond to different audio occlusion coefficients (or sound insulation volume), so that the attenuation degree of the sound is different. For example, the audio occlusion coefficient of the occlusion material of glass is often greater than the audio occlusion coefficient of the occlusion material of metal. And the audio occlusion coefficient of the occlusion material of metal is greater than the audio occlusion coefficient of the occlusion material of rubber or wood. Those skilled in the art can know that, in the case of other factors being the same, the greater the audio occlusion coefficient of the occlusion object, the greater the attenuation degree of the sound, and the smaller the loudness and the more blurred the sound received by the user from the sound emitter C. Therefore, the embodiment can perform audio parameter adjustment on the original audio to be output according to the audio propagation related attribute corresponding to the virtual reality environment, i.e., the occlusion material of the occlusion object existing between the user and the sound emitter C, to obtain target audio matched with the occlusion material.

[0053] As still another example, when the virtual reality environment is a virtual reality scene in which the user hears a sound emitted by a virtual sound emitter (e.g., sound emitter D). In this case, the audio propagation related attribute corresponding to the virtual reality environment can be the virtual indoor space type in which the user is located. It should be noted that, in the case of other factors being the same, different virtual indoor space types correspond to different sound field expansion effects of the sound in the space corresponding to the virtual indoor space type. For example, if the sound emitter D or the user emits sound in a small room with closed walls, the sound reflected from the walls can cause short and natural reverberation signals. Large concert halls or cathedrals can cause longer reverberation. At the same time, the reverberation of the audio signal can present various sound characteristics based on the position or orientation of the surface reflecting the signal, or based on the material of the surface. For example, the reverberation reflected by a ceramic tile wall sounds different from the reverberation reflected by brick, carpet, plasterboard or other materials. The user can use these reverberation characteristics to audibly understand the size, shape and material composition of the virtual indoor space in which the user is located. Therefore, the embodiment can perform audio parameter adjustment on the original audio to be output according to the audio propagation related attribute corresponding to the virtual reality environment, i.e., the virtual indoor space type in which the user is located, to obtain target audio matched with the virtual indoor space type.

[0054] In step S200, the original audio to be output is adjusted according to the audio propagation related attribute to obtain target audio.

[0055] The audio parameters can include frequency, amplitude, period, etc. of the audio.

[0056] Step S300, render the target audio to the virtual reality environment.

[0057] The embodiment dynamically monitors the current virtual reality environment, determines the audio propagation correlation attribute corresponding to the virtual reality environment, wherein the audio propagation correlation attribute can include the distance between the user and the virtual sound source in the virtual reality environment, the sound emission orientation of the virtual sound source, the occlusion material of the occlusion existing between the user and the virtual sound source, and the virtual indoor space type in which the user is located, and then the embodiment adjusts the audio parameters of the original audio to be output according to the audio propagation correlation attribute corresponding to the virtual reality environment to obtain target audio, and renders the target audio to the current virtual reality environment, so that the actual played audio signal is adapted to the current virtual reality environment, so that the virtual sound is like a real sound naturally generated in the virtual reality environment, and thus the audio signal sounds more real and natural, and the immersion of the user to the virtual reality environment is improved. Compared with the existing XR system which extracts the pre-stored audio corresponding to the image frame of the current virtual reality environment from the storage area for synchronous playback to render the current virtual reality environment, the embodiment also considers the influence of environmental changes on the propagation effect of sound or audio parameters, identifies the virtual reality environment in the XR scene, and adjusts the audio parameters of the pre-stored audio (i.e. the original audio to be output) according to the audio propagation correlation attribute corresponding to the virtual reality environment. After the audio parameter adjustment, the user is played, so that the user can hear the sound that matches the XR scene better, the sound effect rendered in the virtual reality environment is accurately matched with the virtual reality environment, the immersion of the user to the XR scene is enhanced, a more real and more immersive spatial audio experience is provided for the user, and thus the technical problem of how to improve the accuracy of sound effect rendering of the extended reality application is solved.

[0058] In an implementable manner, please refer to Figure 2 The step of determining the audio propagation correlation attribute corresponding to the virtual reality environment comprises:

[0059] Step S110, determining the sound source position of the original audio in the virtual reality environment, and picking up the sound pickup position of the original audio;

[0060] It is easy to understand that, in general, in the virtual reality environment, the sound source position of the original audio refers to the position of the virtual sound source. And the sound pickup position of the original audio refers to the position of the user.

[0061] Step S120, determining the audio propagation path of the original audio in the virtual reality environment according to the sound source position and the sound pickup position;

[0062] In this embodiment, it is known to those skilled in the art that, after the sound source position and the sound pickup position are determined, an audio propagation path through which the sound propagates between the sound source position and the sound pickup position can be constructed according to the sound source position and the sound pickup position. It should be noted that, when the audio propagation path is constructed, the influence of the virtual barrier on the audio propagation path is not considered.

[0063] In step S130, a barrier attribute corresponding to the virtual barrier in the audio propagation path is identified, and the barrier attribute is taken as an audio propagation correlation attribute corresponding to the virtual reality environment.

[0064] In this embodiment, the barrier attribute can be at least one of a barrier material, a barrier thickness, and a barrier shape. It is easy to understand that different types of virtual barriers often correspond to different barrier materials, barrier thicknesses, or barrier shapes, and any one of the barrier material, the barrier thickness, and the barrier shape can affect the audio attenuation coefficient. It can be known that, under the condition that other factors are the same, different barrier materials correspond to different audio attenuation coefficients, so that the attenuation degree of the sound is different. For example, the audio attenuation coefficient of the barrier material of glass is often greater than the audio attenuation coefficient of the barrier material of metal. In addition, under the condition that other factors are the same, the thicker the barrier thickness, the greater the audio attenuation coefficient. It can be understood that different barrier shapes correspond to different audio attenuation coefficients. For example, the barrier shape has an inward cavity facing the virtual sound source, and compared with the barrier shape of a smooth entity or a smooth flat plate (the barrier range formed by the virtual barrier perpendicular to the direction of the audio propagation path is the same), the audio attenuation coefficient of the barrier is relatively greater.

[0065] If the sound effect rendering mode of the sound blocking effect is set only by judging whether there is a virtual barrier between the sound source position and the sound pickup position, without considering the actual properties of the virtual barrier, the sound blocking effect cannot accurately reflect the characteristics of the physical world in the virtual reality, and thus the sound heard by the user in the extended reality application is not realistic enough. That is, if the barrier properties such as the material and shape of the virtual barrier are not judged, the physical characteristics in the virtual reality world cannot be accurately reflected. Therefore, the embodiment determines the sound source position where the original audio is generated in the virtual reality environment, and the sound pickup position where the original audio is picked up, and then determines the audio propagation path of the original audio in the virtual reality environment according to the sound source position and the sound pickup position, and identifies the barrier properties of the virtual barrier in the audio propagation path. The barrier properties are taken as the audio propagation correlation properties of the virtual reality environment, so that the actual properties of the virtual barrier in the virtual reality environment are considered to allow more detailed and more credible presentation of the audio parameter signal, so that the sound heard by the user in the extended reality application is more realistic and natural, ensuring that the rendered sound effect accurately matches the virtual reality environment, and greatly improving the immersion experience of the user in the virtual reality environment.

[0066] In a possible implementation, the step of adjusting the original audio to be output according to the audio propagation correlation properties to obtain target audio includes:

[0067] In step A10, the audio blocking coefficient corresponding to the barrier property is queried from the first preset mapping data table, wherein the barrier property is at least one of a barrier material, a barrier thickness, and a barrier shape.

[0068] In the embodiment, the first preset mapping data table stores barrier properties and audio blocking coefficients mapped with the barrier properties, wherein the barrier properties are at least one of a barrier material, a barrier thickness, and a barrier shape. When the barrier property corresponding to the virtual barrier in the audio propagation path is identified, the audio blocking coefficient corresponding to the barrier property can be queried from the first preset mapping data table.

[0069] In the embodiment, the mapping relationship between the barrier properties and the audio blocking coefficients stored in the first preset mapping data table can be calibrated by the smart wearable device before leaving the factory through experimental research, and then stored in the system of the smart wearable device. Alternatively, the mapping relationship can be output to the system of the smart wearable device by the user after the smart wearable device leaves the factory.

[0070] In this embodiment, the occlusion attribute of the virtual occlusion can be a default configuration of the XR system based on specific application scenarios, or it can be customized by the user according to their personal needs, thereby achieving different spatial audio occlusion effects. This allows users to adjust the occlusion effect according to specific application scenarios or usage habits, thereby flexibly adjusting the listening effect of spatial audio in the entire virtual reality world.

[0071] Step A20: Determine the audio attenuation coefficient of the original audio in the virtual reality environment based on the audio occlusion coefficient, and adjust the audio parameters of the original audio to be output based on the audio attenuation coefficient to obtain the target audio.

[0072] It's easy to understand that the larger the audio occlusion coefficient, the greater the degree of sound attenuation. Therefore, the audio attenuation coefficient of the original audio is greater when it propagates in a virtual reality environment.

[0073] In the real world, obstructions attenuate sound, and this attenuation effect is usually measured by sound insulation. The acoustic characteristics of sound insulation (i.e., the audio blocking coefficient in this embodiment) are primarily related to the incident sound frequency and direction, and secondarily to the material's density, thickness, porosity, damping, etc. Sound insulation can be measured by the transmission coefficient τ, and the sound insulation amount R can be measured by the following formula (1):

[0074]

[0075] The smaller the transmittance coefficient, the greater the sound insulation and the better the sound insulation effect.

[0076] For a single-layer homogeneous wall (shading object), assuming the wall area is infinitely large, the influence of boundary conditions can be ignored. Furthermore, the wall is considered a compliant board without stiffness, and the actions of each part are independent, thus the internal stress and elasticity of the wall can be neglected. The sound insulation of the wall depends on its weight per unit area and the frequency of the incident sound wave. When sound waves are randomly incident on a finite-sized wall panel, the sound insulation R of the wall can be measured by the following formula (2):

[0077] R = 20log 10 fm+k=20log 10 f+20log 10 m+k (2)

[0078] Where f represents the frequency of the incident sound wave (Hz), and m represents the surface density of the wall (kg / m³). 2 ), k is a constant, and when the sound wave is randomly incident, k = -48.

[0079] Based on the above formula, the audio blocking coefficient O in this embodiment is positively correlated with the sound insulation R, as can be seen from formula (3):

[0080] O = nR (3)

[0081] Wherein n>1, is a constant; when the incident sound wave is consistent, for different material walls, the audio shielding coefficient is only related to the surface density of the wall, that is, related to the material density of the wall. Generally, the greater the density of the wall material, the greater the shielding sound insulation volume, the greater the audio shielding coefficient, and the more obvious the shielding effect. That is, the audio shielding coefficient and the material density establish a corresponding relationship, and different audio shielding coefficients correspond to different materials.

[0082] In the embodiment, the audio shielding coefficient corresponding to the shielding object attribute is obtained by querying from the first preset mapping data table, wherein the shielding object attribute is at least one of a shielding object material, a shielding object thickness, and a shielding object shape, and the audio attenuation coefficient of the original audio in the virtual reality environment is determined according to the audio shielding coefficient, and the audio parameter adjustment is performed on the original audio to be currently output according to the audio attenuation coefficient to obtain target audio, so that the embodiment starts from the attenuation effect of the shielding object on sound in the real world, designs a corresponding relationship between the shielding object attribute (such as the shielding object material, the shielding object thickness, or the shielding object shape) and the audio shielding coefficient in the spatial audio, and further improves the accuracy of sound effect rendering of the extended reality application, and strengthens the immersion of the user in the virtual reality environment.

[0083] In a possible implementation, the step of obtaining the audio shielding coefficient corresponding to the shielding object attribute from the first preset mapping data table includes:

[0084] Step B10, if there is no virtual shielding object in the audio propagation path, determining the audio propagation distance between the sound source position and the sound pickup position, determining the audio attenuation coefficient of the original audio in the virtual reality environment according to the audio propagation distance, and performing the step of adjusting the audio parameter of the original audio to be currently output according to the audio attenuation coefficient to obtain target audio.

[0085] Step B20, if there is a virtual shielding object in the audio propagation path, performing the step A10 of obtaining the audio shielding coefficient corresponding to the shielding object attribute from the first preset mapping data table.

[0086] Since there can also be no virtual occlusion in the audio propagation path in the virtual reality environment, in order to further ensure that the rendered sound effect is accurately matched with the virtual reality environment and improve the robustness of the sound effect rendering method of the embodiment, the embodiment determines the audio attenuation coefficient of the original audio in the virtual reality environment according to the audio propagation distance between the sound source position and the pickup position when it is identified that there is no virtual occlusion in the audio propagation path, so as to facilitate subsequent audio parameter adjustment of the original audio to be output according to the audio attenuation coefficient, so as to obtain target audio matched with the audio attenuation coefficient, and then provide the user with more real and more immersive spatial audio experience.

[0087] In an implementable manner, the occlusion attribute is an occlusion material, and the step of identifying the occlusion attribute corresponding to the virtual occlusion in the audio propagation path comprises:

[0088] Step C10, collecting a surface texture map rendered on the virtual occlusion in the audio propagation path, determining an occlusion material mapped by the surface texture map, and taking the mapped occlusion material as the occlusion material corresponding to the virtual occlusion; or,

[0089] In the embodiment, it is easy to understand that the surface texture map should have the appearance attribute corresponding to the mapped occlusion material. For example, the mapped occlusion material is blue translucent glass, and the surface texture map is a blue translucent glass appearance image. For another example, the mapped occlusion material is stainless steel metal, and the surface texture map is a stainless steel appearance image with metal luster.

[0090] In the embodiment, the one-to-one mapping relationship data between each surface texture map and occlusion material can be pre-stored in the system, so as to determine the occlusion material mapped by the current surface texture map according to the mapping relationship data, and then identify the occlusion material corresponding to the virtual occlusion in the virtual reality environment.

[0091] Step C20, determining a current scene environment corresponding to the virtual reality environment, determining an occlusion material matched with the current scene environment, and taking the matched occlusion material as the occlusion material corresponding to the virtual occlusion.

[0092] As an example, in step C20, the step of determining the occlusion material matched with the current scene environment comprises:

[0093] Step D10, if the current scene environment is an indoor room, determining that the occlusion material matched with the current scene environment is a concrete material;

[0094] Step D20, if the current scene environment is a forest, determining that the occluder material matched with the current scene environment is a tree material;

[0095] Step D30, if the current scene environment is an ice field, determining that the occluder material matched with the current scene environment is an ice material.

[0096] The embodiment can realize automatic identification of the texture map material of the virtual occluder in the extended reality application, realize the spatial audio occlusion effect matched with the texture map material, and further provide the user with a more realistic and more immersive spatial audio experience.

[0097] The embodiment can also determine the occluder material matched with the current scene environment according to different scene environments, realize the spatial audio occlusion effect matched with the occluder material, and further make the sound heard by the user in the extended reality application more realistic and natural, and ensure that the rendered sound effect is accurately matched with the virtual reality environment.

[0098] Embodiment Two

[0099] Based on the first embodiment of the present application, the same or similar contents as the above-mentioned embodiment one can refer to the above introduction, and the subsequent description will not be repeated. On this basis, please refer to Figure 3 The occluder attribute further includes an occlusion range formed by the occluder in a direction perpendicular to the audio propagation path, and the step of adjusting the audio parameters of the original audio to be currently output according to the audio propagation association attribute to obtain the target audio includes:

[0100] Step S210, obtaining an audio propagation area covered by the original audio in the audio propagation path, and determining an audio occlusion area covered by the virtual occluder in the audio propagation path according to the occlusion range;

[0101] In the embodiment, it should be noted that when the audio propagation area is constructed, the influence of the virtual occluder on the audio propagation area is not considered.

[0102] Step S220, determining an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio occlusion area and the audio propagation area, and adjusting the audio parameters of the original audio to be currently output according to the audio attenuation coefficient to obtain the target audio.

[0103] As an example, in step S220, the step of determining the audio attenuation coefficient of the original audio in the virtual reality environment according to the audio blocking area and the audio propagation area includes:

[0104] Step E10, calculate the ratio of the audio blocking area to the audio propagation area to obtain an audio blocked area ratio; according to the audio blocked area ratio, query the audio attenuation coefficient associated with the audio blocked area ratio from the second preset mapping data table, and take the associated audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment.

[0105] The embodiment calculates the ratio of the audio blocking area to the audio propagation area to obtain an audio blocked area ratio, and then queries the audio attenuation coefficient associated with the audio blocked area ratio from the second preset mapping data table according to the audio blocked area ratio, and takes the associated audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment, so as to facilitate subsequent audio parameter adjustment of the original audio to be output according to the audio attenuation coefficient, obtain target audio matching the audio attenuation coefficient, and further provide the user with a more realistic spatial audio experience, and ensure that the rendered sound effect is accurately matched with the virtual reality environment.

[0106] As another example, in step S220, the step of determining the audio attenuation coefficient of the original audio in the virtual reality environment according to the audio blocking area and the audio propagation area includes:

[0107] Step E20, subtract the audio blocking area from the audio propagation area to obtain an actual sound propagation area of the original audio in the audio propagation path; determine a sound propagation angle of the original audio propagated from the sound source position to the pickup position through the sound propagation area, and determine a pickup offset angle of the pickup position to pick up the original audio according to the sound propagation angle, query the audio attenuation coefficient corresponding to the pickup offset angle from a third preset mapping data table, and take the corresponding audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment.

[0108] Compared with the method of directly determining the audio attenuation coefficient by calculating the proportion of the audio blocked area in the step E10, the embodiment also considers the influence of the pickup deviation angle on the audio attenuation. It is easy to understand that the pickup orientation when the pickup position is directly opposite the sound source position is the target pickup orientation, and the angle by which the pickup orientation of the pickup position deviates from the target pickup orientation is the pickup deviation angle. It is easy to understand that the smaller the pickup deviation angle is, the louder and clearer the sound received by the pickup position is. The larger the pickup deviation angle is, the quieter and blurrier the sound received by the pickup position is. Therefore, the embodiment obtains the actual sound propagation area of the original audio in the audio propagation path by subtracting the audio blocked area from the audio propagation area, determines the sound propagation angle of the original audio propagated from the sound source position to the pickup position through the sound propagation area, determines the pickup deviation angle of the pickup position to pick up the original audio according to the sound propagation angle, queries the audio attenuation coefficient corresponding to the pickup deviation angle from the third preset mapping data table, and takes the corresponding audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment, so as to facilitate more accurate audio parameter adjustment of the original audio to be output subsequently, and further improve the immersion and authenticity in the auditory aspect in the extended reality application.

[0109] Embodiment three

[0110] The embodiment of the application further provides an audio effect rendering device, which is applied to a smart wearable device, and the audio effect rendering device comprises:

[0111] A monitoring module is configured to dynamically monitor a current virtual reality environment and determine audio propagation related attributes corresponding to the virtual reality environment.

[0112] An adjusting module is configured to perform audio parameter adjustment on original audio to be output currently according to the audio propagation related attributes, to obtain target audio.

[0113] A rendering module is configured to render the target audio to the virtual reality environment.

[0114] Optionally, the monitoring module is further configured to:

[0115] determine a sound source position generating the original audio and a pickup position picking up the original audio in the virtual reality environment;

[0116] determine an audio propagation path of the original audio in the virtual reality environment according to the sound source position and the pickup position;

[0117] identify a blocking object attribute corresponding to a virtual blocking object in the audio propagation path, and take the blocking object attribute as the audio propagation related attribute corresponding to the virtual reality environment.

[0118] Optionally, the adjusting module is further configured to:

[0119] query, from a first preset mapping data table, an audio occlusion coefficient corresponding to the occluder attribute, wherein the occluder attribute is at least one of an occluder material, an occluder thickness, and an occluder shape.

[0120] determine an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio occlusion coefficient, and perform audio parameter adjustment on the original audio to be currently output according to the audio attenuation coefficient to obtain target audio.

[0121] Optionally, the adjusting module is further configured to:

[0122] if the audio propagation path does not include a virtual occluder, determine an audio propagation distance between the sound source position and the sound pickup position, determine an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio propagation distance, and perform the step of performing audio parameter adjustment on the original audio to be currently output according to the audio attenuation coefficient to obtain target audio.

[0123] if the audio propagation path includes a virtual occluder, perform the step of querying, from a first preset mapping data table, an audio occlusion coefficient corresponding to the occluder attribute.

[0124] Optionally, the occluder attribute further includes an occlusion range of the occluder formed in a direction perpendicular to the audio propagation path, and the adjusting module is further configured to:

[0125] obtain an audio propagation area covered by the original audio in the audio propagation path, and determine an audio occlusion area covered by the virtual occluder in the audio propagation path according to the occlusion range.

[0126] determine an audio attenuation coefficient of the original audio in the virtual reality environment according to the audio occlusion area and the audio propagation area, and perform audio parameter adjustment on the original audio to be currently output according to the audio attenuation coefficient to obtain target audio.

[0127] Optionally, the adjusting module is further configured to:

[0128] calculate a ratio of the audio occlusion area to the audio propagation area to obtain an audio blocked area ratio, query, from a second preset mapping data table, an audio attenuation coefficient associated with the audio blocked area ratio according to the audio blocked area ratio, and use the associated audio attenuation coefficient as the audio attenuation coefficient of the original audio in the virtual reality environment; or

[0129] Subtract the audio blocking area from the audio propagation area to obtain an actual sound propagation area of the original audio in the audio propagation path; determine a sound propagation angle of the original audio propagated from the sound source position to the sound pickup position through the sound propagation area, and determine a sound pickup offset angle of the sound pickup position picking up the original audio according to the sound propagation angle; query, from a third preset mapping data table, an audio attenuation coefficient corresponding to the sound pickup offset angle to take the corresponding audio attenuation coefficient as an audio attenuation coefficient of the original audio in the virtual reality environment.

[0130] Optionally, the attribute of the barrier is a barrier material, and the monitoring module is further configured to:

[0131] acquire a surface texture map rendered on the virtual barrier in the audio propagation path, determine a barrier material mapped by the surface texture map, and take the mapped barrier material as the barrier material corresponding to the virtual barrier; or

[0132] determine a current scene environment corresponding to the virtual reality environment, determine a barrier material matched with the current scene environment, and take the matched barrier material as the barrier material corresponding to the virtual barrier.

[0133] The sound effect rendering device provided by the embodiments of the present application adopts the sound effect rendering method in the above-mentioned embodiment one or embodiment two, and solves the technical problem of how to improve the accuracy of sound effect rendering in extended reality application. Compared with the prior art, the beneficial effects of the sound effect rendering device provided by the embodiments of the present application are the same as those of the sound effect rendering method provided by the above-mentioned embodiments, and other technical features in the sound effect rendering device are the same as those disclosed in the previous embodiment method, which will not be repeated here.

[0134] Embodiment four

[0135] The embodiments of the present application provide an electronic device, which comprises at least one processor and a memory in communication connection with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the sound effect rendering method in the above-mentioned embodiment one.

[0136] Reference will be made to the following Figure 4The diagram illustrates a structural schematic of an electronic device suitable for implementing embodiments of the present disclosure. The electronic devices in the embodiments of the present disclosure may include, but are not limited to, Mixed Reality (MR) devices (e.g., MR glasses or MR helmets), Augmented Reality (AR) devices (e.g., AR glasses or AR helmets), Virtual Reality (VR) devices (e.g., VR glasses or VR helmets), Extended Reality (XR) devices, or some combination thereof, and other smart wearable devices. Figure 4 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0137] like Figure 4 As shown, the electronic device may include a processing unit 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM 1002) or a program loaded from a storage device into a random access memory (RAM 1004). The RAM 1004 also stores various programs and data required for the operation of the electronic device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. Input / output (I / O) interfaces are also connected to the bus 1005.

[0138] Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the electronic device to communicate wirelessly or wiredly with other devices to exchange data. Although electronic devices with various systems are shown in the figures, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0139] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of embodiments of this disclosure.

[0140] The electronic device provided by the present application adopts the sound effect rendering method in the above embodiment one or embodiment two, and solves the technical problem of how to improve the accuracy of sound effect rendering in extended reality application. Compared with the prior art, the electronic device provided by the embodiment of the present application has the same beneficial effects as the sound effect rendering method provided by the above embodiment one, and other technical features in the electronic device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0141] It should be understood that parts of the present disclosure can be realized by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0142] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0143] Embodiment five

[0144] The embodiment of the present application provides a computer readable storage medium having computer readable program instructions stored thereon, and the computer readable program instructions are used to execute the sound effect rendering method in the above embodiment one.

[0145] The computer readable storage medium provided by the embodiment of the present application may, for example, be a U disk, but is not limited to an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, system or device, or any combination of the above. More specific examples of computer readable storage media can include, but are not limited to: an electric connection with one or more conductive wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with an instruction execution system, system or device. The program code contained on the computer readable storage medium can be transmitted by any appropriate medium, including but not limited to: electric wire, optical cable, RF (radio frequency), etc., or any suitable combination of the above.

[0146] The above computer readable storage medium can be contained in the electronic device; or it can exist separately without being assembled into the electronic device.

[0147] The computer readable storage medium described above carries one or more programs, when the one or more programs are executed by the electronic device, cause the electronic device to: dynamically monitor a current virtual reality environment, and determine an audio propagation related attribute corresponding to the virtual reality environment; perform audio parameter adjustment on original audio to be output currently according to the audio propagation related attribute, to obtain target audio; and render the target audio to the virtual reality environment.

[0148] Computer program code for carrying out operations of the present disclosure can be written in one or more programming languages or combinations of languages including object oriented programming languages such as Java, Smalltalk, C++ or conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0149] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur in a different order than that noted in the figures. For example, two blocks noted in succession can in fact be executed substantially concurrently or in the opposite order, depending on the functionality involved. It should also be noted that each block in the block diagrams and / or flow diagrams, and combinations of blocks in the block diagrams and / or flow diagrams, can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0150] The modules described in the embodiments of the present disclosure can be implemented in the form of software, or can be implemented in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0151] The computer readable storage medium provided by the application stores computer readable program instructions for executing the sound effect rendering method, and solves the technical problem of how to improve the accuracy of sound effect rendering of the extended reality application. Compared with the prior art, the beneficial effects of the computer readable storage medium provided by the embodiment of the application are the same as those of the sound effect rendering method provided by the first embodiment or the second embodiment, and are not described here.

[0152] Embodiment six

[0153] The embodiment of the application further provides a computer program product, comprising a computer program, and the computer program is executed by a processor to realize the steps of the sound effect rendering method.

[0154] The computer program product provided by the application solves the technical problem of how to improve the accuracy of sound effect rendering of the extended reality application. Compared with the prior art, the beneficial effects of the computer program product provided by the embodiment of the application are the same as those of the sound effect rendering method provided by the first embodiment or the second embodiment, and are not described here.

[0155] The above is only the preferred embodiment of the application, and does not limit the patent scope of the application, and any equivalent structure or equivalent process transformation using the content of the specification and the drawings, or direct or indirect application in other related technical fields, are also included in the patent processing scope of the application.

Claims

1. A sound effect rendering method, characterized in that, The sound effect rendering method is applied to smart wearable devices, and the steps of the sound effect rendering method include: The system dynamically monitors the current virtual reality environment and determines the location of the sound source that generates the original audio in the virtual reality environment, as well as the pickup location for picking up the original audio. Based on the sound source location and the sound pickup location, determine the audio propagation path of the original audio in the virtual reality environment; Identify the occlusion attributes corresponding to the virtual occlusion in the audio propagation path, wherein the occlusion attributes include the occlusion range formed by the virtual occlusion in the direction perpendicular to the audio propagation path; Obtain the audio propagation area covered by the original audio in the audio propagation path, and determine the audio occlusion area covered by the virtual occlusion object in the audio propagation path based on the occlusion range; Subtracting the audio obstruction area from the audio propagation area yields the actual sound propagation area of ​​the original audio in the audio propagation path. The sound propagation angle of the original audio from the sound source location through the sound propagation area to the pickup location is determined, and the pickup offset angle of the pickup location in picking up the original audio is determined according to the sound propagation angle. The pickup offset angle is the angle by which the pickup orientation of the pickup location deviates from the target pickup orientation, and the target pickup orientation is the pickup orientation when the pickup location is directly facing the sound source location. The audio attenuation coefficient corresponding to the pickup offset angle is obtained by querying the third preset mapping data table; The original audio is adjusted according to the audio attenuation coefficient to obtain the target audio. The target audio is rendered into the virtual reality environment.

2. A sound effect rendering device, characterized in that, The sound effect rendering device is used in smart wearable devices, and the sound effect rendering device includes: The monitoring module is used to dynamically monitor the current virtual reality environment and determine the location of the sound source that generates the original audio in the virtual reality environment, as well as the pickup location of the original audio; determine the audio propagation path of the original audio in the virtual reality environment based on the sound source location and the pickup location; identify the occlusion attributes corresponding to virtual occlusions in the audio propagation path, the occlusion attributes including the occlusion range formed by the virtual occlusion in the direction perpendicular to the audio propagation path; An adjustment module is used to obtain the audio propagation area covered by the original audio in the audio propagation path, and determine the audio occlusion area covered by the virtual occlusion object in the audio propagation path based on the occlusion range; subtract the audio occlusion area from the audio propagation area to obtain the actual sound propagation area of ​​the original audio in the audio propagation path; determine the sound propagation angle of the original audio from the sound source position through the sound propagation area to the pickup position, and determine the pickup offset angle of the pickup position picking up the original audio based on the sound propagation angle, wherein the pickup offset angle is the angle by which the pickup orientation of the pickup position deviates from the target pickup orientation, and the target pickup orientation is the pickup orientation when the pickup position is directly facing the sound source position; query the audio attenuation coefficient corresponding to the pickup offset angle from a third preset mapping data table; and adjust the audio parameters of the original audio according to the audio attenuation coefficient to obtain the target audio. The rendering module is used to render the target audio to the virtual reality environment.

3. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the steps of the sound effect rendering method of claim 1.

4. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a program that implements the sound effect rendering method. The program that implements the sound effect rendering method is executed by a processor to implement the steps of the sound effect rendering method as described in claim 1.

Images