Audio output method and device, readable storage medium and product

Abstract

The invention discloses an audio output method and device, a readable storage medium and a product, and relates to the technical field of spatial audio, the audio output method is applied to a game engine, an audio shielding component is preset in the game engine, and the audio output method comprises the following steps: obtaining all objects in a preset display scene, determining a shielding object in each object; fitting a curved surface equation of the shielding object through the audio shielding assembly; and inputting the curved surface equation and an original signal to be output into a preset audio shielding algorithm to obtain a processed audio signal, and outputting the audio signal, the preset audio shielding algorithm being used for calculating the signal attenuation intensity of the sound on the curved surface equation, and adjusting the original signal based on the signal attenuation intensity. The immersion of audio output by the game engine is improved.

Description

Technical Field

[0001] This application relates to the field of spatial audio technology, and in particular to an audio output method, device, readable storage medium and product. Background Technology

[0002] A game engine is a software framework designed to simplify the development process of video games. It provides developers with a suite of tools and features needed to create games, eliminating the need for developers to write low-level code from scratch.

[0003] However, game engines only support simple audio processing such as volume and tone adjustment, resulting in poor audio immersion and failing to provide players with a good immersive experience.

[0004] Therefore, how to improve the immersiveness of the audio output by game engines is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The main purpose of this application is to provide an audio output method, device, readable storage medium, and product, which aims to solve the technical problem of how to improve the immersiveness of audio output by game engines.

[0006] To achieve the above objectives, this application provides an audio output method, wherein the game engine has a pre-configured audio occlusion component, and the audio output method includes the following steps:

[0007] Obtain all objects in the preset display scene and identify the occluders in each object;

[0008] The surface equation of the obstruction is fitted by the audio occlusion component;

[0009] The surface equation and the original signal to be output are input into a preset audio occlusion algorithm to obtain a processed audio signal, and the audio signal is output. The preset audio occlusion algorithm is used to calculate the signal attenuation intensity of the sound on the surface equation and adjust the original signal based on the signal attenuation intensity.

[0010] In one embodiment, the step of fitting the surface equation of the occlusion using the audio occlusion component includes:

[0011] The target intersection point between the sound source ray emitted by the preset virtual sound source and the occluder is obtained through the audio occlusion component, and the surface equation of the occluder is fitted based on the target intersection point, wherein the target intersection point is the intersection point formed between the sound ray emitted by the preset virtual sound source and the occluder during the propagation to the preset object.

[0012] In one embodiment, the step of fitting the surface equation of the occluder based on the target intersection point includes:

[0013] Select a preset number of target intersection points as selected intersection points, and fit the candidate surface equation of the occluder based on the selected intersection points;

[0014] The unselected target intersection points are determined as unselected intersection points, and the distance between the coordinate position of each unselected intersection point and the candidate surface equation is calculated;

[0015] Calculate the sum of all the distances, reselect a preset number of target intersection points as selected intersection points, and return to the step of fitting the candidate surface equation of the occluder based on the selected intersection points;

[0016] Once the preset fitting termination condition is met, the candidate surface equation corresponding to the minimum value among all the sums is selected as the surface equation of the occluder.

[0017] In one embodiment, the step of selecting a preset number of target intersection points as selected intersection points includes:

[0018] Density clustering is performed on each of the target intersection points to obtain clusters;

[0019] A preset number of target intersection points are selected from the clusters.

[0020] In one embodiment, the step of fitting the candidate surface equation of the occluder based on the selected intersection points includes:

[0021] Establish a centroid coordinate system with the centroid of the obstruction as the origin, and calculate the coordinate positions of each selected intersection point in the centroid coordinate system.

[0022] Based on the coordinate positions of each selected intersection point, a surface fitting is performed to obtain the candidate surface equation, and the candidate surface equation is transformed into the world coordinate system to obtain the surface equation of the occlusion.

[0023] In one embodiment, after the step of determining the obstructions in each of the objects, the method further includes:

[0024] The occlusion is subjected to regular object recognition to obtain the recognition result;

[0025] If the recognition result indicates that the occluder is an irregular object, then the occluder is divided into multiple sub-occluders, wherein at least one of the sub-occluders is a regular object;

[0026] Based on each of the sub-occluders, the step of fitting the surface equation of the occluder using the audio occlusion component is performed.

[0027] In one embodiment, after the step of performing regular object recognition on the occluded object to obtain the recognition result, the method further includes:

[0028] If the recognition result indicates that the occluder is a regular object, then the description information of the occluder is obtained, wherein the description information includes position, orientation and size;

[0029] The description information and the original signal are input into a preset audio occlusion algorithm to obtain a processed audio signal, and the audio signal is output.

[0030] In addition, to achieve the above objectives, this application also provides an audio output device, the audio output device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio output method as described above.

[0031] In addition, to achieve the above objectives, this application also provides a readable storage medium, which is a computer-readable storage medium, on which a program implementing an audio output method is stored, and the program implementing the audio output method is executed by a processor to implement the steps of the audio output method as described above.

[0032] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the audio output method described above.

[0033] One or more technical solutions proposed in this application have at least the following technical effects:

[0034] An audio occlusion component is pre-configured in the game engine. This component acquires all objects in a preset display scene and identifies occlusions within each object. The component then fits the surface equation of the occlusion. This surface equation, along with the original signal to be output, is input into a pre-configured audio occlusion algorithm to obtain a processed audio signal, which is then output. The pre-configured audio occlusion algorithm calculates the signal attenuation intensity of the sound on the surface equation and adjusts the original signal based on this attenuation intensity. Considering the potential presence of occlusions between the sound source and the user, and the fact that these occlusions have a certain occlusion effect on the sound, the game engine itself lacks a third-party interface for processing audio, meaning it cannot call other audio processing algorithms through a third-party interface to process the audio signal. Thus, in this embodiment, an audio occlusion component is pre-set in the game engine. After determining that there is an obstruction between the sound source and the preset object (i.e., the user), the audio occlusion component fits the surface equation of the obstruction, so that the required data is transmitted to the third-party preset audio occlusion algorithm through the audio occlusion component, thereby achieving the purpose of pseudo-calling the third-party preset audio occlusion algorithm. In this way, the surface equation represents the object, and the preset audio occlusion algorithm can calculate the signal attenuation intensity of the sound on the surface equation (actually, the signal attenuation intensity of the sound on the object represented by the surface equation). Then, the original signal is adjusted based on the signal attenuation intensity to obtain the audio signal, so that the output audio signal can simulate the occlusion effect of the obstruction on the sound, thereby improving the immersiveness of the audio output by the game engine. Attached Figure Description

[0035] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0036] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic flowchart of the first embodiment of the audio output method of this application;

[0038] Figure 2 This is a schematic diagram of the target intersection point involved in an embodiment of the audio output method of this application;

[0039] Figure 3 This is a schematic diagram of density clustering involved in an embodiment of the audio output method of this application;

[0040] Figure 4 This is a brief flowchart illustrating the audio output method of this application;

[0041] Figure 5 This is a schematic diagram of the device structure of the audio output device of this application;

[0042] Figure 6 This is a schematic diagram of the hardware operating environment of the audio output device in the embodiments of this application.

[0043] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0044] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] With the rapid development of consumer electronics, people have increasingly higher demands for quality of life, placing greater emphasis on the functionality and immersive experience of consumer electronics. These devices not only need to meet basic needs but also require higher performance. In game engines, limited audio processing capabilities prevent a truly immersive experience for players / users. The engines themselves lack comprehensive built-in audio functions and third-party interfaces for audio processing.

[0046] The traditional way to implement audio masking in an engine is to set a fixed masking strength, that is, all the obstructions attenuate the audio signal strength with the same signal attenuation strength, resulting in poor immersion of the audio signal output by the engine.

[0047] Based on this, the main solution of this application is: to acquire all objects in a preset display scene and identify the occluders in each object; to fit the surface equation of the occluders using the audio occlusion component; to input the surface equation and the original signal to be output into a preset audio occlusion algorithm to obtain the processed audio signal, and to output the audio signal, wherein the preset audio occlusion algorithm is used to calculate the signal attenuation intensity of the sound on the surface equation and adjust the original signal based on the signal attenuation intensity. Considering that there may be occluders between the sound source and the user, and that these occluders have a certain occlusion effect on the sound, and that the game engine itself lacks a third-party interface for processing audio, i.e., it cannot call other audio processing algorithms through a third interface to process the audio signal.

[0048] This application pre-sets an audio occlusion component in the game engine. After determining that there is an obstruction between the sound source and the preset object (i.e., the user), the audio occlusion component fits the surface equation of the obstruction. This allows the audio occlusion component to pass the required data to a third-party preset audio occlusion algorithm, achieving the purpose of pseudo-calling the third-party preset audio occlusion algorithm. By representing the object through the surface equation, the preset audio occlusion algorithm can calculate the signal attenuation intensity of the sound on the surface equation (actually, the signal attenuation intensity of the sound on the object represented by the surface equation). Then, based on the signal attenuation intensity, the original signal is adjusted to obtain the audio signal, so that the output audio signal can simulate the occlusion effect of the obstruction on the sound, thereby improving the immersiveness of the audio output by the game engine.

[0049] It should be noted that the execution subject of the various embodiments of the audio output method of this application can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an audio output device capable of realizing the above functions, such as AR (Augmented Reality) glasses, VR (Virtual Reality) glasses, AR helmets, VR helmets, etc. Exemplarily, the various embodiments of the audio output method of this application are described and explained using AR glasses as the execution subject.

[0050] Based on this, this application proposes an audio output method according to a first embodiment. The audio output method is applied to a game engine, which has a pre-configured audio occlusion component. Please refer to... Figure 1 The audio output method includes steps S10 to S30:

[0051] Step S10: Obtain all objects in the preset display scene and determine the occluders in each object;

[0052] Understandably, AR glasses are equipped with a display screen, such as an HMD (Head-Mounted Display), and the preset display scene can specifically be a scene displayed on the AR glasses' display screen. In one specific implementation, for example, if a user wears AR glasses to play a game, and the AR glasses' display screen displays the game scene, then the preset display scene is the game scene displayed on the display screen.

[0053] It should be noted that the objects in the preset display scene may include one or more objects displayed in the preset display scene, which may be actual objects around the user or virtual objects superimposed on the real world through AR technology.

[0054] Furthermore, if a virtual sound source exists in the preset display scene, then all obstructions among the objects can be identified. If no virtual sound source exists in the preset display scene, the original signal to be output can be directly output. The virtual sound source can be an object in the preset display scene from which sound should be output (it could be an object displayed in the preset display scene, or it could be a fictitious object that is not displayed). For example, in a game scene, when a game character speaks, that game character can be considered a virtual sound source; similarly, in a game scene, when an object is struck, the struck object can be considered a virtual sound source; similarly, in a game scene, when a musical instrument is played, that instrument can be considered a virtual sound source; similarly, when narration is played, the virtual sound source can be determined to be a preset default virtual sound source (or it can be determined that there is no virtual sound source at this time). This preset default virtual sound source can be located at any position in the preset display scene.

[0055] If there is a virtual sound source in the preset display scene, then an obstruction in the preset display scene is identified. It should be noted that the obstruction is an object located between the user and the virtual sound source. Specifically, if a sound ray from the virtual sound source intersects both the user and an object, and intersects the object first and then the user, then that object is the obstruction.

[0056] It should be noted that the obstruction is an object located between the user and the virtual sound source. Specifically, the obstruction can be located between the user's virtual character in the preset display scene and the virtual sound source.

[0057] Furthermore, the method of emitting sound rays from the virtual sound source can be set. For example, the virtual sound source can be set to emit spherical sound rays uniformly in all directions, and the obstruction can be determined based on the sound ray emission method set by the virtual sound source.

[0058] Step S20: Fit the surface equation of the occlusion object using the audio occlusion component;

[0059] It should be noted that the audio occlusion component is a component pre-set in the game engine to fit the surface equation of an object. Since the game engine does not have an interface to call third-party algorithms, this embodiment sets up an audio occlusion component in the game engine to fit the surface equation of an object, so as to represent the real object by fitting the surface equation, and thus facilitate the subsequent simulation of the occlusion effect of the real object represented by this surface equation on the sound.

[0060] Furthermore, the audio occlusion component can also be used to obtain the object's descriptive information, which includes the object's size, location, material, and orientation information, so as to more accurately determine the physical object represented by the surface equation by combining the descriptive information.

[0061] Furthermore, the size and orientation information can be assigned to the audio occlusion component through the parent class function in the game engine, thereby avoiding the repeated acquisition of size and orientation information and improving the acquisition speed of size and orientation information.

[0062] Furthermore, for regular and uniformly textured objects, the position information can be the location of the object's midpoint; for irregular or non-uniformly textured objects, the position information can be the location of the object's center of mass. Specifically, the location of the center of mass (M) can be easily obtained from physics: M = ∫ρ(r)dτ / ∫ρ(r)rdτ, where ρ(r) is the mass density of the particle system, and dτ is the volume element.

[0063] Step S30: Input the surface equation and the original signal to be output into a preset audio occlusion algorithm to obtain the processed audio signal, and output the audio signal. The preset audio occlusion algorithm is used to calculate the signal attenuation intensity of the sound on the surface equation and adjust the original signal based on the signal attenuation intensity.

[0064] The preset audio occlusion algorithm is used to calculate the signal attenuation intensity of sound on the surface equation, that is, to calculate the signal attenuation intensity of sound on the physical object represented by the surface equation, and to adjust the original signal based on the signal attenuation intensity. Specifically, after calculating the signal attenuation intensity, the original signal intensity of the original signal can be attenuated by the calculated signal attenuation intensity to obtain the audio signal.

[0065] It should be noted that after obtaining the surface equation, the domain of the surface equation can also be input into the preset audio occlusion algorithm. For example, the range of values ​​of the occluding object on each coordinate axis can be used as the domain of the surface equation and input into the preset audio occlusion algorithm to avoid the parts outside the domain being considered as having objects and affecting the simulation effect of audio occlusion.

[0066] To calculate the signal attenuation intensity of sound on a surface equation, denoted as F(x,y,z)=0, a ray tracing algorithm can be used to trace a path from the virtual sound source to the user. The path is then checked to see if it intersects the surface equation F(x,y,z)=0 to determine if there is any obstruction. For unobstructed paths, the straight-line propagation of sound is simulated, meaning there is no signal attenuation. For obstructed paths, the surface equation of the object is used to simulate sound reflection and diffraction. The reflection path and attenuation of sound on the object's surface are calculated. Specifically, based on the object's material information and the sound's incident angle, the reflection and absorption losses on the object's surface are calculated, which represent the signal attenuation intensity.

[0067] In this embodiment, an audio occlusion component is pre-configured in the game engine. This component acquires all objects in a preset display scene and identifies occlusions within each object. The audio occlusion component then fits the surface equation of the occlusion. The surface equation and the original signal to be output are input into a preset audio occlusion algorithm to obtain a processed audio signal, which is then output. The preset audio occlusion algorithm calculates the signal attenuation intensity of the sound on the surface equation and adjusts the original signal based on this attenuation intensity. Considering that there may be obstructions between the sound source and the user, and that these obstructions have a certain occlusion effect on the sound, and that the game engine itself lacks a third-party interface for processing audio (i.e., it cannot call other audio processing algorithms through a third interface to process the audio signal), this approach addresses the issue of obstructions between the sound source and the user. Thus, in this embodiment, an audio occlusion component is pre-set in the game engine. After determining that there is an obstruction between the sound source and the preset object (i.e., the user), the audio occlusion component fits the surface equation of the obstruction. This allows the necessary data to be passed to a third-party preset audio occlusion algorithm, achieving the purpose of pseudo-calling the third-party preset audio occlusion algorithm. By representing the object through the surface equation, the preset audio occlusion algorithm can calculate the signal attenuation intensity of the sound on the surface equation (actually, the signal attenuation intensity of the sound on the object represented by the surface equation). Based on the signal attenuation intensity, the original signal is adjusted to obtain the audio signal, so that the output audio signal can simulate the occlusion effect of the obstruction on the sound, thereby improving the immersiveness of the audio output by the game engine.

[0068] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the step of fitting the surface equation of the occlusion object using the audio occlusion component includes:

[0069] Step A10: Obtain the target intersection point between the sound source ray emitted by the preset virtual sound source and the occluder through the audio occlusion component, and fit the surface equation of the occluder based on the target intersection point, wherein the target intersection point is the intersection point formed between the sound ray emitted by the preset virtual sound source and the occluder during the propagation to the preset object.

[0070] The preset virtual sound source is the same as the virtual sound source described in the above embodiments, and will not be repeated in this embodiment. The preset object can specifically be an object that receives the sound source, such as a virtual character of a user wearing AR glasses in a preset scene.

[0071] The target intersection point is the point where the sound ray emitted by the preset virtual sound source intersects with the obstruction as it propagates to the preset object. After being emitted along a straight line, the target intersection point will eventually intersect with the preset object.

[0072] The surface equation is fitted based on all target intersection points. For example, refer to... Figure 2 As shown, the target intersection points are A1, A2, A3, A4, A5, A6... A surface equation can be fitted based on the plane equation method. Specifically, the basic form of the plane equation is A(x-x1) + B(y-y1) + C(z-z1) = 0; where A, B, and C are the normal vectors of the plane; (x1, y1, z1) are the coordinates of A1. By substituting the coordinates of A2, A3, A5, A6... into this plane equation, multiple normal equations are obtained, thus forming the surface equation. If multiple points are coplanar, then a plane equation can be determined by these points. However, if these points are not coplanar, they can determine a surface equation. This is because a plane can be determined by three non-collinear points, while a surface requires more information points to define its shape. Using this method, a surface equation can be constructed based on the coordinate information of the given target intersection points, and this equation can describe the surface where these points lie.

[0073] In this embodiment, the surface equation is fitted based on the intersection of the occluder and the preset virtual sound source. In other words, only the surface equation of the occluded part of the occluder that has an occlusion effect on the audio is fitted, without having to fit the complete surface equation of the occluder. This reduces the fitting complexity and computational load of the surface equation.

[0074] In one possible implementation, the step of fitting the surface equation of the occluder based on the target intersection point includes:

[0075] Step B10: Select a preset number of target intersection points as selected intersection points, and fit the candidate surface equation of the occlusion based on the selected intersection points;

[0076] The preset number can be any value less than or equal to the total number of target intersection points; this embodiment does not impose specific restrictions on it. For ease of subsequent explanation and illustration, this preset number is denoted as N.

[0077] Step B20: Determine the unselected target intersection points as unselected intersection points, and calculate the distance between the coordinate position of each unselected intersection point and the candidate surface equation;

[0078] After arbitrarily selecting N points to generate the known surface equation F(X,Y,Z)=0, F is a ternary function. Calculate the distance PQ between a point (Q) outside the surface and the nearest point (P) on the surface. The vector PQ should be perpendicular to the surface's normal vector. This can be achieved by calculating the gradient vector of the surface equation. To achieve, specifically, Where K is a preset scalar multiple used to ensure that the PQ vector and the gradient vector are consistent. The direction is consistent, that is, K is used to scale the gradient vector so that PQ is consistent with... Same direction.

[0079] Step B30: Calculate the sum of the distances, reselect a preset number of target intersection points as selected intersection points, and return to the step of fitting the candidate surface equation of the occluder based on the selected intersection points;

[0080] It should be noted that the number of intersection points selected each time can be the same or different. Relevant personnel can set the number of intersection points selected each time according to actual needs. This embodiment does not impose specific restrictions on this.

[0081] Step B40: After the preset fitting termination condition is met, select the candidate surface equation corresponding to the minimum value among all the sums as the surface equation of the occluder.

[0082] The preset fitting termination condition can be a termination condition set in advance by relevant personnel, such as the number of fitting attempts reaching a certain value, the fitting time reaching a certain time, or the minimum sum being less than or equal to a certain value. This embodiment does not impose specific restrictions on this. The number of fitting attempts refers to the number of times the intersection points are selected to fit the surface equation, that is, the number of candidate surface equations obtained by fitting.

[0083] Considering that the number of target intersection points between the preset virtual sound source and the occluder is usually large, fitting the surface equation based on all target intersection points would take a long time. Therefore, in this embodiment, a preset number of intersection points are selected for fitting the surface equation, which can improve the fitting efficiency. For unselected intersection points, this embodiment calculates the sum of distances between all unselected intersection points and the fitted candidate surface equation, and selects the candidate surface equation with the smallest sum of distances as the surface equation of the occluder, which can improve the fitting accuracy of the surface equation.

[0084] In one possible implementation, the step of selecting a preset number of target intersection points as selected intersection points includes:

[0085] Step C10: Perform density clustering on each of the target intersection points to obtain clusters;

[0086] A feasible density clustering method can be shown below:

[0087] 1) Initialization: Mark all target intersections as unvisited.

[0088] 2) Traverse the sample set: Iterate through each target intersection point p in turn. If p has been visited, skip the point; otherwise, mark p as visited and obtain the number of sample points in its preset radius neighborhood. The preset radius neighborhood can be an ε-neighborhood.

[0089] 3) Core object: If the number of sample points in the neighborhood of the preset radius is greater than or equal to the preset value, then p is marked as a core object and p is added to a new cluster.

[0090] 4) Density direct access: For a core object p, add the target intersection points within its preset radius neighborhood to the cluster, and recursively add the target intersection points within their preset radius neighborhoods to the cluster as well. The target intersection points added to the cluster are marked as visited.

[0091] 5) Density Connectivity: For a non-core object q, if q is located within a preset radius neighborhood of a core object p in a certain cluster, then q is added to that cluster.

[0092] 6) Clustering complete: The clustering process ends when all sample points have been marked as visited.

[0093] An exemplary clustering process can be referred to Figure 3 As shown, the specific implementation of density clustering can refer to existing technologies. This embodiment only provides a possible density clustering method and does not limit the density clustering method.

[0094] Step C20: Select a preset number of target intersection points from the cluster as selected intersection points.

[0095] The selected intersection points can be a preset number of target intersection points in the cluster. Specifically, all target intersection points in the cluster can be selected as the selected intersection points, or a preset number of target intersection points can be randomly selected from the cluster.

[0096] It should be noted that each time intersection points are reselected, the same clustering method can be used to reselect them. Furthermore, the clustering parameters can be different each time intersection points are selected, such as different cluster radii and / or cluster densities, to select different intersection points. Alternatively, a preset number of target intersection points can be reselected from the clusters obtained in a single clustering process.

[0097] In this embodiment, density clustering is performed on the target intersection points, and the intersection points are selected from the clusters. This can reduce the impact of outlier noise points on surface fitting and improve the fitting accuracy of the surface equation.

[0098] In one possible implementation, the step of fitting the candidate surface equation of the occluder based on the selected intersection points includes:

[0099] Step D10: Establish a centroid coordinate system with the centroid of the occluder as the origin, and calculate the coordinate positions of each selected intersection point in the centroid coordinate system.

[0100] Step D20: Based on the coordinate positions of each selected intersection point, perform surface fitting to obtain candidate surface equations, and transform the candidate surface equations into the world coordinate system to obtain the surface equation of the occluder.

[0101] In this embodiment, a centroid coordinate system is established with the centroid of the occluded object as the origin. The coordinate positions of each selected intersection point in the centroid coordinate system are calculated. Based on the coordinate positions of the points in the centroid coordinate system, surface fitting is performed to obtain candidate surface equations. In the centroid coordinate system with the centroid as the origin, the complex mass distribution inside the object can be ignored, thereby reducing the complexity of the calculation and reducing the number of operations.

[0102] Based on the first and / or second embodiments of this application, in the third embodiment of this application, the content that is the same as or similar to the first and second embodiments described above can be referred to the above description and will not be repeated hereafter. In addition, after the step of determining the obstructions in each of the objects, the method further includes:

[0103] Step E10: Perform regular object recognition on the occluded object to obtain the recognition result;

[0104] Perform regular object recognition on occluded objects to determine whether the occluded objects are regular objects.

[0105] In one implementation, the contour recognition result of the occluded object can be calculated, and the similarity between the contour recognition result and the contour of a preset regular object can be calculated. If the similarity is greater than or equal to a preset similarity threshold, the occluded object is determined to be a regular object; otherwise, if the similarity is less than the preset similarity threshold, the occluded object is determined to be an irregular object. The preset regular object contour can be a regular object contour set in advance by relevant personnel, such as a spherical contour, a cylindrical contour, etc., and this embodiment does not impose specific limitations on it.

[0106] Step E20: If the recognition result indicates that the occluder is an irregular object, then the occluder is divided into multiple sub-occluders, wherein at least one regular object exists in each sub-occluder.

[0107] If the recognition result indicates that the occluder is an irregular object, the occluder is divided into multiple sub-occluders.

[0108] Step E30: Based on each of the sub-occluders, perform the step of fitting the surface equation of the occluder through the audio occlusion component.

[0109] For each sub-occluder, subsequent step S20 can be performed to fit the surface equation of each sub-occluder. In step S30, the surface equations of all sub-occluders can be input into a preset audio occlusion algorithm to calculate the signal attenuation intensity of sound on the physical object represented by each surface equation.

[0110] Alternatively, step S20 can be performed only for irregular sub-occluded objects to fit the surface equation of the sub-occluded object. For regular sub-occluded objects, steps F10 to F20 can be performed.

[0111] In this embodiment, irregular objects are divided into at least one regular sub-occluder. It is understood that the fitting complexity of the surface equation of regular objects is lower, thereby improving the fitting efficiency of the surface equation.

[0112] In one possible implementation, after the step of performing regular object recognition on the occluded object to obtain a recognition result, the method further includes:

[0113] Step F10: If the recognition result indicates that the occluder is a regular object, then obtain the description information of the occluder, wherein the description information includes position, orientation and size;

[0114] Understandably, for regular objects, a three-dimensional model of the object can be fitted based on its position, orientation, and size, thus eliminating the need to calculate the surface equations of occluders.

[0115] Step F20: Input the description information and the original signal into a preset audio occlusion algorithm to obtain the processed audio signal, and output the audio signal.

[0116] It should be noted that a 3D model of the obstruction can be constructed based on its description information. Then, a ray tracing algorithm is used to trace a path from the virtual sound source to the user based on this 3D model. The path is checked for intersection with the 3D model to determine if an obstruction exists. For unobstructed paths, the straight-line propagation of sound is simulated. For obstructed paths, the surface equations of the object are used to simulate sound reflection and diffraction, calculating the sound reflection path and attenuation on the object's surface. Specifically, based on the object's material information and the sound's incident angle, the sound reflection and absorption loss on the object's surface are calculated. This sound reflection and absorption loss on the object's surface is also the signal attenuation intensity.

[0117] For example, to aid in understanding the technical concept or principle of the audio output method after combining this embodiment with the first and second embodiments, a specific embodiment is provided below. In this specific embodiment, a new audio occlusion component (hereinafter referred to as the component) is added to the game engine. This component should contain descriptive information such as the object's size, position, material, and orientation. Since the object's motion state is uncertain, it needs to be acquired every frame. The size and orientation can be overridden in the parent class function and directly assigned to the component. For regular objects, the position corresponds to the object's midpoint; for irregular or unevenly textured objects, data processing is required to obtain a suitable centroid. Based on this, refer to... Figure 4 As shown, the audio output process is as follows:

[0118] 1. Identify the obstructions between the user and the preset virtual sound source.

[0119] 2. Perform object recognition on the occluded object to identify whether it is a regular object and whether it is an object with uniform texture.

[0120] 3. If the occluding object is a regular object with uniform texture, then the object's size, position, and orientation description information are passed to the preset audio occlusion algorithm to output the audio signal.

[0121] 4. If the occlusion is an irregular object, the occlusion is divided into multiple sub-occlusions, and components are added to each sub-occlusion. That is, the description information of each sub-occlusion is obtained, and the surface equation of each sub-occlusion is fitted.

[0122] 5. If the occluding object is a regular object with uneven texture, calculate the centroid of the object and fit the surface equation of the occluding object.

[0123] 6. Input the fitted surface equation and its domain into the preset audio occlusion algorithm to output the audio signal.

[0124] 7. Output audio signals through the game engine.

[0125] In this specific embodiment, the listener can dynamically monitor the occluded object through the engine, calculate the surface equation of the occluded object as accurately as possible, provide the algorithm with more realistic occluded objects, and make the listener's audio experience more realistic and immersive.

[0126] It should be noted that the above examples are only for the purpose of assisting in understanding this application and do not constitute a limitation on the audio output method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0127] Furthermore, embodiments of this application also propose an audio output device, referring to... Figure 5As shown, the audio output device includes:

[0128] The acquisition module 10 is used to acquire all objects in the preset display scene and determine the occluders in each of the objects;

[0129] Fitting module 20 is used to fit the surface equation of the occluder using a preset audio occlusion component;

[0130] The occlusion module 30 is used to input the surface equation and the original signal to be output into a preset audio occlusion algorithm to obtain a processed audio signal, and output the audio signal. The preset audio occlusion algorithm is used to calculate the signal attenuation intensity of the sound on the surface equation and adjust the original signal based on the signal attenuation intensity.

[0131] In one embodiment, the fitting module 20 is further configured to:

[0132] The target intersection point between the sound source ray emitted by the preset virtual sound source and the occluder is obtained through the audio occlusion component, and the surface equation of the occluder is fitted based on the target intersection point, wherein the target intersection point is the intersection point formed between the sound ray emitted by the preset virtual sound source and the occluder during the propagation to the preset object.

[0133] In one embodiment, the fitting module 20 is further configured to:

[0134] Select a preset number of target intersection points as selected intersection points, and fit the candidate surface equation of the occluder based on the selected intersection points;

[0135] The unselected target intersection points are determined as unselected intersection points, and the distance between the coordinate position of each unselected intersection point and the candidate surface equation is calculated;

[0136] Calculate the sum of all the distances, reselect a preset number of target intersection points as selected intersection points, and return to the step of fitting the candidate surface equation of the occluder based on the selected intersection points;

[0137] Once the preset fitting termination condition is met, the candidate surface equation corresponding to the minimum value among all the sums is selected as the surface equation of the occluder.

[0138] In one embodiment, the fitting module 20 is further configured to:

[0139] Density clustering is performed on each of the target intersection points to obtain clusters;

[0140] A preset number of target intersection points are selected from the clusters.

[0141] In one embodiment, the fitting module 20 is further configured to:

[0142] Establish a centroid coordinate system with the centroid of the obstruction as the origin, and calculate the coordinate positions of each selected intersection point in the centroid coordinate system.

[0143] Based on the coordinate positions of each selected intersection point, a surface fitting is performed to obtain the candidate surface equation, and the candidate surface equation is transformed into the world coordinate system to obtain the surface equation of the occlusion.

[0144] In one embodiment, the audio output device further includes a recognition module, the recognition module being configured to:

[0145] The occlusion is subjected to regular object recognition to obtain the recognition result;

[0146] If the recognition result indicates that the occluder is an irregular object, then the occluder is divided into multiple sub-occluders, wherein at least one of the sub-occluders is a regular object;

[0147] Based on each of the sub-occluders, the step of fitting the surface equation of the occluder using the audio occlusion component is performed.

[0148] In one embodiment, the identification module is further configured to:

[0149] If the recognition result indicates that the occluder is a regular object, then the description information of the occluder is obtained, wherein the description information includes position, orientation and size;

[0150] The description information and the original signal are input into a preset audio occlusion algorithm to obtain a processed audio signal, and the audio signal is output.

[0151] Furthermore, this application also proposes an audio output device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio output method described above.

[0152] In addition, refer to Figure 6The diagram illustrates a structural schematic of an audio output device suitable for implementing embodiments of this application. The audio output device in the embodiments of this application may also include, but is not limited to, mobile terminals such as AR glasses, VR glasses, AR headsets, VR headsets, mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The audio output device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0153] like Figure 6 As shown, the audio output device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the audio output device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the audio output device to communicate wirelessly or wiredly with other devices to exchange data. Although audio output devices with various systems are shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0154] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0155] The audio output device provided in this application, employing the audio output method described in the above embodiments, can solve the technical problem of how to improve the immersiveness of audio output by a game engine. Compared with the prior art, the beneficial effects of the audio output device provided in this application are the same as those of the audio output method provided in the above embodiments, and other technical features of this audio output device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0156] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0157] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0158] In addition, to achieve the above objectives, embodiments of this application also provide a readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the audio output method in the above embodiments.

[0159] The computer-readable storage medium provided in this application embodiment may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0160] The aforementioned computer-readable storage medium may be included in an audio output device; or it may exist independently and not assembled into an audio output device.

[0161] The aforementioned computer-readable storage medium carries one or more programs that, when executed by an audio output device, cause the audio output device to: acquire all objects in a preset display scene and determine occluders among each object; fit the surface equation of the occluder using a preset audio occlusion component; input the surface equation and the original signal to be output into a preset audio occlusion algorithm to obtain a processed audio signal, and output the audio signal, wherein the preset audio occlusion algorithm is used to calculate the signal attenuation intensity of the sound on the surface equation and adjust the original signal based on the signal attenuation intensity.

[0162] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0163] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0165] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described audio output method, thereby solving the technical problem of how to improve the immersiveness of audio output by a game engine. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the audio output method provided in the above embodiments, and will not be repeated here.

[0166] Furthermore, embodiments of this application also propose a computer program product, including an audio output program, which, when executed by a processor, implements the steps of the audio output method described above.

[0167] The specific implementation of the computer program product in this application is basically the same as the embodiments of the audio output method described above, and will not be repeated here.

[0168] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0169] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software sensor. This computer software sensor is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0171] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An audio output method, characterized in that, The audio output method is applied to a game engine, which has a pre-configured audio occlusion component. The audio output method includes the following steps: Obtain all objects in the preset display scene and identify the occluders in each object; The surface equation of the obstruction is fitted by the audio occlusion component; The surface equation and the original signal to be output are input into a preset audio occlusion algorithm to obtain a processed audio signal, and the audio signal is output. The preset audio occlusion algorithm is used to calculate the signal attenuation intensity of the sound on the surface equation and adjust the original signal based on the signal attenuation intensity.

2. The audio output method as described in claim 1, characterized in that, The step of fitting the surface equation of the occlusion using the audio occlusion component includes: The target intersection point between the sound source ray emitted by the preset virtual sound source and the occluder is obtained through the audio occlusion component, and the surface equation of the occluder is fitted based on the target intersection point, wherein the target intersection point is the intersection point formed between the sound ray emitted by the preset virtual sound source and the occluder during the propagation to the preset object.

3. The audio output method as described in claim 2, characterized in that, The step of fitting the surface equation of the occluder based on the target intersection point includes: Select a preset number of target intersection points as selected intersection points, and fit the candidate surface equation of the occluder based on the selected intersection points; The unselected target intersection points are determined as unselected intersection points, and the distance between the coordinate position of each unselected intersection point and the candidate surface equation is calculated; Calculate the sum of all the distances, reselect a preset number of target intersection points as selected intersection points, and return to the step of fitting the candidate surface equation of the occluder based on the selected intersection points; Once the preset fitting termination condition is met, the candidate surface equation corresponding to the minimum value among all the sums is selected as the surface equation of the occluder.

4. The audio output method as described in claim 3, characterized in that, The step of selecting a preset number of target intersection points as the selected intersection points includes: Density clustering is performed on each of the target intersection points to obtain clusters; A preset number of target intersection points are selected from the clusters.

5. The audio output method as described in claim 4, characterized in that, The step of fitting the candidate surface equation of the occluder based on the selected intersection points includes: Establish a centroid coordinate system with the centroid of the obstruction as the origin, and calculate the coordinate positions of each selected intersection point in the centroid coordinate system. Based on the coordinate positions of each selected intersection point, a surface fitting is performed to obtain the candidate surface equation, and the candidate surface equation is transformed into the world coordinate system to obtain the surface equation of the occlusion.

6. The audio output method as described in claim 1, characterized in that, After the step of determining the obstructions in each of the objects, the method further includes: The occlusion is subjected to regular object recognition to obtain the recognition result; If the recognition result indicates that the occluder is an irregular object, then the occluder is divided into multiple sub-occluders, wherein at least one of the sub-occluders is a regular object; Based on each of the sub-occluders, the step of fitting the surface equation of the occluder using the audio occlusion component is performed.

7. The audio output method as described in claim 6, characterized in that, After the step of performing regular object recognition on the occluded object to obtain the recognition result, the method further includes: If the recognition result indicates that the occluder is a regular object, then the description information of the occluder is obtained, wherein the description information includes position, orientation and size; The description information and the original signal are input into a preset audio occlusion algorithm to obtain a processed audio signal, and the audio signal is output.

8. An audio output device, characterized in that, The audio output device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the audio output method as claimed in any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, on which a computer program is stored, and when executed by a processor, the computer program implements the steps of the audio output method as described in any one of claims 1 to 7.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the audio output method as described in any one of claims 1 to 7.

Images