An acoustic image rendering method, device, equipment and computer readable storage medium

Abstract

The application discloses a sound image rendering method, device and equipment and a computer readable storage medium, and applies to the field of audio technology, and comprises the following steps: setting a virtual loudspeaker in a preset direction in the absence of a physical loudspeaker satisfying a preset direction distance condition, obtaining an extended loudspeaker set comprising the virtual loudspeaker and the physical loudspeaker; projecting three-dimensional coordinates of each loudspeaker in the extended loudspeaker set to a unit sphere or a unit circle according to a rendering mode, obtaining a loudspeaker projection point set; performing a convex hull calculation on the loudspeaker projection point set to determine a loudspeaker rendering area; determining a target loudspeaker rendering area according to a sound image direction of a sound image to be rendered, and performing sound image rendering according to a corresponding loudspeaker combination. The application can supplement the distribution of the physical loudspeaker in the corresponding direction, and make the loudspeaker projection point set suitable for the convex hull calculation, thereby automatically generating the loudspeaker rendering area and improving the stability of the sound image rendering.

Description

Technical Field

[0001] This invention relates to the field of audio technology, and in particular to a sound and image rendering method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] With the development of spatial audio technology, sound image rendering methods based on vector basis amplitude translation have been widely used in home theaters, audio renderers, and other spatial audio terminal devices. When performing sound image rendering, it is usually necessary to pre-divide the speaker rendering area according to the positional relationship between multiple speakers involved in the rendering, so that different sound image directions can be matched with corresponding speaker combinations, and the rendering gain of each speaker is determined based on the matched speaker combinations.

[0003] Current technology typically involves technicians manually dividing the speaker rendering area based on the actual speaker arrangement. When there are many speakers involved in the rendering or the arrangement is complex, manual division requires a large amount of work, is inefficient, and the results are easily affected by the operator's experience. While some automatic division methods can reduce manual work, under different speaker arrangements, the division results may still result in areas being missed, overlapping, or having unstable area structures. This can lead to difficulties in accurately matching the corresponding speaker combinations for some sound image directions, thus affecting sound image localization and rendering effects.

[0004] Therefore, how to automatically determine the speaker rendering area under different speaker arrangement forms, and improve the stability of the area division results and the rendering effect, is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of the present invention is to provide a sound and image rendering method, apparatus, device and computer-readable storage medium, which solves the problems of poor stability of region division results and poor rendering effect in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides a sound and image rendering method, comprising: Based on the three-dimensional coordinates of the physical loudspeaker, determine whether there is a physical loudspeaker in the preset direction that meets the preset distance condition; A virtual speaker is set in a preset direction where no physical speaker meets the preset directional distance condition, thereby obtaining an extended speaker set; the extended speaker set includes the virtual speaker and the physical speaker; According to the rendering mode, the three-dimensional coordinates of each speaker in the extended speaker set are projected onto a unit sphere or unit circle to obtain the speaker projection point set; According to the rendering mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation result. Based on the sound image direction of the sound image to be rendered, a target speaker rendering area is determined from the speaker rendering area, and the sound image to be rendered is rendered according to the speaker combination corresponding to the target speaker rendering area.

[0007] Optionally, based on the three-dimensional coordinates of the physical speaker, determine whether there exists a physical speaker in a preset direction that satisfies the preset directional distance condition, including: Based on the three-dimensional coordinates of each physical loudspeaker, determine the unit direction vector of each physical loudspeaker; Calculate the spatial angle between the unit direction vector of each physical speaker and the unit direction vector of each preset direction to obtain the directional distance between each physical speaker and each preset direction; When the directional distance between each of the physical loudspeakers and the target preset direction is not less than the detection threshold corresponding to the target preset direction, it is determined that there are no physical loudspeakers in the target preset direction that satisfy the preset directional distance condition; The target preset direction is any one of the directions of front, back, and top, and the detection thresholds corresponding to the front and back directions are less than the detection thresholds corresponding to the top direction.

[0008] Optionally, after setting a virtual speaker in a preset direction where no physical speaker meets the preset directional distance condition, the method further includes: Based on the orientation of the virtual speaker and the orientation of each of the physical speakers, determine the two physical speakers adjacent to the virtual speaker from among the physical speakers; Based on the unit direction vectors of the two physical loudspeakers and the unit direction vector of the virtual loudspeaker, the gain coefficients of the virtual loudspeaker corresponding to the two physical loudspeakers are determined by vector basis amplitude translation interpolation. The gain coefficients are normalized to obtain the gain mapping relationship corresponding to the virtual speaker, and the gain mapping relationship is associated with and stored with the virtual speaker.

[0009] Optionally, depending on the rendering mode, the three-dimensional coordinates of each speaker in the extended speaker set are projected onto a unit sphere or unit circle to obtain a speaker projection point set, including: When the rendering mode is 3D mode, a preset positive offset is added to the height coordinate of each speaker in the three-dimensional coordinate system of the extended speaker set, and the three-dimensional coordinates after adding the preset positive offset are normalized to obtain a set of three-dimensional projection points located on the unit sphere. When the rendering mode is 2D mode, the horizontal and vertical coordinates of each speaker in the extended speaker set are normalized to obtain a set of two-dimensional projection points located on the unit circle.

[0010] Optionally, when the rendering mode is 3D mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation result, including: Four non-coplanar projection points are selected from the set of three-dimensional projection points to construct an initial tetrahedron. The geometric center of the initial tetrahedron is determined as the internal reference point, and the internal reference point is kept unchanged during the three-dimensional convex hull update process. The remaining projection points in the three-dimensional projection point set are processed sequentially, and the signed distance between the current projection point and each triangular facet in the current three-dimensional convex hull is calculated. Based on the signed distance, it is determined whether the current projection point is located outside the current three-dimensional convex hull. When the current projection point is located outside the current 3D convex hull, determine the visible face corresponding to the current projection point and the boundary edge between the visible face and the non-visible face, delete the visible face, and generate new face connecting the current projection point and each of the boundary edges respectively. The outer normal direction of each new facet is determined based on the internal reference point to obtain the updated three-dimensional convex hull. Obtain the height coordinates of the speaker corresponding to the three vertices of each triangle in the updated 3D convex hull before adding the preset positive offset. When the absolute value of the height coordinates of the speaker corresponding to the three vertices is less than the preset filtering threshold, delete the corresponding triangle and determine the remaining triangle as the speaker rendering area.

[0011] Optionally, when the rendering mode is 2D mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation result, including: Collinearity detection is performed on the projection points in the two-dimensional projection point set; When not all projection points in the two-dimensional projection point set are collinear, the projection points in the two-dimensional projection point set are sorted according to their horizontal coordinates. The upper and lower convex hulls are determined based on the sorted projection points, and the upper and lower convex hulls are merged to obtain a sequence of convex hull vertices arranged according to the two-dimensional convex hull boundary. By sequentially connecting adjacent projection points in the convex hull vertex sequence, and connecting the first and last projection points in the convex hull vertex sequence, multiple convex hull edges are obtained; The line segment region formed by the two speakers corresponding to each convex hull is defined as a speaker rendering region.

[0012] Optionally, based on the sound image direction of the sound image to be rendered, a target speaker rendering area is determined from the speaker rendering area, and the sound image to be rendered is rendered according to the speaker combination corresponding to the target speaker rendering area, including: When the rendering mode is 3D mode, the vector basis amplitude translation inverse matrix is ​​obtained by pre-calculating the unit direction vectors of the three speakers corresponding to the current speaker rendering area, and the candidate gain vector is determined according to the vector basis amplitude translation inverse matrix and the sound image direction; when all three components of the candidate gain vector are non-negative, the current speaker rendering area is determined as the target speaker rendering area, and the candidate gain vector is determined as the initial rendering gain of each speaker in the speaker combination; the current speaker rendering area is any area in the speaker area; When the rendering mode is 2D mode, the target speaker rendering area where the sound image direction is located is determined according to the azimuth angle of the sound image direction and the azimuth angle of the two speakers corresponding to the speaker rendering area in each line segment form; the initial rendering gain of each speaker is determined according to the unit direction vector of each speaker in the speaker combination corresponding to the target speaker rendering area and the sound image direction. When the speaker combination includes a virtual speaker, according to the gain mapping relationship corresponding to the virtual speaker, the initial rendering gain of the virtual speaker is decomposed into the gain components of the two adjacent physical speakers, and each gain component is added to the initial rendering gain of the corresponding physical speaker to obtain the rendering gain of each physical speaker. The rendering gain of each physical speaker is energy normalized, and the rendering gain after energy normalization is used to control each physical speaker to render the sound image to be rendered.

[0013] The present invention also provides an audio-visual rendering apparatus, comprising: The determination module is used to determine whether there is a physical speaker that meets the preset distance condition in the preset direction based on the three-dimensional coordinates of the physical speaker; An extension module is used to set up a virtual speaker in a preset direction where there is no physical speaker that meets the preset directional distance condition, thereby obtaining an extended speaker set; the extended speaker set includes the virtual speaker and the physical speaker; The projection module is used to project the three-dimensional coordinates of each speaker in the extended speaker set onto a unit sphere or unit circle according to the rendering mode, so as to obtain a set of speaker projection points; The calculation module is used to perform convex hull calculation on the speaker projection point set according to the rendering mode, and determine the speaker rendering area based on the convex hull calculation result; The rendering module is used to determine the target speaker rendering area from the speaker rendering area according to the sound image direction of the sound image to be rendered, and to render the sound image to be rendered according to the speaker combination corresponding to the target speaker rendering area.

[0014] The present invention also provides an audio-visual rendering device, comprising: Memory, used to store computer programs; A processor for implementing the audio-visual rendering method described above when executing the computer program.

[0015] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the audio-visual rendering method described above.

[0016] The present invention also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the above-described audio-visual rendering method.

[0017] As can be seen from the above technical solution, the present invention determines whether there is a physical speaker that meets the preset direction distance condition based on the three-dimensional coordinates of the physical speaker; sets a virtual speaker in the preset direction where no physical speaker meets the preset direction distance condition, thereby obtaining an extended speaker set; the extended speaker set includes virtual speakers and physical speakers; according to the rendering mode, projects the three-dimensional coordinates of each speaker in the extended speaker set onto a unit sphere or unit circle, thereby obtaining a speaker projection point set; according to the rendering mode, performs convex hull calculation on the speaker projection point set, and determines the speaker rendering area based on the convex hull calculation result; according to the sound image direction of the sound image to be rendered, determines the target speaker rendering area from the speaker rendering area, and renders the sound image to be rendered according to the speaker combination corresponding to the target speaker rendering area. The beneficial effects of this invention are as follows: By setting virtual speakers in directions where no physical speakers meet the preset directional distance conditions, the speakers participating in the region division complement each other in the corresponding directions, reducing the impact of uneven distribution of physical speakers on the geometry of the rendering region; simultaneously, the three-dimensional coordinates of the physical and virtual speakers are projected onto a unit sphere or unit circle according to the rendering mode, eliminating the distance information of the speakers while retaining the directional information used for sound image rendering, and making the projection point set applicable to the convex hull calculation of the corresponding dimension. Therefore, the speaker rendering region can be automatically generated based on the convex hull calculation results, reducing region omissions, region overlaps, and structural instability caused by manual division, and enabling the sound image to be rendered to match the corresponding speaker combination according to the sound image direction, thereby improving the stability of sound image rendering.

[0018] In addition, the present invention also provides an audio-visual rendering apparatus, device, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0020] Figure 1 A flowchart of an audio-visual rendering method provided in an embodiment of the present invention; Figure 2 An example diagram of a speaker layout provided in an embodiment of the present invention; Figure 3 An example diagram of a three-dimensional projection provided for an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an audio-visual rendering device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an audio-visual rendering device provided in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0022] The core idea of ​​VBAP (Vector Base Amplitude Panning) is to represent the sound image direction vector as a weighted combination of several speaker direction vectors, and obtain the gain of each speaker by solving a system of linear equations. In 3D scenes, the classic VBAP rendering strategy follows the principle of minimum activation, generally activating only 3 speakers to form a triangular rendering area; in 2D scenes, only 2 speakers are activated to form a line segment rendering area. Regardless of whether it is a 2D or 3D scene, the premise of VBAP rendering is to divide the speaker set into several rendering areas, each area consisting of 2 (2D) or 3 (3D) speakers. The quality of the rendering area division directly affects the VBAP rendering effect: the division result must cover all possible sound image directions, there should be no overlap between areas, and the area of ​​the areas should be as uniform as possible to avoid sound image localization deviation. However, existing methods for dividing the rendering region of loudspeakers have the following shortcomings: First, traditional methods rely on manual division, which is time-consuming and prone to omissions or overlaps when there are many loudspeakers (e.g., more than 10). Second, manual division cannot guarantee geometric optimality, i.e., complete coverage, no overlap, and minimum area. Third, 2D and 3D layouts usually require different division strategies, lacking a unified framework. Fourth, when loudspeakers exhibit degenerate conditions such as coplanarity, collinearity, or close overlap, existing automatic division methods are prone to generating degenerate triangles or calculation failures. Fifth, existing automatic division methods often employ exhaustive combinations or greedy strategies, resulting in high algorithm complexity and failing to meet real-time requirements. To address these issues, this invention provides a method for automatic rendering region division, applicable to AV amplifiers, audio renderers, home theater systems, and other spatial audio rendering terminal devices based on the VBAP framework.

[0023] Please refer to the details. Figure 1 , Figure 1 A flowchart illustrating an audio-visual rendering method provided in an embodiment of the present invention. The method may include: S101: Based on the three-dimensional coordinates of the physical speaker, determine whether there is a physical speaker in the preset direction that meets the preset distance condition.

[0024] The audio-visual rendering method in this embodiment can be executed by an electronic device with data processing capabilities, such as an AV amplifier, audio renderer, home theater host, server, or portable terminal. This electronic device can acquire the three-dimensional coordinates and rendering mode of multiple physical speakers involved in the rendering, and perform speaker rendering area division when the speaker configuration changes. Upon receiving the audio-visual image to be rendered, it calls the division results for real-time rendering. The three-dimensional coordinates of the physical speakers are used to characterize the spatial position of the physical speakers relative to the listening position. The three-dimensional coordinates can be input by the user, read from a speaker configuration file, or obtained through speaker position calibration. The preset direction is a pre-set spatial direction, and the preset direction distance condition is used to determine the proximity between the direction of the physical speaker and the corresponding preset direction. The preset directions and their number can be set according to the audio-visual rendering scenario; this embodiment does not limit this.

[0025] S102: Set up a virtual speaker in a preset direction where there is no physical speaker that meets the preset direction distance condition, and obtain an extended speaker set; the extended speaker set includes virtual speakers and physical speakers.

[0026] When physical speakers are missing in a preset direction, the rendering area divided by the convex hull algorithm will exhibit left-right asymmetry. For example, the left and right speakers may form triangles with adjacent speakers at different distances, resulting in inconsistent sizes and shapes of the rendering areas at symmetrical positions. This leads to differences in sound image rendering gain in the left-right symmetrical direction, affecting the symmetry and stability of sound image localization. Therefore, this embodiment injects virtual speakers in the preset direction to provide symmetrical anchor points for the convex hull, enabling the left and right speakers to form symmetrical rendering areas with the same virtual speaker, thereby ensuring the left-right symmetry of the division result and improving the subsequent rendering effect. The injection of virtual speakers adopts an adaptive strategy, that is, it is only injected when physical speakers are missing in the preset direction to avoid redundancy; the position of the virtual speaker is fixed at standardized key direction coordinates, and its gain is calculated by interpolation of adjacent physical speakers using VBAP.

[0027] It should be noted that virtual speakers are logical speakers involved in rendering area partitioning and gain calculation. They have corresponding spatial directions or three-dimensional coordinates, but it is not required to place actual sound-emitting devices at these locations. This step only sets up virtual speakers if there are no physical speakers that meet the preset distance conditions in the corresponding preset direction; if there are physical speakers that meet the conditions in the corresponding preset direction, virtual speakers do not need to be set up. Thus, the expanded speaker set can be adaptively formed according to the actual arrangement of physical speakers. Virtual speakers can be set to be consistent with the corresponding preset direction, and gain mapping relationships between virtual speakers and one or more physical speakers can be pre-established. During real-time rendering, virtual speakers do not directly output audio signals, but instead convert the gain allocated to virtual speakers to the corresponding physical speakers.

[0028] S103: Based on the rendering mode, project the three-dimensional coordinates of each speaker in the extended speaker set onto a unit sphere or unit circle to obtain the speaker projection point set.

[0029] The rendering mode indicates the spatial dimension used for the current audio-visual rendering. Rendering modes can include 3D mode and 2D mode. In 3D mode, multiple speakers are often distributed on the same horizontal or near-horizontal plane around the listener. These points are coplanar or nearly coplanar in three-dimensional space. Directly performing a 3D convex hull algorithm on the coplanar point set will degenerate into a two-dimensional problem, failing to generate effective triangular facets. Therefore, this embodiment normalizes and projects the speaker positions onto a unit sphere, "expanding" the coplanar points into a non-coplanar point set on the sphere, enabling the 3D convex hull algorithm to work properly. Each triangular facet of the convex hull naturally corresponds to a rendering area composed of three speakers.

[0030] In other words, this embodiment addresses the situation where speaker locations are often distributed on the same or nearly the same plane, making direct region partitioning using 3D convex hull algorithms impossible. It proposes a method to map speaker locations onto a unit sphere (3D mode) or a unit circle (2D mode), transforming the originally coplanar speaker locations into a set of non-coplanar points on the sphere. This allows the convex hull algorithm to automatically generate non-overlapping, fully covered triangles (3D) or line segments (2D) rendering regions, replacing traditional manual partitioning. In 3D mode, the 3D coordinates of each speaker in the extended speaker set can be normalized so that the resulting projection points lie on the unit sphere. In 2D mode, the coordinates of each speaker in the horizontal plane can be normalized so that the resulting projection points lie on the unit circle. This step of projecting the 3D coordinates of the speakers onto the unit sphere or unit circle eliminates the distance differences between different speakers and the listening position while preserving directional information related to sound image rendering. The projection process also converts the extended speaker set into a set of speaker projection points suitable for convex hull calculation in the corresponding dimension.

[0031] S104: Based on the rendering mode, perform convex hull calculation on the speaker projection point set, and determine the speaker rendering area based on the convex hull calculation results.

[0032] In 3D mode, the convex hull calculation result can include multiple triangular faces, each corresponding to a speaker combination consisting of three speakers; in 2D mode, the convex hull calculation result can include multiple convex hull edges, each corresponding to a speaker combination consisting of two speakers. Electronic devices can define the directional range defined by the triangular faces or convex hull edges as the corresponding speaker rendering area.

[0033] This embodiment does not specifically limit the representation of the speaker rendering area. For example, the area data of the speaker rendering area may include the speaker index, area boundary parameters, and parameters used to calculate the speaker gain. The above convex hull calculation and speaker rendering area data generation can be performed once when the speaker configuration changes, and the calculation results can be stored for subsequent real-time audio-visual rendering in step S105.

[0034] S105: Based on the sound image direction of the sound image to be rendered, determine the target speaker rendering area from the speaker rendering area, and render the sound image to be rendered according to the speaker combination corresponding to the target speaker rendering area.

[0035] The sound image orientation is used to characterize the target spatial orientation of the sound image relative to the listening position. In 3D mode, the electronic device can traverse each speaker rendering area in the form of triangular patches. For each speaker rendering area, a candidate gain vector is determined using the pre-calculated vector basis amplitude translation inverse matrix and the sound image orientation. The speaker rendering area where all three components of the candidate gain vector are non-negative is determined as the target speaker rendering area. Since the triangular patch speaker rendering areas do not overlap and cover all sound image orientations, except in boundary cases, there is one and only one speaker rendering area whose candidate gain vector satisfies that all three components are non-negative. In boundary cases, any speaker rendering area that meets the condition can be selected as the target speaker rendering area. In 2D mode, the electronic device can determine the target speaker rendering area based on the azimuth angle of the sound image orientation and calculate the initial rendering gain based on the speaker combination corresponding to the target speaker rendering area. When the speaker combination includes virtual speakers, the initial rendering gain corresponding to the virtual speakers can be allocated to the corresponding physical speakers according to the pre-established gain mapping relationship; then the rendering gain of each physical speaker is normalized by energy, and the audio signal of the sound image to be rendered is weighted by the energy-normalized rendering gain to control the output of the corresponding audio signal of each physical speaker.

[0036] The audio-visual rendering method provided by this invention comprises the following steps: S101: Determine whether a physical speaker exists in a preset direction that satisfies a preset distance condition based on the three-dimensional coordinates of the physical speaker; S102: Set up a virtual speaker in the preset direction where no physical speaker satisfies the preset distance condition, thereby obtaining an extended speaker set; the extended speaker set includes virtual speakers and physical speakers; S103: Project the three-dimensional coordinates of each speaker in the extended speaker set onto a unit sphere or unit circle according to the rendering mode, thereby obtaining a speaker projection point set; S104: Perform convex hull calculation on the speaker projection point set according to the rendering mode, and determine the speaker rendering area based on the convex hull calculation result; S105: Determine the target speaker rendering area from the speaker rendering area according to the audio-visual direction of the audio-visual to be rendered, and render the audio-visual to be rendered according to the speaker combination corresponding to the target speaker rendering area. This invention sets up virtual speakers in directions where no physical speakers meet preset directional distance conditions, allowing the speakers participating in region division to complement each other in the corresponding directions, thus reducing the impact of uneven physical speaker distribution on the geometry of the rendering area. Simultaneously, based on the rendering mode, the three-dimensional coordinates of the physical and virtual speakers are projected onto a unit sphere or unit circle, eliminating speaker distance information while retaining directional information for sound image rendering. This also allows the projected point set to be applicable to convex hull calculations in the corresponding dimensions. Therefore, speaker rendering areas can be automatically generated based on the convex hull calculation results, reducing region omissions, region overlaps, and structural instability caused by manual division. Furthermore, the rendering of sound images can be matched with appropriate speaker combinations based on the sound image direction, thereby improving the stability of sound image rendering.

[0037] Based on the above embodiments, the determination process of the preset direction distance condition is further explained. The determination of whether a physical speaker exists in a preset direction that satisfies the preset direction distance condition based on the three-dimensional coordinates of the physical speaker may include the following steps: determining the unit direction vector of each physical speaker based on its three-dimensional coordinates; calculating the spatial angle between the unit direction vector of each physical speaker and the unit direction vector of each preset direction to obtain the directional distance between each physical speaker and each preset direction; when the directional distance between each physical speaker and the target preset direction is not less than the detection threshold corresponding to the target preset direction, it is determined that no physical speaker exists in the target preset direction that satisfies the preset direction distance condition; wherein, the target preset direction is any direction among directly in front, directly behind, and directly above, and the detection thresholds corresponding to directly in front and directly behind are less than the detection threshold corresponding to directly above.

[0038] It should be noted that the convex hull algorithm automatically constructs the envelope surface based on the geometric position of the point set. When there are no speakers in the front, back, or top directions, the left and right speakers will form triangles with their nearest neighbors. Since the distance and angle of the nearest neighbors on the left and right sides may differ, the size and shape of the rendering areas at symmetrical positions will be inconsistent. This asymmetry directly affects the VBAP rendering effect: when the sound image moves in the symmetrical direction, the sound image gain will experience asymmetrical jumps due to the geometric differences in the rendering areas on both sides, affecting the symmetry and stability of sound image localization. To solve this problem, this embodiment injects virtual speakers in key directions such as front, back, and top. The role of the virtual speakers is to provide symmetrical anchor points for the convex hull: after injecting virtual speakers, the left and right speakers form rendering areas with the same virtual speaker, thereby ensuring that the rendering areas at symmetrical positions have a symmetrical geometric structure and eliminating the left-right asymmetry problem caused by the lack of speakers in key directions. For details, please refer to the following: This embodiment can use a right-handed Cartesian coordinate system with the listening position as the origin, where the horizontal x-axis points to the right, the vertical y-axis points forward, and the height z-axis points upward. The three-dimensional coordinates (Cartesian coordinates) of the i-th physical speaker (excluding the subwoofer (LFE channel)) can be represented as: Where N represents the total number of physical speakers involved in the rendering. Indicates the left and right coordinates. Indicates the coordinates of the forward and backward directions. This represents the up and down coordinates. Based on the three-dimensional coordinates of each physical speaker, the position vector of each physical speaker can be normalized to obtain the unit direction vector of each physical speaker: The preset directions can include directly in front, directly behind, and directly above. In the coordinate system described above, the unit direction vectors for directly in front, directly behind, and directly above can be represented as follows: For details, please refer to Table 1, which defines the preset directions.

[0039] Table 1 Definition of Preset Direction

[0040] Among them, azimuth angle Angle of elevation .

[0041] For each physical speaker and each preset direction, the spatial angle between the unit direction vector of the physical speaker and the unit direction vector of the preset direction can be defined as the directional distance: Where k represents the identifier of the preset direction, and d represents the directional distance between the physical speaker and the corresponding preset direction. For a target preset direction, when the directional distance between all physical speakers and the target preset direction is not less than the detection threshold corresponding to that target preset direction, it can be determined that there are no physical speakers satisfying the preset directional distance condition in the target preset direction, i.e.: Where α represents the detection threshold for the corresponding preset direction. The acoustic image localization directly in front and behind is relatively sensitive to directional deviation, while the acoustic image localization directly above has a relatively large tolerance. Therefore, the detection thresholds for directly in front and behind can be smaller than the detection threshold for directly above. Based on the detection results, virtual speakers are only injected in key directions where coverage is deemed insufficient. The specific decision rule is as follows: if there is insufficient coverage directly in front, then a virtual speaker is injected directly in front. If there is a lack of coverage directly behind, then inject a virtual speaker directly behind. If there is a lack of coverage directly above, then inject a virtual speaker directly above. When the physical speaker layout already naturally covers a key direction, redundant virtual speakers are not injected to avoid increasing unnecessary computational overhead and rendering complexity. It should be noted that the detection threshold can be preset based on speaker density, sound field size, or sound image localization accuracy requirements, or it can be adjusted based on historical rendering results.

[0042] This embodiment uses directional distance and detection threshold to form a quantifiable judgment standard, which can accurately determine the preset direction in which the virtual speaker needs to be set, avoiding the need to repeatedly set up the virtual speaker when the existing physical speaker is close to the corresponding direction.

[0043] Based on the above embodiments, the gain mapping relationship corresponding to the virtual speaker is further explained. After setting a virtual speaker in a preset direction where no physical speaker meets the preset directional distance condition, the gain mapping relationship between the virtual speaker and adjacent physical speakers can be predetermined. Specifically, this can include: determining two physical speakers adjacent to the virtual speaker from among the physical speakers based on the direction of the virtual speaker and the direction of each physical speaker; determining the gain coefficients of the virtual speaker corresponding to the two physical speakers respectively through vector basis amplitude translation interpolation based on the unit direction vectors of the two physical speakers and the unit direction vector of the virtual speaker; normalizing the gain coefficients to obtain the gain mapping relationship corresponding to the virtual speaker, and associating and storing the gain mapping relationship with the virtual speaker.

[0044] It should be noted that the three-dimensional coordinates of the virtual speaker can be determined based on the unit direction vector of the corresponding preset direction: Here, R represents the normalized radius, which can be a unit value or the average distance from each physical speaker to the listening position. The coordinates of the virtual speaker are fixed and standardized, independent of the specific physical speaker layout. See reference... Figure 2 , Figure 2 This is an example diagram of a speaker layout provided in an embodiment of the present invention. Figure 2 The circles in the diagram represent actual physical speakers, and the triangles represent injected virtual speakers. For a virtual speaker positioned directly in front or behind (within the horizontal plane), the two adjacent physical speakers located on its left and right sides can be determined based on the azimuth angle. and For a virtual loudspeaker positioned directly overhead (vertically), the two physical loudspeakers adjacent to the zenith direction can be determined based on the elevation angle. and Here, two adjacent physical speakers are used. and The construction process of the 2DVBAP direction vector matrix is ​​explained as follows: The unit direction vectors of two adjacent physical loudspeakers are constructed into a two-dimensional vector basis amplitude translation direction vector matrix: Based on the unit direction vector and direction vector matrix L of the virtual loudspeaker, the gain coefficient of the virtual loudspeaker corresponding to two adjacent physical loudspeakers can be determined by vector basis amplitude translation interpolation. By normalizing the obtained gain coefficients, the normalized gain mapping relationship for the virtual speaker can be obtained. The expression for the normalized gain effect is as follows: Electronic devices can associate and store normalized gain mappings with virtual speaker identifiers. When a subsequent target speaker rendering area contains the virtual speaker, the rendering gain of the virtual speaker is decomposed into gain components of two adjacent physical speakers using this gain mapping.

[0045] In this embodiment, without setting up actual sound-generating devices, virtual speakers can participate in the division of the rendering area, and the corresponding sound image output can be achieved through adjacent physical speakers.

[0046] Based on the above embodiments, the projection process of the three-dimensional coordinates of the loudspeakers is further explained. The rendering mode can be specified by the user or determined according to the spatial arrangement of the physical loudspeakers. The 3D mode corresponds to three-dimensional spatial sound image rendering, and the 2D mode corresponds to sound image rendering in a horizontal plane. According to the rendering mode, the three-dimensional coordinates of each loudspeaker in the extended loudspeaker set are projected onto a unit sphere or unit circle to obtain a set of loudspeaker projection points. Specifically, this may include: (1) The rendering mode is 3D mode: add a preset positive offset to the height coordinate of each speaker in the three-dimensional coordinate of the extended speaker set, and normalize the three-dimensional coordinates after adding the preset positive offset to obtain a set of three-dimensional projection points on the unit sphere.

[0047] refer to Figure 3 , Figure 3 This is an example diagram of a three-dimensional projection provided in an embodiment of the present invention. The unit sphere is a non-developable curved surface. Coplanar points, after normalization, are distributed across the sphere. It is almost impossible for any four spherical points to be coplanar; therefore, the three-dimensional convex hull algorithm can function correctly, overcoming the fundamental obstacle that the convex hull algorithm cannot be directly applied to speaker rendering region partitioning. Projection simultaneously eliminates the distance difference between the speaker and the listener, retaining only directional information, which is consistent with the characteristic of VBAP relying solely on the speaker's direction vector. When the rendering mode is 3D, a preset positive offset can be added to the height coordinates of each speaker in the extended speaker set: Where ε represents a preset positive offset, such as, but not limited to, [other values]. The reason for this is that, for a loudspeaker that is precisely distributed on the horizontal plane (z=0), after normalization, all projected points will fall on the equator of the sphere. While spherical projection transforms coplanar points into spherical points, the points on the equator are still approximately coplanar (all located near the equatorial plane where z≈0), and the convex hull may still degenerate. By adding an offset, all projected points are strictly deviated from the equatorial plane and located in the upper hemisphere, thus generating a valid three-dimensional envelope surface in subsequent convex hull calculations.

[0048] Normalizing the 3D coordinates after adding a preset positive offset yields the 3D projection points located on a unit sphere: .

[0049] (2) The rendering mode is 2D mode: the horizontal and vertical coordinates of each speaker in the extended speaker set are normalized to obtain a set of two-dimensional projection points on the unit circle.

[0050] When the rendering mode is 2D, normalization can be performed using only the horizontal and vertical coordinates of each speaker to obtain the two-dimensional projection points located on the unit circle: .

[0051] When the denominator of the normalized calculation is less than the preset zero vector threshold, the projection point in 3D mode can be set to the preset unit direction vector (i.e., set to...). ), set the projection point in 2D mode to the preset default point (i.e. ), to avoid zero vector normalization.

[0052] This embodiment can eliminate the distance difference between the speaker and the listening position by projecting a unit sphere or a unit circle, while retaining the speaker's directional information, so that different speaker arrangements can be divided into regions in a unified directional space.

[0053] Based on the above embodiments, the process of convex hull calculation and speaker rendering region determination in 3D mode is further explained. When the rendering mode is 3D, the above-mentioned convex hull calculation is performed on the speaker projection point set, and the speaker rendering region is determined based on the convex hull calculation results. Specifically, this may include: selecting four non-coplanar projection points from the 3D projection point set to construct an initial tetrahedron, determining the geometric center of the initial tetrahedron as the internal reference point, and keeping the internal reference point unchanged during the 3D convex hull update process; sequentially processing the remaining projection points in the 3D projection point set, calculating the signed distance between the current projection point and each triangular facet in the current 3D convex hull, and determining whether the current projection point is located outside the current 3D convex hull based on the signed distance; when the current projection point is located outside the current 3D convex hull... When the 3D convex hull is outside, determine the visible face corresponding to the current projection point and the boundary edges between the visible and non-visible faces, delete the visible face, and generate new faces that connect the current projection point and each boundary edge respectively; determine the outward normal direction of each new face based on the internal reference point to obtain the updated 3D convex hull; obtain the height coordinates of the speaker corresponding to the three vertices of each triangle face in the updated 3D convex hull before adding the preset positive offset, and when the absolute value of the height coordinates of the speaker corresponding to the three vertices is less than the preset filtering threshold, delete the corresponding triangle face and determine the remaining triangle face as the speaker rendering area.

[0054] In this embodiment, four non-coincident, non-collinear, and non-coplanar projection points are sequentially selected from the set of 3D projection points to construct an initial tetrahedron, and the geometric center (centroid c) of the initial tetrahedron is determined as the internal reference point. The internal reference point remains unchanged during subsequent 3D convex hull updates, always located inside the convex hull, and is used to unify the outward normal direction of each triangular facet. For a triangular facet composed of three projection points, the unit normal vector of the facet can be determined based on the three projection points: The plane constant of the plate can be expressed as: The formula is Let P be any point on the convex hull. For any projection point to be judged, calculate the signed distance between the projection point and the triangular facet. If the outward normal of the facet points towards the outside of the convex hull, a signed distance greater than zero indicates that the current projection point is located outside the facet. The electronic device processes the remaining projection points in the 3D projection point set sequentially. When the current projection point is located outside the current 3D convex hull, determine the triangular facets visible to the current projection point and define the edges between visible and invisible facets as boundary edges. Then, delete the visible facets, connect the current projection point to each boundary edge to generate new facets, and adjust the normal direction of each new facet according to the internal reference point. When generating new facets, the area of ​​the new facets can be calculated, and facets with an area less than a preset area threshold can be skipped to avoid adding facets with zero or near-zero area to the 3D convex hull. After processing all projection points, the updated 3D convex hull is obtained. Furthermore, the original height coordinates of the speakers corresponding to the three vertices of each triangular facet in the 3D convex hull before adding a preset positive offset can also be obtained. When all three original height coordinates meet the threshold (e.g., ...), The corresponding triangular facets can be identified as degenerate facets near the equator and deleted, as degenerate facets do not constitute a valid 3D rendering area. The remaining triangular facets after filtering are then identified as speaker rendering areas in 3D mode, with each triangular facet corresponding to a speaker combination consisting of three speakers.

[0055] This embodiment can unify the normal direction of the facets and improve the effectiveness and stability of the 3D rendering area by keeping the internal reference point unchanged, updating the convex hull based on the boundary edge and filtering degenerate facets.

[0056] Based on the above embodiments, the process of convex hull calculation and speaker rendering region determination in 2D mode is further explained. When the rendering mode is 2D, the above-mentioned convex hull calculation is performed on the speaker projection point set, and the speaker rendering region is determined based on the convex hull calculation results. Specifically, this may include: performing collinearity detection on the projection points in the 2D projection point set; when not all projection points in the 2D projection point set are collinear, sorting the projection points in the 2D projection point set according to their horizontal coordinates; determining the upper and lower convex hulls based on the sorted projection points, and merging the upper and lower convex hulls to obtain a sequence of convex hull vertices arranged according to the 2D convex hull boundary; sequentially connecting adjacent projection points in the convex hull vertex sequence, and connecting the first and last projection points in the convex hull vertex sequence to obtain multiple convex hull edges; and determining the line segment region formed by the two speakers corresponding to each convex hull edge as a speaker rendering region.

[0057] In this embodiment, collinearity detection is first performed on the projection points in the two-dimensional projection point set. If all projection points are collinear, an error message is output and the current convex hull calculation is terminated. If not all projection points are collinear, they are sorted by horizontal coordinate from smallest to largest; if the horizontal coordinates are the same, they are sorted by vertical coordinate. Based on the sorted projection points, each projection point is processed sequentially from left to right to form the lower convex hull, and then processed sequentially from right to left to form the upper convex hull. During the formation of the upper or lower convex hull, the upper and lower convex hulls are merged to obtain a sequence of convex hull vertices arranged according to the two-dimensional convex hull boundary order. Connecting adjacent projected points in the convex hull vertex sequence sequentially, and then connecting the first and last projected points, yields multiple convex hull edges connected end-to-end. The rendering region corresponding to the k-th convex hull edge can be represented as: .

[0058] Where H represents the number of vertices in the 2D convex hull, and h represents the speaker index corresponding to each vertex of the convex hull. Two speakers corresponding to each convex hull edge constitute a speaker combination, and the directional interval between the two speakers constitutes a 2D speaker rendering region. Optionally, a unified data structure containing a third index can be used to store the 2D speaker rendering region, and the third index can be set to a preset special value (e.g., the third vertex index is set to -1) to distinguish between 2D line segment regions and 3D triangle regions.

[0059] This embodiment uses the above-mentioned two-dimensional convex hull processing, which can automatically form a line segment rendering area connected end to end according to the direction sequence of the speaker in the horizontal plane, so that the 2D mode and the 3D mode can share the speaker projection, area storage and subsequent sound and image rendering framework.

[0060] Based on the above embodiments, the determination of the target speaker rendering area and the process of sound image rendering based on speaker combinations are further explained. Specifically, determining the target speaker rendering area from the speaker rendering area according to the sound image direction of the sound image to be rendered, and rendering the sound image to be rendered according to the speaker combination corresponding to the target speaker rendering area, may include: Step 1: When the rendering mode is 3D mode, obtain the vector basis amplitude translation inverse matrix pre-calculated based on the unit direction vectors of the three speakers corresponding to the current speaker rendering area, and determine the candidate gain vector based on the vector basis amplitude translation inverse matrix and the sound image direction; when all three components of the candidate gain vector are non-negative, determine the current speaker rendering area as the target speaker rendering area, and determine the candidate gain vector as the initial rendering gain of each speaker in the speaker combination; the current speaker rendering area is any area in the speaker area.

[0061] It should be noted that the direction of the sound image to be rendered can be represented by a unit direction vector with the listening position as the origin. When the rendering mode is 3D, for the speaker rendering area in the form of a triangular facet, a direction vector matrix can be constructed based on the unit direction vectors of the three speakers corresponding to the triangular facet, and the inverse matrix of the direction vector matrix can be pre-calculated to obtain the vector basis amplitude translation inverse matrix G corresponding to the speaker rendering area. f The electronic device traverses the speaker rendering areas in the form of triangular facets, and follows... Calculate the candidate gain vector wf corresponding to the f-th speaker rendering region. Where G f P represents the vector basis amplitude translation inverse matrix corresponding to the f-th speaker rendering region. src w represents the direction of the sound image to be rendered. f This represents the candidate gain vector corresponding to the f-th speaker rendering region. When w f When all three components are non-negative, it indicates that the sound image direction is projected outward to the f-th speaker rendering area, and the f-th speaker rendering area is determined as the target speaker rendering area. Since the speaker rendering areas in the form of triangular patches do not overlap and cover the entire sound image direction, except in the boundary case, there is one and only one candidate gain vector corresponding to a triangular patch whose three components are all non-negative; in the boundary case, any triangular patch that meets the conditions can be selected as the target speaker rendering area.

[0062] Specifically, for the target speaker rendering area in 3D mode, a direction vector matrix can be constructed based on the unit direction vectors of the three corresponding speakers: And pre-calculate the vector basis amplitude translation inverse matrix corresponding to this direction vector matrix: .

[0063] Step 2: When the rendering mode is 2D, determine the target speaker rendering area where the sound image direction is located based on the azimuth angle of the sound image direction and the azimuth angle of the two speakers corresponding to the speaker rendering area in each line segment form; determine the initial rendering gain of each speaker based on the unit direction vector and sound image direction of each speaker in the speaker combination corresponding to the target speaker rendering area.

[0064] It should be noted that when the rendering mode is 2D, the azimuth angle of the sound image direction can be determined, and this azimuth angle can be compared with the azimuth angles of the two speakers corresponding to the speaker rendering areas of each line segment. The adjacent speaker direction interval where the sound image direction is located is determined as the target speaker rendering area.

[0065] For the target speaker rendering area in 2D mode, a direction vector matrix can be constructed based on the unit direction vectors of the two corresponding speakers. And the corresponding vector basis amplitude translation inverse matrix: Based on the vector basis amplitude translation inverse matrix Harmony image direction P src The initial rendering gain of the two speakers can be determined using the following formula: Where w2 represents the initial rendering gain vector corresponding to the target speaker rendering area in 2D mode.

[0066] Step 3: When the speaker combination includes virtual speakers, according to the gain mapping relationship corresponding to the virtual speakers, the initial rendering gain of the virtual speakers is decomposed into the gain components of the two adjacent physical speakers, and each gain component is added to the initial rendering gain of the corresponding physical speaker to obtain the rendering gain of each physical speaker.

[0067] During real-time rendering, the virtual speaker is merely a mathematical anchor point used to ensure convex hull symmetry and correct region partitioning; it does not correspond to any physical hardware and cannot directly produce sound. Therefore, when a virtual speaker is included in the target speaker combination, its initial rendering gain cannot be directly output. It must be proportionally allocated to its two adjacent physical speakers using the gain mapping relationship pre-stored in the preprocessing stage to obtain the corresponding gain components. Subsequently, these gain components are added to the initial rendering gains of the two physical speakers themselves to obtain the total rendering gain that each physical speaker should actually output, i.e., the unnormalized gain vector. This process seamlessly transfers the acoustic energy from the virtual speaker to the real physical speaker, maintaining the energy intensity of the original sound image direction and ensuring that all driving signals are output from the physical speaker during final playback. This ensures that the rendering result is physically achievable while preserving the left-right symmetry and sound image positioning accuracy obtained from convex hull partitioning.

[0068] Step 4: Normalize the rendering gain of each physical speaker, and control each physical speaker to render the sound image to be rendered based on the normalized rendering gain.

[0069] You can refer to the following formula: The electronic device weights the audio signals of the sound image to be rendered according to the rendering gain after energy normalization, and outputs the weighted audio signals to the corresponding physical speakers.

[0070] This embodiment combines region matching, loudspeaker gain calculation, and virtual loudspeaker gain decomposition to maintain the correspondence between the sound image direction and the actual sound-emitting loudspeaker under different loudspeaker arrangement forms, thereby improving the continuity and stability of sound image localization.

[0071] The audio-visual rendering apparatus provided in the embodiments of the present invention will be described below. The audio-visual rendering apparatus described below can be referred to in correspondence with the audio-visual rendering method described above.

[0072] Please refer to the details. Figure 4 , Figure 4 A schematic diagram of the structure of an audio-visual rendering device provided in an embodiment of the present invention may include: The determining module 100 is used to determine whether there is a physical speaker that meets the distance condition of the preset direction in the three-dimensional coordinates of the physical speaker. The expansion module 200 is used to set up a virtual speaker in a preset direction where there is no physical speaker that meets the preset directional distance condition, thereby obtaining an expanded speaker set; the expanded speaker set includes the virtual speaker and the physical speaker; The projection module 300 is used to project the three-dimensional coordinates of each speaker in the extended speaker set onto a unit sphere or unit circle according to the rendering mode, so as to obtain a set of speaker projection points. The calculation module 400 is used to perform convex hull calculation on the speaker projection point set according to the rendering mode, and determine the speaker rendering area based on the convex hull calculation result. The rendering module 500 is used to determine a target speaker rendering area from the speaker rendering area according to the sound image direction of the sound image to be rendered, and to render the sound image to be rendered according to the speaker combination corresponding to the target speaker rendering area.

[0073] Based on the above embodiments, the determining module 100 may include: The unit direction vector determination unit is used to determine the unit direction vector of each physical loudspeaker based on the three-dimensional coordinates of each physical loudspeaker. The directional distance calculation unit is used to calculate the spatial angle between the unit directional vector of each physical speaker and the unit directional vector of each preset direction, so as to obtain the directional distance between each physical speaker and each preset direction. The determining unit is configured to determine that there is no physical speaker satisfying the preset directional distance condition in the target preset direction when the directional distance between each of the physical speakers and the target preset direction is not less than the detection threshold corresponding to the target preset direction; The target preset direction is any one of the directions of front, back, and top, and the detection thresholds corresponding to the front and back directions are less than the detection thresholds corresponding to the top direction.

[0074] Based on the above embodiments, the audio-visual rendering apparatus may further include: An adjacent speaker determination unit is used to determine two physical speakers adjacent to the virtual speaker from among the physical speakers, based on the direction of the virtual speaker and the direction of each physical speaker; The gain coefficient determination unit is used to determine the gain coefficient of the virtual speaker corresponding to the two physical speakers respectively by vector basis amplitude translation interpolation based on the unit direction vectors of the two physical speakers and the unit direction vector of the virtual speaker; The normalization unit is used to normalize the gain coefficients to obtain the gain mapping relationship corresponding to the virtual speaker, and to associate and store the gain mapping relationship with the virtual speaker.

[0075] Based on the above embodiments, the projection module 300 may include: The unit projection point set acquisition unit is used to add a preset positive offset to the height coordinate of each speaker in the three-dimensional coordinates of the extended speaker set when the rendering mode is 3D mode, and to normalize the three-dimensional coordinates after adding the preset positive offset to obtain the three-dimensional projection point set located on the unit sphere. The two-dimensional projection point set acquisition unit is used to obtain the two-dimensional projection point set located on the unit circle by normalizing the horizontal and vertical coordinates of each speaker in the three-dimensional coordinates of the extended speaker set when the rendering mode is 2D mode.

[0076] Based on the above embodiments, the calculation module 400 may include: An internal reference point determination unit is used to select four non-coplanar projection points from the three-dimensional projection point set to construct an initial tetrahedron, determine the geometric center of the initial tetrahedron as the internal reference point, and keep the internal reference point unchanged during the three-dimensional convex hull update process; The judgment unit is used to process the remaining projection points in the three-dimensional projection point set in sequence, calculate the signed distance between the current projection point and each triangular facet in the current three-dimensional convex hull, and determine whether the current projection point is located outside the current three-dimensional convex hull based on the signed distance. A patch generation unit is used to determine the visible patch corresponding to the current projection point and the boundary edge between the visible patch and the non-visible patch when the current projection point is located outside the current three-dimensional convex hull, delete the visible patch, and generate new patches that connect the current projection point and each of the boundary edges respectively. The convex hull update unit is used to determine the outward normal direction of each new facet based on the internal reference point, so as to obtain the updated three-dimensional convex hull. The first region determination unit is used to obtain the height coordinates of the speakers corresponding to the three vertices of each triangle facet in the updated 3D convex hull before the preset positive offset is added. When the absolute values ​​of the height coordinates of the speakers corresponding to the three vertices are all less than the preset filtering threshold, the corresponding triangle facet is deleted and the remaining triangle facet is determined as the speaker rendering area.

[0077] Based on the above embodiments, the calculation module 400 may include: Collinearity detection unit, used to perform collinearity detection on the projection points in the two-dimensional projection point set; The sorting unit is used to sort the projection points in the two-dimensional projection point set according to their horizontal coordinates when not all projection points in the two-dimensional projection point set are collinear. The merging unit is used to determine the upper convex hull and the lower convex hull respectively according to the sorted projection points, and merge the upper convex hull and the lower convex hull to obtain the convex hull vertex sequence arranged according to the two-dimensional convex hull boundary; The convex hull edge acquisition unit is used to sequentially connect adjacent projection points in the convex hull vertex sequence, and connect the first projection point and the last projection point in the convex hull vertex sequence to obtain multiple convex hull edges; The second region determination unit is used to determine the line segment region formed by the two speakers corresponding to each convex hull as a speaker rendering region.

[0078] Based on the above embodiments, the rendering module 500 may include: The first target loudspeaker rendering area determination unit is used to, when the rendering mode is 3D mode, obtain the vector basis amplitude translation inverse matrix pre-calculated based on the unit direction vectors of the three loudspeakers corresponding to the current loudspeaker rendering area, and determine the candidate gain vector based on the vector basis amplitude translation inverse matrix and the sound image direction; when all three components of the candidate gain vector are non-negative, determine the current loudspeaker rendering area as the target loudspeaker rendering area, and determine the candidate gain vector as the initial rendering gain of each loudspeaker in the loudspeaker combination; the current loudspeaker rendering area is any area in the loudspeaker area; The second target speaker rendering area determination unit is used to determine the target speaker rendering area where the sound image direction is located based on the azimuth angle of the sound image direction and the azimuth angle of the two speakers corresponding to each line segment form speaker rendering area when the rendering mode is 2D mode; and to determine the initial rendering gain of each speaker based on the unit direction vector of each speaker in the speaker combination corresponding to the target speaker rendering area and the sound image direction. The rendering gain acquisition unit is used to, when the speaker combination includes a virtual speaker, decompose the initial rendering gain of the virtual speaker into the gain components of two adjacent physical speakers according to the gain mapping relationship corresponding to the virtual speaker, and add each of the gain components to the initial rendering gain of the corresponding physical speaker to obtain the rendering gain of each physical speaker. The rendering unit is used to normalize the rendering gain of each physical speaker and control each physical speaker to render the sound image to be rendered based on the normalized rendering gain.

[0079] It should be noted that the order of the modules and units in the aforementioned audio-visual rendering device can be changed without affecting the logic.

[0080] An audio-visual rendering apparatus provided in this embodiment of the invention comprises: a determining module 100, used to determine whether a physical speaker exists in a preset direction that satisfies a preset directional distance condition based on the three-dimensional coordinates of the physical speaker; an expansion module 200, used to set up a virtual speaker in a preset direction where no physical speaker satisfies the preset directional distance condition, thereby obtaining an expanded speaker set; the expanded speaker set includes the virtual speaker and the physical speaker; a projection module 300, used to project the three-dimensional coordinates of each speaker in the expanded speaker set onto a unit sphere or unit circle according to a rendering mode, thereby obtaining a speaker projection point set; a calculation module 400, used to perform convex hull calculation on the speaker projection point set according to the rendering mode, and determine the speaker rendering area based on the convex hull calculation result; and a rendering module 500, used to determine a target speaker rendering area from the speaker rendering area according to the audio-visual direction of the audio-visual to be rendered, and render the audio-visual to be rendered according to the speaker combination corresponding to the target speaker rendering area. This device sets up virtual speakers in directions where no physical speakers meet preset directional distance conditions, allowing the speakers involved in region division to complement each other in the corresponding directions. This reduces the impact of uneven physical speaker distribution on the geometry of the rendering area. Simultaneously, based on the rendering mode, the three-dimensional coordinates of the physical and virtual speakers are projected onto a unit sphere or unit circle. This eliminates the speaker distance information while retaining the directional information used for sound image rendering, and also makes the projected point set suitable for convex hull calculations in the corresponding dimensions. Therefore, speaker rendering areas can be automatically generated based on the convex hull calculation results, reducing region omissions, region overlaps, and structural instability caused by manual division. Furthermore, the device allows the rendered sound image to match the corresponding speaker combination according to the sound image direction, thereby improving the stability of sound image rendering.

[0081] The audio-visual rendering device provided in the embodiments of the present invention will be described below. The audio-visual rendering device described below can be referred to in correspondence with the audio-visual rendering method described above.

[0082] Please refer to Figure 5 , Figure 5 A schematic diagram of the structure of an audio-visual rendering device provided in an embodiment of the present invention may include: Memory 10 is used to store computer programs; Processor 20 is used to execute computer programs to implement the above-described audio-visual rendering method.

[0083] The memory 10, processor 20, and communication interface 31 all communicate with each other through the communication bus 32.

[0084] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions: Based on the three-dimensional coordinates of the physical loudspeaker, determine whether there is a physical loudspeaker in the preset direction that meets the preset distance condition; A virtual speaker is set in a preset direction where no physical speaker meets the preset directional distance condition, resulting in an extended speaker set; the extended speaker set includes virtual speakers and physical speakers. Depending on the rendering mode, the three-dimensional coordinates of each speaker in the extended speaker set are projected onto a unit sphere or unit circle to obtain the speaker projection point set; Based on the rendering mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation results. Based on the direction of the sound image to be rendered, the target speaker rendering area is determined from the speaker rendering area, and the sound image to be rendered is rendered according to the speaker combination corresponding to the target speaker rendering area.

[0085] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0086] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0087] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0088] Communication interface 31 can be an interface for the communication module, used to connect with other devices or systems.

[0089] Of course, it should be noted that, Figure 5 The structure shown does not constitute a limitation on the audio-visual rendering device in the embodiments of the present invention. In practical applications, the audio-visual rendering device may include devices that are more advanced than those described above. Figure 5 More or fewer components as shown, or combinations of certain components.

[0090] It is understood that if the audio-visual rendering method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk or optical disk, and other media capable of storing program code.

[0091] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-described audio-visual rendering method.

[0092] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0093] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed audio-visual rendering method.

[0094] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0095] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0096] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0097] The above provides a detailed description of the audio-visual rendering method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A sound and image rendering method, characterized in that, include: Based on the three-dimensional coordinates of the physical loudspeaker, determine whether there is a physical loudspeaker in the preset direction that meets the preset distance condition; A virtual speaker is set in a preset direction where no physical speaker satisfies the preset directional distance condition, thereby obtaining an extended speaker set; The extended speaker set includes the virtual speaker and the physical speaker; According to the rendering mode, the three-dimensional coordinates of each speaker in the extended speaker set are projected onto a unit sphere or unit circle to obtain the speaker projection point set; According to the rendering mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation result. Based on the sound image direction of the sound image to be rendered, a target speaker rendering area is determined from the speaker rendering area, and the sound image to be rendered is rendered according to the speaker combination corresponding to the target speaker rendering area.

2. The audio-visual rendering method according to claim 1, characterized in that, Based on the three-dimensional coordinates of the physical loudspeaker, determine whether a physical loudspeaker exists in a preset direction that satisfies a preset directional distance condition, including: Based on the three-dimensional coordinates of each physical loudspeaker, determine the unit direction vector of each physical loudspeaker; Calculate the spatial angle between the unit direction vector of each physical speaker and the unit direction vector of each preset direction to obtain the directional distance between each physical speaker and each preset direction; When the directional distance between each of the physical loudspeakers and the target preset direction is not less than the detection threshold corresponding to the target preset direction, it is determined that there are no physical loudspeakers in the target preset direction that satisfy the preset directional distance condition; The target preset direction is any one of the directions of front, back, and top, and the detection thresholds corresponding to the front and back directions are less than the detection thresholds corresponding to the top direction.

3. The audio-visual rendering method according to claim 1, characterized in that, After setting a virtual speaker in a preset direction where no physical speaker meets the preset directional distance condition, the method further includes: Based on the orientation of the virtual speaker and the orientation of each of the physical speakers, determine the two physical speakers adjacent to the virtual speaker from among the physical speakers; Based on the unit direction vectors of the two physical loudspeakers and the unit direction vector of the virtual loudspeaker, the gain coefficients of the virtual loudspeaker corresponding to the two physical loudspeakers are determined by vector basis amplitude translation interpolation. The gain coefficients are normalized to obtain the gain mapping relationship corresponding to the virtual speaker, and the gain mapping relationship is associated with and stored with the virtual speaker.

4. The audio-visual rendering method according to claim 1, characterized in that, According to the rendering mode, the three-dimensional coordinates of each speaker in the extended speaker set are projected onto a unit sphere or unit circle to obtain a speaker projection point set, including: When the rendering mode is 3D mode, a preset positive offset is added to the height coordinate of each speaker in the three-dimensional coordinate system of the extended speaker set, and the three-dimensional coordinates after adding the preset positive offset are normalized to obtain a set of three-dimensional projection points located on the unit sphere. When the rendering mode is 2D mode, the horizontal and vertical coordinates of each speaker in the extended speaker set are normalized to obtain a set of two-dimensional projection points located on the unit circle.

5. The audio-visual rendering method according to claim 4, characterized in that, When the rendering mode is 3D mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation result, including: Four non-coplanar projection points are selected from the set of three-dimensional projection points to construct an initial tetrahedron. The geometric center of the initial tetrahedron is determined as the internal reference point, and the internal reference point is kept unchanged during the three-dimensional convex hull update process. The remaining projection points in the three-dimensional projection point set are processed sequentially, and the signed distance between the current projection point and each triangular facet in the current three-dimensional convex hull is calculated. Based on the signed distance, it is determined whether the current projection point is located outside the current three-dimensional convex hull. When the current projection point is located outside the current 3D convex hull, determine the visible face corresponding to the current projection point and the boundary edge between the visible face and the non-visible face, delete the visible face, and generate new face connecting the current projection point and each of the boundary edges respectively. The outer normal direction of each new facet is determined based on the internal reference point to obtain the updated three-dimensional convex hull. Obtain the height coordinates of the speaker corresponding to the three vertices of each triangle in the updated 3D convex hull before adding the preset positive offset. When the absolute value of the height coordinates of the speaker corresponding to the three vertices is less than the preset filtering threshold, delete the corresponding triangle and determine the remaining triangle as the speaker rendering area.

6. The audio-visual rendering method according to claim 4, characterized in that, When the rendering mode is 2D mode, convex hull calculation is performed on the speaker projection point set, and the speaker rendering area is determined based on the convex hull calculation result, including: Collinearity detection is performed on the projection points in the two-dimensional projection point set; When not all projection points in the two-dimensional projection point set are collinear, the projection points in the two-dimensional projection point set are sorted according to their horizontal coordinates. The upper and lower convex hulls are determined based on the sorted projection points, and the upper and lower convex hulls are merged to obtain a sequence of convex hull vertices arranged according to the two-dimensional convex hull boundary. By sequentially connecting adjacent projection points in the convex hull vertex sequence, and connecting the first and last projection points in the convex hull vertex sequence, multiple convex hull edges are obtained; The line segment region formed by the two speakers corresponding to each convex hull is defined as a speaker rendering region.

7. The audio-visual rendering method according to any one of claims 1 to 6, characterized in that, Based on the sound image direction of the sound image to be rendered, a target speaker rendering area is determined from the speaker rendering area, and the sound image to be rendered is rendered according to the speaker combination corresponding to the target speaker rendering area, including: When the rendering mode is 3D mode, the vector basis amplitude translation inverse matrix is ​​obtained by pre-calculating the unit direction vectors of the three speakers corresponding to the current speaker rendering area, and the candidate gain vector is determined according to the vector basis amplitude translation inverse matrix and the sound image direction; when all three components of the candidate gain vector are non-negative, the current speaker rendering area is determined as the target speaker rendering area, and the candidate gain vector is determined as the initial rendering gain of each speaker in the speaker combination; the current speaker rendering area is any area in the speaker area; When the rendering mode is 2D mode, the target speaker rendering area where the sound image direction is located is determined according to the azimuth angle of the sound image direction and the azimuth angle of the two speakers corresponding to the speaker rendering area in each line segment form; the initial rendering gain of each speaker is determined according to the unit direction vector of each speaker in the speaker combination corresponding to the target speaker rendering area and the sound image direction. When the speaker combination includes a virtual speaker, according to the gain mapping relationship corresponding to the virtual speaker, the initial rendering gain of the virtual speaker is decomposed into the gain components of the two adjacent physical speakers, and each gain component is added to the initial rendering gain of the corresponding physical speaker to obtain the rendering gain of each physical speaker. The rendering gain of each physical speaker is energy normalized, and the rendering gain after energy normalization is used to control each physical speaker to render the sound image to be rendered.

8. An audio-visual rendering device, characterized in that, include: The determination module is used to determine whether there is a physical speaker that meets the preset distance condition in the preset direction based on the three-dimensional coordinates of the physical speaker; An extension module is used to set up a virtual speaker in a preset direction where there is no physical speaker that meets the preset directional distance condition, thereby obtaining an extended speaker set; The extended speaker set includes the virtual speaker and the physical speaker; The projection module is used to project the three-dimensional coordinates of each speaker in the extended speaker set onto a unit sphere or unit circle according to the rendering mode, so as to obtain a set of speaker projection points; The calculation module is used to perform convex hull calculation on the speaker projection point set according to the rendering mode, and determine the speaker rendering area based on the convex hull calculation result; The rendering module is used to determine the target speaker rendering area from the speaker rendering area according to the sound image direction of the sound image to be rendered, and to render the sound image to be rendered according to the speaker combination corresponding to the target speaker rendering area.

9. An audio-visual rendering device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the audio-visual rendering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the audio-visual rendering method as described in any one of claims 1 to 7.

Images